Interesting one, I'd not heard of the robots page before so I've doen a bit of digging - from the looks of things the robots.txt page is an option only available if you run your own server (not just have webspace) and want to EXCLUDE certain areas from search engine spiders.
Quote:
|
You'll also want to check your robots.txt file. This is a standard file for Web servers that sits at the root of your site and excludes unwelcome indexing robots or restricts them from crawling certain directories. If you run your own server, you control this file; otherwise, your host server administrator controls it. Make sure that this file exists and that it allows at least your search indexer robot to access your directories.
|
Therefore its not something you need to worry about. If you want to know more follow this link for a few pages with some relavant information:
http://search.hotwired.com/webmonkey...ery=robots.txt