robots.txt?

[Red Mancunian]

Ok, I know what this is, but acan anyone please tell me what the robots.txt file should look like for my website?

I notice a lot of spiders looking for it, and I'm sure if I bothered to put one in then it would help with listing my site on the search engines.

When replying, please remember 2 things:

(1) I'm thick
(2) I'm thick

Thanks!

[Tigger]

Interesting one, I'd not heard of the robots page before so I've doen a bit of digging - from the looks of things the robots.txt page is an option only available if you run your own server (not just have webspace) and want to EXCLUDE certain areas from search engine spiders.

You'll also want to check your robots.txt file. This is a standard file for Web servers that sits at the root of your site and excludes unwelcome indexing robots or restricts them from crawling certain directories. If you run your own server, you control this file; otherwise, your host server administrator controls it. Make sure that this file exists and that it allows at least your search indexer robot to access your directories.

Therefore its not something you need to worry about. If you want to know more follow this link for a few pages with some relavant information:

http://search.hotwired.com/webmonkey...ery=robots.txt

[Red Mancunian]

Sh*te. I think I'll take yer advice and just ignore the bugger.

Spiders? Web? Crawling?

Fekk me, is it a computer or a frikkin creepy-crawly house? :p

[fugjostle]

Search engines use a technique called spidering (web... get it). The program (or spider as its called) goes trawling through the web following links on all the web pages it finds and builds a database of all the web pages it finds (database?, well yep, thats what google is... and altavista.. and yahoo... and etc).

Some sites have pages that they don't want found in google or altavista. For example, development pages, private members pages, etc.

The robots.txt file is a simple text file that tells the spider not to check the pages your text file specifies. The various spiders always check for that file so that they don't include anything you don't want it to.

If your site is www.redmanc.com and you didn't want the pages within your private or tmp directory appearing in google then you could create a robots.txt file that would look like this:

User-agent: *
Disallow: /tmp/
Disallow: /private/

The web spider would check for www.redmanc.com/robots.txt and obey by ignoring www.redmanc.com/tmp and www.redmanc.com/private

The "user-agent" thing is quite handy as you can allow certain spiders and disallow others:

User-agent: WebCrawler
Disallow:

User-agent: *
Disallow: /

This would only allow the webcrawler spider to index your site and deny all others.

Have a look at http://www.robotstxt.org/wc/exclusion-admin.html for some more ideas of what you can do.

Hope this helps

---
Fug

[Red Mancunian]

Cheerz fug fella.

I should have just kept my Commodore 64, shouldn't I? j/k

That site's got loads of useful reading on it mate.

[fugjostle]

Good Luck mate

When you finish your site you should run it through the world wide web (w3) consortiums validater... it will check your html and tell you if its compliant or not. Compliance means that any web browser can view it.

http://validator.w3.org/

---
Fug

[Red Mancunian]

Nice one mate. I will.

Cheers