I have had nothing but problems due to high server load since switching to a dedicated server. I noticed that in my largest forum, Inktomibot was crushing me. (50 guests, 1 member... all the guests were Inktomibot)

That's one of Yahoo's bots, and it was doing multi-threaded, nonstop indexing on my site, resulting in server slowdowns and high-load induced server crashes.

ANYWAY, I added the following to my robots.tx t file and was immediately rewarded with dramatically lower server load.


The first puts a limit on Slurp (another name for Yahoo's bot) and the second puts a limit on all bots (including Google, etc.)
They can still spider you... it'll just happen at a much slower rate to avoid server load issues.

PS - Anybody with more information is highly encouraged to chime in on this topic.


Jake

Sorry to reply to such an old post, but I just came across it. But anyway:
According to http://tool.motoricerca.info/robots-checker.phtml I get the following message:

"You specified both the generic user-agent "*" and specific user-agents for this block of code; this could be misinterpreted."

Also that website says that it's better to have an empty line between the User Agent blocks.
Like this: