Inktomi and Yahoo's a spiderin?
I thought some one would send Yahoo a note to please cut down RR forum's bandwidth bill by not running around old posts..
Wonder if Enki or someone else as an endless /dev/null to sink them into..
404 page that links to /dev/null... Where is my electronic whirlpool when I need it?
The amount of bandwidth used by the spiders is negligible, and this forum only uses around 1% of the VM's total compute power. I would rather see us get hits of those poor bastards stuck using Yahoo for search. Perhaps we could edumacate them a bit

Unless the crawler is bringing one of my machines to it's knees, I would always rather see our content in their index than try to hide it with a robots file.
Also, if I couldn't build a site that could handle crawlers, I wouldn't be much of an Admin.