I just got hit by an ill-formed spider originally written by Microsoft. Called MSNPTC, it hammers the server badly enough that Lunarpages disabled my wiki. I found a list of IP addresses it's been known to use at http://www.botsvsbrowsers.com/details/4800/index.html, and used the IP deny manager, which put

deny from 202.96.51.
deny from 207.46.
deny from 219.142.53.

in my .htaccess file. I've also entered this spider in robots.txt, but it doesn't seem to honor that file. Just an FYI.