As many of you know, the robots.txt file is the file that search engines look at for "instructions" on which files and folders they should or should not index. Looking around for a good SMF-inspired robots.txt file, I found this:
User-agent: *
Disallow: index.php?action=search*
Disallow: index.php?action=calendar*
Disallow: index.php?action=login*
Disallow: index.php?action=register*
Disallow: index.php?action=profile*
Disallow: index.php?action=stats*
Disallow: index.php?action=arcade*
Disallow: index.php?action=printpage*
Disallow: index.php?PHPSESSID=*
Disallow: index.php?*rss*
Disallow: index.php?*wap*
Disallow: index.php?*wap2*
Disallow: index.php?*imode*
This will tell the search engines to stop following those directories and links, and not to index the content within. If you are using a plugin or service that re-writes your addresses to make your links more "SEO friendly" the above code might need to be edited some, but you get the basic idea of what needs to be done.
Just wanted to share my find, just in case it helps somebody else out. Anybody have any suggestions on how to do it better?
