Actually, googlebot just killed my bandwidth for the month.  Repeatingly downloaded some very large files that I thought were already protected, but my robots.txt file was missing (probably accidentally deleted it ).  My traffic spiked the last 5 days and went through about 30gigs of it.

You might want one, even if you leave it blank, so as to avoid 404 errors. When the good bots (that is, the ones that respect robots.txt instructions) come by, they will look for the robots.txt file. If it isn't there, you'll get a 404 error in your logs, due to the bot having requested a file that doesn't exist. I have a client whose site was getting an astonishing number of 404 errors, and they were all due to not having a robots.txt file.

I believe this is correct. Since the file only blocks (good) bots from files, it would do nothing to encourage those bots to access your site in the first place. But if you block all bots, then your site wouldn't get crawled; this would prevent your getting ranked (since, as far as the bot is concerned, you don't exist).

I don't know that you'd want every spider; that will depend on what you've got to protect from spidering. I think the robots.txt file is more often used to "hide" things like private directories (a family page, say, that you don't care to see in the search engines). That is, you would use the robots.txt file to "hide" just a part of your site. You can still leave the part you want in the search engines open to all the bots.

Eliz.

hmmm... very interesting the stuff whitehouse.gov wants to hide from bots...

I know the intention of robots.txt but I'm just giving people a heads-up that if they don't use other means of protecting their directories, robots.txt is the equivalent of announcing to people where their "hidden" stash is.

Hi, I did what you said to do in Notepad, but when I open up WS_FTP LE, I can't find Notepad so that I can transfer the file over to my public.html file. Please tell me what to do.

Disallow: /dirA/ will affect all paths that begin with /dirA/, so that includes child directories.

What you have been looking at are probably ones like this:

User-agent: *
Disallow: /folder/subfolder1
Disallow: /folder/subfolder2
Disallow: /folder/subfolder3

That would indicate that those 3 subfolders should not be crawled, but /folder/subfolder4 or /folder/ can be crawled.

I was just doing some research on just what a "robots.txt" file is (along with what else is supposed to be inside public_html and what's not) and came across this thread. Now I know I can leave this one in my public_html when I delete my current site's folders and add my newly rebuilt site !

Since I am currently using Mambo, my robots.txt file contains the following:

User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /components/
Disallow: /editor/
Disallow: /help/
Disallow: /images/
Disallow: /includes/
Disallow: /language/
Disallow: /mambots/
Disallow: /media/
Disallow: /modules/
Disallow: /templates/
Disallow: /installation/

Thanks for this link James !!

RAT

Using robots.txt will keep well behaved robots from indexing your page, but there are robots that ignore robots.txt, so your content could still wind up being generally available.

The next step in security would be put your content in a folder of your site that has no link to it from the home page (or anywhere else). Give the name of the folder to anyone you want to have access.

That is still not actually protected, however. If your blog will contain truly personal info that you don't want republished anywhere, the only solution is to password protect the folder it's in and give the password to selected people. Even that solution is only as trustworthy as the people to whom you give the password.