A lot of people like to protect certain files and folders from being indexed by search engines and other robots...
WHAT IS A ROBOTS.TXT FILE?A robots.txt file is an instruction to the robots and search engines that travel the web and add your site to their directories, for people to search for them.
Before you begin, you need to know how to write the .txt file.
Prepare it in a text editor such as Notepad. Don't attempt it in Word or an HTML editor such as FrontPage. When you're finished, save it as "robots.txt".
If you want to disallow all robots from all your files and folders you would use the following:
User-agent: *
Disallow: / If you only want to dissalow certain search engines like Google or webcrawler you must add the following lines to your robot.txt file
User-agent: Google
User-agent: WebCrawlerto disallow all search engines from access the files folders your going to specify you must use the command:
User-agent: *To disallow them from searching your folders the command is:
Disallow: /foldername/or
Disallow: /folder/foldername/ (if your folder is inside another)
For this to work your robots.txt file must be uploaded using ASCII format, depending on what program you use to upload your files depends on how you do this.
Hope This Helps
Admins/Moderators, Feel Free To Modify This Post
Dilber MC Theme by HarzeM