Web Hosting Forum | Lunarpages
News: October 6, 2008 - Submit Your Site for the October 2008 Site of the Month!
 
Home Help Search Calendar Login Register
*
Welcome, Guest. Please login or register.
Did you miss your activation email? 		October 12, 2008, 11:43:56 PM


Login with username, password and session length


Web Hosting Forum | Lunarpages > Lunarpages Web Hosting - Advanced Assistance > Lunarpages - Search Engine Optimization > Using Robots.txt To Protect Files/Folders
Pages: 1 ... 3 4 [5]   Go Down
« previous next »
  Print  
Author Topic: Using Robots.txt To Protect Files/Folders  (Read 15273 times)
netsavy006
Andy - Junkie
Spaceship Navigator
*****
Offline Offline

Posts: 82



WWW
« Reply #60 on: October 18, 2007, 11:42:29 AM »
Only thing that I want to know about it is how would you modify the files to tell search engines to index all your pages and follow links.
I want the robots to be able to find my site...

I've heard of something where it does:

index,follow
noindex,follow
index,nofollow
noindex,nofollow
all

or something like that.  I'd like to know how to tell the search engines to index the main page and index everything else as well.

I hope it's all clear and makes sence.
Andy...
« Last Edit: October 18, 2007, 11:52:15 AM by netsavy006 » Logged
sonic-promotion.us
lpfollower
Trekkie
**
Offline Offline

Posts: 16


WWW
« Reply #61 on: March 03, 2008, 10:38:07 AM »
If you place the robot.txt in the root directory, will it affect the subfolders and subdirectories?
Logged
Pictyors - Pick your Best
Jason2
Intergalactic Superstar
*****
Offline Offline

Posts: 199


WWW
« Reply #62 on: March 07, 2008, 02:18:31 PM »
If you are using vBulletin forum software, then create/add this to your robots.txt:

Code:
User-agent: *
Disallow: /forum/acp/
Disallow: /forum/announcement.php
Disallow: /forum/calendar.php
Disallow: /forum/cron.php
Disallow: /forum/editpost.php
Disallow: /forum/faq.php
Disallow: /forum/joinrequests.php
Disallow: /forum/login.php
Disallow: /forum/member.php
Disallow: /forum/misc.php
Disallow: /forum/mcp/
Disallow: /forum/moderator.php
Disallow: /forum/newreply.php
Disallow: /forum/newthread.php
Disallow: /forum/online.php
Disallow: /forum/printthread.php
Disallow: /forum/private.php
Disallow: /forum/profile.php
Disallow: /forum/register.php
Disallow: /forum/search.php
Disallow: /forum/sendmessage.php
Disallow: /forum/showgroups.php
Disallow: /forum/showpost.php
Disallow: /forum/subscription.php
Disallow: /forum/subscriptions.php
Disallow: /forum/threadrate.php
Disallow: /forum/usercp.php

Change /forum/ to your forums directory.

I won't go into big detail on why you need to add this, but if you look at the above, then you will see that most don't need crawling, such as sendmessage.php -- why let a bot crawl your contactus page, when it can be crawling your threads/posts.

Regards Jason Smile
Logged
"Anyone who has never made a mistake has never tried anything new." -- Albert Einstein
Mitch
Lunarpages Traffic Cop
Senior Moderator
Berserker Poster
*****
Offline Offline

Posts: 7949



WWW
« Reply #63 on: March 10, 2008, 05:50:14 AM »
Very handy list Jason!   Thumbs Up
Logged
Spot On Click Here for the Lunarforums FAQ - Read This BEFORE Posting!

+ Submit Your Site for the Oct 08 Web Site of the Month Award!
+ Also Read the SEPTEMBER 2008 Lunarpages Newsletter!

Mitch the Moderator - Please Respect & Review the Lunarforums Rules Before Posting!
Lunarpages Web Hosting | Lunarpages Forums | Lunarpages Support Wiki
Need something to read?  Try Lunartics & the Lunarpages Newsletter!
Pages: 1 ... 3 4 [5]   Go Up
  Print  
« previous next »
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.6 | SMF © 2006-2008, Simple Machines LLC

Valid XHTML 1.0! Valid CSS! Dilber MC Theme by HarzeM