Web Hosting Forum | Lunarpages


*
Welcome, Guest. Please login or register.
Did you miss your activation email?


Login with username, password and session length
May 25, 2012, 12:44:56 PM

Pages: [1]   Go Down
« previous next »
  Print  
Author Topic: robots.txt question  (Read 346 times)
h00
Newbie
*
Offline Offline

Posts: 4
« on: July 19, 2011, 10:49:36 AM »
I have multiple domains hosted at lunarpages in subdirectories. I want to allow search engines at these domains. BUT I don't want the main/root domain to be indexed. If I block search engines at the root domain, can the other domains be indexed still if they have their own urls?

Here's the tree: mymaindomain has myregulardomain1 and myregulardomain2 as subdirs the way it works at lunarpages.


mymaindomain.com (I want to block search engines)
myregulardomain1.com (search engines OK)
myregulardomain2.com (search engines OK)
Logged
wektech
Master Jedi
*****
Offline Offline

Posts: 1031



WWW
« Reply #1 on: July 19, 2011, 11:42:24 AM »
To quote google "When a spider finds a URL, it takes the whole domain name (everything between 'http://' and the next '/'), then sticks a '/robots.txt' on the end of it and looks for that file. If that file exists, then the spider should read it to see where it is allowed to crawl".

So this should mean that if you tell robots.txt to not index anything found at mymaindomain.com it would not index mymaindomain.com/myregulardomain1/ but it would index the exact same content at myregulardomain1.com because it would not find the robots.txt existant in the invisible parent directory.
Logged
Money savings tips for real people by real people!
Pages: [1]   Go Up
  Print  
« previous next »
 
Jump to: