Web Hosting Forum | Lunarpages
News: July 14, 2008 - New Contest! - Submit Your WordPress Theme Designs, Win BIG!
September 1, 2008 - Submit Your Web Site for the Sept 2008 Site of the Month!
 
*
Welcome, Guest. Please login or register.
Did you miss your activation email?
September 07, 2008, 04:33:12 PM


Login with username, password and session length


Pages: [1]   Go Down
  Print  
Author Topic: Addon Domain & Robots.txt Question  (Read 928 times)
boatman
Trekkie
**
Offline Offline

Posts: 16


WWW
« on: June 12, 2007, 07:15:03 AM »

I have a question for the group.

I have a primary domain and an addon domain.  The addon domain is a directory off the primary, such as: www.primary.com/addon/.

Question is this: if I write a robots.txt to disallow indexing on the "addon" directory, will that prevent robots from indexing the "correct" domain, www.addon.com?

I definitely want to have the www.addon.com domain indexed, but prefer not to have www.primary.com/addon/ indexed.  Does that make sense?  I just want users to see the correct domain, not the directory under the primary.com domain.
Logged
Mitch
Lunarpages Traffic Cop
Senior Moderator
Berserker Poster
*****
Offline Offline

Posts: 7577



WWW
« Reply #1 on: June 12, 2007, 07:56:24 AM »

Hello - as I understand it - As long as your not linking to yourdomain.com/addon/ anywhere on your primary domain, this should not be an issue because there is no way for the spider to crawl to that address.
Logged

boatman
Trekkie
**
Offline Offline

Posts: 16


WWW
« Reply #2 on: June 12, 2007, 08:01:11 AM »

Ok good; I do not have any links from the primary domain to the primary.com/addon/ directory.  Any links go directly to: www.addon.com

So, does that mean I don't need to use the robots.txt at all, assuming that was the only thing I didn't want indexed?
Logged
Mitch
Lunarpages Traffic Cop
Senior Moderator
Berserker Poster
*****
Offline Offline

Posts: 7577



WWW
« Reply #3 on: June 12, 2007, 08:13:39 AM »

Well it would be helpful for other reasons, but for this I don't think you really need it.  Lets say you had a page filled with images that you wanted to make sure a search engine didn't crawl but is still linked up on your web site.  That would be a time setting up a robots.txt would be helpful.
Logged

boatman
Trekkie
**
Offline Offline

Posts: 16


WWW
« Reply #4 on: June 12, 2007, 08:19:01 AM »

okay, perfect -- that makes sense.

I'm now searching the forums to learn how to create a Google sitemap using Python!  Hopefully that will work like a charm and I'll at least be at a basic level to get indexed properly by Google.
Logged
TranzNDance
Princess of Naboo
Berserker Poster
*****
Offline Offline

Posts: 11809



WWW
« Reply #5 on: June 12, 2007, 07:14:02 PM »

To be on the safe side, you should do a 301 redirect from the addon folder path to the addon domain.
Logged

Grr..!! Luff Ya Grr..!! Luff Ya Grr..!! Luff Ya
MagickCrafter
Long live VI
Intergalactic Superstar
*****
Offline Offline

Posts: 156

VI-VI-VI, the Editor of the Beast


WWW
« Reply #6 on: June 12, 2007, 08:58:30 PM »

Yeah in your htaccess of your /addon/ directory add this:

Code:
RewriteEngine on
RewriteCond %{HTTP_HOST} ^addon\.domain\.com$ [OR]
RewriteCond %{HTTP_HOST} ^domain\.com$ [OR]
RewriteCond %{HTTP_HOST} ^www\.domain\.com$
RewriteRule ^(.*) http://addon.com/$1 [R=301,L]

That should work if I'm not mistaken.
Logged

http://designandprogram.com/

^^ My programming blog ^^
Chrys
Newbie
*
Offline Offline

Posts: 3


« Reply #7 on: June 15, 2007, 04:49:10 AM »

Sorry to come in your thread, but could someone please tell me if I've understood this correctly, because I want to do a similar thing? At the moment I just have a primary domain, and I don't want any part of it indexed. I have a robots.txt in the root directory which disallows everything (User-agent: * Disallow: /). I'm thinking of getting an addon domain, but I would like that to be indexed, as long as there's no way for webcrawlers to index it through the primary domain. The robots.txt I have already would disallow that, is that right? So the addon domain would get indexed, but only through the name www.addon.com? Or would my robots.txt file prevent indexing of the addon too? I find the whole thing a bit confusing (webmistress newbie!), and I'm not sure I've really understood the answers I've seen on the forum. Thanks to anyone willing to set me straight!
Logged
TranzNDance
Princess of Naboo
Berserker Poster
*****
Offline Offline

Posts: 11809



WWW
« Reply #8 on: June 15, 2007, 07:11:23 AM »

You're right, Chrys. The addon won't be indexed via the primary, only via addon.com.
Logged

Grr..!! Luff Ya Grr..!! Luff Ya Grr..!! Luff Ya
Chrys
Newbie
*
Offline Offline

Posts: 3


« Reply #9 on: June 16, 2007, 09:10:33 AM »

That's great - thanks very much for the reply!
Logged
NevadaSam
The Queen of Cut & Paste
Galactic Royalty
*****
Offline Offline

Posts: 213



« Reply #10 on: June 28, 2007, 06:16:40 AM »

I think I am looking for the same thing Boatman is. So instead of starting another thread, I would like to tag on to this one.

For me the main purpose of Primary.com is to host  AddOn.com
So I have
Primary.com/addon_1
Primary.com/addon_2
Primary.com/addon_3

... etc.

Each addon.com has it own robot.txt file. I have very good search engine placement and I don't want to mess anything up. As with posters above, I never use Primary.com/addon to link any sites to each other. I always use addon.com.  Even in the index.html of Primary.com uses a redirect to addon_1.com to move visitors away from my Primary.com space.

This is what I want to do: I want to change my robot.txt file in Primary.com to this:
Code:
# go away
User-agent: *
Disallow: /
Disallow: *.*
I understand this will send all robots away from Primary.com without indexing anything. And from what I read above this will not affect any Addons.com


Sam
;
Logged

Today I ask questions... tomorrow I will have answers.
Pages: [1]   Go Up
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.3 | SMF © 2006-2007, Simple Machines LLC
Seo4Smf v0.2 © Webmaster's Talks


Valid XHTML 1.0! Valid CSS! Dilber MC Theme by HarzeM