Web Hosting Forum | Lunarpages


*
Welcome, Guest. Please login or register.
Did you miss your activation email?


Login with username, password and session length
February 09, 2012, 05:04:19 PM

Pages: [1]   Go Down
« previous next »
  Print  
Author Topic: Googlebot stopped crawling my web site  (Read 1873 times)
SitNGo Wizard
Newbie
*
Offline Offline

Posts: 1
« on: November 07, 2008, 12:27:34 PM »
Around the middle of October, Googlebot stopped crawling my web site. The result is that my web site has now dropped completely out of Google search results.

I submitting a sitemap to Google webmaster tools to help track the problem. Google reported a ” Network unreachable: robots. txt unreachable” error. The robots.txt file exists and had 577 permissions. Plus the robots.txt file has not changed in over a year.

I asked for help in the Google webmaster forums. They suspect that Lunarpages is blocking or limiting Googlebot. Lunarpages support says they do not block Googlebot in any way.

Does anybody here have a suggestion I can try to track down the problem?
Logged
Mitch
Berserker Poster
*****
Offline Offline

Posts: 12838


WWW
« Reply #1 on: November 07, 2008, 12:40:25 PM »
It is very unlikely that we would be blocking access for Googlebot to you for any reason at all.  One thing you noticed, that I wanted to help you correct is your robots.txt file should be set to 644, not 577.  It does need executable permissions. 

Have you checked your web site logs for the exact date and time to see when that stopped hitting your site with their robots? 

I would also suggest signing up for Google webmaster tools, from there - Google can provide you directly with more information if they have any issues or errors when crawling your web site. 

http://wiki.lunarpages.com/Google_Webmaster_Tools
Logged
New to Web Site Hosting? Check Out the Lunarpages Blog Hosting Guide!

Follow us @lunarpages on Twitter!
Important Threads: Read This Before Posting! | Lunarforums Rules! | Mitch's Link of the Day!
Also, be sure to check out and subscribe to the Lunartics Blog and the Lunarpages Newsletter !

Need Web Hosting Help? Check out the Lunarpages Web Hosting Wiki. It has tons of tips, tutorials and resources!
cemedia
Intergalactic Superstar
*****
Offline Offline

Posts: 160


WWW
« Reply #2 on: November 14, 2008, 07:43:51 AM »
I have a similar issue with the sites on a basic plan , sitemaps are submitted through the google webmaster tools and the error is a time out/not reachable

On my business plan and dedicated plan the issue doesn't exist. Submitted sitemaps get fetched by googlebot without any problems.

When this did occur on the basic plan, since the troublesome move to the SD datacenter.
Logged
fooey
Trekkie
**
Offline Offline

Posts: 19
« Reply #3 on: November 15, 2008, 07:18:02 AM »
for many of you your robots.txt file should look like:

User-agent: *
Disallow:
sitemap: http://www.YOURSITE.com/sitemap.xml
Logged
cemedia
Intergalactic Superstar
*****
Offline Offline

Posts: 160


WWW
« Reply #4 on: November 15, 2008, 08:51:07 AM »
@fooey

Search engines get pinged when a new sitemap is available, they are also listed in the google webmaster tools, so google 'tries' to fetch them every day. And my robots.txt files are ok don't worry.

Now from google webmaster tools, crawl info.



That's def not normal and the error google gives is a time out

All this since the move to the SD datacenter, as it looks like I'm not the only one affected by this.
« Last Edit: November 15, 2008, 09:02:58 AM by cemedia » Logged
fooey
Trekkie
**
Offline Offline

Posts: 19
« Reply #5 on: November 15, 2008, 08:59:15 AM »
how old is your site?
Logged
cemedia
Intergalactic Superstar
*****
Offline Offline

Posts: 160


WWW
« Reply #6 on: November 15, 2008, 09:05:29 AM »
majority of my sites are +3y old

the ss of the above one, is like a 30 months old site

So my sites are known by google.
Logged
fooey
Trekkie
**
Offline Offline

Posts: 19
« Reply #7 on: November 15, 2008, 09:17:01 AM »
well if you have had regular activity from google for 30 months, then a sudden drop off something is up.

it could be LP it could be something to do with your site. hard to say.

is google still having trouble getting to your sitemaps? if so you may want to make 100% sure that it's setup properly.

then make sure your robots.txt is a-ok. a simple typo could easily torpedo your site. also check permissions.

if everything comes back okay, i'd then run xenu link sleuth on your site just make sure it wasn't hacked. also run a "site:YOURSITE.com" in google to make sure the right pages are indexed.

at a last resort maybe try putting up an add-on domain/site on that account/server and see if you get the same results...
Logged
cemedia
Intergalactic Superstar
*****
Offline Offline

Posts: 160


WWW
« Reply #8 on: November 15, 2008, 09:43:15 AM »
all tried before and yes the 10 add-on domains on that account suffer also from the same issue

so it's def something to do with the Lunarpage SD Data center imo
Logged
fooey
Trekkie
**
Offline Offline

Posts: 19
« Reply #9 on: November 15, 2008, 09:59:43 AM »
well, if you are finding the same results on other add-on sites then sounds like something got screwed up somewhere. it still could turn out to be a issue on your end but obviously email support with your facts and findings.
Logged
cemedia
Intergalactic Superstar
*****
Offline Offline

Posts: 160


WWW
« Reply #10 on: November 20, 2008, 04:15:38 PM »
My issue got fixed, the firewall of the LP server was blocking google ....
Logged
valdeam
Pong! (the videogame) Master
*****
Offline Offline

Posts: 20
« Reply #11 on: December 10, 2008, 08:01:18 PM »
What was it doing to block Google??
Logged
"If you always do what you've always done, how can you ever be any better than you've always been?" 
Pages: [1]   Go Up
  Print  
« previous next »
 
Jump to: