Web Hosting Forum | Lunarpages
News: July 14, 2008 - New Contest! - Submit Your WordPress Theme Designs, Win BIG!
September 1, 2008 - Submit Your Web Site for the Sept 2008 Site of the Month!
 
*
Welcome, Guest. Please login or register.
Did you miss your activation email?
September 07, 2008, 09:49:55 PM


Login with username, password and session length


Pages: 1 [2]   Go Down
  Print  
Author Topic: Web listings I don't want  (Read 1358 times)
Martijn
Intergalactic Superstar
*****
Offline Offline

Posts: 180


WWW
« Reply #15 on: April 27, 2006, 01:22:18 PM »

Are you sure Google re-indexed your site? If you want to be sure, change the date at the bottom of your page (the part where it says "It was last updated on April 19, 2006") to today. As soon as the cached version of your page at Google displays the new date, you're sure that the Googlebot came across your site.

Your robots.txt file seems to be fine, so I guess you'll just have to wait a little longer.
Logged
SteveW
Master Jedi
*****
Offline Offline

Posts: 1394


WWW
« Reply #16 on: April 27, 2006, 01:32:00 PM »

When people say that Google has "done a crawl", they're speaking rather loosely, since Google obviously can't start at one end of the web and keep going until they reach the "other end".  Their crawlers are constantly visiting various websites, and with different frequencies for different websites, so the fact that they've visited this forum thread isn't that significant.  You might check your usage logs to see if they've recently re-encountered the area of your site that contains the pages you don't want crawled.  If they crawled the other pages from that area, but not the ones you wanted excluded, then maybe robots.txt is working.  Even so, it's only been about a week since your original post, and I would think it might take several weeks to two months for your pages to drop out, assuming that it takes about the same amount of time for them to drop out as to get in in the first place.  Maybe even longer, since obviously they would be worried about too quickly dropping pages that really do still exist, due only to a server error at the time they crawled. 
Logged





Mt. Shasta
photo gallery.


Don't forget Lunarpages 24/7/365 support documentation:
Flash Tutorials, Knowledge Base FAQ Articles, cPanel Manual, Glossary/Dictionary, Support Tickets,
and
Forum Search.

Tom Veik
Intergalactic Cowboy
*****
Offline Offline

Posts: 61



« Reply #17 on: February 10, 2007, 05:23:50 PM »

My problem is that I have many outdated dead links that are still in the Google index that have been dead for close to four months now.  That Google help page says that dead links are removed automatically.  Maybe, but it sure is a slow process.  I get hundreds of 404 errors from search engines each week trying to index my dead links.
It's been over a year now since I totally revamped my website.  I deleted hundreds of HTMLs and jpgs near the end of 2005 and put in new content.  There are no erroneous links in my new content to that old deleted stuff.  The search engins are still trying to index that old stuff that was deleted more than a year ago.  It's tapered off quite a bit, but they're still trying to index some of it.  Why?  It makes my 404 error report just about useless.  The worst offender is Inktomi.
Logged
SteveW
Master Jedi
*****
Offline Offline

Posts: 1394


WWW
« Reply #18 on: February 10, 2007, 09:31:17 PM »

The answer I've seen elsewhere, at least regarding Google, is that they are very reluctant to purge URLs that they know once existed. Ok, make that very, very reluctant. It could also have to do with their making backup archives and failing to purge those. If they ever have to revert to the archive for some reason, your URLs are resurrected. In a previous post I mentioned "several weeks to two months". Since then I saw a complaint somewhere that URLs that had been dead for 2 years were still being crawled.
Logged





Mt. Shasta
photo gallery.


Don't forget Lunarpages 24/7/365 support documentation:
Flash Tutorials, Knowledge Base FAQ Articles, cPanel Manual, Glossary/Dictionary, Support Tickets,
and
Forum Search.

Pages: 1 [2]   Go Up
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.3 | SMF © 2006-2007, Simple Machines LLC
Seo4Smf v0.2 © Webmaster's Talks


Valid XHTML 1.0! Valid CSS! Dilber MC Theme by HarzeM