|
Parity
|
 |
« on: April 19, 2006, 09:03:54 AM » |
|
Strangely, Google is not only listing my Web site but also listing my index file, forum, statistics about my forum, etc. All I want is my Web site. How can I get rid of all these extra listings, which are nobody's business and just clutter up the page?
|
|
|
|
|
Logged
|
|
|
|
|
TranzNDance
|
|
« Reply #1 on: April 19, 2006, 09:15:03 AM » |
|
Do you have example listing? I'm not sure what you want. Your index file is part of your web site. There is code you can put in meta tags or in robots.txt so that robots behave a certain way but you need to be more specific.
I don't see how it's strange. A lot of website owners wish more of their web sites were indexed by the search engine.
|
|
|
|
|
Logged
|
|
|
|
|
|
|
Martijn
|
|
« Reply #3 on: April 19, 2006, 11:13:23 AM » |
|
If you don't want your forum to be indexed, create a file called robots.txt, and place it in your public_html directory. Add the following to this file to stop search engines from storing your forum contents:
User-agent: *
Disallow: /forum/
|
|
|
|
|
Logged
|
|
|
|
|
Parity
|
|
« Reply #4 on: April 19, 2006, 11:31:41 AM » |
|
Thanks so much for this information! I'll try it this evening. I've got to go back to work now.
|
|
|
|
|
Logged
|
|
|
|
|
Parity
|
|
« Reply #5 on: April 19, 2006, 03:40:53 PM » |
|
Hi, this is Parity again.
I tried what you suggested, but it doesn't seem to have worked. How long does it take to work? Then I read in another thread that you have to make the robots.txt file in Notepad. I've now done that, but I don't know how to upload the Notepad file into my public.html file. The Notepad file doesn't show up with my other files in the left-hand side of WS-FTP LE.
Thanks for your help.
|
|
|
|
|
Logged
|
|
|
|
|
bryantrv
|
|
« Reply #6 on: April 19, 2006, 03:45:45 PM » |
|
That only works to keep the next crawl of the website out, not on already crawled (indexed) information. You need to look at http://www.google.com/webmasters/remove.html for info on removing already crawled stuff. |
|
|
|
|
Logged
|
|
|
|
|
Parity
|
|
« Reply #7 on: April 19, 2006, 04:07:03 PM » |
|
Thanks. I finally located and uploaded by Notebook file. I'll just have to be patient until the next crawl, because I don't know how to to Meta tags.
|
|
|
|
|
Logged
|
|
|
|
|
easycall
|
|
« Reply #8 on: April 20, 2006, 12:35:04 PM » |
|
Since the pages are already indexed, putting up a robots.txt file will not remove them. It will just keep the bots from re-indexing them. You need to use Google's remove tool to get the pages removed, but be very careful or you might accidentally get your whole site removed.
|
|
|
|
|
Logged
|
|
|
|
|
Martijn
|
|
« Reply #9 on: April 20, 2006, 12:57:51 PM » |
|
Google will remove them at the next crawl. Bots always scan for robots.txt and follow the rules. This is actually the only way to be pretty sure the listings will be removed.
|
|
|
|
|
Logged
|
|
|
|
|
Tom Veik
|
|
« Reply #10 on: April 20, 2006, 06:19:51 PM » |
|
My problem is that I have many outdated dead links that are still in the Google index that have been dead for close to four months now. That Google help page says that dead links are removed automatically. Maybe, but it sure is a slow process. I get hundreds of 404 errors from search engines each week trying to index my dead links.
|
|
|
|
|
Logged
|
|
|
|
|
bryantrv
|
|
« Reply #11 on: April 21, 2006, 06:50:33 AM » |
|
You do have a custom 404 page set up, don't you? One with navigation leading back in to the main site.
I changed a bunch of urls a while ago (from .htm to .html ), and put up permanent redirects, which didn't take long to propagate.
|
|
|
|
|
Logged
|
|
|
|
|
Tom Veik
|
|
« Reply #12 on: April 21, 2006, 06:31:11 PM » |
|
You do have a custom 404 page set up, don't you? One with navigation leading back in to the main site.
Yes on both counts. And it does properly return the 404 error in the header by the way. |
|
|
|
« Last Edit: April 21, 2006, 06:39:30 PM by tlveik »
|
Logged
|
|
|
|
|
easycall
|
|
« Reply #13 on: April 21, 2006, 10:24:07 PM » |
|
Recently Google tried to index a page that I haven't had for about 4 years. It is a problem that many have reported on Webmasterworld.com - When I find Google trying to find a page that doesn't exist, I usually put up a 301 redirect to either the new page, or one that is similar. That solves the problem as long as you keep the 301 redirect. I had that redirect up 4 years ago, but took it down sometime since then and Google still went looking for it again just this month. Very weird.
|
|
|
|
|
Logged
|
|
|
|
|
Parity
|
|
« Reply #14 on: April 27, 2006, 12:02:44 PM » |
|
Hii, I'm the one who started this thread. Can you believe that Google now shows a link to this thread? Obviously, they've done a crawl since I posted it, but my robots.txt file seems not to have worked: I'm still getting unwanted listings. I put it into Notepad and uploaded it into my public.html file as an ASCI file. I have only one page (unless the forum counts as a page). Would redirecting even help here? Exactly how do you do it?
|
|
|
|
|
Logged
|
|
|
|
|
Dilber MC Theme by HarzeM