Web Hosting Forum | Lunarpages


*
Welcome, Guest. Please login or register.
Did you miss your activation email?



Login with username, password and session length
May 25, 2012, 12:43:34 AM

Pages: [1]   Go Down
  Print  
Author Topic: Spiders  (Read 372 times)
Trail Hunter
Spacescooter Operator
*****
Offline Offline

Posts: 36



WWW
« on: August 01, 2003, 04:32:18 PM »

I'm just curious if anyone has any knowledge of what types of links spiders follow. On one of my pages, I realized I was using a form simply to navigate to another page on the website. I'm guessing that spiders don't follow form buttons so I have since changed them to regular text links just to be safe. Does anyone have any idea on that one?

Second, do spiders follow links that have variables in them for example, 'index.php?all=true'? I'm just curious because I fear that much of my site isn't getting indexed because of this. Does this simply vary between search engines?  Thanks so much.
Logged

http://www.4x4TrailHunters.com - 4x4 Trails, Forum and Reader's Rigs
TWebMan
Quantum Encyclopedia Writer
*****
Offline Offline

Posts: 3112



WWW
« Reply #1 on: August 02, 2003, 08:15:36 AM »

Spiders will generally follow links, and may follow query strings, though I don't believe they do the query strings all that much these days.

http://www.robotstxt.org/wc/robots.html
Logged

"Computers cause people to make more mistakes than any other invention in history, with the possible exception of handguns and tequila."  - Unknown
"Liberty of any kind is seldom lost all at once." - D. Hume
Every day is an Ode to Joy
The planet will be fine... and so will your site
Adrian
Galactic Royalty
*****
Offline Offline

Posts: 283



WWW
« Reply #2 on: August 02, 2003, 11:42:54 AM »

In so far as I am aware from my own experience with google, dynamic URLs will only be followed if the links are on 'ordinary' non-dynamic pages.

As long as you have a sitemap HTML page then you should get everything indexed.
Logged

Pages: [1]   Go Up
  Print  
 
Jump to: