|
Lupine1647
|
 |
« Reply #15 on: December 20, 2004, 07:41:58 PM » |
|
Works for me:
Result: Page will not be crawled - failed robots.txt check
|
|
|
|
|
Logged
|
|
|
|
|
Lupine1647
|
|
« Reply #16 on: December 20, 2004, 07:43:02 PM » |
|
Ok, i found out that it doesn't work if you put www.domain.combut if you put domain.com/ It works. |
|
|
|
|
Logged
|
|
|
|
|
leighsww
|
|
« Reply #17 on: December 20, 2004, 07:49:22 PM » |
|
Ok, i found out that it doesn't work if you put www.domain.combut if you put domain.com/ It works. Hmmmm, I was using the http://domain.com (NOT the "www") from the beginning and it just doesn't work for me. I'm setting this up for my Dad's site and it's strictly going to be used for client communication and file transferring, so I don't want to have him crawled. I'm gonna have to investigate this further why it isn't working for his site. Thanks for testing it out on your end for me there, Lupus!   |
|
|
|
|
Logged
|
|
|
|
|
Lupine1647
|
|
« Reply #18 on: December 20, 2004, 07:53:27 PM » |
|
Put the slash at the end  He probabley doesn't have the PHp script set up to accept URLs without the slash. |
|
|
|
|
Logged
|
|
|
|
|
leighsww
|
|
« Reply #19 on: December 20, 2004, 07:59:25 PM » |
|
Put the slash at the end He probabley doesn't have the PHp script set up to accept URLs without the slash. That did it!!!  For you!! --> At least now that you've figured that out, anybody trying nibbler's link will not go nuts like I did  |
|
|
|
|
Logged
|
|
|
|
|
Nibbler
|
|
« Reply #20 on: December 20, 2004, 09:25:39 PM » |
|
Hmm, that script seems to have been overwritten by an older version, it should be ok now. The code expects to be given a file to check so just make one up if you want to check a path - it doesn't need to actually exist.
|
|
|
|
|
Logged
|
Missing since 1983  |
|
|
|
JamesG
|
|
« Reply #21 on: December 21, 2004, 01:04:55 AM » |
|
and here's me thinking it was my fault |
|
|
|
|
Logged
|
|
|
|
|
Nibbler
|
|
« Reply #22 on: December 21, 2004, 11:28:28 AM » |
|
It's ok, I'll blame you anyway  |
|
|
|
|
Logged
|
Missing since 1983 |
|
|
|
Lupine1647
|
|
« Reply #23 on: December 21, 2004, 11:30:01 AM » |
|
Everyone blames Gravey and a couple other people.
|
|
|
|
|
Logged
|
|
|
|
|
JamesG
|
|
« Reply #24 on: December 21, 2004, 12:54:49 PM » |
|
i cant blame Nibbler, i may need his help in the future  |
|
|
|
|
Logged
|
|
|
|
|
TranzNDance
|
|
« Reply #25 on: December 21, 2004, 02:41:25 PM » |
|
Yup, we must bow to Nibbler. |
|
|
|
|
Logged
|
|
|
|
|
JamesG
|
|
« Reply #26 on: December 22, 2004, 02:39:41 AM » |
|
better keep him away from Leigh incase she scares him away  |
|
|
|
|
Logged
|
|
|
|
|
varianet
|
|
« Reply #27 on: January 30, 2005, 08:00:29 AM » |
|
Can password protected dirs be accessed by bots  |
|
|
|
|
Logged
|
I need a Java break.
|
|
|
|
Nibbler
|
|
« Reply #28 on: January 30, 2005, 08:06:33 AM » |
|
Maybe, but only if they know the password and support the authentication method.
|
|
|
|
|
Logged
|
Missing since 1983 |
|
|
|
FST2005
|
|
« Reply #29 on: March 05, 2005, 01:23:41 PM » |
|
Hello,
Hope this does not come off like a wrong question, but just want to clear things up.
I have read two things with the robot.txt. One of which is creating a file as mentioned in this posting. The other is to only add the following:
<META name="ROBOTS" content="INDEX,FOLLOW">
It has said that you only need one of them, not both. So which is correct, is having the above pasted code all I need to do the same effect as having a robot.txt file, or not?
Thanks for the time,
|
|
|
|
|
Logged
|
|
|
|
|
Dilber MC Theme by HarzeM