Google and Yahoo Slurp spiders. Robot text
 

Running vb 4.15

Integrated Photopost 7.1

Use Spider-friendly URLs? = Yes

The Google and Yahoo Slurp spiders seem to be hanging up trying to index our Photopost and Reviewpost. What directories and files can we block them from? Right now I've got /photopost/ and /reviewpost/ disallowed in our robot .txt file and that doesn't seem to be blocking them.

Bing, Lycos, Google FeedFetcher Spider and other spiders seem to be indexing our forum okay.

Thanks

The robots.txt file in our application lists which files not to allow indexing of.

I know nothing of google or yahoo spiders on how they index things so I doubt I could tell you anything about those spiders. That would be something you can ask google or yahoo on there stuff.

Our application is working urls are formatted correctly and all content is viewing correctly right?

Thanks Chuck. Yes, Photopost and Reviewpost are working fine, I'm just seeing those spiders spending what seems like a lot of time trying to index in there.

yeah I hear I just can not tell you anything about those spiders not our area of expertise we did not write them. ;)