Wanted: Service spiders, how to block?

[stevey]

hi, was wondering if there was a way to stop spiders picking up urls such as:

www.mysite.com/index.php?url=fdsf.sdfer,sader
http://www.mysite.com/index.php?url=jjj.tbbtyr.pbz

(billions of combinations of /index.php?url=*code* so blocking each individually isnt a realistic option)

yet i still want the spiders to crawl www.mysite.com/index.php

anyone have any surgestions or ideas that would work?

thanks,
steve

i'll give 1,000 DNF$ to anyone who can solve this problem

 

[DogFaceBoy]

In your Robots.txt just disallow:

http://www.mysite.com/index.php?url=*

 

[stevey]

thanks for the help

 

[DogFaceBoy]

Np thanks for payment .

 

[kokopelli]

I'm not too sure your suggested robots syntax is correct. I did a test robots.txt file as per your suggestion and ran it through an online validator and this is what I got:

Disallow: http://www.mysite.com/index.php?url=*
The "*" wildchar in file names is not supported by (all) the user-agents addressed by this block of code. You should use the wildchar "*" in a block of code exclusively addressed to spiders that support the wildchar (Eg. Googlebot).
You can't use an absolute URL. Please remove the "http://" and the domain name and insert just a file/directory full path, starting from the root directory (Example: /pagename.html).

The Disallow field has an inherent wildcard nature. The standard dictates that /bob would disallow /bob.html and /bob/index.html (both the file bob and files in the bob directory will not be indexed). Another example, Disallow: /help disallows both /help.html and /help/index.html, whereas Disallow: /help/ would disallow /help/index.html but allow /help.html

So perhaps your robots.txt file should rather just read:

User-agent: *
Disallow: /index.php?

References:

• http://www.robotstxt.org/wc/norobots.html
• http://www.searchengineworld.com/robots/robots_tutorial.htm