► Stupid and Abusive Sedo Bot ► sedo_parking_robot

[InfoGuy]

91.195.241.219 - - [06/Apr/2009:04:33:22 -0500] "GET / HTTP/1.1" 200 13897 "-" "sedo_parking_robot"
91.195.241.219 - - [06/Apr/2009:04:33:22 -0500] "GET / HTTP/1.1" 200 13897 "-" "sedo_parking_robot"
91.195.241.219 - - [06/Apr/2009:04:33:23 -0500] "GET / HTTP/1.1" 200 13897 "-" "sedo_parking_robot"
91.195.241.219 - - [06/Apr/2009:04:33:23 -0500] "GET / HTTP/1.1" 200 13897 "-" "sedo_parking_robot"
91.195.241.219 - - [06/Apr/2009:04:33:23 -0500] "GET / HTTP/1.1" 200 13897 "-" "sedo_parking_robot"
91.195.241.219 - - [06/Apr/2009:04:33:23 -0500] "GET / HTTP/1.1" 200 13897 "-" "sedo_parking_robot"

91.195.241.219 - - [06/Apr/2009:04:33:24 -0500] "GET /index,follow HTTP/1.1" 404 13904 "-" "sedo_parking_robot"
91.195.241.219 - - [06/Apr/2009:04:33:24 -0500] "GET /index,follow HTTP/1.1" 404 13904 "-" "sedo_parking_robot"
91.195.241.219 - - [06/Apr/2009:04:33:24 -0500] "GET /index,follow HTTP/1.1" 404 13904 "-" "sedo_parking_robot"
91.195.241.219 - - [06/Apr/2009:04:33:24 -0500] "GET /index,follow HTTP/1.1" 404 13904 "-" "sedo_parking_robot"
91.195.241.219 - - [06/Apr/2009:04:33:24 -0500] "GET /index,follow HTTP/1.1" 404 13904 "-" "sedo_parking_robot"
91.195.241.219 - - [06/Apr/2009:04:33:24 -0500] "GET /index,follow HTTP/1.1" 404 13904 "-" "sedo_parking_robot"

91.195.241.219 - - [06/Apr/2009:04:33:25 -0500] "GET /global HTTP/1.1" 404 13904 "-" "sedo_parking_robot"
91.195.241.219 - - [06/Apr/2009:04:33:25 -0500] "GET /global HTTP/1.1" 404 13904 "-" "sedo_parking_robot"
91.195.241.219 - - [06/Apr/2009:04:33:25 -0500] "GET /global HTTP/1.1" 404 13904 "-" "sedo_parking_robot"
91.195.241.219 - - [06/Apr/2009:04:33:25 -0500] "GET /global HTTP/1.1" 404 13904 "-" "sedo_parking_robot"
91.195.241.219 - - [06/Apr/2009:04:33:25 -0500] "GET /global HTTP/1.1" 404 13904 "-" "sedo_parking_robot"
91.195.241.219 - - [06/Apr/2009:04:33:25 -0500] "GET /global HTTP/1.1" 404 13904 "-" "sedo_parking_robot"

91.195.241.219 - - [06/Apr/2009:04:33:25 -0500] "GET /text/html;charset=iso-8859-1 HTTP/1.1" 404 13904 "-" "sedo_parking_robot"
91.195.241.219 - - [06/Apr/2009:04:33:25 -0500] "GET /text/html;charset=iso-8859-1 HTTP/1.1" 404 13904 "-" "sedo_parking_robot"
91.195.241.219 - - [06/Apr/2009:04:33:25 -0500] "GET /text/html;charset=iso-8859-1 HTTP/1.1" 404 13904 "-" "sedo_parking_robot"
91.195.241.219 - - [06/Apr/2009:04:33:25 -0500] "GET /text/html;charset=iso-8859-1 HTTP/1.1" 404 13904 "-" "sedo_parking_robot"
91.195.241.219 - - [06/Apr/2009:04:33:25 -0500] "GET /text/html;charset=iso-8859-1 HTTP/1.1" 404 13904 "-" "sedo_parking_robot"
91.195.241.219 - - [06/Apr/2009:04:33:25 -0500] "GET /text/html;charset=iso-8859-1 HTTP/1.1" 404 13904 "-" "sedo_parking_robot"

1) Why is Sedo's dumb bot hammering my server with multiple requests (6 every time) for the same page from the same IP within 1 second?

2) Sedo's bot is also making numerous unnecessary requests, because it can't differentiate a meta tag from a anchor text link.

I do not appreciate stupid bots that eat up my bandwidth for no good reason. The crawling patterns of this bot lead me to believe that there is no legitimate reason for it crawling my site, other than to spam me with Sedo's parking service. A Sedo representative is more than welcome to shed some light on this matter.

 

[Sonny Banks]

Can we have a Sedo Bot Support? :lol:

 

[mvl]

What kind of sites does it spider?

 

[Theo]

What kind of sites does it spider?

Presumably it checks to see whether the domains are still parked.

 

[SedoCoUk]

1) Why is Sedo's dumb bot hammering my server with multiple requests (6 every time) for the same page from the same IP within 1 second?

2) Sedo's bot is also making numerous unnecessary requests, because it can't differentiate a meta tag from a anchor text link.

I do not appreciate stupid bots that eat up my bandwidth for no good reason. The crawling patterns of this bot lead me to believe that there is no legitimate reason for it crawling my site, other than to spam me with Sedo's parking service. A Sedo representative is more than welcome to shed some light on this matter.

Hello InfoGuy,

I just sent you a PM for more information.

Thanks,

Lisa

 

[BobDiGiTaL]

Lisa, why so secretive? post your response below.

 

[SedoCoUk]

Lisa, why so secretive? post your response below.

Hello everyone,

Yes, we do have checks in place to detect domains that are parked with us. I will certainly post a response to this thread, however at this time I am waiting for further information from InfoGuy.

Lisa

 

[InfoGuy]

Lisa,

My domain is irrelevant. It is the Sedo bot that is not behaving properly.

1) There is no reason for your bot to request the same page 6 times within a 1-second interval. Once every 10 days is considered appropriate for spidering a site.
2) If the Sedo bot is just checking to see if a domain is still parked at Sedo, it should only request Robots.txt and the main index page. There is absolutely no reason for it to deep crawl other pages on a site. Given how poorly Sedo's bot crawls sites, I wouldn't be surprised if it triggered fraudulent clicks on non-Sedo parking pages.

Nevertheless, I have banned Sedo's bot from my server with .htaccess to prevent future issues.

To anyone interested in blocking sedo_parking_robot, add the following to your .htaccess file:

RewriteCond %{REMOTE_ADDR} ^91\.195\.(24[0-1])\. [OR]
RewriteCond %{HTTP_USER_AGENT} sedo_parking_robot
RewriteRule .* - [F,L]

- or -

SetEnvIf Remote_Addr ^91\.195\.(24[0-1])\. ban

<Files ~ "^.*$">
order allow,deny
allow from all
deny from env=ban
</Files>

 

[namenut]

Lisa,

My domain is irrelevant. It is the Sedo bot that is not behaving properly.

1) There is no reason for your bot to request the same page 6 times within a 1-second interval. Once every 10 days is considered appropriate for spidering a site.
2) If the Sedo bot is just checking to see if a domain is still parked at Sedo, it should only request Robots.txt and the main index page. There is absolutely no reason for it to deep crawl other pages on a site. Given how poorly Sedo's bot crawls sites, I wouldn't be surprised if it triggered fraudulent clicks on non-Sedo parking pages.

Nevertheless, I have banned Sedo's bot from my server with .htaccess to prevent future issues.

Nicely done InfoGuy .... note how "Lisa" wants to get more info from InfoGuy before she releases any of her own. Obviously, or so seemingly, not wanting shed light on anything more than she/SEDO absolutely has to. Indicative of the services and practices at SEDOn't. :yes:



nn

 

[SedoCoUk]

Lisa,

My domain is irrelevant. It is the Sedo bot that is not behaving properly.

1) There is no reason for your bot to request the same page 6 times within a 1-second interval. Once every 10 days is considered appropriate for spidering a site.
2) If the Sedo bot is just checking to see if a domain is still parked at Sedo, it should only request Robots.txt and the main index page. There is absolutely no reason for it to deep crawl other pages on a site. Given how poorly Sedo's bot crawls sites, I wouldn't be surprised if it triggered fraudulent clicks on non-Sedo parking pages.

Nevertheless, I have banned Sedo's bot from my server with .htaccess to prevent future issues.

Hello,

As I do not have direct access to the information needed to research your issue I spoke with our Technical Team. In order for them to look into this more closely they would need the domain name to review the logs. From your post it was not clear if this involved one domain or several. Unfortunately without this information we cannot identify the issue to provide you with an answer.

If you would like to PM the domain we can look into this further.


Thanks,

Lisa