yahoo crawler

[csitenet]

Woo, well. Yesterday i noticed 900 people on my forum which has about 10-15 on average online usually... then i checked the ip. all 66.196.**.***

I thought it was a ddos. had to trace each ip one by one because most just failed! finally found it, Yahoo, crawling my site. i thought when a search engine crawls your site / forum they use 1 ip, not like 100 to check all pages at once.

Anyone else have this issue?

 

[Domagon]

See the same thing all the time from Google too. The big SEs use numerous servers for their crawlers and thus the numerous IPs you're seeing.

If doing filtering of SEs, you need to do so by IP ranges, not by individual IPs.

On a related topic ... one can NOT trust the data returned when doing a reverse lookup on an IP unless one also does a forward lookup (*assumes the reverse isn't cached; that can sometimes lead to false results on some systems).

Best way to determine who controls an IP address/range is to use ARIN http://www.arin.net/ ,or one of its counterparts APNIC, RIPE, etc, to lookup IP delegation - important note: the results shown often won't indicate exactly who is currently using an IP, but rather who they are delegated to - that gives folks a point of contact to research an IP/range further.

Ron

 

[OgieOgalthorpe]

Its quite common at many forums.