Message Board

Tracking Harvesters/Spammers

harvester with a rotating user agent

Author: J.Westmark (10 May 05 1:45pm)

I just detected a bot/harvester that rotated it's user agent making it impossible to filter out using htaccess.

The bot IP address is 205.209.134.60 (which I blocked the entie range). I've filed a complaint with their ISP and will follow it up with a phone call but I've never seen one do this before. I also use spider/bot traps on my site and it didn't trip any of them.

-jay

Below are a few entries from my log file:

205.209.134.60 - - [10/May/2005:12:53:29 -0400] "GETml HTTP/1.1" 302 - "-" "Mozilla/4.8 [en]C-BMY (Windows NT 5.0; U)"
205.209.134.60 - - [10/May/2005:12:53:35 -0400] "GET 48974 "-" "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; METL)"
205.209.134.60 - - [10/May/2005:12:53:37 -0400] "GET 200 50194 "-" "Mozilla/4.0 (compatible; MSIE 5.5; Windows 95; FREETELECOM)"
205.209.134.60 - - [10/May/2005:12:53:38 -0400] "GETTTP/1.1" 200 50553 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Linux 2.4.20 i686) Opera 6.1 [en]"
205.209.134.60 - - [10/May/2005:12:53:44 -0400] "GETP/1.1" 200 57459 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Win 9x 4.90; SYMPA; Hotbar 4.3.1.0)"
205.209.134.60 - - [10/May/2005:12:53:59 -0400] "GEThtm HTTP/1.1" 200 46519 "-" "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; T312461; IE55232Ba)"
205.209.134.60 - - [10/May/2005:12:53:59 -0400] "GET 200 51308 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; version6)"
205.209.134.60 - - [10/May/2005:12:54:07 -0400] "GET0 51109 "-" "Mozilla/4.0 (compatible; MSIE 5.5; Windows 95; Wanadoo 5.2; Wanadoo 5.5)"
205.209.134.60 - - [10/May/2005:12:54:07 -0400] "GET1458.htm HTTP/1.1" 200 60453 "-" "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; Wanadoo 5.6)"
205.209.134.60 - - [10/May/2005:12:54:27 -0400] "GET 200 50315 "-" "Mozilla/4.0 (compatible; MSIE 4.01; Windows 98; DigExt)"

Re: harvester with a rotating user agent

Author: C.Dijkgraaf (11 May 05 1:29am)

I've seen such behaviour before, the one I observed came from 217.20.113.110
and 217.20.114.85 (both netdirekt.de). It was also fairly dumb, as it would follow anchor URL's and fetch the same page multiple times. For 207 hits it used 70 unique user agents. Same examples below.

217.20.114.85 - - [02/Apr/2005:01:35:42 +1200] "GET /klootwijk.htm#63 HTTP/1.1" 200 14022 "-" "Mozilla/5.0 (Windows; U; Win95; de-AT; rv:1.6) Gecko/20040113"
217.20.114.85 - - [02/Apr/2005:01:35:43 +1200] "GET /heuvel.htm#15 HTTP/1.1" 200 18551 "-" "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; DT)"
217.20.114.85 - - [02/Apr/2005:01:35:44 +1200] "GET /pligt.htm#122 HTTP/1.1" 200 4650 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Q312461)"
217.20.114.85 - - [02/Apr/2005:01:35:45 +1200] "GET /legerste.htm#242 HTTP/1.1" 200 16624 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de-DE; rv:1.4)"
217.20.114.85 - - [02/Apr/2005:01:35:46 +1200] "GET /legerste.htm#243 HTTP/1.1" 200 16624 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Win 9x 4.90)"
217.20.114.85 - - [02/Apr/2005:01:35:47 +1200] "GET /dijkgraa.htm#1a HTTP/1.1" 200 17596 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; DT)"
217.20.114.85 - - [02/Apr/2005:01:36:06 +1200] "GET /rolffs.htm#11 HTTP/1.1" 200 3615 "-" "psbot/0.1 (+http://www.picsearch.com/bot.html)"