Message Board

Tracking Harvesters/Spammers

Learning bots

Author: C.Desguin (17 Sep 12 4:09am)

In addition to the HoneySpot plugin for Spip, I use an automatic blocking script that is based upon the number of hits per minute. I can see in the log of this blocking script that some bots, after having been blocked once, are apparently able to change their IP then reset the hit counter every ten hits, so they never reach the blocking limit. An example is 180.62.xxx.xxx (Japan). Maybe there is a human behind the screen...

Re: Learning bots

Author: E.Langheinrich (17 Sep 12 9:07am)

Interesting.

When a visitor hits the block limit what do you do with their requests?

Re: Learning bots

Author: C.Desguin (17 Sep 12 11:48am)

Simple: any IP that hits more than "x" pages of the site within 60 seconds is appended to a blacklist table. Spip bans all blacklisted IP's and returns a message "Your IP is banned for abuse". There is a link allowing the visitor to ask for removal from the blacklist, if she/he is actually a bona fide human.

The "x" parameter is currently set to 15: human visitors are not supposed to hit more than 15 pages per minute. Nobody sent a removal request so far; however, more than 1,000 IP's have been blocked over the past six months.

Note: IP's belonging to well-known search engines are not blocked.

List of 978 blacklisted IP's: http://is.gd/palaqu (some were removed manually)

Post Edited (17 Sep 12 11:41pm)

Re: Learning bots - addendum

Author: C.Desguin (28 Oct 12 10:31am)

Remark: unfortunately, this extra protection apparently prevents the HoneyPot plugin from detecting and denouncing some unwanted IP's, as they are blocked by the script before they are caught by the HoneyPot plugin.

However, I will keep my little script, as I experienced some undesirable visits before I installed it, in spite of the previously installed HoneyPot protection.