Message Board

http:BL Use/Development

Repel http:BL for Apache

Author: T.Norderhaug (20 Jan 09 2:40am)

Repel http:BL is used with the Apache webserver to identify friendly search engines and detect malicious web bots:

http://repel.in-progress.com/

The application is free and comes with open source licensed under LGPL. It is activated using Apache rewrite rules and works with Python 2.3.5 or later. Editing the code of the implementation is not required.

Repel decodes the http:BL responses into a format suitable for regular expression matching in Apache rewrite conditions. It provides keywords for the different variations of bots, and reports how many days the bots have been dormant and to what extent they represent a threat. Common search engines are also labeled.

Repel can create a log of IP addresses and decoded http:BL responses. This log is in the same format as used by text based Apache rewrite maps. The application can optionally be used for batch processing.

Repel caches DNS queries to minimize redundant lookups. Key parameters can be set on the command line to optimize performance and tailor the functionality. The conversion rules to decode the http:BL responses are customizable by modifying regular expressions in an external file.

Re: Repel http:BL for Apache

Author: J.Yard2 (27 Feb 09 5:38pm)

Windows XP SP3
Python 2.6.1

C:\Repel>repel.py
Traceback (most recent call last):
File "C:\Repel\repel.py", line 63, in <module>
import syslog
ImportError: No module named syslog

C:\Repel>

Python 3.0.1 also throws a syntax error on line 199.

Re: Repel http:BL for Apache

Author: J.Yard2 (28 Feb 09 12:14am)

Won't the following create a rewriting loop?

RewriteCond ${repel:%{REMOTE_ADDR}|OK} Harvester|CommentSpammer
RewriteRule ^.* /cgi-bin/honeypot.py [L]

Request some page
RewriteCond true
Rewrite to /cgi-bin/honeypot.py
Request /cgi-bin/honeypot.py
RewriteCond true
Rewrite to /cgi-bin/honeypot.py
Request /cgi-bin/honeypot.py
and so on.

Maybe the thing to do would be to add some exclusion options.
exclude_path
exclude_file_name
exclude_file_extension

Post Edited (28 Feb 09 12:32am)

Re: Repel http:BL for Apache

Author: T.Norderhaug (28 Feb 09 2:32am)

Thanks for the feedback!

Syslog is a UNIX module and is not available in Python on Windows. It is not actually needed, so the following line can be removed from the repel.py file:

import syslog

The syntax error in line 199 is due to a change in Python from version 2.x. The line reads:

except Exception, ex:

In Python 3.x, it should instead be:

except Exception as ex:

I will add fixes for these issues in the next version of the Repel distribution.

Re: Repel http:BL for Apache

Author: T.Norderhaug (28 Feb 09 2:41am)

The following Apache rewrite instruction for Repel will NOT create a rewriting loop:

RewriteCond ${repel:%{REMOTE_ADDR}|OK} Harvester|CommentSpammer
RewriteRule ^.* /cgi-bin/honeypot.py [L]

The reason is that the [L] (last) flag tells Apache to stop rewriting if the rule was applied.

Re: Repel http:BL for Apache

Author: J.Yard2 (28 Feb 09 3:40am)

It creates a rewite loop here, because the rule is then applied again to the rewritten uri.

Adding an additional RewriteCond to exclude (!) the new file from the rewrite fixes it though.

RewriteCond ${repel:%{REMOTE_ADDR}|OK} Harvester|CommentSpammer
RewriteCond %{REQUEST_URI} !^.*/cgi-bin/honeypot.py(/)?.*$ [NC]
RewriteRule ^.* /cgi-bin/honeypot.py [L]

P.S. tried posting in your support fourm but 'post' page comes up blank.

Update:
Located cause of rewrite loop. Page I'm rewitting to contains some additional items, so when the client request them the rewrite is invoked and sends the page again. And the cycle repeats...

Post Edited (28 Feb 09 12:30pm)

Re: Repel http:BL for Apache

Author: J.Yard2 (28 Feb 09 2:54pm)

How can the http:BL response code be obtained from Repel with PHP?
The one last item I need before moving to my produciton site.

Re: Repel http:BL for Apache

Author: T.Norderhaug (1 Mar 09 4:38am)

To pass the Repel http:BL response code from Apache to PHP, you may consider reading up on RewriteCond backreferences and how these can be used in the Apache rewriterule to forward values from the condition.