Message Board

Newbie/Basic Questions

Catching data harvesters, not just email

Author: S.Sanu (29 Dec 15 1:07am)

I am launching a new website with a database with a lot of painstakingly collected data. I don't want this to be crawled and scraped though I do want legitimate crawlers like Google.

It seems like waiting for the email to be spammed could take a long time and they would collect a lot of data from the site. Is there any pro-active way to block. Using currently blocked IPs may not work either. I am looking for honey-trap solutions that lead to hidden pages e.g. that can then be automatically IP detected and the spider blocked instantly. Do this project help in that. Sorry, I'm a newbie here ...

Thanks.

Re: Catching data harvesters, not just email

Author: E.Geier (5 Jan 16 9:53am)

Try out the HTTP Blacklist (http://www.projecthoneypot.org/services_overview.php).
In addition, check out these CDN services:
- https://www.incapsula.com/
- https://www.cloudflare.com/