Message Board

Installing Honey Pots

robots.txt?

Author: X.Robin (30 Sep 10 7:09am)

Should I add a line in robots.txt in order to stop legitimate bots requesting the page?

Re: robots.txt?

Author: D.Daster (11 Oct 10 10:38pm)

I can´t imagine why you should, a legitimate bot would reach the page, but in theory nothing else would happen.

Also, i figure this would be a rather big "red alert" for spammers; if people started adding honeypots to their robots.txt i reckon spammers would just start heeding robots.txt and the problem would be solved.

Re: robots.txt?

Author: R.Waldock (24 Nov 10 5:42pm)

My recommendation is not to include it. That would seriously decrease the effectiveness of your honeypot, since most spam crawlers do heed robots.txt in order to avoid bait addresses.

If you're concerned about the page being indexed in search engines or other archives, don't be: the generated honeypot document includes a robots meta tag in the head section that provides a noarchive,noindex directive. Any legitimate robot heeds this and does not archive the document or include it in its index.

-RW

Re: robots.txt?

Author: C.Smith25 (24 Jan 11 5:06am)

Thanks for this. I've just installed a honeypot and was wondering the same thing - I'll leave robots.txt alone.

Post Edited (24 Jan 11 5:07am)

Re: robots.txt?

Author: M.Ball2 (25 Jan 11 2:54am)

"...since most spam crawlers do heed robots.txt in order to avoid bait addresses."

I wish this was true. Then perhaps I wouldn't be getting comment spam in my guestbook (or at least no one would be trying - at the moment I have sufficient protection in place to stop it all on submission).

I installed my honeypot in the same directory as my guestbook submission form. Trick here was to call the honeypot "guestbook" and change the name of the page for the actual guestbook to something non-typical. The whole directory is disallow in robots.txt but still the spammers come. Fortunately, now all the spammers go to the honeypot and only human users navigating through the website hit the real guestbook.