Message Board

Installing Honey Pots

Older Posts ]   [ Newer Posts ]
 Should we add our honeypot in robots.txt?
Author: A.Bv   (16 Jul 13 11:39am)
I have a honeypot script up on our site - it's been catching bad guys for years. Today I was reviewing our robots.txt and it occurred to me, "Hey - should I put this file in the robots.txt as an exclusion?" My guess is that I should not - it's already linked from multiple pages using the techniques you suggest - but I thought I'd ask.

 Re: Should we add our honeypot in robots.txt?
Author: A.Bv   (17 Sep 13 1:16pm)
 Re: Should we add our honeypot in robots.txt?
Author: H.User1325   (17 Sep 13 4:05pm)
Suggested reading:

 Re: Should we add our honeypot in robots.txt?
Author: R.Readhimer   (12 Oct 13 12:29am)
 Re: Should we add our honeypot in robots.txt?
Author: S.Byrne   (25 Jun 14 2:06am)
As many forums and blogs add the login URLs to the robots.txt file to keep out legit bots such as Google and Bing, generally spam bots check the those links anyway, since they are not going to avoid spamming a forum just because the robots.txt file says bots are not allowed to access the login link.

So in theory, the same holds true where the honeypot URL is in the robots.txt file.

As I'm sure many spammers are aware that forums place login and comment links in the robots.txt file, putting the honeyput URL in the robots.txt file could actually lead to bots landing in the honeypot that test every URL in the robots.txt file, that otherwise may not follow empty 'a href' links or links it determines are not visible.

If you do this, you will likely need to add a rel="nofollow" to every link pointing to the honeypot. Although Google does not follow links blocked by the robots.txt file, it does tend to index pages within those locations that are linked to, so could end up leading to Google indexing the honeypot link with some sort of title, since it would not be able to crawl that page to see the 'noindex' header. Should this happen, the rel="nofollow" should stop it getting any ranking value.

do not follow this link

Privacy Policy | Terms of Use | About Project Honey Pot | FAQ | Cloudflare Site Protection | Contact Us

Copyright © 2004–17, Unspam Technologies, Inc. All rights reserved.

contact | wiki | email