I wanted to set up a honey pot for bots which intentionally crawl links which are disallowed in the robots.txt file. I have no idea if there are any, I bet there are.

The implementation is simple, add Disallow: /bad-bots to the /robots.txt file and log the user agents which visit. Additionally, I have included a visually hidden link to the honey pot with rel="nofollow"

Now we wait...

Hall Of Shame

What did we catch?

2    Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/605.1.15 (KHTML, like Gecko) Chrome/89.0.4389.82 Safari/605.1.15
2    Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36
2    Mozilla/5.0 (X11; Linux i686; rv:146.0) Gecko/20100101 Firefox/146.0
1    Mozilla/5.0 (X11; Linux x86_64; rv:146.0) Gecko/20100101 Firefox/146.0
1    Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:146.0) Gecko/20100101 Firefox/146.0
1    Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36
1    Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:132.0) Gecko/20100101 Firefox/132.0
1    Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36
1    Mozilla/5.0 (compatible; archive.org_bot +http://archive.org/details/archive.org_bot) Zeno/d3712ae warc/v0.8.99