IP Address Inspector

J.Doe48 commented...

My logs are filled to the rim with the blocked MJ12bot (by robots.txt).

Conclusion:
-MJ12 does not obey robots.txt
-spoof bots pretending to be MJ12

unfortunately this doesn't work:
RewriteCond %{HTTP_USER_AGENT} MJ12bot [NC]
RewriteRule ^.* - [F,L]
December 02 2012 10:40 AM

F.Mertz commented...

A bot from this IP address disobeyed my robots.txt file on 2012-08-06. It didn't actually make a request for robots.txt according to my HTTP server logs. I informed MajesticSEO about it, suggesting that perhaps their bot encountered a network error and so shouldn't have gone anywhere near my site until robots.txt was either successfully retrieved or 404'd, and their Alex Chudnovsky advised me: "it's just if bot could not get robots.txt then it could not obey it :(".

My take on it is that Alex knows the crawler is broken but doesn't care to fix it.
August 07 2012 06:22 PM

B.Simpson2 commented...

It doesn't hurt to also notify all those companies hosting Majestic.
I'm sure the hosts don't appreciate the waste on their resources.
February 26 2012 04:53 AM

J.Judish commented...

Well, Mr. Chudnovsky,maybe there is nothing wrong with your bot; just the peole who use it.

My site just got hammered, again, by one of yor robots, making dozens of calls for pages that don't exist and trying every which way to generate a list of users.

You're blocked.
February 04 2012 04:44 PM

A.Chudnovsky commented...

Hi,

I am the founder of Majestic-12 Distributed Search Engine project that created MJ12bot.

We do obey robots.txt from start of the project and details shown on our bot's page: http://majestic12.co.uk/bot.php

In some rare cases when we had reports of it not obeying we found out that webmasters acually made mistakes in writing robots.txt OR used non-standard approach to disallow bots: we ended up supporting many of those non-standard robots.txts in the middle of 2011.

Our bot email on that page is right on top of the page and we respond very quickly to any queries, don't believe me? Try sending an email to check.

Anyone who has got a problem with our bot please contact us via bot's page provided above.

Best regards,

Alex Chudnovsky
Managing Director

Email: alexc A-T majestic12.co.uk

Majestic-12 Ltd (t/a Majestic-SEO)
Faraday Wharf, Holt Street
Birmingham Science Park, Aston
Birmingham, B7 4BB
United Kingdom
November 01 2011 06:35 PM

J.Woody commented...

BAD BOT(Ignores Robots.txt)
Suspicious behaviour

173.242.125.206 - United States - VolumeDrive

SMALL SAMPLE:
173.242.125.206 - - [29/Oct/2011:17:13:43 +0100] "GET /function.require HTTP/1.0" 403 1224 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.0; http://www.majestic12.co.uk/bot.php?+)"
October 29 2011 04:29 PM