Message Board

http:BL Use/Development

How to allow google bots

Author: S.Grayban2 (9 Nov 08 11:04pm)

Been trying to figure this all out but I simply don't understand how to allow google to crawl our clients websites. Would someone be able to help me out on this?

Re: How to allow google bots

Author: M.Prince (10 Nov 08 11:09am)

The IPs used by search engines return a special code from http:BL. Specifically, the IP address that http:BL returns will end in a zero if it's a known IP used by a search engine. For example:

127.0.1.0

The first octet (127) and second octet (0) have no meaning. The third octet (1) is a serial number which corresponds to the particular search engine that is visiting. The final octet (0) means it is a search engine.

So long as you don't block visitors when a response where the fourth octet is a zero, the Google bot and other search engines should still be able to visit your website.

One caveat is that we do not know of all the IPs being used by search engines. If you find an IP address that is used by a search engine but not returning an accurate http:BL code, please let us know and we'll update our systems.

If the specific implementation of http:BL that you are using does not support search engines, write to the implementation's author and ask him or her to update the implementation's code.

Matthew.