Editing Google News:Googlebot-news (section)

==Identifying Googlebot-News==

A typical Apache access.log entry of a crawling process made by Googlebot-News looks like this:

<code><nowiki>66.249.71.168 - - [31/Mar/2010:07:07:44 -0500] "GET /index.php HTTP/1.1" 200 5404 "-" "Googlebot-News"</nowiki></code>

To identify the crawler, you can do a <code>host</code> look-up for the IP address on a UNIX-like shell, for example:

<code>host 66.249.71.168</code>

Running the above command should provide something like:

<code>[root@google]# host 66.249.71.168</code>

<code>168.71.249.66.in-addr.arpa domain name pointer '''crawl-66-249-71-168.googlebot.com'''.</code>

Since the above information can still be spoofed, you may want to do the check again, but this time for the hostname provided by the previous command (bold above):

<code>[root@google]# host crawl-66-249-71-168.googlebot.com</code>

<code>crawl-66-249-71-168.googlebot.com has address 66.249.71.168</code>

If the IP address from the first command matches the IP address thrown back by the second command, you know that crawler is legitimate.