A while back I wrote about how claiming to be Googlebot is now a bad idea , where I noted that there were (reports of) malicious crawlers out there impersonating Googlebot and other legitimate big crawlers and at the same time, Google and other crawler operators published the IP address ranges their crawlers used. You could put these two together to block these impersonators:

Anything claiming to be Googlebot that is not from a listed Google IP is extremely suspicious and in this day and age of increasing anti-crawler defenses, blocking all 'Googlebot' activity that isn't from one of their listed IP ranges is an obvious thing to do.

After I wrote that entry, I quietly went and added support for blocking crawler impersonators to DWiki , the wiki-engine that powers Wandering Thoughts , and set it up for a few big crawlers with published IP address ranges. When I did this, I didn't expect to block very much, and for months that was indeed what happened; I'd get a few attempts once in a while. Then, this June, the floodgates opened.

For weeks, I've been seeing hundreds of requests a day claiming to be Googlebot (on a few days, thousands of requests). The requests come from a variety of IP addresses at a variety of providers, which I think are mostly or entirely cloud and hosting providers. The top sources by ASN are a rogue's gallery of places that I was already having problems with, like HostRoyale, M247, Latitude.sh, and Web2Objects. But AWS is in the collection as well, and there are a lot of other relatively mainstream providers. Many IPs seem to make only a few requests as Googlebot, and at least some of them immediately retry with another User-Agent value (which also generally doesn't work).

My guess is that this isn't a bunch of different abusive crawlers who've all spontaneously decided to try forging Googlebot to see if it gets them anywhere. Instead, I suspect that this is a large scale campaign by a single abusive crawler, run by people who can afford to obtain a lot of servers at a lot of different hosting providers (or who are prepared to commit various sorts of criminal fraud on a large scale). Ironically, if they'd picked a different tactic, I might not have noticed them among the background radiation of crawl attempts. Forging Googlebot and other known big crawlers is generally sufficiently rare that I actually bother looking at my logs to see it happening.

(I also suspect I'm not the only website this is happening to.)

PS: It feels somewhat ironic that this is happening at the same time as me wondering if I should allow Googlebot at all .