One piece of technology news of the time interval is that Google is no longer going to be providing Internet search, it's going to be providing 'answers' . On the Fediverse, I had a reaction to that :

Given Google's apparent change to what Google Search will be, I'm wondering if I should even allow Googlebot to crawl my techblog. If they're going to AI slop up anything they tell people after a search, I'd rather have my writing excluded entirely rather than be garbled.

(I have no idea how much traffic I get from people using Google Search, and if Google is going to do this I feel like I should punish them by pushing people away from them, even a little bit.)

To echo what Paul Cantrell said on the Fediverse , we tolerated and even embraced Googlebot crawling our websites as part of a social bargain. Allowing Googlebot was a big part of how people found our sites and our work, which is to say through searches. That social bargain has been fading as Google put more and more things inline, but at least they were still providing links and directly showing our words (I know, usually).

But now Google has said out loud that Googlebot is just the front end crawling ingester to an LLM system, much like all of the other LLM crawlers that are hammering our sites. Google is no longer in the search business, where they provide links to people; they're in the 'answers' business (which is to say, the probabilistic text generation business). Much as with HTTP requests from cloud provider IPs , this raises the question of whether we should care about allowing Googlebot to access our websites or whether it's now a source of undesired crawl load, or at least of no meaningful benefit to our sites.

(A hasty clarification: by 'we' I mean people running small web servers and web sites, as with requests from cloud provider IPs .)

It feels practically heretical to say this; as recently as last November I was saying that people needed Google to crawl them . However, here we are. It no longer feels at all obvious that I'm going to get future benefits from allowing Google to crawl Wandering Thoughts . For that matter I don't know if I'm getting current benefits (well, if people are in general), since you can't really tell any more when people come to your site from Google Search.

(If you can still tell from the Referer HTTP header, that suggests that almost no one is coming here from Google Search. Googlebot is still (re)crawling portions of Wandering Thoughts on an ongoing basis; in fact, now that I look it regularly fetches the front page, although it politely uses HTTP conditional GET.)