Once upon a time, something like the following was a perfectly decent User-Agent header string for a web crawler or a web fetching agent:

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36 (compatible; Yourbot; +https://some/url)

You weren't hiding, after all, you called yourself 'Yourbot', and for the rest, you were asking for people to serve you pages like you were Chrome. Well, I'm not too sad to say, those days are over.

They're over because an increasing number of websites are increasingly requiring that anything that looks like a browser in its User-Agent also act like a browser, in specific the browser and browser version it's saying it is, and there are a lot of picky details around other HTTP headers ( also ). For example, often simply having 'Mozilla' in your User-Agent will cause Anubis to challenge your crawler ( cf ). And the version of Chrome being asserted here is new enough that it should be reporting a Sec-CH-UA-Platform header , among Sec-CH- headers.

( Claiming to be a really old version of Chrome without those features is likely to be worse .)

Now, you can certainly pin your hopes on the idea that people who are writing header checking code will pay attention to the presence of the 'compatible;' and the URL in your User-Agent, and realize that you're not actually a browser despite you having a fairly good imitation of a Chrome User-Agent. However, you're not Google(bot) . People have to make exceptions for Googlebot (to some degree), but they don't have to make exceptions for you and they probably won't.

The User-Agent you should instead use today is something like, for example:

Fedithing/4.5.1 (library/1.2.3; +https://some/url)

You don't start with a superstitious invocation of 'Mozilla/5.0', you don't claim to be be like any version of any browser, and you put in the basics of identifying your software and yourself so no one can accuse you of hiding. No one is going to match your User-Agent against detectors for old versions of browsers, or things claiming to be browser but lacking their headers, and so on, because you haven't put in the names of any browsers.

PS: Googlebot and Bingbot and a few others still use User-Agent strings very much like my first example, but they're Googlebot (and Bingbot) and to a fair extent they do get their HTTP headers relatively authentic.