It's hopefully not news to people that there is a plague of disguised web crawlers that are imitating web browsers (and not infrequently crawling from residential IPs, through various extremely questionable methods). However, many of these crawlers have only a skin-deep imitation of browsers, primarily done through their HTTP User-Agent header . This creates a situation where some of these crawlers can currently be detected (and blocked) because they either lack entirely or have non-browser values for other HTTP headers. I've been engaged in a little campaign to reduce the crawler presence here on Wandering Thoughts , so I've been experimenting with a number of HTTP header checks.

Headers I'm currently looking at include:

• The CF-Worker header is set for all requests from Cloudflare Workers. Anubis blocks all requests with this header set by default ( cf ), and I decided to copy it. This occasionally blocks things trying to scrape Wandering Thoughts .

• As I discovered, you can't block requests with X-Forwarded-For headers because people really do set these headers on real, non-malicious requests.

• The Sec-Fetch-Mode header is sent by every modern browser and is sent by almost no bad crawlers. However, checking things claiming to be Safari is a little bit complicated, since Sec-Fetch-Mode support was only added in early 2023 (in 16.4) and there are still older Safari versions out there (including earlier 16.x versions). This is a quite effective check in my environment.

  (I got this trick from here , although apparently there may be trouble with mobile WebView interfaces , which might come about through in-app navigation if someone sends a URL around.)

• Every mainstream browser sends an Accept-Encoding header and has for a long time. If it's missing for a fetch of a regular HTML page, you have an imposter. Unless you like maintaining a list of old browsers and other programs that don't send Accept-Encoding, you probably want to limit requiring the header to things claiming to be at least a bit like mainstream browsers.

• Some bad bots are sending an Accept-Encoding of 'identity' in what is apparently an attempt to avoid being fed compression bombs by people (I can't find my source for this). No mainstream browser should do this and in general most things fetching web pages from you should accept compressed responses if they advertise an Accept-Encoding at all.

  Sadly, the exception to this is syndication feed fetchers, some of which refuse to do compression. Whether you keep supporting such feed fetchers is up to you. Wandering Thoughts still does so far, although it's getting tempting to say that enough is enough, especially with the size of syndication feeds here.

• Some or perhaps many bad crawlers set a HTTP Accept header of '*/*' on HTML requests, which isn't something that real browsers do ( source ). Unfortunately, browser-based syndication feed fetchers will send this value, so you can only do this check on HTML pages, and also bingbot and Googlebot (at least) will sometimes also send this Accept value. Some things seem to not end an Accept header at all, too.

  Based on monitoring the results so far, there may be something funny going on; I've seen the same IP and User-Agent making an initial request that is fine and then one or more re-requests for the same URL that have ' Accept: */* ' and fail

• A number of bad crawlers make HTTP/1.0 requests while claiming to be mainstream browsers, all of which have supported HTTP/1.1 for a very long time, and these days I block such requests . Although it's tempting to reject all HTTP/1.0 requests, some text-mode browsers still make them (the ones I know of are Lynx and w3m, including inside GNU Emacs). The HTTP version isn't really a HTTP header, but close enough.

Some of these checks overlap with each other. For example, the crawler with a bad Accept: HTTP header wasn't sending Sec-Fetch-Mode either.

Many of these HTTP headers are only sent by relatively mainstream browsers and environments that have added support for recent HTTP headers. For example, people still use text-based browsers and most of them don't send headers like Sec-Fetch-Mode ; other programs that make HTTP requests through various packages and libraries probably won't either.

There are probably other useful header differences between crawlers imitating mainstream browsers and actual browsers (and, apparently, between headless browsers being driven by automation and real ones being used by people). You could probably discover some of them by collecting enough of a data set of request headers and then doing some sort of statistical analysis to discover correlations and clusters.

PS: The big offenders for requesting uncompressed syndication feeds appear to be Tiny Tiny RSS, Selfoss, and Nextcloud-News. Some browser based syndication feed readers also appear to do it, as do some curl-based syndication feed fetching that people are doing here.

Sidebar: What is a (mainstream) browser-like User-Agent?

It depends on how restrictive you want to be. There are a lot of options:

• Just look for "Mozilla/5.0 (" at the start of the User-Agent.
• Also look for " Chrome/", " Firefox/", or " AppleWebKit/" in the User-Agent
• Try to specifically match a Firefox or Webkit based browser User-Agent format, which will cause you to learn a lot about what Webkit-based user agents appear in your logs.

• Potentially exclude things that mark themselves as robots or crawlers, for example by having 'compatible;' in their User-Agent, or 'robot', or a URL. Anything with these markers is not trying to exactly be a browser User-Agent, although they may be looking generally like one.

I use different versions of these for different checks in DWiki 's steadily growing pile of hacks to detect bad crawlers. Currently the most specific matching is reserved for blocking claimed browsers from cloud/server space , which catches a significant amount even with a limited selection of cloud and VPS provider space that it applies to.

( Some cloud space is blocked entirely ; blocking only things that claim to be browsers is a lesser step.)