These days, there's an unusually large plague of web crawlers, many of them attributed to LLM activities and most of them acting anonymously, with forged user agents and sometimes widely distributed source IPs. Recently I've been using two tools more and more to try to identify and assess suspicious traffic sources.

The first tool is Anarcat's asncounter . Asncounter takes IP addresses, for example from your web server logs, and maps them to ASNs (roughly who owns an IP address) and to CIDR netblocks that belong to those ASNs (a single ASN can have a lot of netblocks). This gives you information like:

count   percent ASN     AS
1460    7.55    24940   HETZNER-AS, DE
[...]
count   percent prefix  ASN     AS
1095    5.66    66.249.64.0/20  15169   GOOGLE, US
[...]
85      0.44    49.13.0.0/16    24940   HETZNER-AS, DE
85      0.44    65.21.0.0/16    24940   HETZNER-AS, DE
82      0.42    138.201.0.0/16  24940   HETZNER-AS, DE
71      0.37    135.181.0.0/16  24940   HETZNER-AS, DE
68      0.35    65.108.0.0/16   24940   HETZNER-AS, DE
[...]

While Hetzner is my biggest traffic source by ASN, it's not my biggest source by 'prefix' (a CIDR netblock), because this Hetzner traffic is split up across a bunch of their networks. Since most software operates by CIDR netblocks, not by ASNs, this difference can be important (and unfortunate if you want to block all traffic from a particular ASN).

The second tool is grepcidr . Grepcidr will let you search through a log file, such as your web server logs, for traffic from any particular netblock (or a group of netblocks), such as Google's '66.249.64.0/20'. This lets me find out what sort of requests came from a potentially suspicious network block, for example 'grepcidr 49.13.0.0/16 /var/log/...'. If what I see looks suspicious and has little or no legitimate traffic, I can consider taking steps against that netblock.

Asncounter is probably not (yet) packaged in your Linux distribution. Grepcidr may be, but if it's not it's a C program and simple to compile.

(It wouldn't be too hard to put together an 'asngrep' that would cut out the middleman, but I've so far not attempted to do this.)

PS: Both asncounter and grepcidr can be applied to other sorts of logs with IP addresses, for example sources of SSH brute force password scans. But my web logs are all that I've used them for so far.