I mentioned recently that we'd moved a very popular data set from our main web server to a new one that only handled that data set . On the new web server, we found it necessary to set an absurdly high Apache connection limit of 4,000 concurrent requests, because Apache could run out otherwise (and could even run out at 4,000, it was just less often).

Our main web server has a much lower concurrent connection limit but it didn't experience these problems (and not because we'd imposed connection limits in Apache; we'd turned that off in December of 2022 ). But as we (I) wrangled with the new web server to get it to stop running out of connections and so on, and even after I got mod_qos working on it , I never paused to ask myself why the new web server had so many problems with this when the main web server had run for years without explosions (or at least very infrequent ones).

(Part of this was because the main web server had started exploding sometimes; that was why we'd moved this data set to its own server. But even those explosions had been less severe than what I was seeing.)

The answer is that for years, our perimeter firewall has had per-IP connection rate limits for HTTP and HTTPS connection to our main web server (among other per-IP connection rate limits, for example for SSH connections). These predate the modern popularity of this data set and were added to stop other abuse, but it turns out that a lot of the connection volume for this data set was coming from a few IPs that were opening up a ton of rapid-fire and often simultaneous connections. Once we applied per-IP limits on both the number of simultaneous connections you could have (in mod_qos) and the rate at which you could make connections (in the firewall), the new web server's connection count dropped like a stone (but people still kept pulling data from it as fast as they could).

In retrospect, the change in the web server's behavior when we moved this data set to a new host was a signal. We'd had only occasional problems on the old web server host one (despite it being actively used for other things) and we had constant ones on the new server (dedicated only to the data set). So we could have asked what was different, and then investigated, and then found the perimeter firewall issue. But on the other hand, this is sort of hindsight bias speaking. Such changes in behavior are a signal, but as system administrators we're drowning in signals and we have to sort out what's meaningful and what's either a coincidence or a consequence of something else (for example, a sudden increase in demand for this data set, which would have also explained why we were suddenly seeing problems even on the main web server).

PS: There's some recent evidence that there was a real but temporary shift in demand for this data set, in large part from people (or software) that make extremely inefficient requests . If these people are done now, or have improved their software, that would be nice. Perhaps all of the connection blocking and limited bandwidth have encouraged them to download things only once and then keep local caches.