Staged rollouts of things still have limitations

One of the commonly suggested remedies for deploying things that can gowrong is to do staged rollouts, where you deploy to only a subset ofthe things at a time and look for problems before proceeding. Stagedrollouts are in general a good idea, but it's important to understandthat there are limits on how much they can improve the situation,especially if the staged rollouts are going out to outside people('customers') instead of internally, within your organization inenvironments that you control.

The first limitation


The speed of updates for signatures of bad things matters (a lot)

These days (and for a long time), most spam, phish, malware, andso on (in email and other things) is recognized not through generalrules, patterns, and processes ( eg ),but by seeing if the content matches any known signatures . Sometimesthis is literally matching cryptographic hashes, but more oftenthere's some sort of signature matching engine involved with variousmatching operators, conditions for combining them, and so on. ClamAV is one example that's mostly a matching engine ,


A surprise with the temperature distribution in our machine room

Our primary machine room is quite old and is set up in an old fashioned way, so that we don't really haveseparate 'hot aisles' and 'cold aisles'; the closest we come is oneaisle where both sides are the the front of servers. We have somelong standing temperature monitoring in this machine room , and recently (for reasons outside thescope of this entry) we put a second (trustworthy) temperaturemonitoring unit into the room. The first temperature sensor isrelatively near the room


Modern web PKI (TLS) is very different than it used to be

In yesterday's entry on the problems OCSP Stapling always faced , I said that OCSP Stapling felt likesomething from an earlier era of the Internet. In a way, this isliterally true. The OCSP Stapling RFC was issued inJanuary 2011, so the actual design work is even older. In 2011, Let's Encrypt was a year away from being started and theSnowden leaks about pervasive Internet interception (and 'SSL addedand removed here') had not yet happened.


OCSP Stapling always faced a bunch of hard problems

One reaction to my entry about how the Online Certificate StatusProtocol (OCSP) is basically dead is to askwhy OCSP Stapling was abandoned along with OCSP and why it didn't catch on. The answer,which will not please people who liked OCSP Stapling, is that OCSPStapling was always facing a bunch of hard problems and it's notreally a surprise that it failed to overcome them.

If OCSP Stapling was to really deliver serious security improvements,it had to be mandatory in the long run. Otherwise


We may want /usr/bin/python to be Python 3 sooner than I expected

For historical reasons, we stillhave a '/usr/bin/python' that is Python 2 on our Ubuntu 22.04machines. Yes, we know, Python 2 isn't supported any more, but ourusers have had more than a decade where /usr/bin/python was Python2 and while Ubuntu continued to ship a Python 2, we didn't feellike breaking their '#!/usr/bin/python' lines in scripts by eitherremoving /usr/


On not automatically reconnecting to IPMI Serial-over-LAN consoles

One of the things that the IPMI (network) protocol supports is Serial over LAN , which can be usedto expose a server's serial console over your BMC's managementnetwork. These days, servers are starting to drop physical serialports , making IPMI SOL your only wayof getting console serial ports. The conserver serial console managementsoftware supports IPMI SOL (if built with the appropriate libraries),and you can directly access SOL serial consoles with IPMI programs.However, as I mentioned in passing in yesterday


Handling (or not) the serial console of our serial console server

We've had a central serial console server for a long time. It has two purposes; it logs all of the (serial)console output from servers and various other pieces of hardware(which on Linux machines includes things like kernel messages, cf ), and it allows us to log in tomachines over their serial console. For a long time this server wasa hand built one-off machine, but recently we've been rebuildingit on our standard Ubuntu framework ( much like our central syslogserver


Our slowly growing Unix monoculture

Once upon a time, we ran UbuntuLinux machines, OpenBSD machines, x86 Solaris machines, and whatwas then RHEL machines (in the days of our first generation ZFSfileservers ). Over time, Solarischanged to OmniOS (and RHEL to CentOS) ,but even at the time it was clear that both of those hadn't caughton here and after a while we replaced the OmniOSfileservers and CentOS iSCSI backends with our third generationUbuntu-based fileservers . Thenrecently, the final pieces of CentOS


Backward compatibility, even for settings, has real costs

When I wrote about how I felt GNU Emacs configuration bankruptcywas inevitable in the longer term , oneof the reasons was that Emacs configuration settings have changedover time (and will change more in the future), so that your oldway of configuring something (like the C indentation style) wouldhave to be updated periodically. One reaction to this is to say thatopen source software should keep backward compatibility for suchsettings. Unfortunately and as usual, such backward compatibility wouldhave real costs; it would effectively act as