Switching Let's Encrypt clients is currently quite disruptive

On Twitter, I said :

At the moment, changing between Let's Encrypt clients appears tobe about as disruptive as changing to or from Let's Encrypt andanother CA. Certificate paths change, software must be uninstalled andinstalled, operational practices revised, and nothing can be movedover easily.

I didn't mention that you are probably going to have to get reissuedcertificates unless you like doing a lot of work, but that's truetoo. Let's Encrypt makes this easy, but


Django 1.11 has a bug that causes intermittent CSRF validation failures

Over on Twitter, I said :

People say that Django version upgrades are easy and reliable. Thatis why our web app, moved from 1.10 to 1.11, is now throwing CSRFerrors on *a single form* but only when 'DEBUG=False' which, you know,doesn't help debug the issue.

Last week I updated our Django webapplication from Django 1.10.7 to 1.11.22.Today, one of its users


Go's proposed try() will be used and that will change how code is written

One of the recent commotions in Go is over the Go team's proposed try() built-in error check function ,which is currently planned to be part of Go 1.14 ( cf ). To simplify, ' a, [...] :=try(f(...)) ' can be used to replace what you would today have towrite as:

a, [...], err := f(...)if err !=

ZFS on Linux still has annoying issues with ARC size

I'll start with my tweet :

One of the frustrating things about operating ZFS on Linux is thatthe ARC size is critical but ZFS's auto-tuning of it is opaque andapparently prone to malfunctions, where your ARC will mysteriouslyshrink drastically and then stick there.

Linux's regular filesystem disk cache is very predictable; if youdo disk IO, the cache will relentlessly grow to use all of yourfree memory. This sometimes disconcerts people when free reportsthat there's very little memory actually free


We're going to be separating our redundant resolving DNS servers

We have a number of OpenBSD machines in various roles; they're ourfirewalls, our resolving DNS servers as well as our public authoritativeDNS server, and so on. For pretty much all of these, we actuallyhave two identical servers per role in a hot spare setup, so thatwe can rapidly recover from various sorts of failures. For ourfirewalls, switching from one to another takes manual action (wehave to change which one is plugged into the live network, althoughtheir firewall state is synchronized with


Our switches can wind up in weird states after a power failure

We've had two power failuresso far this year, which is two more than we usually have. Each hasbeen a learning experience, because both times around our overallenvironment failed to come back up afterward. The first time around the problem was DNS, due to a circular dependency that we stilldon't fully understand . The second timearound, what failed was much more interesting.

Three things failed to come back up after the second power failure.The more understandable and less fatal problem was that our


Reflections on almost entirely stopping using my (work) Yubikey

Several years ago ( back in 2016 ), work got Yubikeys for a number of usfor reasons beyond the scope of this entry. I got designated as theperson to figure out how to work with them, and in my usual waywith new shiny things, I started using my Yubikey's SSH key forlots of additional things over and above their initial purpose (and I added things to my environment to make that work well ). For a long time since then,I've had


Browers can't feasibly stop web pages from talking to private (local) IP addresses

I recently read Jeff Johnson's A problem worse than Zoom ( via ), in whichJohnson says:

[...] The major browsers I've tested — Safari, Chrome, Firefox —all allow web pages to send requests not only to localhost but also toany IP address on your Local Area Network! Can you believe that? I'mboth astonished and horrified.

(Johnson mostly means things with private IP addresses , which is the onlysense of 'on your local and private network'


I brought our Django app up using Python 3 and it mostly just worked

I have been worrying for some time about the need to eventually get our Django web application running underPython 3; most recently I wrote about being realistic about ourfuture plans , which mostly amounted tonot doing anything until we had to. Well, guess what happened sincethen .

For reasons beyond the scope of this entry, last Friday I ended upworking on moving our app from Django 1.10.7 to 1.11.x, which was enlivened by the usual problem . After Ihad


Systemd services that always restart should probably set a restart delay too

Ubuntu 18.04's package of the Prometheus host agent comes with a systemd.service unit that is set with ' Restart=always '(something that comes from the Debian package , cf ).This is a perfectly sensible setting for the host agent for a metricsand monitoring system, because if you have it set to run at all,you almost always want it to be running all the time if at allpossible. When we set up a local version of the host agent, I started