How we sort of automate updating system packages across our Ubuntu machines
Every place with more than a handful of Unix systems has to figureout a system for keeping them up to date, because doing it entirelyby hand is too time consuming and error prone. We're no exception,so we've wound up steadily evolving our processes into a decentlyfunctional but somewhat complicated setup for doing this to ourUbuntu machines.
The first piece is a cron job that uses apt-show-versions and a state file to detect new updates for a machine and send emaillisting them off
Linux's iowait statistic and multi-CPU machines
Yesterday I wrote about how multi-CPU machines quietly complicatethe standard definition of iowait ,because you can have some but not all CPUs idle while you haveprocesses waiting on IO. The system is not totally idle, which iswhat the normal Linux definition of iowait is about,but some CPUs are idle and implicitly waiting for IO to finish.Linux complicates its life because iowait is considered to be aper-CPU statistic, like user, nice, system, idle, irq, softirq,and the other per
The problem of Unix iowait and multi-CPU machines
Various Unixes have had a 'iowait' statistic for a long time now(although I can't find a source for where it originated; it's not in 4.xBSD, so it may have come through System V and sar ). The traditionaland standard definition of iowait is that it's the amount of time thesystem was idle but had at least one process waiting on disk IO. Ratherthan count this time as 'idle' (as you would if you had a three-
Unix's iowait% is a narrow and limited measure that can be misleading
For a long time, iowait% has been one of my standard Unix systemperformance indicators to check, both for having visible iowait%and for not having it. As I interpreted it, a machine with high orappreciable iowait% clearly had potential IO issues, while a machinewith low or no iowait% was fine as far as IO went, including NFS(this is on display in, for example, my entry on the elevatedload average of our main IMAP server ). Unfortunately, I'verecently
One impact of the dropping of Python 2 from Linux distributions
Due to uncertainty over the future of the Python 2 interpreterin future Linux distributions , I've beenlooking at some of our Python 2 code, especially the larger programs.This caused me to express some views over on Twitter , whichcame out long enough that I'm recycling them here with additionalcommentary:
Everyone's insistence on getting rid of Python 2 is magicallytransforming all of this perfectly functional and useful Python 2 codewe have from an asset to a liability . You can imagine how I feel about
More or less what versions of Go support what OpenBSD releases (as of March 2020)
In light of recent entries, such as the situation with Go onOpenBSD , I've decided to write down what I canfind out about what versions of Go theoretically work on whatreleases of OpenBSD. This is not actually tested for various reasonsincluding the lack of suitable OpenBSD machines (for example, onesthat are not in production use as important firewalls).
The Go people have a wiki page on OpenBSD that lists both the Goversions in OpenBSD's ports tree and the built from source versionsthat are
The situation with Go on OpenBSD
Over in the fediverse, Pete Zaitcev had a reactionto my entry on OpenBSD versus Prometheus for us :
Ouch. @cks himself isn't making thatclaim, but it seems to me that anything written in Go is unusable onOpenBSD.
I don't think the situation is usually that bad. Our situation withPrometheus is basically a worstcase scenario for Go on OpenBSD, and most people will have much betterresults, especially if you stick to supported OpenBSD versions.
If you stick to supported OpenBSD versions,
OpenBSD versus Prometheus (and Go)
We have a decent number of OpenBSD machines that do important things(and that have sometimes experienced problems like running out ofdisk space), and we have a Prometheus based metrics and monitoringsystem . The Prometheus host agent has enough supportfor OpenBSD to be able to report on critical metrics, includingthings like local disk space. Despite all of this, after someinvestigation I've determined that it's not really sensible to eventry to deploy the host agent on our OpenBSD machines. This is dueto a
One reason for Go to prefer providing indexes in for ... range loops
I was recently reading Nick Cameron's Early Impressions of Gofrom a Rust Programmer ( via ).One of the things Cameron did not like about Go was, well, let mequote directly from the article:
for ... rangereturns a pair of index/value. Getting just theindex is easy (just ignore the value) but getting just the valuerequires being explicit. This seems back-to-front to me since I needthe value and not the index in most cases.
Implicitly,
Some alert inhibition rules we use in Prometheus Alertmanager
One of the things you can do with Alertmanager is to make onealert inhibit notifications for some other alerts. This is set upthrough inhibition rules . Youcan find some generic examples in sample Alertmanager configurationfiles, and today I'm writing up two specific inhibition rules thatwe use.
First off, inhibition rules are very closely tied to the structureof your alerts, your labels, and your overall system; they can'tbe understood or written outside of that. This is because all ofthose determine both