Some practical notes on the systemd cgroups/units hierarchies

Systemd organizes everything on your system into a hierarchy ofcgroups, or if you prefer a hierarchy of units that happen to beimplemented with cgroups. However, what this hierarchy is (or isgoing to be) isn't always obvious, and sometimes what shows upmatters, for example because you're generating per-cgroup metricsand might hit a cardinality explosion . So here are some noteson things you may see in, for example, systemd-cgls or ' systemctlstatus ' (or if you'


Our varied approaches to upgrading machines with local state

Our normal approach for distribution version upgrades for ourservers is to build a new version of the server on new hardware , then swap it into place by renaming thenew server. Sometimes, as for our SLURM cluster ,we have sufficiently cattle-like machines that we will take one or a few of them out of service and reinstallthem in place. All of this is straightforward for machines thathave basically no local state, but this isn't the case for all ofour machines. Servers like our ZFS-


More use of Rust is inevitable in open source software

Recently, I saw a poll on the Fediverse about making Rust a harddependency for fwupd .This got me to post a lukewarm take of my own :

Lukewarm take: the spread of Rust in open source software isinevitable, because nothing else fills the niche for 'C/C++ butstrongly safe'. We need a replacement C because in general we can'twrite safe C/C++ at scale.

Rust isn't really my thing , but this shift impliesthat sooner or later


Sorting out PC chassis power switches for ATX power supplies


The systemd journal is primarily time-based

Recently, I encountered a surprising systemd behavior :

Dear systemd: apparently using absolute system date (at boot?!)to determine 'the system boot before this one' and so on is notnecessarily a great idea, given that systems can boot with scrambledsystem time. If I want to look at 'the previous boot', I want that towork even given bad time. Especially if the system booted with badtime, ran with bad time for a bit, and then rebooted again.

(There are


Handling numbers in Vim when they have a dash in front of them

Over on the Fediverse, I mentioned a Vim situation I'd run into :

Here's a situation Vim's Ctrl-A feature for handily incrementingnumbers doesn't readily handle: if you have machines called eg'something-1' and want to make that 'something-2', 'something-3',and so on. Vim will interpret the '-1' bit as the number -1 andauto-increment it to 'something0'. There is probably a clever


The power of URLs you can use with query parameters and a HTTP GET request

I recently wrote about some aspects of my dmenu setup , including using a custom$PATH that contains a bunch of little utility scripts . These little scripts are animportant part of making my dmenu setup so useful and a crucialbuilding block of my environment , but inturn a lot of them are enabled by something else. In practice, alot of what I do with dmenu is to open a Firefox window on somesort of URL, and in turn this is relies on being able to createURLs that do useful


The Prometheus cardinality issues with systemd unit-related metrics

Over on the Fediverse I said something about cAdvisor's Prometheusmetrics :

Unless I'm missing something, cAdvisor's Prometheus metrics exporthas no way to limit what cgroups it generates metrics for. This isfatal on systems used by users, since it leads to a label cardinalityexplosion as the cgroups reported on will include 'session-NNN.scope'cgroups, with a constantly increasing and basically never re-used NNN.

This is a general issue for any Prometheus metrics that are relatedto particular systemd units


Detecting missing or bad Go modules and module versions

I recently wrote about the case of a half-missing Go import , where a particular version of a module hadbeen removed upstream but was still available through the (default)Go module proxy , so things didn't reallynotice. This raises the interesting question of how you find thissort of thing, in all of at least three variations. Unfortunately,right now there are no great answers, but here's what I can see.

The most straightforward case is if a module has been marked asentirely


Systemd unit templates don't provide a native way to have multiple parameters

We have a collection ofApache servers, and as part of our Prometheus metricssystem we indirectlyscrape metrics from them, using a third party Apache exporter that queries Apache's mod_status information and converts it to Prometheus metrics. Because the exporter'sauthor chose to write it that way, you need to run one instance of theexporter per Apache server you want to extract metrics from. Ordinarypeople would probably run these exporter instances on the Apache hoststhemselves. We opted to instead run all of the exporters