Detecting absent Prometheus metrics without knowing their labels

When you have a Prometheus setup ,one of the things you sooner or later worry about is importantmetrics quietly going missing because they're not being reportedany more. There can be many reasons for metrics disappearing onyou; for example, a network interface you expect to be at 10Gspeeds may not be there atall any more, because it got renamed at some point, so now you'renot making sure the new name is at 10G.

(This happened to us with one


Our probably-typical (lack of) machine inventory situation

As part of thinking about how we configure machines to monitorand what to monitor on them , I mentioned inpassing that we don't generate this information from some centralmachine inventory because we don't have a single source of truthfor a machine inventory .This isn't to say that we don't have any inventory of our machines;instead, the problem is that we have too many inventories, eachserving somewhat different purposes.

The core reason that we have wound up with many different lists of


How to make your GNU Emacs commands 'relevant' for M-X

Today I learned about the M-X command (well, key binding) ( via ), which "[queriesthe] user for a command relevant to the current mode, and thenexecute it". In other words it's like M-x but it restricts whatcommands it offers to relevant ones. What is 'relevant' here? Toquote the docstring:

[...] This includes commands that have been marked as being speciallydesigned for the current major mode (and enabled minor modes


Open source culture and the valorization of public work

A while back I wrote about how doing work that scales requiresbeing able to scale your work , whichin the open source world requires time, energy, and the willingnessto engage in the public sphere of open source regardless of theother people there and your reception. Not everyone has this sortof time and energy, and not everyone gets a positive reception byopen source projects even if they have it.

This view runs deep in open source culture, which valorizes publicwork even at the cost of stress and time


The Go 'range over functions' proposal and user-written container types

In Go 1.22, the Go developers have made available a "range overfunction" experiment, as described in the Go Wiki's "RangefuncExperiment" . Recently Iread a criticism of this, Richard Ulmer's Questioning Go'srange-over-func Proposal ( via ).As I read Ulmer's article, it questions the utility of the rangeover func (proposed) feature based on the grounds that this isn'ta significant enough improvement in standard library functions likestrings


Fixing my problem of a stuck 'dnf updateinfo info' on Fedora Linux

I apply Fedora updates only by hand, and as part of this I like tolook at what ' dnf updateinfo info ' will tell me about why they'rebeing done. For some time, there's been an issue on my work desktopwhere 'dnf updateinfo info' would report on updates that I'd alreadyapplied, often drowning out information about the updates that Ihadn't. This was a bit frustrating, because my home Fedora machinedidn't do this but I couldn't


A recent abrupt change in Internet SSH brute force attacks against us

It's general wisdom in the sysadmin community that if you expose aSSH port to the Internet, people will show up to poke at it, andby 'people' I mean 'attackers that are probably mostly automated'.For several years, the pattern to this that I've noticed was anapparent combination of two activities. There was a constantbackground pitter-patter of various IPs each making a probe once aminute or less (but for tens of minutes or longer), and then periodicbursts


What ZIL metrics are exposed by (Open)ZFS on Linux

The ZFS Intent Log (ZIL) is effectivelyZFS's version of a filesystem journal, writing out hopefully briefrecords of filesystem activity to make them durable on disk beforetheir full version is committed to the ZFS pool. What the ZIL isdoing and how it's performing can be important for the latency (andthus responsiveness) of various operations on a ZFS filesystem,since operations like fsync() on an important file must wait forthe ZIL to write out ( commit ) their information before they canreturn


NetworkManager won't share network interfaces, which is a problem

Today I upgraded my home desktop to Fedora 39. It didn't entirelygo well; specifically, my DSL connection broke because Fedorastopped packaging some scripts with rp-pppoe and Fedora'sold ifup , which is used by my very old-fashioned setup still requires those scripts. After I gotback on the Internet, I decided to try an idea I'd toyed with,namely using NetworkManager to handle (only) my DSL link . Unfortunately this did not go well:

audit: op=


The flow of activity in the ZFS Intent Log (as I understand it)

The ZFS Intent Log (ZIL) is a confusing thingonce you get into the details, and for reasons beyond the scope ofthis entry I recently needed to sort out the details of some aspectsof how it works. So here is what I know about how things flow into theZIL, both in memory and then on to disk.

(As always, there is no single 'ZFS Intent Log' in a ZFS pool. Eachdataset (a filesystem or a zvol) has its own logically separateZIL.