When metrics disappear on updates with Prometheus Pushgateway

To simplify, Prometheus has a notion of current versus stale metrics. As you'd want, straightforwardPrometheus queries (for instance, for your current CPU usage ) return only current metrics. There are alsoa number of ways to push metrics into Prometheus from places likescripts, such as the node exporter 's textfile collector ,where your script can write files to a magic directory on a machinethat is already running the node exporter, and the Pushgateway , where your script canuse curl to just poke metrics straight


Quickly bashing together little utilities with Python is nice

One of the reasons that I love Python, and one of the things thatI love using it for, is its ability to quickly and easily bashtogether little utility programs. By their nature, these thingsdon't get talked about very much (they're so small and often soone-off), but this time I have a couple of examples to talk about.

As part of writing yesterday's entry on external email deliverydelays we see , I found myselfwanting to turn Exim's


Garbage collection and the underappreciated power of good enough

Today I read The success of Go heralds that of Rust ( via ). Init the author argues that Rust is poised to take over from Go becauseRust is a more powerful language environment. As one of their argumentsagainst Go, the author says:

All the remaining issues with Go stem from three design choices:

  • It’s garbage collected, rather than having compile time definedlifetimes for all its resources. This harms performance, removesuseful concepts (like move semantics and destructors) and makescompile-time error

The external delivery delays we see on our central mail machine

These days, our central mail machine almost always has a queueof delayed email that it's trying to deliver to the outside world.Sometimes this is legitimate email and it's being delayed becauseof some issue on the remote end, ranging from the remote end beingdown or unreachable to the destination user being over quota. Butquite a lot of time it is for some variety of email that we didn'tsuccessfully recognize as spam and are either forwarding to someplace that does (but that hasn'


Getting a CPU utilization breakdown in Prometheus's query language, PromQL

A certain amount of Prometheus 's querylanguage is reasonably obvious, but once you start getting into thedetails and the clever tricks you wind up needing to wrap your mindaround how PromQL wants you to think about its world. Today I want to tackle oneapparently obvious thing, which is getting a graph (or numbers)of CPU utilization.

Prometheus's host agent (its 'node exporter') gives us per-CPU, permode usage stats as a running counter of seconds in that mode (which


How Prometheus's query steps (aka query resolution) work

Prometheus and the combination of Prometheus and Grafana have many darkcorners and barely explained things that you seem to be expectedto just understand. One of them is what is variously called queryresolution or query steps (in, for example, the Grafana documentationfor using Prometheus ). Hereis what I think I understand about this area, having poked at anumber of things and scrutinized the documentation carefully.

In general, when you write a simple Prometheus PromQL query, it is evaluated at some point in time (normally the


My unusual use for Firefox's Private Browsing mode

I've written before about how I use my browsing history to keeptrack of what I've read , although that onlypartly works in this era of websites making visited links look thesame as unvisited ones. However, there is a little problem withusing the 'visited' status to keep track of what I've read, andthat is that visiting a web page doesn't necessarily correspondwith actually reading it. Specifically, sometimes I run across apotentially interesting link but I'm not sure


Some notes on Prometheus's Blackbox exporter

To make a long story short, I'm currently enthusiastically experimentingwith Prometheus . As part of this I'mtrying out Prometheus's support for 'black box' status and healthchecks, where you test services and so on from the outside (insteadof the 'white box' approach of extracting health metrics from themdirectly). The Prometheus people don't seem to be too enthusiasticabout black box metrics, so it's perhaps not surprising that theofficial Prometheus blackbox exporter is somewhatunderdocumented and


Even systemd services and dependencies are not self-documenting

I tweeted :

I'm sure that past-me had a good reason for configuring my Wireguardtunnel to only start during boot after the VMWare modules had beenloaded. I just wish he'd written it down for present-me.

Systemd units are really easy to write, straightforward to read,and quite easy to hack on and modify. But, just like everythingelse in system administration, they aren't really self documenting.Systemd units will generally tell you clearly what they're doing,but


Something systemd is missing for diagnosing oddly slow boots

In yesterday's entry , I mentioned thatmy office machine has always been oddly slow to boot (and this goesback years, as far back as when I switched to systemd's networkd ). Over the years I have made various attemptsto figure out why this was so and what I could do about it, generallynot getting anywhere with them. As part of this, I have of coursepoked around at what systemd-analyze could tell me, looking atthings like ' systemd-analyze blame '