I recently wrote about how systemd's auto-restart of units canhide problems , where we discoveredthis was hiding failures of the Prometheus host agent itself. This raisesthe question of how and if we can monitor for this sort of thinghappening with our Prometheus setup . The answer turns out tobe more or less yes.

The host agent has a systemd collector, which as of 1.6.1 isn'tenabled by default (you enable it with '--collector.systemd').

Over on the Fediverse, I said :

Firefox tip: do you hate that the built-in PDF viewer opens thespace-eating sidebar all the time (I do)? If so, go to about:configand set pdfjs.sidebarViewOnLoad to '0' from its default. As far as Iknow this is not exposed as a Preferences setting, so you have to useabout:config (and search the magic numbers in the Firefox source).

Apparently this behavior may actually be something that defaultsto

Today, more or less by coincidence, I discovered that the Prometheushost agent on ourLinux machines was periodically crashing with an internal Go runtimeerror (which had already been noticed by other people and filed as issue #2705 ).You might wonder how we could not notice the host agent for ourmonitoring, metrics, and alerting system doing this, and part of theanswer is that the systemd service has a setting of ' Restart=always '.

(We inherited this setting from the Ubuntu package

A while ago I wrote about how an obvious problem isn't necessarilyobvious , where one thing I said was thatin many situations, there are too many obvious problem causes forpeople to keep track of them all. A corollary to this is that thereare too many things that could go wrong on your systems to monitorand alert on all of them. In fact, I'm not convinced that we couldeven identify all of the possible things that could go wrong. Amongother issues, systems can fail

A while back I discovered that Prometheus will make persistentconnections to its scrape targets .Julien Pivotto added the important additional note thatPrometheus only does a DNS lookup when it makes a new connection;once the connection is established, it doesn't re-check DNS. Oneof the consequences of this is that ~~these persistent connectionswill stay up even if your DNS resolution falls over (which can beuseful in trying to understand such a failure ). However, there is an importantqualification on this. The

When I wrote about the 'eval' pattern for interpreting simplelanguages , where you parse the languageinto a syntax tree, give each node an 'Eval()' method, and call thetop level Eval() to evaluate (interpret) the tree, I mentioned thatthis approach got increasingly complex if you needed to deal withloops and other control flow. However, I didn't explain what theproblem was. That's what this entry is about.

The ideal situation for an 'eval(

One of the things that's quite popular with people out in the worldis being able to set up a Linux server and then leave it be for thebetter part of a decade without having to reinstall it or upgradethe distribution. I believe this is a significant reason peopleused CentOS , and it's popular enough to supportsimilar things in other distributions. I'm not fond of these oldzombie distribution versions , but even wehave some of them (running CentOS 7). However, I'

The news of the time interval is that Red Hat has stopped makingRed Hat Enterprise Linux source code generally available ,although just as with the switch to 'CentOS Stream' from CentOS their article doesn't put it that way.This created difficulties for at least two CentOS replacementdistributions, forcing AlmaLinux to change what they are . I don't havemuch to say on this specific topic, but it has sparked a series ofexchanges about, for example, the history of RHEL rebuilds ( via ). As

A bit over five years ago, I wrote about the increasingly surprisingspeed limits on our Amanda backups . Atthat time we were using spinning rust hard drives on both theAmanda backup servers and our fileserversof the time , 10G networking onthe Amanda servers, and 1G networking on the fileservers ( forreasons ). Generally this madeeither the Amanda 'holding disk' (whereit streams backups to) or the fileserver 1G networking the limiton our backup speeds. Since then there have been two