A YAML syntax surprise and trick in Prometheus Alertmanager configuration
In a comment on my entry on doing reboot notifications with Prometheus , Simon noted:
Just a note to say that since Alertmanager v0.16.0, it is possible togroup alerts by all labels using "group_by: [...]".
When I saw this syntax in the comment, my eyebrows went up, becauseI'd never seen any sort of ... syntax in YAML before; I had noidea it was even a thing you could do in YAML,
Some additional information on ZFS performance as you approach quota limits
My first entry on this subject gotsome additional information from Allan Jude and others on Twitter, which I'mgoing to replicate here:
@alanjude :re:
] - Basically, when youare close to the quota limit, ZFS will rate-limit incoming writes asit has to be sure you won't go over your quota. You end up having towait for the pending transactions to flush to find out how much roomyou have left I was turned on to the issue by
Limiting the size of things in a filesystem is harder than it looks
Suppose, not entirely hypothetically , that you want toartificially limit the size of a filesystem or perhaps somethingwithin it, for example the space used by a particular user. Thesesort of limits usually get called quotas . Although you mightinnocently think that enforcing quotas is fairly straightforward,it turns out that it can be surprisingly complicated and hard evenin very simple filesystems. As filesystems become more complicated,it can rapidly become much more tangled than it looks.
Let's start with a simple filesystem with no holes in
How we implement reboot notifications when our machines reboot in Prometheus
I wrote yesterday about why we generate alerts that our machineshave rebooted , but not about how we do it. Itturns out that there are a few little tricks about doing this inPrometheus, especially in an environment where you're using physicalservers.
The big issue is that Prometheus isn't actually designed to sendnotifications; it's designed to have alerts. The difference betweena notification and an alert is that you send a notification onceand then you're done, while an alert is raised
Why we generate alert notifications about our machines having rebooted
Part of our Prometheus alerts is an alertthat triggers whenever a machine has been recently rebooted. Myimpression is that having such alerts these days is unusual, sotoday I'm writing up the two reasons why we have this alert.
(This is an 'alert' in the sense that all of the output from ourPrometheus and Alertmanager is an 'alert', but it is not an alertin the sense of bothering someone outside of working hours. All ofour alerts go only to email, and we
Automating our 'bookable' compute servers with SLURM has created generic 'cattle' machines
I'll lead with the thing I realized. Several years ago I wrote about how all of our important machines were 'pets' instead of 'cattle' . One of the reasons for that was that peoplelogged in to specific machines by name in order to use them, andso they cared if a particular machine went down (which is my viewof the difference between pets and cattle ).Due to recent changes in how we run a bunch of our compute servers,we've more or less transformed
The wikitext problem with new HTML elements such as <details>
I recently wrote about my interest in HTML5's
Vim, its defaults, and the problem this presents sysadmins
One of Vim's many options is '
hidden' , whichmay be off or on. The real thing that it does, behind the thicketof technical description, is thathiddencontrols whether or notyou can casually move away from a modified Vim buffer to anotherone. In most editors this isn't even an option and you always can(you'll get prompted if you try to exit with unsaved changes). InVim, for historical reasons, this is an option and for furtherhistorical reasons
Making changes to multiple files at once in Vim
We recently finished switchingthe last of our machines to a different client for Let's Encrypt,and as part of that switch the paths to our TLS certificates hadto be updated in all of the configuration files using them. On alot of our machines there's only a single configuration file, buton some of our Apache servers we have TLS certificate paths inmultiple files. This made me quite interested in finding out howto do the same change across multiple files in Vim. It turns outthat Vim
It's useful to record changes that you tried and failed to do
Today, for reasons beyond the scope of this entry, I decided to tryout enabling HTTP/2 on our support site . We already have HTTP/2 enabledon another internal Apache server, and both servers run Ubuntu18.04, so I expected no problems. While I could enable everythingfine and restart Apache, to my surprise I didn't get HTTP/2 onthe site. Inspecting the Apache error log showed the answer:
[http2:warn] [pid 104
Menu