Some implications of using offset instead of delta() in Prometheus

I previously wrote about how delta() can be inferior to subtractionwith offset , because delta() has toload the entire range of metric points and offset doesn't. Inlight of the issue I ran into recently with stale metrics andrange queries , there turn out tobe some implications and complexities of using offset in placeof delta() , even if it lets you make queries that you couldn'totherwise do.

Let's start with the basics, which is that ' delta(mymetric


One of my problems with YAML is its sheer complexity

YAML is unarguably a language,with both syntax (how you write and format it) and semantics (whatit means when you write things in particular ways). This is notunusual for configuration files; in fact, you could say that it'sabsolutely required to represent a configuration in text in anyway. However, most configuration file languages are very simpleones, with very little syntax and not much semantics.

YAML is not a simple language. YAML is a complex language, withboth a lot of


My problem with YAML's use of whitespace

Over on Mastodon, there was a little exchange:

@ceejbot :
current status: yaml

please send your thoughts & prayers to be with me at this difficult time

@cks :
It's impressive how yaml took Python's significant whitespace andsomehow made it far worse.

I've been thinking about my remark since I said it, and I thinkI've finally put my finger on a large part of why I feel that wayabout the YAML I've written for Prometheus .As best as I


Notifications and interruptions, and my view on them

On Twitter, I got irritated at Apple News :

Congratulations, Apple News: for allowing some random news serviceI've never indicated any interest in (or even looked at) to pusha notification (complete with a sound), you have completely andpermanently lost all notification privileges.

(For bonus points this was on the lock screen, which means thatApple News made a noise to attract my attention to an otherwiseinactive device.)

I'm not the only person that Apple News is doing this to


A gotcha with stale metrics and *_over_time() in Prometheus

We monitor the available disk space on our fileservers through custom metrics, and weallow people to set alerts on various levels of low disk space,which are evaluated through a Prometheus alert rule. Because severalpeople are allowed to each set alerts on the same filesystem, wehave to use group_right in the alert expression. A simplifiedversion of this expression is:

cslab_avail_bytes <= on (filesystem)  group_right(fileserver, pool) cslab_alert_minfree

Every so often we migrate


Some useful features of (GNU) date for things like time conversion

As part of using shell scripts to generate Prometheus metrics, and also sometimes wanting tointeract with Prometheus's API through the command line ( cf ), I've wound up gettingdeeper into the date command than I usually do.

(I've done some things with GNU date before, such as using' date -d ' , including for working outLinux kernel timestamps , somethingwhich is now mostly obsolete since ' dmesg -T ' will do that foryou.)

If you have a timestamp of


Various aspects of Python made debugging my tarfile problem unusual

I was recently thinking about what I like when I use Python, andin the process I wound up reflecting about how working out that the tarfile module is too generous about what is a tar file was made different and easier by variousaspects of Python. I'm not going to say that I couldn't haveworked out a similar problem in, say, Go ,but if I had, I think it would have been a relatively differentexperience.

One aspect of CPython specifically is that a lot of the


Brief notes on making Prometheus instant queries with curl

Every so often, I wind up wanting to directly extract some informationfrom Prometheus without going through a Grafana dashboard or eventhe Prometheus web UI. At this point I have stubbed my toes on thesame issues enough times (ie, more than once) that I'm going towrite down some how-to information for my future reference.

In general, the Prometheus HTTP API itself is tersely documentedin the obvious spot in the documentation , andsee also Extracting raw samples from Prometheus .Since it returns JSON,


How we're making updated versions of a file rapidly visible on our Linux NFS clients

Part of our automounter replacement is a file with a master list of all NFS mounts that client machinesshould have, which we hold in our central administrative filesystemthat all clients NFS mount .When we migrate filesystems from our old fileservers to our new fileservers , one of the steps is to regenerate thislist with the old filesystem mount not present, then run a mountupdate on all of the NFS clients to actually unmount the filesystemfrom the old fileserver. For a long time, we almost always had towait a bit


The appeal of using plain HTML pages

Once upon a time our local support site was a wiki, for all of thereasons that people make support sites and other things into wikis.Then using a wiki blew up in our faces . You mightreasonably expect that we replaced it with a more modern CMS , or perhaps a static site generator of somesort (using either HTML or Markdown for content and some suitabletheme for uniform styling). After all, it's a number of interlinkedpages that need a consistent style and consistent navigation, whichis