For a long time, Linux has perhaps been in a transition from thevenerable iptables to nftables . Ihaven't been terribly enthused about nftables in the past ( thelast time I considered it I was persuaded to take a different path ) for an assortment of reasons. My recentuse of nftables wasn't because I've changed my spots; instead, it'sbecause Ubuntu 22.04 has chosen to make nftables the native interface.If you install a stock 22.04

Recently we upgraded ourPrometheus server from Ubuntu 18.04 to 22.04, moved it to newhardware, and migrated it from using a mirrored pair of 4 TB HDDsto a new mirrored pair of 20 TB HDDs .One of our goals in this migration was for the server to be downfor as short as possible, because when the server is down we're notcollecting metrics (including ones that we alert on, such as thetemperature of the department's machine rooms

The simplest way to set up Grafana Loki is as a single server that handlesboth ingesting logs and querying them, much like Prometheus . Unlike Prometheus, Loki is a 'push' systeminstead of a 'pull' one, where clients send logs to Loki (via HTTP)instead of Loki reaching out to collect logs from them. This mattersbecause of access permissions; it's one thing to allow a system to talkto Loki to send logs, but another thing to let it query everyone'slogs

In yesterday's entry on a Grafana Loki cardinality issue with themetadata you can harvest from systemd log messages , I said that I thought we'd windup dealing with the issue by sending systemd journal messages to Loki in the journal's JSON format.This JSON format preserves all of the available fields and Loki can process it to extract them when you access the logs.Having actually tried doing this, I've now given up on the idea.

In theory, sending JSON to Loki is the correct

Grafana Loki ( documentation ) is described as 'Prometheusfor logs', or to quote its website, it's 'a log aggregation systemdesigned to store and query logs from all your applications andinfrastructure'. Similar to Prometheus, it has the idea of datapoints having both labels and a value (and a timestamp); where inPrometheus the value was always a number, in Loki the 'value' isthe log message. On a modern Linux system, the obvious easy way toget started with

When I wrote yesterday's entry on producing and parsing a commonUnix log timestamp format ,I expected to casually find a bunch of examples of programs usingthat format. What I found instead was a veritable garden of timestampformats, and today I feel like inventorying them. My examples willuse real timestamps plucked from our logs; representing them in Go or C time formatting is up to you.

(I thought about generating Go formats for all of these then realizedI would need to test them all. That'

One common form for timestamps in Unix log files is timestamps thatcome out looking like '2022-07-16 20:50:35', which is to say thedate (in a sensible order) and then the time, with no timezone.Unless the program writing the logs is perverse, the timestamp isin the system's local time (whatever that is), not something fixedlike UTC (of course the system's local timezone may be UTC

Today I learned something that I could have noticed if I'd paidenough attention, which is that the default ' iptables -L ' outputdoesn't show you interface matches that are part of rules . Inorder to see this in 'iptables -L' you have to make it verbose (with-v), which also shows you the volume counters. There is a storyhere.

Normally I look at my iptables rules with ' iptables -vnL ' (andoften a particular chain, like INPUT

Here's something that hadn't really occurred to me until I saw ithappen: an IMAP server can have a significant imbalance between the levels of diskIO and network IO (network traffic). This imbalance can go in bothways; you can have a lot of read or write IO with low networktraffic, and less commonly you can have a lot of network trafficwith low disk IO. I think an imbalanced situation is especiallylikely with disk writes, because of the nature of the IMAP protocol.