Some notes on Grafana Loki's new "structured metadata" (as of 3.0.x)

Grafana Loki somewhat bills itself as "Prometheus for logs", andso it's unsurprising that it started with a data model much likePrometheus. Log lines in Loki are stored with some amount of metadatain labels, just as Prometheus metrics values have labels (includingthe name of the metric, which is sort of a label). UnfortunatelyLoki made implementation choices that caused this data model to berelatively catastrophic for system logs (either syslog logs or thesystemd journal). Unlike Prometheus, Loki stores each set


Flaky alerts are telling you something

Sometimes, monitoring and alerting systems have flaky alerts, eitherin the form of flapping alerts (where the alert will repeatedlytrigger and then go away) or alerts that go off when there is noproblem. Broadly speaking, these flaky alerts aren't just noise;they're telling you something.

To put it one way, flaky monitoring system alerts are like flakytests in programming. Each of these is telling you that yourunderstanding of things is incorrect or that something odd andunusual is going on, and


Reasons to not expose Go's choice of default TLS ciphers

When I wrote about the long-overdue problem people are going tohave with go:linkname in Go 1.23 , thespecific case that caused me to notice this was something tryingto access crypto/tls 's'defaultCipherSuitesTLS13' variable. As its name suggests, thisvariable holds the default cipher suites used by Go for TLS 1.3.One reaction to this specific problem is to ask why Go doesn'texpose this information as part of crypto/tls 's API.

One


The long-overdue problem coming for some people in Go 1.23

Right now, if you try to build anything using the currently releasedversion of github.com/quic-go/quic-go with the current developmentversion of Go (such as this DNS query program ), you will probably encounter thefollowing error:

link: github.com/quic-go/quic-go/internal/qtls: invalid reference to crypto/tls.defaultCipherSuitesTLS13

Experienced Go developers may now be scratching their heads abouthow quic-go/internal/qtls is referring tocrypto


There are multiple uses for metrics (and collecting metrics)

In a comment on my entry on the overhead of the Prometheust hostagent's 'perf' collector ,a commentator asked a reasonable question:

Not to be annoying, but: is any of the 'perf data' you collect herehonestly ' actionable data ' ? [...]In my not so humble opinion, you should only collect the type of datathat you can actually act on.

It's true that the perf data I might collect isn't actionable data(and thus not actionable


The Prometheus host agent's 'perf' collector can be kind of expensive

Today I looked at some system statistics for the first time in quitea while and discovered that both my office and home desktops were handling about 12,000to 15,000 interrupts a second. On my office desktop this used about1.5% of the overall (multi-)CPU for IRQ handling; on my home desktopit was just over 4%. Eventually I traced this down to me havingenabled the Prometheus host agent's 'perf' collector.This collector


Go's old $GOPATH story for development and dependencies

As people generally tell the story today, Go was originally developed without support for dependency management.Various community efforts evolved over time and then were sweptaway in 2019 by Go Modules , which finally added coresupport for dependency management. I happen to feel that this storyis a little bit incomplete and sells the original Go developersshort, because I think they did originally have a story for how Godevelopment and dependency management was supposed to work. To me,one of the fascinating bits in Go's


The power of using external authentication information in a web application

Recently, a colleague at work asked me if we were using theuniversity's central authentication system to authenticate accessto our Grafana server .I couldn't give them a direct answer because we use Apache HTTPBasic Authentication with a local password file , but I could give them a pointer.Grafana has the useful property that it can be configured to takeauthentication information from a reverse proxy through a HTTP header field, and you can set up Apache with Shibboleth authentication so that it uses the institutional authenticationsystem (with


My GNU Emacs MH mail folder completion in MH-E

When I wrote about understanding the orderless package , I mentioned that orderless doesn't work well withhierarchical completions such as file names, which are completedone component at a time. I also said this mattered to me because MH-E completed the names of mail folders in this part by part manner,but I didn't feel like rewriting MH-E's folder completion system tofix it. Well, you can probably guess what happened next.

In the GNU Emacs way, I didn't so


Realizing the hidden complexity of cloud server networking

We have our first cloud server .This cloud server has a public IP address that we can talk to, whichis good because we need it and feels straightforward; we have lotsof machines with public IP addresses. This public IP address has afirewall that we have to set rules for, which feels perfectly normal;we have firewalls too. Although if I think about it, the cloudprovider is working at a much bigger scale, which makes it harderand more impressive. Except that our actual cloud server has