What I think goes wrong periodically with our Grafana Loki on restarts

We have now had two instances where restarting Grafana Loki caused it to stop working.Specifically, shortly after restart, Loki began logging a flood ofmysterious error messages of the form:

level=warn ts=2024-01-29T19:01:30.[…]Z caller=logging.go:123 [...] msg="POST /loki/api/v1/push (500) 148.

Servers are (probably) starting to drop serial ports

One of the things that we havehad for a long time is a serial console server ,which is to say a server that collects and logs serial consoleoutput from all of our regular servers (this is primarily Linuxkernel messages , as we've movedaway from making the serial port the 'real' kernel console ). To date we've done this through actualserial ports on the servers, which we set up as an additionalconsole. However, this has an obvious issue, which is that your


Getting the Python LSP server working with venvs the brute force way

Recently I wound up doing some Django work using a Python venv , since this is theeasiest way to get a self-contained Python environment that hassome version of Django (or other applications) installed. However,one part of the experience was a little bit less than ideal. Inormally write Python using GNU Emacs and the Python LSP server , and this environmentwas complaining about being unable to find Django modules to docode intelligence things with them. A little thought told me why;GNU Emacs was running my


Histogram data is most useful when they also provide true totals

A true histogram isgenerated from raw data. However, in things like metrics, we generallydon't have the luxury of keeping all of the raw data around; insteadwe need to summarize it into histogram data. This is traditionallydone by having some number of buckets with either independent orcumulative values . A lotof systems stop there; for example OpenZFS provides its histogramdata this way . Unfortunatelyby itself this information is incomplete in an annoying way.

If you're generating histogram data, you should go


In Go, I'm going to avoid using 'any' as an actual type

As modern Go programmers know, when Go introduced generics it alsointroduced a new ' any ' type. This is officially documented as:

For convenience, the predeclared type any is an alias for the emptyinterface.

The 'any' type (alias) exists because it's extremely common in codethat's specifying generic types to want to be able to say 'anytype', and the way this is done in generics is 'interface{}', theempty interface. This makes generic code clearly


The cooling advantage that CPU integrated graphics has

Once upon a time, you could readily get basic graphics cards,generally passively cooled and certainly single-width even if theyhad to have a fan in order to get you dual output support; this is,for example, more or less what I had in my 2011 era machines . These days these cards are mostlyextinct, so when I put together my current office desktop I wound up with a dual width, definitelyfan-equipped card that wasn't dirt cheap. For some time I


CGI programs have an attractive one step deployment model

When I wrote about how CGI programs aren't particularly slow thesedays , one of the reactions I saw was to suggest thatone might as well use a FastCGI system to run your 'CGI' as a persistent daemon, saving you theoverhead of starting a CGI program on every request. One of thepractical answers is that FastCGI doesn't have as simple a deploymentmodel as CGIs generally offer, which is part of their attractions .

With many models of CGI usage and configuration, installing a CGI,removing


Desktop PC motherboards and the costs of extra features

My current office desktop and homedesktop are now more than five years old(although they've had some storage tuneups since then), so I'vebeen looking at PC hardware off and on. As it happens, PC desktopmotherboards that have the features I'd like also not infrequentlyinclude extra features that I don't need, such as built in wificonnectivity. I'm somewhat of a hardware minimalist so in the pastI've reflexively attempted to avoid these features. The obviousreason


The expected size of a gap in a Prometheus range vector (sometimes)

In yesterday's example of delta() extrapolating to cover a fulltime range , we saw that an examplefifteen minute range vector for a metric actually covered a timerange less than fifteen minutes. In fact, it covered fifteen secondsless than fifteen minutes, and the scrape interval for the metricin question was fifteen seconds. In thinking about it, I've realizedthat this isn't a coincidence and in fact I believe that nearly allof the time, many range vectors for many time ranges will


An example of how Prometheus's delta() function will extrapolate time ranges

Recently, someone came to the Prometheus mailing list with aninteresting issue they were having, where they were using Prometheus's delta() function to look at the amount of change over some time range, butwere getting results that they didn't expect. They had a relativelyslow changing metric where they could look at the value at the startof a fifteen minute time interval, the value at the end of it, andthe delta() result for 'metric[15m]', but