We need to start getting some experience with using Ubuntu 20.04

Under normal circumstances, we would have a decent number of machinesrunning Ubuntu 20.04 by now, probably including our login servers.But the situation is not normal, because ongoing world and localevents still haveus working from home, making it not so simple to install and deploya new physical server with a new version of Ubuntu. However, itreally looks like this is the new normal so we should start dealingwith it.

It may or may not make sense to spend limited in-office


A potential Prometheus issue for labeled metrics for infrequent events

One of the things you often get to do with Prometheus is to designyour own custom metrics for things (which may be generated andexposed in a number of ways, for example using mtail to extract them from log files).One piece of advice for designing metrics that I've seen is to groupclosely related measurements together under one metric name usinglabel values to differentiate between them. The classical examplehere is counting HTTP requests with different return codes usingone metric name with a label for the return code,


Go is gaining the ability to trace init calls on program startup

Go packages can have init() initialization functions, which arecalled when a Go program starts as part of package initialization . One of thepractical issues with init functions in Go so far is that theirperformance and even their existence is relatively opaque, so thatit's hard to tell how much of an impact they have on the startuptime of your programs.

The good news is that the Go team is moving to change this lack ofvisibility, as tracked through this issue and recently landed inthe development version


Go packages can have more than one init() function

Go has some surprisingly complex rules for how packagesare initialized, partly because package level variables can beinitialized based on the value returned from function and methodcalls (and then other variables can be initialized from them). Aspart of package initialization, you can have an initializationfunction, called init() , that will be called.

Or at least that's what I would have told you before I actually hada reason to read that section of the Go language specificationtoday. In fact, the specification is


As an outsider, I prefer issue tracking to be in its own application

I recently read Josef "Jeff" Sipek's Email vs. Tool du Jour ( via ).I broadly agree with Sipek's call to generally use email to othermechanisms, but after mulling over my reaction to one portion ofit, I disagree with Sipek over issue tracking, at least some ofthe time. Sipek says:

First, what about issue tracking? How does that tie into myemail-centric world? Well, you can keep your issue tracker, but in myopinion, the comments


If you send automated email, you should scan it with anti-spam software

Partly in light of Microsoft SharePoint's problem with spam , here is an obvious thing:

If you send automated email to the outside world, you should alwaysrun it through some anti-spam system and raise big alarms if theyever trigger.

(If I was doing this today I would use ClamAV and rspamd becausethey're free and what I'm used to, but anything will do.)

You should do this even if you aren't allowing external people toinitiate the email and put content


Microsoft SharePoint is being used to send spam

I'm paying more attention to what our mail system detects as spamand where it's coming from than usual, so I'm getting to noticethings (or, in the alternate phrasing, being forced to noticethings). Today's thing that I noticed is that to no one's surprise,Microsoft SharePoint is currently being used as a spam sendingvector. I say 'to no one's surprise' because it's a long standingrule that anything that can be used to


Our current usage and views of UPSes (late 2020 edition)

Over the time I've been here ,we've had rather mixed experiences with UPSes. Initially we usedthem relatively freely on machines that we felt were important,such as our first generation ZFS fileservers . Unfortunately, after a while theUPSes themselves caused us problems (for example, a spontaneousUPS reset that power cycled the machine attached to it ), and we grew much more disenchantedwith them.

These days we still use UPSes on some machines, such as ourfileservers , but pretty much onlyon machines


Wanting to be able to monitor for electrical power quality issues

Today, we had what you could call a "power event" at work. Therewas some sort of power blip in the electrical feeds from TorontoHydro to multiple university buildings, and a number of our serversrebooted (not all of them, though). We've seen this kind of thinghappen before so in one sense it's not surprising, but this timearound the actual experience was rather alarming and confusingbecause it happened during the working day and we were all workingremotely, so we couldn


Whether extra disks should be live or spare now depends on HDs versus SSDs

Suppose, not entirely hypothetically, that you have a server withat least three or four spare drive bays and you want to build amirrored storage setup that can maintain redundancy without requiringan in-person drive swap should a drive fail. Let's say you go withthree drives in the system in total. Obviously two of them have tobe mirrored in order to get your basic redundancy, but the thirdone could be used either to make a three way mirror or held backas a (configured) hot