Sometimes it actually is the network: a war story
We've recently been having mysterious problems getting some of ourbackups to perform well . Also I recently wrote about how we'd wound up with a web server that frequently saturated itsoutgoing 1G interface with traffic (and it was a feature that itdidn't have a faster network link). These two things turn out to notbe as unrelated as we'd like, and there's a story or two there.
The problematic backups aren't done through our usual Amanda-basedbackup
What buffer size (OpenSSH) <code>ssh</code> seems to use for streaming output
Suppose that you're generating and transferring a file over ssh, forexample to create a tar archive of something on a remote server andsave it locally:
ssh [...] rem-server 'cd /var/log && tar -cf - .' >/backup/file.tar
If you're experiencing IO problems in this backup process , an interesting question is what buffer size ssh uses for its writes, perhaps because you'd like to make a few largewrites to disk
Go maps, hashes of map keys, and pointers: a little surprise
Go's maps are famouslyimplemented as hash tables, which is the only reasonable choice. Theimplementation has gotten somewhat more complicated since I lookedat how maps store their values and keys due to themove to swiss tables , and thesedays you find the comments about how they work in internal/runtime/maps/map.go ,but the core is still the same. Recently, a documentation commit landed in the Go development tree that opened my eyes to a bit ofsubtle complexity I hadn't considered
Discovering rsync's -W option and our use for it
Suppose, not hypothetically, that you use rsync to push an encrypted backup file from themachine it's created on to a fileserver , where it will be backed up by yourregular backup system . Because this encryptedbackup file is backed up every day, you only need one copy of it onthe filesystem, so you use and reuse a fixed name for the file. Inother words, we're using rsync somewhat as if it was scp , but with better control over what remote files can be written
I'm only interested in "native" installation systems
There are a variety of ways to automatically or semi-automaticallyinstall systems, especially Linux systems and especially over thenetwork. People have built a whole raft of them over the years, oftenwith relatively impressive capabilities. A number of them havesignificant levels of automation and control, including for things wemight want like sophisticated automatic disk setup . Despite that, we'reinterested in approximately none of them. As a practical matter, weonly want to use the standard, native install systems for whateverwe're
Our long path from IPMI remote installs to network installs
Once upon what's now a long while ago, we had a bunch of SunFire X2100and X2200 1U servers (for example, they were used in our firstgeneration ZFS fileserver ). One of thereasons that I loved these servers back in the days was that for free,they came with a full IPMI / BMC setup that supported both KVM over IP and virtual media. I installedany number of servers this way from the comfort of my office, usingthe
As expected, using kexec to switch to a new Ubuntu kernel works
For a long time, I ignored kexec both for myown personal machines and at work .I knew it existed but I never attempted to use it. That changedrecently when I realized we could use kexec to start a network(re)install environment withoutneeding our servers to actually have network booting enabled (whichthey mostly don't currently, for historical reasons). This has led meto do some additional experimentation with kexec .
The most recent experiment was updating to a new Ubuntu kernel andthen using '
There's a plague of Googlebot impersonation going on (in June 2026)
A while back I wrote about how claiming to be Googlebot is now a badidea , where I noted that there were (reportsof) malicious crawlers out there impersonating Googlebot and otherlegitimate big crawlers and at the same time, Google and other crawleroperators published the IP address ranges their crawlers used. Youcould put these two together to block these impersonators:
Anything claiming to be Googlebot that is not from a listed GoogleIP is extremely suspicious and in this day and age of increasinganti-crawler defenses, blocking all
Go interfaces, reflection, and binary size
Recently an interesting series of commits landed in Prometheus with the goal of reducing the size of thePrometheus binary by allowing the Go linker to remove more unused code( something it's quite good at in general ,although the linker is also deliberately limited in this ). The commit with the message that's mostinformative about what is going on and why is discovery/gce: keep[Google Cloud] Compute SD client from defeating dead-code elimination ,and you can read the full details in it. The
How your Ubuntu 26.04 server boots with a software RAID array root filesystem
One answer to how your Ubuntu 26.04 server boots when its rootfilesystem is on a software RAID array is that it just does and youdon't need to think about it. Unfortunately this wasn't the case inthe pre-beta version . That'sbeen fixed since ( contrary to what I thought until now ), but that raisesthe question of what changed between 24.04 LTS and 26.04 LTS to causeand then fix this.
At some