You can't always trust a BMC's inventory of the server's hardware
Many BMCs will tell you what hardware yourserver or other system has in it. This is a useful, even valuablefunction, but you shouldn't necessarily trust what your BMC is saying. We recently had a very vividdemonstration of that, where the BMC of a server confidently reportedthat the hardware had a number of inexpensive NVMe SSDs, a bunch of(expensive) RAM, and a pair of processors, but mysteriously the serverwouldn't power on at all. When we opened up the server,
Some things on '<code>systemctl kexec</code>' as compared to '<code>kexec -e</code>'
Suppose, unfortunately not hypothetically, that you have some machinesthat the Ubuntu 26.04 LTS installer kernel sometimes gets a kerneloops during anetwork based reinstall (we don't think this is a hardware flaw, butwho knows; these machines were stable on 24.04). Further suppose thatyou're not network booting thesemachines but instead you're using kexec to boot them into theinstaller environment . This createsan awkward situation, where the over the network installer may have
Two versions of a 'is SSH up on a machine' check
One of my standard little scripts is something I call sshup , which waits for a machine to be 'up' byperiodically checking to see if its SSH port is responding. Asmentioned in my original entry on sshup , Iactually have two versions of this script and recently I discoveredthat the difference is quietly important.
One version of the script uses Netcat, on our Ubuntu machines. The specific Netcatcommand line it uses is:
nc -w3 -q3 -z "$1" ssh >/
The quiet issue of lurking settings (and how it bit me)
Recently I wrote an entry about a simple but difficult wish I hadfor a certain sort of terminal pager that handled emoji . In comments, people suggested thatthe venerable less would do whatI wanted, which was something I'd already tried and discovered it hadbehavior I didn't want. Except, well, let me quote my eventualcomment:
It turns out that my testing of less's behavior was beingcontaminated by my usual
$LESSsettings, and I think that 'less-XRn' does
Configuration is a liability, just like code
One of the broadly accepted things among at least systemadministrators is that our own code is a liability (some programmersmay resist this idea somewhat more). If your systems run on acollection of locally developed, bespoke programs, scripts, and so on,someone has to maintain and update all of that ( I sort of wroteabout this long ago , also ).If you use standard programs instead (or at least as much standardcode as possible), hopefully someone else does that, and you canchoose
PyPy and Python 3 for us
Ever since I started using PyPy and then had it quietly work finefor years , I've been keeping it in mind as agenerally easy way to speed up any Python program that could benefitfrom a performance boost (which programs PyPy could accelerate sometimes surprised me ). But that was inthe era when most everything we had was Python 2 based. Now that I'mmoving more and more things to Python 3, there's a little issueopening up that I've been thinking about. That
Limiting web server bandwidth the brute force way
A while back on the Fediverse, I mentioned something about ourdepartmental web server :
Wow, people do some extremely slow downloads from work's main webserver. We roll logs at midnight and Apache wrote the last logrecord to the old file at 03:58 (for a request that had started at23:27).
(For our sins we've become a load-bearing source of ML imagetraining data, for the "CIFAR" image dataset. We average 40
Some reasons why your server may not be doing a UEFI network boot
In theory, network booting with UEFI is fairly straightforward. Unlikethe earlier BIOS based network booting, UEFI defines network bootingin the standard itself (although it'snot required). In practice, vendors of x86 servers have found anynumber of creative ways to put stumbling blocks in your way in theirfirmware (aka 'BIOS'). Here's an incomplete list of reasons that yourserver might not be willing to do a UEFI network boot the way youexpect it.
- The firmware might not
How to correctly not wait for network carrier in Netplan
Yesterday I wrote an entry about why servers running Ubuntu canstall on boot for two minutes ; the shortversion is that in 26.04, a network interface that either has nocarrier or that has nothing else on it will cause systemd-networkd-wait-online to wait for two minutes in the hopes that this changes and theinterface becomes healthy. My core diagnosis of the trigger for theproblem was correct, but I had the wrong fix because in my testing, Imade a classic mistake that
Why servers running Ubuntu can stall on boot for two minutes
Suppose, not hypothetically, that you have some standard physicalservers. As is typical, the servers have more than one networkinterface (two is usually the basics even for 1U servers, and maybeyou put in a 10G-T card in some), but you're only using one networkinterface on one network; the others are just sort of there. Recentlyyou've started putting Ubuntu 26.04 LTS on them, and to your surprise(and displeasure) these machines