For reasons outside of the scope of this entry, we have an increasing number of servers in an inconvenient location ( I called it 'offsite' but that's not quite accurate). Since these servers run Ubuntu LTS, they're going to need to be reinstalled with new versions every so often, starting this summer (as 26.04 comes out), and we really don't want to do that in person, so we've been thinking about our options.

The best option would be to only have servers with full support for remote management through their BMC . By full support, I mean full 'KVM over IP' support for remote access as well as remote media, so that we could continue to use our regular install processes, exactly as if we were physically present to plug in a bootable USB stick and so on. Unfortunately lots of our servers either don't have a BMC at all or have a BMC with restricted and limited features (because we haven't bought an expensive license, for example).

The cheapest solution that is the most work to add and implement is network booting (and this only handles our reinstall issue). Assuming that all of our servers can reliably boot from the network (which isn't a sure thing), I believe that we could set up an environment that booted into our install setup, then had us SSH in to the Ubuntu server installer, where we could handle things more or less as if we were booting and installing locally. In a UEFI environment, in theory we can reliably switch boot entries around to do things like a one-shot network boot. Another nearby group does this for all of their servers (running Debian), so it's definitely possible to handle things this way. This option is arguably the technically correct way to handle installs and reinstalls, but will take much more staff time to set up.

The most general and easy to implement solution is a modern external 'KVM over IP' system. These plug into the video and USB ports of your server or servers, then let you access everything over IP, usually through an embedded web server. Since they use USB for their keyboard and mouse, it's easy for good modern ones to also support 'remote media', presenting either a USB disk or a USB DVD drive to the host. Often you can either upload your image (or images) to the internal storage on the KVM over IP system or stream it from somewhere else. For various reasons this isn't as good as a fully capable BMC, but you can get rack-focused multi-server systems that will let you connect one head unit to 8, 16, or more servers, as well as smaller scale single system units (including FOSS-based ones that can do useful tricks like run WireGuard).

(Most of the small scale systems only support HDMI. The rack systems often support VGA and DVI as well, which is good, because lots of our servers are still VGA. You can get converter dongles but it's better to have fewer pieces of hardware.)

For our purposes we'd like a KVM over IP system that supports more than one server at once but we don't need too many. Our major use is reinstalls and other troubleshooting, and we don't too many of those at once, so it's okay if we have to walk over to the datacenter location to shuffle which eight servers in a rack the KVM over IP system is currently connected to. However, for reasons outside of the scope of this entry it would be very useful if the KVM over IP system was rack-mountable (in a switch style two-post fashion would be fine).

The external KVM over IP option is more expensive in direct up front costs than building a network booting environment, but it requires a lot less staff time to build and maintain. On the other hand, at a university staff time is often considered a sunk cost that's ignored , so we may wind up with network booting even though we could probably get an 8-server KVM over IP solution for a cost that works out to only a few days time of a single system administrator.

(We could get a basic, single server, non-rack-mount, HDMI only KVM over IP box for trivial amounts of money, but that doesn't work as well in this situation.)

(A bit of me would enjoy the challenge of designing and building a network boot install environment, but the rest of me is looking at the amount of regular work we have and the fact that this is a repeatedly solved problem and feeling unenthused. Someone out there may even have written up how to go from having an Ubuntu server installer image to network-booting it on some machine, but good luck finding that writeup on today's Internet.)