Modern (x86) firmware can be really slow to boot, which can be painful

We 've recently wound up with some unusually big (x86) servers, from a vendor we don't normally get servers from. One of the unusual things about these servers is that they take a very long time to initialize on boot (at least by our standards). I haven't actually timed it, but I'm pretty certain there's more than a minute of staring at 'chipset initializing', and it may be as much as several minutes.

In a way this isn't entirely surprising, because modern systems are really complex under the hood ( cf ). Your RAM modules may require 'training' ( also ) to reach reasonable speeds (and these servers have 24 of them each), PCIe busses and devices require initialization and probing (and these servers have a lot of PCIe devices), and so on and so forth. It's not as if we haven't had relatively slow booting x86 servers before, and how much the BIOS tells you during this process is probably partly a design choice (although presumably you have to get a fair distance into the overall hardware initialization process before you can display much video).

(In theory I believe the firmware saves data from 'training' RAM so it doesn't necessarily have to go through the full process every time. Possibly there are circumstances where this doesn't work out in practice, or there are other things going on with these servers.)

In some ways a long firmware boot is just an inconvenience. It means rebooting or powering on one of these servers takes an extra long time, but that's not a big issue for these servers. We're not using them in a situation where how fast they take to reboot is all that important, although it is kind of annoying to start a reboot of a server then have to make a mental note to come back in a few minutes when it might have finished the process (it's easy to get distracted).

Where it gets painful is when you (I) need to interact with the BIOS firmware (or the Linux bootloader) in the brief window between when the firmware initializes enough to start accepting input, when it switches to the bootloader (GRUB for us), and when the bootloader brings up the default kernel, at which point it's too late to affect either. If I need to get into BIOS firmware setup, I get to sit watching the 'chipset initializing' for as long as it takes, and that's tedious (and painful if I miss my window of opportunity).

(I was going to say it would be nice if the server's BMC had an option to go into BIOS setup on a triggered reboot or power-on, but now that I look the BMC has a 'remote BIOS setup' feature. Amusingly, this works through a completely different web environment than the BMC's web environment. For example, the 'remote BIOS setup' sub-site uses HTTP Basic Authentication instead of the BMC's conventional cookie-based login process.)

PS: I'm pretty sure this isn't the slowest-starting firmware I've seen. Long ago I dealt with SGI servers that my memory asserts took at least five minutes to get through their firmware ( cf ), but they had the excuse that they had much, much slower MIPS CPUs.

( 2 comments .)