We had a weekend long power shutdown this past weekend in the building with our main machine room. As is our custom, we powered off the servers before hand (on Friday evening, with some surprises ), and then turned them back on this morning. This isn't the first time we've gone through such a power shutdown (although usually they're shorter), and over time we've written checklists and lessons learned for these things . This time was no exception, so I wrote checklists for both powering down everything and powering it back up (well in advance for once). Then we collectively looked at my nice, detailed, step by step power on checklist and ripped it up.

The issue is that powering things up in our environment is not really an orderly, step by step process. A lot of our systems are both core things and relatively independent of each other, and while there are ordering dependencies ( our fileservers have to be up before any NFS client, for example, and the DNS resolvers need to be up before the fileservers), they're small and at the start. Even in the ordering there's a lot that can be done at once, such as booting up all of the fileservers at once.

This structure, or lack of it, doesn't particularly fit in the traditional checklist format and process, which sort of assumes that you have a real order to things. Our power up process is more anarchic than that; at best it proceeds in stages, and even then there are multiple stages that can be done at once (such as turning on most of the firewalls and turning on the fileservers; neither depends on the other). Adding to the mix is the potential need to either troubleshoot things like failed PDUs or non-booted switches, or to decide to defer them to later.

This isn't the first time I've written up a power up list and had it more or less abandoned in practice (and our retrospective 'what actually happened' worklogs even talked about it). This is just the first time I've really admitted it up front.

I'm not sure what the best form of documentation is for our orderly cold start power up requirements, but it's certainly not a detailed checklist or anything that claims to be a linear narrative. Maybe what we want to do is simply list what everything requires, starting from the machines that don't require anything. Then everyone involved can look at what has all its requirements satisfied and go for it.

(A complication is that there are also some things that are ideally started early but if they're having problems it's not critical. For example, it's nice to have our central syslog server up early to collect everyone's logs right from the start, but it's not essential in the way that, say, our NFS fileservers or our local DNS resolvers are.)