We have a whole building, weekend long power shutdown in the building with our machine room that officially starts tomorrow (Saturday) morning at 5am, which is the motivation for our newly added temporary backup MX . Because we like to be in control of both the shutdown and the startup of our machines, we turn machines off in advance for scheduled outages (there's not much we can do about unscheduled ones ). For various reasons we did the shutdown earlier this evening.

(One reason to start machines under controlled circumstances is that sometimes hardware fails , things go wrong , or you discover unfortunate aspects of your environment ( also ). At least these days we've mostly learned lessons from previous power shutdowns and startups , although there are aspects I hadn't fully absorbed and will write about later.)

During the shutdown, something surprising happened, which is that all of our ZFS fileservers came back to life . We definitely ran ' poweroff ' on each of them and they were off the network for some amount of time, but then my co-workers doing work in the machine room noticed that they were all powered back on. We ran ' poweroff ' on the rebooted servers and they shut down properly, rather than rebooting, so that part's not the problem. After some discussion we decided to deal with the immediate problem by pulling their power plugs, so they can't come back on even if something on board wants them to (all of these servers have BMCs ).

One of the things we did between the fileservers shutting down and them coming back up is that I ran fping to scan the subnet they're on, to see if we'd missed shutting down any machines (and this fping run showed that none of them were on the network at the time). The host I ran fping from was on the same network and would have still had the MAC addresses of the fileservers in its ARP cache, so it could have directly unicast packets to the MAC.

One theory we have is that this triggered some sort of 'Wake on LAN' power up behavior. I wasn't pinging with a WoL 'Magic Packet', but as covered in sources like the Linux ethtool(8) manual pages, your hardware may potentially support a whole host of WoL mechanisms, including 'unicast messages'. This sounds like it might cause a server to wake up if its network interface receives a packet to its hardware MAC. Such as, for example, an ICMP ping packet that didn't need an ARP because the sending host already knew the target's MAC.

(I can't find much documentation on what these Wake on LAN options mean, but see eg here , this chipset documentation , or FreeBSD's ifconfig and its 'wol' options.)

When the power shutdown is over and we bring the fileservers back up on Monday, we'll be looking at what ' ethtool ' reports as their Wake on LAN settings. Since they have fully capable BMCs , we may want to force all of them to have no Wake on LAN active at all. Certainly it seems undesired to have them potentially powering up based on just receiving packets, since there's a whole host of ways they could receive traffic.

PS: We haven't seen this in past power shutdowns, but our fileserver hardware was refreshed between the last one and now.