More on systemd on Ubuntu 16.04 failing to reliably reboot some of our servers
I wrote about how Ubuntu 16.04 can't reliably reboot some ofour servers , then discovered that systemd can shut down the network with NFS mounts still present and speculated this was (and is) one of ourproblems. I've now been able to reliably produce such a reboot failureon a test VM and narrow down the specific component involved.
Systemd shuts down your system in two stages ;the main stage that stops systemd units, and the final stage, donewith systemd-shutdown ,
Shell builtin versions of standard commands have drawbacks
I'll start with a specific illustration of the general problem:
bash# kill -SIGRTMIN+22 1bash: kill: SIGRTMIN+22: invalid signal specificationbash# /bin/kill -SIGRTMIN+22 1bash#
The first thing is that yes, this is Linux being a bit unusual.Linux has significantly extended the usual range of Unix signalnumbers to include POSIX.1-2001 realtime signals ,and then can vary what SIGRTMIN is depending onhow a system is
Putting cron jobs into systemd user slices doesn't always work (on Ubuntu 16.04)
As part of dealing with our Ubuntu 16.04 shutdown problem , we now have our systems set up to putall user cron jobs into systemd user slices so that systemd will terminate them before it starts unmounting NFSfilesystems. Since we made this change, we've rebooted all of oursystems and thus had an opportunity to see how it works in practicein our environment.
Unfortunately, what we've discovered is that pam_systemd apparently doesn't always work right. Specifically, we've
ZFS's recordsize, holes in files, and partial blocks
Yesterday I wrote about using zdb to peer into ZFS's on-diskstorage of files , and in particular Iwondered if you wrote a 160 Kb file, would ZFS really use two128 Kb blocks for it. The answer appeared to be 'no', but I wasa little bit confused by some things I was seeing. In a comment,Robert Milkowski set me right:
In your first case (160KB file with 128KB recordsize) it does actually
Using zdb to peer into how ZFS stores files on disk
If you've read much about ZFS and ZFS performance tuning, one of thethings you'll have run across is the ZFS recordsize . The usualway it's described is, for example (from here ):
All files are stored either as a single block of varying sizes(up to the recordsize) or using multiple recordsize blocks.
For reasons beyond the scope of this entry, I was wondering if this wasactually true. Specifically, suppose you're using the default 128 Kbrecordsize
What I use printf for when hacking changes into programs
I tweeted :
Once again I've managed to hack a change into a program through bruteforce, guesswork, determined grepping, & printf. They'll get you far.
First off, when I say that I hacked a change in, I don't mean thatI carefully analyzed the program and figured out the correct andelegant place to change the program's behavior to what I wanted. Imean that I found a spot where I could add a couple of lines ofcode that reset some variables
Reading code and seeing what you're biased to see, illustrated
Recently I was reading some C code in systemd , one of theLinux init systems. This code is run in late-stage system shutdownand is responsible for terminating any remaining processes. Asimplified version of the code looks like this:
void broadcast_signal(int sig, [...]) { [...] kill(-1, SIGSTOP); killall(sig, pids, send_sighup); kill(-1, SIGCONT); [...]}At
A clever way of killing groups of processes
While reading parts of the systemd source code that handle late stageshutdown , I ran across an oddityin the code that's used to kill all remaining processes. A simplifiedversion of the code looks like this:
void broadcast_signal(int sig, [...]) { [...] kill(-1, SIGSTOP); killall(sig, pids, send_sighup); kill(-1, SIGCONT); [...]}(I've removed
Using a watchdog timer in system shutdown with systemd (on Ubuntu 16.04)
In Systemd, NFS mounts, and shutting down your system , I covered how Mike Kazantsev pointed me at the ShutdownWatchdogSec setting in system.conf as a way of dealing with our reboot hang issues. I also alluded tosome issues with it. We've now tested and deployed a setup usingthis, so I want to walk through how it works and what its limitationsare. As part of that I need to talk about how systemd actually shutsdown your system.
Under systemd, system shutdown happens in two stages
My potential qualms about using Python 3 in projects
I wrote recently about why I didn't use the attrs module recently ; the short version is that it would haveforced my co-workers to learn about it in order to work on my code.Talking about this brings up a potentially awkward issue, namelyPython 3. Just like the attrs module, working with Python 3 codeinvolves learning some new things and dealing with some additionalconcerns. In light of this, is using Python 3 in code for worksomething that's justified?
This issue is relevant