Yesterday I wrote about how on one of our servers, systemd took three seconds to load (user) units as part of setting up your systemd user session. I also mentioned that various operations were unusually slow on the system in general, but I couldn't see anything in our metrics system to suggest slow IO or the like. Today, I solved both problems and it turned out to be our own fault.

This particular server is our SLURM cluster's 'head node' (master server) , which we allow people to log in to because it used to be the only place you could submit SLURM jobs from. However, we had long standing problems with people accidentally running their CPU-consuming jobs on this SLURM head node instead of in a submitted SLURM job, or doing CPU and memory-intensive preparation work on it (including personal package installs, which in some languages and for some sets of packages can be quite resource intensive). In order to discourage this, we wound up setting very low systemd-based CPU and memory limits on people's user-<uid>.slice units ( cf ); for years, the limits were 128 MBytes of RAM and 25% of (one) CPU. It turns out that in Ubuntu 26.04, these limits are sufficiently low that things such as 'dpkg -S' run appreciably slowly.

(I don't know if it's the memory limit, the CPU limit, or the kernel impact of setting a CPU limit below full time use of one CPU. This didn't used to happen on previous Ubuntu versions, at least not to such a noticeable degree, but programs keep growing and so on. Other login servers with per-user limits have much bigger limits and so don't run into this even on 26.04.)

That turned out to be the cause of all of the slowness I saw once I was logged in (in dpkg and other things), but it didn't seem to explain the slowness in systemd user session setup, because these limits are only set after the systemd user session has been created (we set them through pam_exec running a script after pam_systemd has run and made the systemd user session if necessary). And, famously, your user-<uid>.slice unit goes away after you entirely log out, so the old limit settings weren't being carried over through an old user-<uid>.slice unit. Or so I thought, but I was wrong.

Our per-user limits are set with ' systemctl --runtime set-property user-<uid>.slice ...'. As documented, --runtime doesn't just change the running unit in memory, it also writes your changes to files in /run/systemd/system.control/<unit>.d/ , and these written out files are then persistent until the next reboot wipes out /run. When systemd recreated a user-<uid>.slice that had already been set up with limits (for example, because I'd logged in once already), those /run files meant that it re-applied our (low) limits right from the start and then the systemd user session process was constrained and (drastically) slowed by those limits. Raising the limits to more reasonable values completely eliminated the slow systemd user session startup, making systemd's user session startup as fast as expected.

I had vaguely known that --runtime wrote things to /run to persist them, but I hadn't put the pieces together in my mind to realize that the per-user 'user-<uid>.slice' units would immediately have those limits applied again the moment they started. So I believed that all systemd user sessions started out completely unlimited, even if I'd logged in before, and limits were only applied once the session existed. A consequence of this that I want to remember is that stopping setting limits on someone at login time doesn't remove limits existing limits set in past login sessions, even if the person logged out completely. To undo setting per-user limits, you need to clear them somehow (or reboot the machine).

(Systemd may cache things in memory, so I'm not sure if wiping out the /run files is sufficient to completely reset the situation. I believe clearing CPU limits in cgroup v2 is done by setting ' CPUQuotaPerSecUSec ' to 'infinity' and clearing memory limits is done by setting ' MemoryMax ' and/or ' MemoryHigh ' to 'infinity'. Set ' TaskMax ' to some large number; the system default varies from machine to machine.)