Unsurprisingly, the clock in your server's IPMI drifts over time
Most servers these days have an IPMI ,including more or less all of ours. One of the standard IPMI functionsis a 'system event log', which stores 'events' with timestamps,which means that the IPMI needs to have a clock. Some of our IPMIsare both connected to a network and support maintaining their timethrough NTP, but most of our IPMIs are either disconnected or atleast don't support NTP (as far as I know). This means that theirclocks are what's
Some notes on lslocks, the Linux command to list current file locks
Lslocks(8) is the Linux command that you usually use to list current file lockson a machine. Lslocks uses the kernel's /proc/locks to find out aboutlocks, and so is subject to various limitations /proc/locks has . It adds some conveniences to the raw /proc/locksinformation, but also has some limitations of its own on whatinformation it can present when.
Run as root, the normal lslocks output looks something like this:
COMMAND PID TYPE SIZE MODE M START END
Some notes on Linux's /proc/locks listing of file locks
As covered in the proc(5) manual page , /proc/locks lists current flock(2) and fcntl(2) file locks(and also lockf(3) locks, because those are actually fcntl() locks). Unsurprisingly,this /proc file is the fundamental source of information used by lslocks(8) ,and so understanding what appears in /proc/locks and what doesn'tcan tell you something about what lslocks can show you.
The basic format of a /proc/
Document your mistakes and then try to block them in the future
Today we had a somewhat complicated maintenance downtime to moveone of our core filesystems around. As is my habit, I wrote achecklist , evolving it to be very detailed, downto the actual commands I was going to carry out, and subjecting itto as much scrutiny as I could (including looking over the documentationfrom the last time we moved this filesystem). When the time cametonight I faithfully followed the steps I'd written down, and inthe process I committed a couple of small mis-
Getting a fixed baud rate on your serial ports for logins under systemd
Many servers have support for serial consoles and in general forserial ports, either physical ones or virtual ones provided bythings like IPMI Serial over LAN, and Linux (still) supports logins over them. Systemdmakes life very convenient for any serial consoles ( also ) you have; if you configure the kernelwith one or more kernel serial consoles via 'console=' arguments,systemd will automatically start a getty on it (as covered in Gettys on Serial Consoles (and Elsewhere) and themanual page for systemd-
The modern trend of variable DNS results and its effects on troubleshooting
It's DNS. Of course it's DNS, it's always DNS. (Okay, it involves DNS,because the modern Internet loves to give different people differentDNS answers for the same thing and then all your other troubleshootinggoes out the window until you notice.)
That parenthetical bit is important and increasingly irritating.The story is somewhat straightforward; a co-worker reported thatpeople in his group were having problemsdownloading things from Dropbox from inside our networks , although sometimes it would
Prometheus: using gauge-like things as if they were counters
I recently wrote about how I wish ZFS pools kept a persistentcount of errors , instead ofmore or less a current count that you're expected to clear to verifythat you've dealt with your problems. I a comment, Daniel EbdrupJensen said that this (and other things)was better collected and tracked in an event monitoring system. Myfirst reaction was that you couldn't really do this because apersistent, always accumulating count of errors (or whatever) iswhat Prometheus really wants to see
Some problems that Python's cgi.FieldStorage has
In my entry on our limited use of the cgi module ,I praised cgi .FieldStorage as a nice simple way to write Python CGIs that deal with parameters,especially for POST forms. Unfortunately there are some dark sidesto cgi.FieldStorage (apart from any bugs it may have), and infairness I should discuss them. Overall, cgi.FieldStorage isprobably safe for internal usage, but I would be a bit wary ofexposing it to the Internet in hostile circumstances. The ultimateproblem is that in the
Our limited use of Python's cgi module
The news of the time interval is that Python is going to removesome standard library modules ( via ). Thisnews caught my eye because two of the modules to be removed are cgi and its closelyrelated kin cgitb .We have a number of little CGIs in our environment for internal use, and many ofthem are written in Python, so I expected to find us using cgi all over the place. When I actually looked, our usage was muchlower than I expected, except for one thing.
Some of
I need to remember to check for ZFS filesystems being mounted
Over on the Fediverse I said something :
I keep re-learning the ZFS lesson that you want to check not only forthe mount point of ZFS filesystems but also that they're actuallymounted, since ZFS can easily have un-mounted datasets due to egreplication in progress.
We have a variety of management scripts on our fileservers that do things on 'all ZFSfilesystems on this fileserver' or 'a specific ZFS filesystem ifit's hosted on this fileserver'. Generally they get their list ofZFS