The Linux kernel's pstore error log capturing system, and ACPI ERST
In response to my entry yesterday on enabling reboot on panic onyour servers , a commentator left the succinctsuggestion of 'setup pstore'. I had never heard of pstore before,so this sent me searching and what I found is actually quiteinteresting and surprising, with direct relevance to quite a fewof our servers.
Pstore itself is a kernel feature that dates to 2011. It providesa generic interface to storage that persists across reboots andgets used to save kernel messages during a crash, as covered
A piece of email malware that wanted to make sure we rejected it
Recently our system for logging email attachment type information recorded aninteresting attachment:
attachment application/octet-stream; MIME file ext: .ace; zip exts: .exe
The .ace extension is for an old archive file format andtoday is mostly used by malware, possibly because tools to lookinside ACE archives are less common for reasons you can read abouton the Wikipedia page (see eg here ). We see a certainamount of .ace attachments all of the time, and we've been rejectingthem
Consider setting your Linux servers to reboot on kernel problems
As I sort of mentioned when I wrote about things you can do tomake your Linux servers reboot on kernel problems ,the Linux kernel normally doesn't reboot if it hits kernel problems.Problems like OOPSes and RCU stalls generally killsome processes and try to continue on; more serious issues causepanics, which freeze the machine entirely.
If your goal is to debug kernel problems, this is great because itpreserves as much of the evidence as possible (although you probablyalso want things like a serial console or at
A little surprise with Prometheus scrape intervals, timeouts, and alerts
Prometheus pulls metrics from metricsources or, to put it in Prometheus terms, scrapes targets. Every scrape configuration and thus every target has a scrape interval and a scrape timeoutas part of its settings; these can be specified explicitly orinherited from global values. In a perfect world where scrapingtargets either completes or fails in zero time, this results insimple timing; a target is scraped at time T, then T + interval,then T + interval + interval, and so on. However, the real world
Things you can do to make your Linux servers reboot on kernel problems
One of the Linux kernel's unusual behaviors is that it often doesn'treboot after it hits an internal problem, what is normally calleda kernel panic. Sometimes this is a reasonable thing and sometimesthis is not what you want and you'd like to change it. FortunatelyLinux lets you more or less control this through kernel sysctlsettings .
(The Linux kernel differentiates between things like OOPSes and RCU stalls,which it thinks it can maybe continue on from, and kernel panics,which immediately freeze the
Two annoyances I have with Python's imaplib module
As I mentioned yesterday , I recently wrotesome code that uses the imaplib module . In the processof doing this, I wound up experiencing some annoyances, one of thema traditional one and one a new one that I've only come to appreciaterecently.
The traditional annoyance is that the imaplib module doesn't wraperrors from other modules that it uses. This leaves you with atleast two problems. The first is that you get to try to catch abunch of exception classes to handle errors:
try
A few notes on using SSL in Python 3 client programs
I was recently writing a Python program to check whether a testaccount could log into our IMAP servers and to time how long ittook (as part of our new Prometheus monitoring ). I used Python becauseit's one of our standard languages and because it includes theimaplib module ,which did all of the hard work for me. As is my usual habit, Iread as little of the detailed module documentation as possibleand used brute force, which means that my first code looked kindof like this:
A surprise potential gotcha with sharenfs in ZFS on Linux
In Solaris and Illumos, the standard and well supported way to setand update NFS sharing options for ZFS filesystems is through the sharenfs ZFS filesystem property. ZFS on Linux sort of supports sharenfs , but itattempts to be compatible with Solaris and in practice that doesn't work well, partly because there areSolaris options that cannot be easily translated to Linux . When we faced this issue for our Linux ZFSfileservers , we decided that we would build an entirely separate system to handle NFS exports that directly invokes exportfs ,which
Linux CPU numbers are not necessarily contiguous
In Linux, the kernel gives all CPUs a number; you can see this numberin, for example, /proc/stat :
cpu0 [...]cpu1 [...]cpu2 [...]cpu3 [...]
Under normal circumstances, Linux has contiguous CPU numbers thatstart at 0 and go up to however many CPUs the system has. However,this is not guaranteed and is not always the case on certain liveconfigurations. It's perfectly possible
Why C uninitialized global variables have an initial value of zero
In C, uninitialized local variables are undefined but uninitializedglobal variables (whether static or not) are defined to start outas zero. This difference periodically strikes people as peculiarand you might wonder why C is this way. As it happens, there is afairly simple answer.
One answer is certainly 'because the ANSI C standard says thatglobal variables behave that way', and in some ways this is theright answer (but we'll get to that). Another answer is 'because Cwas documented