Operating spam and malware filtering is ultimately a social problem
Successfully filtering spam and malware is a technical issue, fullof problems like recognizing new sorts of spam and malware, developingrecognition rules, and not having your servers eaten by expensivecode (even when people send you gigantic files, or malicious onessuch as compressed archives that expand hugely or endlessly).However, operating spam and malware filtering is ultimately asocial problem , because the peopleyou are doing the filtering for need to be happy with what yoursystem does and how you operate it.
No spam and malware filtering
Counting the number of distinct labels in a Prometheus metric
Suppose, not hypothetically , that you'recollecting Prometheus metrics on yourseveral VPN servers, including a per user count of sessions on eachserver. The resulting metric looks like this:
vpn_user_sessions{ user="cks", server="vpn1", ... } 1vpn_user_sessions{ user="fred", server="vpn1", ... } 1vpn_user_sessions{ user="cks", server="vpnThe problem of multiple NVMe drives in a PC desktop today
My current office workstation currentlyhas two 250 GB Samsung 850 EVO SSDs (and some HDs). These weredecent SSDs for their era, but they're now any number of years oldand 250 GB isn't very large, so as part of our general stocking upon 500 GB SSDs at good sale prices, I get to replace them. To mysurprise, it turns out that decent 500 GB NVMe drives can now behad at roughly the same
Capturing command output in a Bourne shell variable as a brute force option
Often, the natural form of generating and then processing somethingin the Bourne shell is as a pipeline:
smartctl -A /dev/$dsk | tr A-Z- a-z_ | fgrep -v ' unknown_' | awk ''timeout 30s ssh somehost npppctl session brief | awk ' ' (Using awk is not necessarily recommended here, but it's theneutral default.)
However, there can be two problems with this. First
Selecting metrics to gather mostly based on what we can use
Partly due to our situation with our L2TP VPN servers , I've set up a system to capturesome basic usage metrics from our VPN servers. They don't directlyprovide metrics; instead we have to parse the output of things likeOpenBSD's npppctl and usethe information to generate various metrics from the raw output.As part of doing this, I had to figure out what metrics we wantedto generate.
I could have generated every single metric I can think of a way toget,
In Prometheus, don't be afraid of high cardinality metrics if they're valuable enough
We generate any number of custom local metrics that we feed intoour local Prometheus metrics and monitoringsetup. Most of them are pretty conventional, but one of them isprobably something that will raise a lot of eyebrows among peoplewho are familiar with Prometheus and set them to muttering about cardinality explosions . Among ourmore conventional metrics about how much disk space is free on our fileserver filesystems,we generate a per-user, per-filesystem disk space usage metric.In the abstract, this looks like:
cslab_user
I use unit tests partly to verify that something works in the first place
A long while ago I read Mostly avoid unit tests ( via ), in whichthe author starts out with:
Most of the value of a unit test comes when you change the original(tested) code, but not the test, and can run the test to make surethat all is still well. [...]
This is, broadly, not my experience with a lot of the unit tests that Iwrite. A lot of the time I write tests in large part to verify that thecode
Timing durations better in Python (most of the time)
As I mentioned in yesterday's entry on timeouts and exceptions , we have some Python programs that check to makesure various things are working, such as that we can log in to our IMAPservers. Since they're generating Prometheus metrics as part of this, one of the metrics they generate is how long ittook.
(My feeling is that if you're going to be generating metrics aboutsomething, you should always include timing information just ongeneral principles even if you don't have an immediate
Thinking about timeouts and exceptions in Python
As part of our overall monitoring and metrics system , I have a Python program that logs in toour IMAP server to make sure that we can at least get that far(because we have broken that sort of thing in the past). The programemits various Prometheus metrics, includinghow long this took. For reasons beyond the scope of this entry, Iwould also like to have some very basic information on how the IMAPserver is performing, such as how long it takes to do an IMAP SELECToperation on
Our problem of checking if our L2TP VPN servers are actually working
We operate some VPN servers tolet people who are currently outside our wired networks have accessto internal networks and internal services (this includes peopleusing our wireless network, for reasons beyond the scope of thisentry). For various reasons, we offer both L2TP and OpenVPN (ondifferent servers). Our OpenVPN servers are pretty reliable, butour current L2TP servers have a little problem where sometimesthey'll just stop responding to people's attempts to establish L2TPVPNs while otherwise looking
Menu
![]()