Time and time again, people have tried to create entirely automatedsystems for detecting, identifying, and dealing with spam ontheir services. Time and time again, they've ultimately failed ; their systems may stop a great deal of spam, butenough gets through despite it.

(Not infrequently the spam that gets through looks, from the outside, asif it should be trivial to recognize. I think there is a deep reason forthis, which we'll get to.)

There is a shallow and

When I read Philip Hollenback's latest entry and it mentioned someonedoing (change/system) management through shell scripts (instead of, say,Puppet), my first thought was 'hey, I've done that'. So I might as wellwrite up how I did it, either for someone to use or in case people wantto marvel at the crazy person.

(Now, a disclaimer: by now this was more than half a decade ago, andsome of my memories of the fine

Recently there was a Stackoverflow question that asked why Tanenbaum was wrong in his predictions for the futurein the Tanenbaum/Torvalds debate. This got me thinking aboutmicrokernels (one of Tanenbaum's predictions was that microkernelswere the future of operating system design). In particular, it gotme thinking that microkernels have a bunch of properties that makethem almost uniquely attractive for academic operating systemresearchers. For background, I recommend Rob Pike's SystemSoftware Research is Irrelevant [PDF] for an idea of the constraints and

Ignoring things like SELinux for the moment, the basic security stateof Unix system calls has always been that root is allowed to becomeany other UID at will, but no one else is allowed to change their UIDor other security attributes (setuid programs then provide an escapehatch from this). But what does it mean to become another user on Unix,beyond just setting your UID? In fact there are a whole series of thingsthat it can mean, some of which you do not necessarily want.

Let'

Jay Kreps' Getting Real About Distributed System Reliability is a very interesting discussion of the reliability of distributedsystems in the real world. He patiently explains that a number ofassumptions normally made to reason about this are in fact wrongin practice, especially the assumption that failures are independent.I'm not going to try to summarize his entry beyond that; go read itinstead.

(I suspect that his logic extends to all real systems, not justdistributed ones, and in any case he has given me a lot

Recently, Aristotle Pagaltzis noticedthat I had misspelled the filename of this entry . When I renamed it to fix this,the WanderingThoughts Atom feed repeated the entry (and the commentsfeed repeated the comments on it); this led to a discussion on Atomentry IDs between Aristotle and I that I am now going to surface asan actual entry so I can discuss DWiki 's problem with Atom entries atlength.

Atom is in general a nice feed format, but it has one awkwardrequirement: it absolutely demands that

Every so often I run across a piece of writing that makes me slapmy forehead because it points out something obvious that I'd stilltotally missed. Today's is Pat Shaughnessy's discussion of Ruby2.0's new garbage collection system ,in which he casually explains the problem with traditionalnon-compacting mark and sweep garbage collection ina modern Unix environment. As usual, I am now going to rephrase thegeneral issue to make sure I have it straight in my head.

Mark and sweep

We had a building power failure today in the building with our mainmachine room (and thus all of our core servers). When we realizedwhat was going on and got to the machine room, we made an extremelyunpleasant discovery; as far as we can tell, the automatic transferswitches in front of our UPSes , well, didn't transfer.Instead they all entered some sort of faulted state where they providedno power.

(The UPSes all at least claimed to have good battery charge and to not

Back in DevOps and the blame problem I notedthat operations has a problem getting praised, because people generallyfeel that the computers should just work. This leads to what I call theheroism problem for ops.

In practice, ops can easily get praised in exactly one situation: whenit's clear to everyone that something exceptional is going on, thatit is not just business as usual. In short, you get praised if you(visibly) fix a panic situation, and the more exceptional your effortsto fix