The Python marshal module versus the cPickle module

The marshal module looks interesting for persisting and retrieving lightweight data,but the big question to me has always been whether in exchangefor constraining your data down to simple structures of primitivetypes you got something that was actually faster than the cPicklemodule .

So today I decided to finally answer the question by doing sometiming tests. I won't claim that these are comprehensive or entirelyscientific, but I do have some results:

  • the speed difference is mostly in dumping things; marshal andcPickle generally load things as fast as

Our experience with Linux's strict overcommit mode

As a follow-up to 64BitDrawback : after we had several machines crashdue to being driven out of memory, dealing with the whole issue suddenlygot a whole lot more urgent and we opted to try to solve it by turningon Linux's strict overcommit mode for swap allocation. At first we didthis only on our compute servers, but after some of our login serversalso OOM'd and crashed, we enabled it on them too.

(Strict overcommit has the great advantage that we don'


Our old mail system's configuration

Before I can talk about more interesting mailer things, I have toexplain how our old mail system was configured.

Our old mail system makes perfect sense once you realize that it wasmore or less designed around the idea that nothing should ever haveto be done over NFS. In order to manage this, each different sort ofprocessing had to be done on the machine that held the relevant files;deliveries to /var/mail were done on the postbox machine, which had /var/mail on local disks,


The arrogance of trying to design for long term storage management

Many systems seem to be not really designed for a long term storagemanagement environment . Instead they seem toopt for a kind of planned obsolescence approach where they assume thatyou will buy them, run them more or less into the ground without reallychanging or upgrading anything, and then replace them wholesale in abig, painful, user-visible bang.

From the perspective of a long term storage management environment this is a crazy thing to do; with no growth and thus no future, thesesystems are basically closed boxes


Why I think identity blurs into authority

In theory we can separate the ideas of identity and authorization,and it is common to present complex computer systems this way. Inpractice I think that many people blur the two together and attemptingto forcefully separate them only leads to confused users and frustratedsecurity people.

I believe that one reason for this is because we rarely think ofpeople alone in the real world; instead we think of them with attachedassociations. It is not 'Chris Siebenmann, who is authorized to', it is'Chris Siebenmann who works


Weekly spam summary on October 13th, 2007

This week, we:

  • got 11,905 messages from 252 different IP addresses.
  • handled 27,710 sessions from 2,367 different IP addresses.
  • received 342,122 connections from at least 124,401 different IPaddresses.
  • hit a highwater of 36 connections being checked at once.

Connection volume seems up a bit from last week ,although it's hard to be entirely sure. Session volume is definitelyup,


Getting your networks to your racks

I'm in the process of getting a new test server installed in a rackin our machine room. This means we needed to set up some networkconnections for it, which involved someone dragging yet more cablearound our machine room and finding switch ports to plug into to getthe necessary networks (which turned out not to be an entirely trivialthing).

As a result, the whole thing got me thinking about the issue of the bestway to get all of your networks to where they need to be in


A gotcha with command order in pipes

Here's a mistake I've made more than once:

tail -f active-log | grep big-filter | grep -v one-thing

(Here the big-filter is something that selects only a small amountof actual interesting logfile output, and then you want to throw awaya small bit more. For example, my most recent version of this wasmonitoring the exim log on our new mail system for failures and otheranomalies, and then wanting to throw away one particular known failure.)


How to properly look up hostnames from IP addresses

Looking up reverse DNS to determine the hostname of an IP address thatyou are talking to is one of those almost simple things that softwarekeeps getting wrong .So here is how you do it, in two variants.

First, if you use DNS and have an (almost) empty /etc/hosts or justdon't care about looking up something in it, is the simple version:

  • gethostbyaddr() the IP address. If it fails, you are done. Otherwiseyou have the claimed hostname .

    (


A silly trick with X

Once upon a time, I was at home and really needed to see somethingthat was displayed on my screen at work. This being Unix, there was noconvenient remote desktop add-on that would have let me mirror my workdisplay to home, but this being Unix, there are ways around that.

(Also, I was connecting over a slow dialup PPP connection , so a live remote desktop thing would havebeen difficult anyways.)

My first attempt was simple; ssh in to the work machine and do