An illustration of the problem of noise

Our backup software was originally writtenin a world in which you backed up to comparatively slow tape drivesand many of its core workings remain in that world even today. Oneabsolutely necessary trick for speeding up tape backups is to firstwrite backups to a 'holding disk' on the backup host, which can be donein parallel, and only later write them serially to tape. Of course ifyou don't have a holding disk, you must make backups one at a time soyou can write them to tape


Linux has at least two ways that disks can die

We lost a disk on one of our iSCSI backends lastnight. Normally when an iSCSI data disk dies on a backend, what happensat the observable system level is that the disk vanishes. If it used tobe, say, sdk , then there is no sdk any more. I'm not quite sure whathappens at the kernel level as far as our iSCSI target software goes,but the reference that the iSCSI target kernel module holds doesn'twork any more. This is basically just the same as


Why I now believe that duck typed metaclasses are impossible in CPython

As I mentioned in my entry on fake versus real metaclasses , I've wound up a bit obsessed with the questionof whether it's possible to create a fully functional metaclassthat doesn't inherit from type . Call this a 'duck typed metaclass'or if you want to be cute, a 'duck typed type' (DTT). As a resultof that earlier entry and some additionalexploration I now believe that it's impossible.

Let's go back to MetaclassFakeVsReal for a moment and


OmniOS (and by extension Illumos) is pretty much Solaris

I've been working on bringing our customized Solaris fileserverenvironment up on OmniOS, which is going to power the new generationof our fileservers . On our current Solarisfileservers we do things that range from relatively standard through fairly crazy , so one of my concernswith OmniOS was how easy these things would be to port over to it(especially the fairly crazy things).

I've been pleasantly surprised by the answer, which is that for themost part OmniOS is Solaris. In fact it is so much Solaris


One cause of Linux's popularity among Unixes

Regardless of what you feel about it, I think that most people can agreethat Linux is winning whatever is left of the Unix wars. It isn't theonly Unix left but for a fairly long time now it's been the leading one,often the default choice . You can attribute this togood PR if you want to, but I happen to think that that's a mistake . Linux has attracted people partly because it hasgenuine attractions.

In light of my rant about the waste inherent in


Building software packages yourself is generally a waste (why package selection matters)

I recently spent a day working on building a version of Amanda for ournew OmniOS fileservers to be. The story of why it took a day is beyondthe scope of this entry, because the point of this entry is that timewas a deadweight loss to my employer . It was in a sense unproductive.

Building software packages is a solved problem. I can't bring anythingnew or novel or particularly valuable to it and as a result I add novalue by doing it myself. Building software myself is pure


Things that affect how much support you get from a Linux distribution

Recently commentators have noted ( here and here ) that you may get less support than youexpect from distributions like Debian and Ubuntu because both splitpackages into multiple sections and may apply different supportpolicies to different sections. In my view, there are several factorsthat affect how much support you get in practice in this situation.

The obvious first factor is what the official support policies arefor the different sections of the package repository. This can bea little bit hard to find out, but Debian's is here and


Why writing sysadmin tools in Go is getting attractive

I don't find Go as nice a language as Python,but it is not terrible; in the past I've called it 'Python withperformance' . What makes it especially attractive for sysadmintools, at least for me, is how it simplifies deployment.Python deployment has three problems: the actual Python versionson various different systems, the availability of additional moduleson those systems (and their versions), and actually packaging upand shipping around a modular Python program ( cf , and loading zipfiles is


The origin of RCS (the version control system)

Let's start with the tweets:

@johnregehr :quick! who remembers a revision control system before RCS?

@thatcks :SCCS. Bonus trivia: RCS exists because getting SCCS required payingAT&T extra money and universities don't have that.

I've mentioned this before in passing ,but I might as well tell the full story (or at least folklore) here.The disclaimer is that this is the story as I heard it, not definitivehistory. A lot of Unix history goes around


Things I want to remember during a security incident

We're having a little security incident right now . I thoughtabout writing an entry about some details and specific thoughts,but then realized I had some more important things to write about,things that I need to keep in mind to keep grounded during ourhandling of the incident.

The first thing I want to remember is that hindsight is too easy. Welive in a noisy world in the present, but when we look back at thepast it's easy to pick out the thread of signal that we