What I know about ZFS and disk write caches

Many people with ZFS experience have probably heard that ZFS is designedto work safely with disks with write caches, and may also have heardthat this is done with cache flush commands. The details of what getdone when and under what circumstances are a bit complicated, though.

First, I'm only talking about the situation where you give ZFS wholedisks, instead of slices. When you do this, ZFS marks the device asbeing a whole disk in the pool configuration, and the remaining magic isdriven from


What an iSCSI Enterprise Target kernel message really means

IET is the Linux iSCSI targetimplementation that we use . Periodically it willspit out somewhat alarming kernel messages that look like this:

iscsi_trgt: scsi_cmnd_start(1045) Unsupported 15
iscsi_trgt: cmnd_skip_pdu(454) e0da9005 1c 15 32

(The numbers in brackets will vary from IET version to version; they areline numbers in the source code of IET's kernel module.)

What


The challenge of analyzing NFS packet traces

Suppose that you are having NFS performance problems on a single NFSclient machine (for simplicity), where every so often the system's loadspikes dramatically, and you want to track down what's going on. In anideal world the client's operating system would have built in monitoringthat you could use to analyze this. We do not live in that ideal world.

(Maybe someday DTrace and SystemTap and so on will ensure that we do,especially if the kernel NFS server and client code is


Focusing on what you actually need in a program

I recently wrote a local program to do more useful job of reportingthe status of our ZFS pools. To simplify slightly, ZFS pools have anumber of flavours of redundancy; none, mirroring, RAID-5, and RAID-6(currently). However, we only use mirrored pools, so for our own usesthe program I was writing didn't need to deal with any other sort ofpool.

So of course I spent a not insignificant chunk of time making my programhandle RAID-5 and


One problem with testing system changes

One of the strange things about system administration as compared todevelopment is the general lack of testing that sysadmins do. I believethat one reason for this is that sysadmins have a hard time testingchanges, especially on a budget.

Now, I will admit that I have a biased viewpoint on this; I work in arelatively complex environment (although one that's fairly small by thestandards of large systems). As is common in multi-machine environments,we effectively have hierarchies of machines and systems, with


The quiet death of postmaster@anywhere

I like following RFCs, by and large, and I like being a good Internetneighbour, and I have been behind one or another postmaster@ addressfor a very long time. But I have to face reality and admit something;regardless of what the RFCs say, the postmaster address is dead andhas been for years, and basically no one would notice if we quietlyturned off ours.

It would be one thing if postmaster addresses didn't get any email atall, but of course this isn't


How to set up your module exceptions to be useful

Suppose that you are writing a Python module that has exceptions as partof its interface, and you want them to actually be useful to people. Fromthe perspective of a sometimes grumpy user of module exceptions, here ismy opinions on what you should do:

  • all exceptions that you expect people to actually catch should descendfrom a common ancestor class, so that people can just do ' exceptYourException, e: ' and be done with it.

    If you have an exception you raise for internal errors and otherimpossible situations


How disk write caches can corrupt filesystem metadata

It's intuitively obvious how (volatile) disk write caches can result inyou losing data in files if something goes wrong; you wrote data, thedisk told your program it had been written, the disk lost power, andthe data vanishes. But it may be less obvious how this can result incorrupted or destroyed filesystems and thus why you need (working) cacheflush operations even just to keep your filesystems intact (never mindwhat user level programs may want).

Consider a filesystem where you have two


A ZFS feature wish: rewriting read errors

Today's missing ZFS feature is most easily described by telling youabout the problem. Suppose that you have a redundant pool (okay, aredundant vdev) and one of the disks in it develops some bad sectorsthat can't be read. My current understanding is that this is not a'replace disk immediately' sign the way that write errors are, andthus can happen on otherwise healthy and usable disks.

(A persistent write error is a 'replace disk immediately' sign becauseit means that the


My sad little irritation with Twitter

I have a confession: every so often, Twitter irritates me. To be honest,there's a bunch of sources of this specific irritation besides Twitter,but Twitter is the most frequent one.

The irritation is that Twitter effectively hides information on who islinking to WanderingThoughts entries. In most linking and re-bloggingenvironments, some amount of this information shows up in the Refererheaders of incoming HTTP requests. Even if the Referer URL doesn't giveyou the exact entry where someone linked to you, it