When is using SQL the right answer?

One of the things I've been thinking about as a result of my too muchSQL war story is how to distinguish between when using SQLis the right answer and when it's the wrong answer. I've just started onthis, but I've come to one obvious sign: data reduction.

The downside to just asking your SQL server for the entire database anddoing all of the processing in your own program is that the SQL serverships you a lot of data. Thus, one important


People forget exceptions

One of the possible replies to the problem of Unix programs exitingnon-zero for clever reasons is that all ofthese special cases and clever reasons are clearly documented in themanpages for the various commands, so all of these problems are not thefault of the authors of the commands.

This view is objectively wrong. The clear, well established truth isthat people do not remember weird little exceptions and funny littlecorner cases in what your program does unless they use your program allof the time (in fact, unless


Unix programs should avoid exiting non-zero for clever reasons

Dan Astoorian's comment on yesterday's entry reminded me of this general issue,which I will phrase this way:

Dear authors of Unix programs, please don't invent clever reasonsto have your programs exit with a non-zero status.

Unless you have a really good reason, your program should only ever exitnon-zero if it has actually failed. Trust me; shell programmers areperfectly competent to check for some property of your output ourselvesif we really want to know.

(Even if you


A gotcha with the Bourne shell's set -e and &&

Suppose that you have the following Bourne shell code:

set -ecmd1 && cmd2 && cmd3echo all done

Now suppose that cmd2 exits with a non-zero status. Do you expectthe script to abort, or to print out 'all done'?

My assumption when I was writing a script recently was that the scriptwould abort; after all, I had set -e turned on. But this is not whathappens, and in fact most Bourne shell manpages spell it out


ZFS scrubs and resilvers are not sequential IO

Here is something that is probably not as widely known as it couldbe: ZFS scrubs and resilvers are not done with sequential IO, theway conventional RAID resynchronization and checking is done.

Conventional RAID resyncs run in linear order from the start of eachdisk to the end. This means that they're strictly sequential IO ifthey're left undisturbed by user IO (which is one reason that you candestroy conventional RAID resync performance by doing enough random userIO).

ZFS scrubs and resilvers don't work


How zpool status reports on ZFS scrubs and resilvers

A recent thread on the ZFS mailing list concerned inaccurate ormisleading reports from zpool status about the progress of scrubs(both bad time estimates and a scrub that wasn't finishing despiteclaiming to be 100% done). As it happens, I know something about howthis works because I recently went digging into what information thekernel actually reports to userland.

The kernel reports the following information:

  • the number of bytes currently allocated in the pool and in each vdev.

  • how many bytes in the pool have been

A thesis: noisy email addresses are dead

Here is a thesis, or at least a theory: any email address that getstoo much noise (spam, bounces, autoresponder replies, and so on) iseffectively dead. This is especially so if the noise reaches real peopleand must be filtered by hand.

(This thesis underlies my belief that postmaster is a dead address .)

The problem with hand filtering is twofold. First, hand filtering isan unpleasant task (and a pain), which means that it will likely bedone reluctantly and as little


Returning to the era of 'duplicated' Ethernet addresses

Once upon a time back in the old days, Sun caused quite a stir bydeciding that they would make the Ethernet addresses for their hardwarebe an attribute of the machine, not of the network interface. Or totranslate, Sun machines used the same Ethernet address on all of theirinterfaces instead of doing what everyone else did, which was to havea different Ethernet address on each interface. This is spec-legal butcaused various sorts of annoyances for system administrators and networkdesigners; among the set of people who


/u, one of our long-standing Unix customizations

One of Unix's small problems is that it has no simple and universal wayto refer to people's home directories, especially other people's homedirectories. Some people are about to pipe up about ~user , but that'snot a universal way; support for it has to be added to every programthat will use it (and in every context where they resolve file names),and inevitably there are programs where it is not supported at all or isnot supported everywhere. Beyond that,


A gotcha with Python's socket.htonl() and ntohl()

Here is something that I ran into the other day :the socket module 's htonl() and ntohl() functions will return signed numbers undersome circumstances, not unsigned ones, which means that for some inputvalues they will return negative numbers.

(Some quick testing suggests that this happens on 32-bit x86 machinesbut not 64-bit x86 Linux machines; all of the 32-bit machines I haveaccess to are running some version of Python 2.5.