Suppose that you have a program that can run other programs, andyou want to find out if it securely runs the other programs . In an ideal world, the program'sdocumentation would tell you, and you could trust it. Sadly we do notlive in an ideal world.

First you need a test environment where you can control what externalprogram your program runs and force it to actually run the program. Inmany cases (such as testing daemons, web servers, MTAs, and so on),

I've mentioned before that we replaced ZFS's normalspares handling with our own system, but I've never actually describedthat system. For background, see my earlier entry onour issues with ZFS's normal spares handling and the challenges we havewith spares in our ZFS fileserver environment .

There are two ways to implement a spares system. Let us call themevent driven and state driven. In an event driven spares systemyou activate spares in response to fault events, and one of yourchallenges is