Formatting information output to make it easy to manage

Suppose that you are designing systems to be managed and you are taking the straightforward approachof dumping out information when external programs ask for it. As ithappens, I have some views on how you should format this information andwhy.

Monitoring systems have two major concerns when they're dealingwith your status output. First, they want to be confident that theyunderstand what you're printing out, ie that they are correctly parsingit. Second, they want to be sure that they understand the meaningof everything


Design systems to be managed

Here is something that I've come to believe strongly: if you're buildinga system today (whether it's just a single program or much larger), youshould be designing it for what I'll call 'manageability'. By this Imean the ability of your system to integrate with other systems and tobe managed by them.

Once upon a time, the state of the art in systems operation was suchthat systems (programs, packages, environments, etc) were basicallystandalone and


A small wish about parked and squatted domains

One of the things that clutters up the modern Internet is all of theparked and (typo-)squatted domains that are floating around, and alongwith them all of the domain name variants that legitimate domain ownershave picked up to defend themselves from the typo-squatters. Sometimesit feels like putting any vaguely word-like name in a browser will windyou up somewhere, often on an add-filled landing page. (And let's notmention what happens to old domains whose registrations are allowed tolapse


Milter tools for Python and an experiment with coding in public

I've decided to buckle down and write some Python code to deal withsendmail milters, in order to satisfy a long standing desire ofmine . The current name of this code is'python-milter-tools', and so far I have encoders and decoders for allv2 milter protocol messages and some code to make it easier to talk themilter protocol over sockets. So far it is mostly focused on the 'MTA'side of the protocol, since that's what I'm interested in


The ZFS opacity problem and its effect on manageability

I've alluded to this before in passing, but one of my great frustrationswith ZFS is that it has basically no public interface or API for gettinginformation about its state; it is an almost completely opaque box. Doyou want to know configuration information about your pools? Do you wantto know state information about how healthy or damaged they are? Tough.You can't have it, or rather your programs and systems can't have it.Not with a public, reliable interface at any rate


Abusing Python classes as namespaces

Suppose, not entirely hypothetically, that you are dealing with binarystructures and so have a bunch of functions to encode Python values tovarious different sorts of binary fields and decode those fields back toPython values. These functions clearly come in pairs.

There are various ways of organizing these functions; forexample, you can just give them names like encode_uint16 and decode_uint16 . One attractive way would be to group the pairs offunctions together in namespaces, for example as uint16.


A basic Namespace metaclass for Python

Suppose that you want to conveniently abuse classes as namespaces . For me, this means avoiding having to splatter @staticmethod all over my class definitions; it's both repetitivemake-work and distracting when I'm looking at the source code.

Clearly the solution to this problem is a metaclass to do all ofthe work for me. Here's one, which is deliberately somewhat simple:

from types import FunctionType
class NSMeta(type):  def __new__(meta, cname, bases,

What I need for stream decoding and encoding binary protocols

I've recently started poking a bit at my milter project . The first part of this is to writesomething that can talk the milter protocol and thus decode and encodeits binary messages. Writing binary parsers and generators by hand isannoying; ideally you'd write a declarative description of all of thebinary messages, feed it to a package or a library, and get a codecback. So I gave it a spin with a Python package nominally intendedfor this purpose. The experience was educational but not successful


It's long since time for languages to provide sensible network IO

You may not have noticed, but stream-based network IO is famously notlike regular file IO. Well, mostly network read IO. Believing that it isis a common socket programming error .

A great deal of the time, programs and programmers do not want to careabout this. When they say 'read ten bytes', they want to get backexactly ten bytes (assuming that the stream is not closed on them) andthey do not care how long they have to wait for those ten bytes to


The Unix shell initialization problem and how shells should work

In general, there are at least three sorts of things that you may wantto do when a shell is started: one time environment initializationsthat can be inherited in subshells, per shell things that cannot beinherited, and anything that you want to do when logging in (you mightsplit this into interactive and non-interactive things under somesituations). Depending on what a shell is being started for, you want itto do all, some, or none of these things.

A regular login shell does all