There's a lot of rules of running large-scale systems, with lots ofmachines. I'll probably be writing up my own version of them at somepoint. But they all really come down to one big trick:

Don't administer individual machines .

That's it. Everything else is in the implementation details. (Ofcourse, the devil is always in the details.)

But what does it mean ? More or less what it says: you should neverdeal with machines one by

A lot of what I think about and do is keeping large computer systemsrunning. In an online discussion recently I've (re)realized that thereare two different sorts of 'large systems', each with very differentchallenges.

One sort of 'large system' is large because it handles a lot, forexample a mail system that handles email for 25,000 people on asingle, beefy machine. Running a single large machine like this ismostly a matter of tuning, and it

Another Saturday, another spam entry. Today I decided to look at ourlogs to see if some of the DNSBls we check had either gone away orweren't giving us any hits. Since our software checks DNSBls insequence instead of in parallel, removing useless DNSBls both reducesquery volume and speeds things up slightly.

(In general, it's wise to do this periodically unless you're currenton news.admin.net-abuse.email and other information sources, andalready know about any changed