In a comment on my entry on universities, email, and the issues of running things in house , I mentioned that our departmental email system has a non-trivial cost in hardware alone to keep going. To better illustrate that, I'll describe all of the servers that our email system currently requires (because it's more than one). Some of these servers exist for historical reasons and may go away at some point, but many of them don't.

Currently, we have:

• A server as our external mail gateway (our DNS MX target). This is separate from other mail servers because it's much simpler to configure and operate this way.

• A server for the (FOSS) anti-spam and anti-virus software we use (and everyone needs some version of). This could be folded into the mail gateway server (and it was in our recent backup MX , but we weren't sure about the software's resource usage and system impact when we set it up. Keeping it separate also means we can move it to a new OS version for more up to date software without having to worry about any changes in new versions of the mailer that the mail gateway runs.

• A server for our central mail machine that handles all aspects of email to local addresses, which for various reasons ( cf ) can include sending email to the outside world. This machine doesn't store any email locally; instead, to simplify slightly, email lives on our general purpose NFS fileservers .

• A separate server to handle forwarding known spam to outside email addresses . We're required to support this by people using our email system and we found it necessary to put this work on a separate machine.

• A server to handle unauthenticated mail submission from inside our networks. Separating mail submission from the central mail machine makes for a simpler configuration for both ( eg ), and we historically started with only an unauthenticated mail submission machine.

• A fairly powerful server to handle IMAP and authenticated SMTP submission , which these days also has /var/mail (where all our inboxes live) on local storage and thus also acts as a NFS server.

• A server for a webmail frontend (to our IMAP server). We put this on a separate server than IMAP for multiple reasons, including resource usage and that it decouples the OS and packaged software version requirements of our webmail (for instance, certain versions of PHP and Apache) from everything else.

We've found it very important for practical reasons to use separate IP addresses for different sorts of outgoing email ( also ). We can do this on a single machine (and we do), but in many ways it's simpler to use separate machines for different sorts of email. It's also simpler to handle things like rate limits if we use different machines for things that need different rate limits.

All of these servers rely on existing elements of our general infrastructure, such as our general purpose NFS fileservers , our local DNS resolvers, and our system of propagating account information . I hope that at some point in the future our IMAP server machine will also wind up relying on our local OIDC identity provider (and indirectly on the LDAP server it uses), but that's currently not possible in practice . I'm mentioning these because a stand-alone mail environment would require some equivalent of all of them; you have to store mailboxes somewhere, get account and authentication information, do DNS resolution, and so on.

Most of these servers are 'basic' 1U servers, which these days means that they have 16 GB to 32 GB of RAM, a mirrored pair of SATA SSDs, a reasonable CPU, and traditionally cost a few thousand dollars each if bought new (their prices are probably higher at the moment). These specifications are good enough that we don't have to worry about the exact resource requirements of each server's job (although we made sure to give the anti-spam software machine 32 GB of RAM and a decent CPU). If we used smaller machines we'd have to be more careful; I'm pretty sure that not all of these roles would be happy with only 8 GB of RAM in practice (much less 4 GB). Basic 1U servers used to be cheaper, and these days we've got a stock of older servers that are good enough for these jobs . But if we were setting up a green field environment from scratch and had to buy all of these new, five or six servers (possibly plus a spare) would be a non-trivial cost.

(Because we're using the same sort of servers for these as we use for everything else, there's no dedicated spare for specific machines; we have spare server hardware in general.)

The one server that is an exception is our IMAP server. The current version has 64 GB, four relatively large SATA SSDs, a decent CPU, and 10G-T networking, and because it's so important we have a spare server ready to be pressed into use immediately in case of a hardware failure. The current hardware is old enough that we'd like to replace it, this time with more memory (so more things get cached) and NVMe SSDs instead of SATA ones. Unfortunately, in the current environment the price quotes we got are jaw dropping and unpleasant (especially since we have to buy two of the basic server to have a spare, although we don't need two sets of the NVMe drives).

All of this serves a department with somewhat over a thousand active people, about 1.5 TBytes of inboxes (if we talk about the likely uncompressed size; since we use ZFS for /var/mail , we have compression turned on), and an inbound mail volume that is probably around 10,000 messages a day. As mail system sizes go, this is modest.

(We have several thousand inboxes (and Unix accounts to go with them), but many of them are inactive for various reasons. The size distribution of inboxes is also extremely uneven, as you might guess.)

(Publication of this entry was delayed by me getting distracted and forgetting to actually publish it last night. I didn't realize it was still sitting in my drafts area until I noticed the stray editor window just now.)