We've recently been having mysterious problems getting some of our backups to perform well . Also I recently wrote about how we'd wound up with a web server that frequently saturated its outgoing 1G interface with traffic (and it was a feature that it didn't have a faster network link). These two things turn out to not be as unrelated as we'd like, and there's a story or two there.

The problematic backups aren't done through our usual Amanda-based backup system; instead, a particular central machine pulls /var/log and other relevant files from FreeBSD and OpenBSD hosts (either through rsync or direct SSH access ) and then writes them over NFS to one of our NFS fileservers (and then they get backed up by the Amanda backups). The problems manifested as terrible NFS performance; the central machine's load average would go to 50 or 70, everything NFS related on it would be slow, and so on. While we like NFS in general we've had quite a share of problems with it ( cf , also ), so we immediately assumed that this was yet another instance of an NFS problem. First we reduced the IO load from rsync s in various ways ( eg ), then we spent quite a while digging at various NFS metrics, both from our metrics system and from the live system during a problem. Nothing really seemed to be the problem; the fileservers had perfectly good performance, the master machine had perfectly terrible actual NFS performance, and especially it didn't seem to be able to write the backup data very fast, in the range of a few MBytes a second.

Since I was pretty sure that the central machine could write to its local disk acceptably fast, I considered switching to a more complex backup scheme where we first rsync'd things to the local disk, packed up this directory tree into a tar archive, then scp 'd it to the relevant fileserver, thereby bypassing any NFS write issues. To check that this would work acceptably fast, I started by scp'ing a test file from the central machine to the fileserver. To my surprise, the scp ran at only about 2 MBytes/s. More testing showed that scp's from this central machine to anywhere ran at 2 MBytes/sec at most, which did rather explain the NFS write problems ( much like the lack of a CPU explains a server's failure to power on ). At this point a penny dropped in my mind.

We're a university department, which means that we don't necessarily have the newest, shiniest stuff around and we keep things in service for a long time. One of the things we've kept in service is basic 1G switches, because quite a lot of servers don't need more than 1G (and as mentioned, sometimes it's a feature that a server only has 1G ). But when I say 'basic 1G switches' I mean switches that are so old that all of their ports are 1G, including the port we use for 'uplinking' them into our overall 10G switch fabric (in contrast to modern 1G switches, many of which have one or two SFP+ ports that can run at 10G, or even a 10G-T port or two). This is fine for our normal 1G servers, which don't generate or receive much traffic even in aggregate, but it breaks down badly the moment you put a high volume 1G server on such a switch. For example, our new high volume web server , which was not only saturating its own outgoing 1G interface but was also saturating the 1G uplink from the switch it was connected to, a switch which unfortunately also had this critical central machine connected to it.

There's a programmer saying that if you think you have a compiler bug, you don't, you have a regular bug that you haven't spotted yet . This saying is almost always true, and it generalizes to other areas, like kernel bugs ( also ) or hardware problems , or networking problems. At our scale, modern networks are reliable, so if we don't have anything obvious wrong (for example, our monitoring system hasn't alerted us that some machine is unexpected at 100 MBit/sec), our network is almost certainly working fine. So for days it didn't occur to us to actually check. Of course the network was working fine, the network is always working fine. It obviously had to be NFS , especially since NFS has been flaky for us in the past. Except that sometimes it is the network and there's even a good explanation for it that becomes obvious once you realize it's the network.

This whole experience has given me some things to think about. On the one hand, years ago I wrote about how an obvious problem isn't necessarily obvious . There are a ton of things that can be wrong and you have to winnow through them somehow. On the other hand, I'm pretty sure that if we'd engaged in systematic troubleshooting from the ground up, we'd have found this pretty early on. For example, the USE method would have had us look at usage, saturation, and critically 'errors', which might well have caused us to look at TCP retransmits on the central machine (which were decidedly high).

Being systematic about troubleshooting is generally a good thing, but at the same time it's tedious. If the problem had really been a NFS problem (as it has been in the past) and I'd followed the USE method from the ground up, I'd have spent a chunk of time verifying that yes, the network was performing fine on both machines (along with the other things I looked at it, like NFS server metrics). Possibly what I should try to do is start out with likely guesses and then when they come up dry (eg, there are no obvious reasons for a NFS performance problem) and I'm getting frustrated, fall back to the USE method or something similar, even though it's possibly tedious.