Suppose, not hypothetically, that you use rsync to push an encrypted backup file from the machine it's created on to a fileserver , where it will be backed up by your regular backup system . Because this encrypted backup file is backed up every day, you only need one copy of it on the filesystem, so you use and reuse a fixed name for the file. In other words, we're using rsync somewhat as if it was scp , but with better control over what remote files can be written and (not) read .

When you push (or pull) a file over rsync, rsync normally attempts to optimize what gets transferred by looking for common blocks in the file (how big a 'block' is depends on the file size, or you can fix it with the ' --block-size ' option, as covered in rsync(1) ). This is a nice potential bandwidth saving, but it creates CPU and IO load on both ends as each of them checks their version of the file. In our specific case, we know that there aren't going to be common blocks; since the whole file is encrypted, it's basically random noise. Recently this backup process had some IO load problems, and today in the process of working on this I discovered rsync's ' -W ' option (also known as ' --whole-file '). As the manual page explains, this 'disables rsync's delta-transfer algorithm'; in other words, it stops looking for common pieces between the two versions of the file. Rsync simply sends the whole file (and the receiver writes the whole file).

Since we know that today's encrypted file has no blocks in common with yesterday's encrypted file (well, it had better not if the encryption is working right), ' -W ' is exactly what we want to stop the receiving rsync daemon from doing unnecessary work (specifically, unnecessary IO). Effectively it turns the 'file copy' part of rsync into scp (although not literally; rsync will normally write the new version of the file to a temporary file and then replace the old version). Now that I know about -W , I'm going to be looking at some of our other uses of rsync to see if we might want to use it more widely.

(For example, we use rsync to back up /var/log from some FreeBSD hosts, and I'm pretty sure that's a good candidate for -W too.)

If you're using ' -W ', you want to avoid using ' --checksum ' and instead rely on the default 'quick check' of file size and modification time. This is because using --checksum requires rsync to read and checksum the whole file before the transfer starts (which is something that the manual page warns you about).

One lesson I've taken from today's experience is that when I use rsync, I should think about what I want to optimize (and what can actually be optimized). Rsync's default behavior is to optimize transfer bandwidth, but sometimes you have enough transfer bandwidth and you want to optimize for lower IO, lower CPU, or both (which is sort of our case for this encrypted backup file, with the extra issue that we know rsync can't reduce the transfer size). Alternately, sometimes you really want to squeeze the bandwidth and maybe ' -S ' and ' -z ' (and perhaps others) are what you want, even though you'll do more work on both ends.

(It's possible that rsync already has a clever encoding for runs of zero bytes and so ' -S ' doesn't save you any transfer bandwidth. I haven't tested.)

(This elaborates on a Fediverse post of mine .)