Suppose, not hypothetically, that you have a web server that's using as much of your server's bandwidth as it can get and you would like it to use less bandwidth than that, so that you can get a word in edgewise (for backups, for example) or just because you don't feel like donating 1/10th of your outgoing bandwidth in apparent perpetuity to people who should be building local caches. There are various ways you might do this, for example using FreeBSD pf on your perimeter firewall, but the lowest impact and risk option is to do it on the (Linux) web server itself with tc(8) , the Linux traffic control system. Conveniently I've already done a tiny bit with tc to fight bufferbloat latency .

There are probably a variety of ways to do this in tc(8) , but what I'm using right now is mostly pulled from the Arch wiki . It goes like this:

1. We have to switch our device, which is eno1 for me, to using Hierarchy Token Bucket as its top level qdisc (queueing discipline), and in the process set a default class that otherwise unclassified packets will be assigned to.

   tc qdisc del dev eno1 root
   tc qdisc add dev eno1 root handle 1: htb default 30 r2q 1000
   

   Following the Arch example, the default class is 1:30 (the root handle of '1' combined with 'default 30'). Because I'm working with high bandwidth, I need to change r2q to make tc happy.

2. Set up a child class to limit bandwidth, here with a limit of more or less half of a 1G Ethernet. Because we're only doing one level bandwidth limiting (ie, we're not sub-dividing it), this can go right under the root parent ('1:').

   tc class add dev eno1 parent 1: classid 1:10 htb rate 400mbit ceil 500mbit prio 10
   

   Pick the bandwidth numbers to taste depending on your irritation (and the interface speed of your server, your outgoing bandwidth in general, and so on).

   The classid is somewhat arbitrary but as the Arch Wiki example shows, there can be a use for setting a numbering hierarchy if you have multiple levels of parent and child classes. In the Arch example, the top level bandwidth limited class is '1:1', then it has child classes '1:10', '1:20', and '1:30' (the default class for traffic).

3. Following the Arch Wiki example, we add a fair queueing qdisc below our bandwidth limit, so traffic flows that fall into this bandwidth limit are (hopefully) a bit better handled. I think this means that one HTTPS reply to a requester on a fast link won't starve all of the others.

   tc qdisc add dev eno1 parent 1:10 handle 10: sfq perturb 10
   

4. Filter outgoing HTTPS traffic into our bandwidth limiting class:

   tc filter add dev eno1 protocol ip parent 1: prio 1 u32 match ip sport 443 0xffff flowid 1:10
   

   This is a u32 match , and it requires some decoding to understand. A u32 match like this is fundamentally 'match <value>/<mask> at <offset>', and if we dump the raw form of this with 'tc filter show dev eno1', we'll get "match 01bb0000/ffff0000 at 20". The 'ip sport 443' format is simply tc-u32's friendlier way of encoding that (well, technically 'ip sport 443 0xffff', since the 0xffff is a load-bearing part of the shorthand). The tc-u32(8) manual page has various cautions about this matching, so I think if you really care you want to use iptables (or nftables) rules to set a firewall mark and then match on that with tc-fw(8) .

5. Create our default catch-all class that effectively has no bandwidth limit, and as with our bandwidth limited class, also attach a tc-sfq(8) qdisc below it.

   tc class add dev eno1 parent 1: classid 1:30 htb rate 1gbit ceil 10gbit prio 1
   tc qdisc add dev eno1 parent 1:30 handle 30: sfq perturb 10
   

   We have two choices for 'prio'. The first option is that we can set this to 'prio 10', the same as our bandwidth limited class; in that case, I believe non-limited traffic will share bandwidth with the bandwidth limited HTTPS traffic, basically getting what's left over. Alternately, we can decide that we want non-limited traffic to have priority over bandwidth limited traffic, in which case we want a lower priority so its packets are sent first. That's what we're doing here.

   We need a classful qdisc , but we'd like one that will automatically use all of the bandwidth available to it. I've used tc-htb(8) here because it's what I was already using, but there are probably better options. The bandwidth I put here is simply 'really big', starting with the 1G interface rate.

This appears to work but the tc-htb(8) rate numbers may not correspond exactly to the wire bandwidth you (I) see. For our purposes, this is good enough; we're not trying to limit things to an exact MBytes/sec value, just something that's in the right ballpark.

Now that I've read the tc-u32(8) manual page, I'm pretty certain I wouldn't want to use it for anything significant where I cared deeply about what traffic was getting sorted into what class. Iptables or nftables is going to be much better at matching network traffic, especially in unusual and weird situations, and you can use that in traffic control rules through tc-fw(8) . Here it doesn't matter too much if some unusual HTTPS traffic 'leaks' outside of my tc filter, as long as there's not too much of it (plus the whole server has an intrinsic bandwidth limit ).

PS: For our purposes it would be nice to do something sophisticated to aggregate all HTTPS requests from a single IP address together in a single 'flow' for tc-sfq(8) . In theory this is possible with tc-flow(8) but in practice I can't figure out a command line that works right (despite consulting eg tc-sfb(8) 's example). I'm sure the documentation makes sense to people who have a deep understanding of Linux traffic control, but I'm not such a person.