Due to blocking until remote CPUs complete the IPI function execution, scheduling latency can increase "dramatically" with the number of remote CPUs and other conditions. Especially for x86_64 where Translation Lookaside Buffer (TLB) flushes are performance via IPIs, process exit and process-mapped page reclamation can lead to waiting on multiple IPI operations.
Bytedance engineers thus took to reworking the kernel SMP code to allow preemption during IPI completion waiting in order to avoid scheduling latency spikes. Bytedance engineers found the existing Linux SMP behavior to be particularly bad for latency-sensitive workloads like DPDK. In their end their optimizations paid off big time with a 90% reduction in P99 latency in testing with DPDK:
Thomas Gleixner summed up the situation elegantly in the SMP pull request:
"Reduce the preemption disabled sections in smp_call_function*().
The various smp call functions keep preemption disabled across the full operation which includes the wait for completion. Especially the latter can take some time when one of the target CPUs is not immediately responding to the IPI, which can result in large latency spikes.
To improve this provide a per task CPU mask to track the CPUs to wait for. That makes the information required for the wait task local and therefore allows to reenable preemption before the wait. While this comes with moderate extra memory cost this reduces SMP function call induced latency measured in a fleet for high priority tasks from ~17ms to ~1.5ms (~90%)."
The pull is now awaiting action by Linus Torvalds. Barring any code nit picks from Linus Torvalds, it's pretty safe to assume it will land this week as we kick off the Linux 7.3 merge window.