Without going into too many details, we'd be amiss to leave out the recent and powerful [e]BPF Linux kernel feature and it's associated frontends; there are a few to specifically measure scheduler and runqueue-related system latencies. (We covered the installation of the [e]BPF tools back in Chapter 1, Kernel Workspace Setup under the Modern tracing and performance analysis with [e]BPF section).
The following table summarizes some of these tools (BPF frontends); all these tools need to be run as root (as with any BPF tool); they show their output as a histogram (with the time in microseconds by default):
| BPF tool | What it measures |
| runqlat-bpfcc | Time a task spends waiting on a runqueue for it's turn to run on the processor |
| runqslower-bpfcc | (read as runqueue slower); time a task spends waiting on a runqueue for it's turn to run on the processor, showing only those threads that exceed a given threshold... |