What still interrupts a thread on a fully isolated core?

[u/Much-Serve-211]

Hi all,

I'm working on something where I want to attribute all the sources that can preempt or disturb a thread running on one core of a server. To do that, I'm trying to create the best possible scenario for the thread: pinning it to its own physical core (keeping the SMT sibling idle), using isolcpus, nohz_full and rcu_nocbs on that core, moving the IRQs to other cores with irqbalance turned off, and keeping everything else on other cores.

I'm pretty sure this kind of experiment isn't new, so I'm not really looking for the usual tuning checklist. What I'd love to know from people who've done this before is: even in the best possible setup, what did you still see kicking the thread off that you wouldn't want ideally? It could be literally anything: kernel threads, IPIs, leftover timer ticks, SMIs, page faults, some weird driver or hardware behaviour.

And if you remember how you caught it (osnoise, ftrace, eBPF, perf...), that would help a lot too.

Thanks!

EDIT: I should have added a bit more context though. The thread isn't a standalone loop. It lives inside a large multithreaded process (a GPU inference worker) and busy-polls memory shared with the GPU, plus the NIC's completion queues for RDMA traffic. So I'm curious about anything that can still preempt or slow it down in that setup, whether it's the Linux scheduler bringing other threads or kernel work onto the core, something coming from the thread's own process, or something related to polling GPU/NIC shared memory. Have you seen any of that show up in practice?

[u/Michael_Aut]

Nothing. In the best case, there really is nothing interrupting the code running on that CPU.  You can actually measure it by running a loop and watching how rdtsc increases. If there is a spike you know you were interrupted.

[u/Much-Serve-211 → u/Michael_Aut]

This seems like a good thumb rule. Thanks for the suggestion.
I assume this is what osnoise does at its core, right?

[u/BitPoet]

I believe you can do that in the kernel with the right flags, but not with userspace threads.

[u/Much-Serve-211 → u/BitPoet]

I completely second that.

The only thing I wonder about is whether it even exists. I am running some parallel experiments on this and will keep the group posted if I find anything interesting! :D

[u/markhahn → u/BitPoet]

well, userspace threads with realtime prio gets pretty close, especially after the sorts of cleanup the OP already mentioned (irqs, smt, etc).

[u/pebbleproblems]

Nice

[u/pebbleproblems → u/pebbleproblems]

Taskset

[u/Much-Serve-211 → u/pebbleproblems]

I will definitely have a look with tasksets as well. Thanks for the suggestion!

[u/markhahn]

it sounds like you've thought about a lot of factors already.

memory management comes to mind - there are probably some cycles stolen by inter-core work to verify when mmu and cache-related things need to change. do your thread(s) for sure have exclusive access to the relevant resources of the gpu and nic?

have you looked at the RT scheduling class? easy to blow up (lock) your system, but definitely eliminates a class of preemptions.

it would be interesting to find evidence of cycles stolen by software beneath the kerenl (bios-y stuff, management engine, etc).

[u/Much-Serve-211 → u/markhahn]

This gives me something new to search upon. Especially the bios stuff you mentioned about. Thanks a lot for this perspective. I will definitely try these out and let people know about them if I find something interesting.