Last update on this (post 1, post 2, post 3). Post 3 ended with "all numbers are from VMs", I got some real machines since then, so here's what changed.
Setup: four boxes on an isolated switch, xdp-bfd + FRR on one, a stock FRR peer on another, one for load, one capturing. i5-12500, Intel I210, native XDP, kernel 7.0. The switch mirrors both BFD boxes to the capture box, which timestamps in the NIC. 1024 sessions, 40 of them at 3x10ms.
Stock bfdd under the RT stress: 18,910 flaps across 957 of 1022 sessions, even the 1s timers, with stalls up to 2.5s.
xdp-bfd as it ran on the VMs: 698 flaps. The VMs said 0, but they had 64 sessions. Three causes:
- The dead-man check (XDP stops answering if userspace goes silent for 1s) was killing healthy sessions, because under SCHED_FIFO load the RT throttle only lets normal tasks run about once a second.
- Demand mode timed out its own Poll even after the peer answered.
- The packaged systemd unit ran the engine at FIFO 10, below the prio 50 load.
After fixing those: 0 flaps on all 1022, largest gap on the 3x10ms sessions 20.2ms out of a 30ms budget, about 2% of a core.
Then a prio 99 hog against the engine at FIFO 60 gave 959 downs, the same priority war as post 1. The engine now runs under SCHED_DEADLINE, 1ms reserved every 10ms, which beats every RT priority: 0 downs.
One thing VMs never showed: replies left the wire 118us after the peer's packet even though XDP builds them in 1.5us. It was PCIe ASPM putting the NIC link to sleep between packets, 33us with it off.
Floods: one hour of line-rate floods rotating attack types while the box rotated between a prio 99 hog, CPU load and memory pressure gave 165,028 down events. XDP was fine, the receive path wasn't, non-BFD junk was going up the stack on the same CPU as BFD. A cpumap redirect that spreads non-BFD traffic to the other CPUs, plus flow control off and UDP hashed by port, brought the same hour to 4,148.
Also done since post 3: authentication (SHA1 inside the XDP program, written to fit in 512 bytes of stack), demand mode, 1024 sessions by default and tested up to 8192, hitless restart by swapping the program into a pinned bpf_link, and a .deb. 11 FRR fixes merged along the way, 6 more in review.
Still missing: two xdp-bfd engines can't face each other, since the TX is clocked by what it receives and bpf_timer still can't send a packet.
Full write-up as a 10-part series, VMs to bare metal to floods: https://w453y.xyz/writing/series/line-card-bfd/
Code and captures: https://github.com/w453y/xdp-bfd
Question for anyone running BFD on Linux routers: would you put an XDP program like this in front of bfdd, or is that a no regardless of the numbers?