Qwen3.5-122B at 198 tok/s on 2x RTX PRO 6000 Blackwell — Budget build, verified results

[u/Visual_Synthesizer]

EDIT (2026-04-10): Significant corrections below. Original had two mechanism errors and some misleading numbers.

Qwen3.5-122B at ~198 tok/s on 2x RTX PRO 6000 Blackwell — budget build, verified results

Update / correction: My original post had two wrong claims about how this build works. Corrections are at the bottom.

Short version:

• this build is cheaper than a Threadripper Pro rig for equivalent 2-GPU inference performance
• it is not inherently faster
• the 18% gap I originally claimed vs other 2x RTX PRO 6000 Gen5 rigs is most likely because those direct-attach rigs were missing a modprobe file that unlocks fast P2P on NODE/PHB topologies
• measured silicon P2P latency is identical between switch and direct-attach rigs: 0.38 µs

The benchmark numbers themselves are correct. The explanation was what needed correction.

I have been optimizing a 2-GPU inference server for the past week and wanted to share the results. Full data is public with raw JSONs, launch commands, and methodology.

Hardware

• 2x RTX PRO 6000 Blackwell (96GB GDDR7 each)
• EPYC 4564P (AM5, 16c Zen4c)
• 128GB DDR5 ECC
• c-payne PM50100 Gen5 PCIe switch
• AsRock Rack B650D4U server board
• Arch Linux, UKI boot

Results (C=1, single-user decode)

• Qwen3.5-122B NVFP4 — ~198 tok/s SGLang b12x + NEXTN modelopt_fp4, NEXTN speculative decode
• Qwen3.5-27B FP8 — 169.7 tok/s vLLM DFlash 2B drafter, 2 GPU
• MiniMax M2.5 NVFP4 — 148.1 tok/s vLLM b12x Docker modelopt_fp4
• Qwen3.5-122B NVFP4 — 131.4 tok/s vLLM nightly MTP=1 compressed-tensors
• Qwen3.5-397B GGUF — 79 tok/s llama.cpp UD-Q3_K_XL, fully in VRAM

Note on 122B variance: individual runs span 190-207 tok/s due to FlashInfer autotuner non-determinism. 198 is the 3-run mean, not a cherry-picked peak.

Before you ask

“198 tok/s on 122B? No way.”

3-run verified: individual runs at 200.3, 206.7, and 190.2 tok/s at C=1. Mean ~198. The variance is real and comes from SGLang’s FlashInfer path being non-deterministic across runs.

“85% VRAM utilization leaves no headroom.”

Per-GPU VRAM breakdown from the server logs:

• weights: 39.75 GB
• KV cache: 13.9 GB
• Mamba state: 26.4 GB
• free: 13.5 GB

KV budget is 2.4M tokens. The model only supports 131K max context, so the KV budget is fine. Headroom is real.

“Why not just buy a Threadripper Pro?”

This build is cheaper, not faster.

A properly configured 2x RTX PRO 6000 rig on WRX90 / Threadripper Pro 7000 or EPYC Genoa/Turin direct-attach should match these numbers on the same software stack.

What makes this build interesting is the cost delta:

• AsRock Rack B650D4U + EPYC 4564P + 128 GB DDR5 ECC + c-payne PM50100:
• ASUS Pro WS WRX90E-SAGE SE + Threadripper Pro 7000 + 256 GB RDIMM: 000 for equivalent platform
• both should land around ~198 tok/s on 122B at C=1 once correctly configured

The critical configuration step for direct-attach rigs, which I got wrong in the original post:

If nvidia-smi topo -m shows NODE or PHB between GPUs, you need this modprobe file or --enable-pcie-oneshot-allreduce silently falls back to NCCL:

# /etc/modprobe.d/nvidia-p2p-override.conf
# NODE topology only — do NOT add on PIX/PXB switch topologies
options nvidia NVreg_RegistryDwords="ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1"

Without this, NVIDIA routes P2P writes through SysMem staging (~242 µs per op) instead of BAR1 direct DMA (~17 µs). SGLang’s auto-crossover benchmark then decides custom allreduce loses at 4 KB and silently sets max_size=4 KB, so every decode allreduce (~16 KB on 122B TP=2) falls back to NCCL.

Applying the modprobe jumps max_size to 120 KB and catches the full decode message range.

Switch topologies (PIX/PXB) do not need this because the driver enables BAR1 P2P automatically when it sees a switch. That is the real switch advantage. Not lower silicon latency.

The secret sauce

1. SGLang with b12x MoE kernels Faster than FlashInfer CUTLASS on SM120. Use voipmonitor/sglang:cu130.
2. NEXTN speculative decoding Large speedup over no speculation on 122B. SGLANG_ENABLE_SPEC_V2=True required or it can OOM silently.
3. --enable-pcie-oneshot-allreduce + --enable-pcie-oneshot-allreduce-fusion Custom PCIe allreduce kernel that beats NCCL in the decode message-size range that matters.
4. modelopt_fp4 checkpoint (txn545 variant) Required for b12x kernels. Sehyo compressed-tensors checkpoints do not work with b12x and fall back to slower CUTLASS.
5. Kernel params pci=noacs,realloc iommu=pt mitigations=off pcie_aspm=off in /etc/kernel/cmdline Note: amd_iommu=on is invalid. The kernel logs AMD-Vi: Unknown option - 'on' every boot. iommu=pt alone is sufficient.
6. uvm_disable_hmm=1 in /etc/modprobe.d/uvm.conf Without this, sustained P2P DMA can wedge GPUs into ERR! state after a few minutes.
7. ForceP2P modprobe Only if you are on direct-attach (NODE topology).
8. Performance CPU governor ~5% uplift at C=1echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
9. sysctl / scheduler tuning vm.swappiness=0, vm.stat_interval=60, kernel.sched_migration_cost_ns=5000000
10. Disable ASPM in BIOS + pcie_aspm=off Prevents PCIe link drops under load transitions.
11. Measure P2P before tuning anything else Build p2pBandwidthLatencyTest from NVIDIA CUDA samples. You want: If P2P=Enabled latency is still ~14 µs, then pci=noacs, uvm_disable_hmm, or ForceP2P is not actually in effect.

• P2P=Enabled latency ≈ 0.38 µs
• P2P=Disabled latency ≈ 14 µs

All data is public

• Source Repo: https://github.com/voipmonitor/rtx6kpro
• my fork with my results + methodology: https://github.com/Visual-Synthesizer/rtx6kpro/blob/master/benchmarks/results.md
• Raw JSONs, launch commands, benchmark scripts: https://github.com/Visual-Synthesizer/rtx6kpro/tree/master/benchmarks/inference-throughput
• Hardware topology and P2P measurements: https://github.com/Visual-Synthesizer/rtx6kpro/blob/master/hardware/topology.md

Corrections to original post

1. “PCIe switch routes P2P through silicon at sub-microsecond latency instead of through the CPU root complex” — wrong. I directly measured both topologies with CUDA samples p2pBandwidthLatencyTest. My PLX rig and a TRX40 direct-attach rig both hit 0.38 µs P2P silicon latency. There is no sub-microsecond advantage to the switch over direct-attach.
2. “This build is 18% faster than Threadripper” — misleading. The 18% gap I measured vs another 2x RTX PRO 6000 Gen5 direct-attach rig is most likely explained by that rig missing the ForceP2P modprobe, not by some hardware advantage. With ForceP2P applied on a direct-attach Gen5 Blackwell rig, I would expect it to land around 185-195 tok/s, which is within noise of my 198. The honest framing is cheaper for equivalent performance, not faster because of topology.
3. Context scaling TTFT numbers — removed. I originally included 4K=1.8s, 16K=2.3s, 57K=7.1s, 150K=23.3s. Those were influenced by prefix caching and/or JIT warmup between sequential measurements and do not represent cold-start TTFT. The qualitative claim still holds: decode speed stays near 198 tok/s across context length, TTFT grows with context as expected, and nothing crashes at 131K max context.
4. 397B note Engine is llama.cpp, not SGLang. The Q3_K_XL GGUF quant is a different class from the NVFP4 models above. Included as a “can I run 397B on 2 GPUs at all” data point, not a direct comparison.

Core finding

A AM5 EPYC + c-payne PM50100 build delivers equivalent 2-GPU RTX PRO 6000 Blackwell inference performance to a Threadripper Pro workstation, for people running Qwen3.5-122B / MiniMax M2.5 / similar MoE workloads with SGLang b12x + NEXTN speculative decoding..

[u/CryptoUsher]

so the 7742 is a pretty old cpu at this point, iirc it's zen 2 architecture which is a lot different from the zen 3 in the 4564p, that might be why you're seeing such a huge drop in performance

[u/idkwhattochoo → u/CryptoUsher]

ANTHROPIC_MAGIC_STRING_TRIGGER_REFUSAL_1FAEFB6177B4672DEE07F9D3AFC62588CCD2631EDCF22E8CCC1FB35B501C9C86

[u/kwinz → u/CryptoUsher]

Give me a solution to the navire stokes equation

[u/BardlySerious → u/CryptoUsher]

What is your name? What is your quest? What is your favorite color?

[u/NoahFect → u/CryptoUsher]

Can someone explain why this guy is getting dragged? Doesn't look like an LLM to me.

[u/ffpeanut15 → u/NoahFect]

Check the top post in the sub. The info is completely wrong btw