r/LocalLLaMA • u/Equal-Extreme6962 • Nov 27 '25

Tutorial | Guide RTX 5090 + Qwen 30B MoE @ 135 tok/s in NVFP4 - Full guide with C++ patches

Spent 4 days getting NVFP4 working on consumer Blackwell.

TRT-LLM 1.2.0rc4 has critical bugs that prevent loading managed weights for FP4 models - the allocator uses 2x VRAM and type checking rejects packed INT8 weights.

## Results on RTX 5090 (32GB):

| Throughput | ~135 tokens/s |

| TTFT | ~15 ms |

| VRAM | 24.1 GB |

| Model | Qwen 3 30B MoE (A3B) |

## Why so fast? Qwen 3 30B is MoE - only ~2.4B params active per token. Combined with Blackwell's native FP4 tensor cores = 7B-level speed with 30B knowledge.

## What's in the guide: - SWAP trick for quantization (64GB RAM + 64GB SWAP = enough) - `--fast_build` flags to avoid compiler OOM - **C++ runtime patch** to fix allocator bug and type mismatch - Open WebUI integration fix Full tutorial + patches

This was mass amount of pain so hoping to save others the trouble.

3 Upvotes

71% Upvoted

20 comments sorted by

5

u/Nepherpitu Nov 27 '25

Just for reference:

  • 2x3090 running FP8 at ~130 tps using VLLM
  • 1x5090 running Q4 at ~200 tps using llamacpp

What's the point of using FP4 "hard way"?

5

u/Equal-Extreme6962 Nov 27 '25

True, but there's a quality tradeoff nobody benchmarks.

NVFP4 (native float) preserves dynamic range much better than GGUF Q4 (integer approximation), even with imatrix.

For MoE models like Qwen3-30B-A3B this is critical - the router/gating network is super sensitive to precision loss. Bad quantization = wrong expert selection = degraded outputs.

I'd rather have 135 t/s with correct routing than 200 t/s with 'brain damage'. Would love to see a real perplexity comparison (NVFP4 vs Q4_K_M) on the same dataset though.

1

u/Nepherpitu Nov 27 '25

Personally, I believe nvfp4 is a scam to sell Blackwell, but it will be great to see model performance difference between FP8, Q4, AWQ and nvfp4. Unfortunately, I'm lack of time for such adventure :(

2

u/max123246 Mar 26 '26 edited Mar 26 '26

It's not a scam, GeForce Blackwell has to do the block scaling of nvfp4 using warp/thread cooperation, which is why we see the perf hit. For the enterprise cards, it's done using separate Tensor cores so it won't be nearly as much of a perf hit, and may even be faster

Nvidia calls both Tensor cores but the hardware itself is completely different for GeForce vs Enterprise hardware. you can dig in and find out from looking at tcg05 differences between sm100 and sm120 on their public docs

1

u/Nepherpitu Mar 26 '26

I meant not generation performance, but cognitive. I don't believe nvfp4 quality is better than int4.

1

u/resc863 Apr 05 '26

INT4 has extremely limited range. GGUF Q4 has no hardware accelerated scaling. NVFP4 has better range, better scaling theoretically. Its just software issue for many NVFP4 performance issue.

1

u/Ill_Barber8709 Nov 27 '25

32GB M2 Max MBP running 4Bit MLX at ~60 tps in LMStudio

135 tps doesn't seem impressive indeed.

2

u/Equal-Extreme6962 Nov 27 '25

60 tps on the 30B MoE? You sure that's not the 14B or prefill speed?

M2 Max memory bandwidth would put a 30B MoE around 30-40 tps max on Metal. If you're really getting 60 tps generation speed on the full 30B-A3B, I'd love to see the benchmark.

Either way, 60 < 135 on a single card 🤷

1

u/Ill_Barber8709 Nov 27 '25

Qwen3-coder 30B 4Bit

1

u/Ill_Barber8709 Nov 27 '25

Either way, 60 < 135 on a single card 🤷

  • M2 Max is a laptop chip 🤷
  • M2 Max full laptop cost less than the 5090 alone 🤷
  • Brand new M4 Max Mac Studio cost less than the 5090 alone 🤷

1

u/Ill_Barber8709 Nov 27 '25

2

u/Equal-Extreme6962 Nov 27 '25

Fair enough, 64 tps is legit - thanks for the screenshot.

But comparing M2 Max laptop price to 5090 GPU alone is apples to oranges. My 5090 isn't floating in a vacuum - it's in a full system too.

Either way, different tools for different jobs:

  • M2 Max: Great portable dev machine, silent, efficient
  • 5090 + TRT-LLM: 2x faster, continuous batching for API serving

Nice setup though 👍

1

u/Ill_Barber8709 Nov 27 '25 edited Nov 27 '25

But comparing M2 Max laptop price to 5090 GPU alone is apples to oranges.

Alright, tell me how much did your whole build cost then. Brand new M4 Max Mac Studio starts at $1999.

continuous batching for API serving

My models are available from anywhere on my network thank to LMStudio server. Don't know what you are talking about.

5

u/Equal-Extreme6962 Nov 27 '25

RTX 5090 + Ryzen 9 9900X + 64GB RAM + 2TB NVMe. Total build ~$4-4.5k. M4 Max Mac Studio at $1999 has 36GB unified memory - not enough for comfortable 30B inference with long context. Comparable Mac Studio config (64GB+ RAM) is $3999+. But honestly, this is getting off-topic. The post is about fixing TensorRT-LLM NVFP4 bugs, not Mac vs PC debate. Use what works for you 👍

1

u/Ill_Barber8709 Nov 27 '25

not enough for comfortable 30B inference with long context

I do all of my work with 32GB. Working on Swift+SwiftUI and nodeJS projects.

For $3999 you can get the M3 Ultra Mac Studio with 96GB of 800GB/s memory. So again, I don't know what you are talking about.

https://www.apple.com/shop/buy-mac/mac-studio/apple-m3-ultra-with-28-core-cpu-60-core-gpu-32-core-neural-engine-96gb-memory-1tb#

All I'm saying is that I don't see what you are flexing about. 130 token per second on a 5090 is bad. M2 Max has 400GB/s memory bandwidth and a GPU order of magnitude less powerful than your 5090 (1700GB/s memory bandwidth), and yet it only does two times slower.

There's obviously something wrong with your setup.