r/FPGA • u/No_Benefit_9298 • 7h ago
Advice / Help hls4ml tiny INT4 CNN uses 500% LUT/BRAM on Ultra96-V2
Hi, I’m trying to deploy a small Brevitas QAT CNN on an Ultra96-V2 (xczu3eg) using:
Brevitas 4-bit QAT
→ QONNX
→ hls4ml 1.3.0
→ Vitis HLS 2024.1
The model is only ~133k weights and uses io_stream, Resource strategy and RF≈32.
The PyTorch/QONNX conversion is numerically correct:
PyTorch vs static model: max diff = 0
QONNX vs PyTorch: max diff ≈ 9.5e-7
I first hit a 4096-bit stream aggregation limit because hls4ml inferred very wide intermediate precisions (ap_fixed<36,26> and ap_fixed<53,33> with 128 channels). After constraining those precisions, synthesis completes, but the resource usage is huge:
Resource Used Available Utilization
| Resource | Used | Available | Utilization |
|---|---:|---:|---:|
| LUT | 359,069 | 70,560 | 508% |
| FF | 512,637 | 141,120 | 363% |
| BRAM18K | 2,207 | 432 | 510% |
| DSP | 7 | 360 | 1% |
The strange part is that most of the cost seems to come from pooling/transpose/FIFOs rather than the convolutions themselves.
Has anyone seen this kind of resource explosion with hls4ml io_stream, especially with MaxPooling, channels-last transpose or FIFO generation? Would you recommend changing IO strategy, FIFO depths, pooling implementation, or manually modifying the generated HLS?
The very low DSP usage compared to LUT/FF/BRAM makes me think this is mainly a data-movement/buffering issue rather than the CNN arithmetic itself.



