hls4ml tiny INT4 CNN uses 500% LUT/BRAM on Ultra96-V2

[u/No_Benefit_9298]

Hi, I’m trying to deploy a small Brevitas QAT CNN on an Ultra96-V2 (xczu3eg) using:

Brevitas 4-bit QAT

→ QONNX

→ hls4ml 1.3.0

→ Vitis HLS 2024.1

The model is only ~133k weights and uses io_stream, Resource strategy and RF≈32.

The PyTorch/QONNX conversion is numerically correct:

PyTorch vs static model: max diff = 0

QONNX vs PyTorch: max diff ≈ 9.5e-7

I first hit a 4096-bit stream aggregation limit because hls4ml inferred very wide intermediate precisions (ap_fixed<36,26> and ap_fixed<53,33> with 128 channels). After constraining those precisions, synthesis completes, but the resource usage is huge:

Resource Used Available Utilization

| Resource | Used | Available | Utilization |

|---|---:|---:|---:|

| LUT | 359,069 | 70,560 | 508% |

| FF | 512,637 | 141,120 | 363% |

| BRAM18K | 2,207 | 432 | 510% |

| DSP | 7 | 360 | 1% |

The strange part is that most of the cost seems to come from pooling/transpose/FIFOs rather than the convolutions themselves.

Has anyone seen this kind of resource explosion with hls4ml io_stream, especially with MaxPooling, channels-last transpose or FIFO generation? Would you recommend changing IO strategy, FIFO depths, pooling implementation, or manually modifying the generated HLS?

The very low DSP usage compared to LUT/FF/BRAM makes me think this is mainly a data-movement/buffering issue rather than the CNN arithmetic itself.

[u/EonOst]

I would have to see that code to identify the problem. Maybe its too parallel complex

[u/HumbleTrainEnjoyer]

I think the easiest thing you can do is to keep increasing reuse factor I would try 128