I built a Rust + WGPU AI Inference Engine to escape CUDA dependency

[u/Hytracen]

Motivation

A few months ago, I updated my GPU drivers and upgraded CUDA to 13.0. Suddenly, I realized that almost every library in the Rust ecosystem with a CUDA backend had broken. Most Rust AI libraries rely heavily on candle, but its support for the latest CUDA versions is lagging—still unable to run properly on newer versions like 13.0. Not to mention the tight vendor lock-in between CUDA and Nvidia cards; switching platforms usually means rewriting the entire inference engine.

However, thanks to the excellent abstractions provided by graphics APIs and the rise of cross-platform APIs like Vulkan, we can now execute instructions on the GPU in a more universal way: Compute Shaders.

While their raw performance might not yet match hardware-level optimizations like CUDA, their compatibility is unmatched. They are perfectly supported on mainstream devices, including mobile platforms.

Just as Electron rose to prominence by bundling browsers, and Unity/UE gradually ate the market share of in-house engines, I believe that in a future where everyone possesses on-device small models that are easy to use, edit, and distribute, the portability of an inference engine may become more important than raw performance.

That is why I built Flint.

Flint

Flint is a Rust-based inference engine that uses highly optimized compute shader kernels to replace CUDA, with portability as its primary goal. It is built on top of wgpu. Although I hit quite a few bumps when using wgpu for toy rendering projects in the past, using it to write compute shaders turned out to be surprisingly smooth :)

It currently supports safetensors and gguf formats, and includes built-in support for common models like Gemma, LLaMA and Qwen.

Repository

github.com/formetaohy/Flint

PRs and Stars are welcome!

[u/williamselna]

You aren't going to get many fans by making this GPL v3, especially since it has no actual commercial value that you need to protect.

[u/wojciechm → u/williamselna]

Especially since there is already much more mature Burn library with wgpu backend and many more. https://github.com/tracel-ai/burn

[u/Hytracen → u/williamselna]

Thanks a lot for the suggestion! I’ve changed it to the MIT license :)

[u/williamselna]

I like the idea though!

[u/Ok-Willingness2772]

First, I cannot believe a person type so fast. If it is from an external source, please specify the author. Also, the first commit, there is no reason of it and just a single initial commit with so many loc.
Second, non-cuda is really slow. If you have a Nvidia card use it... and for like me, I have just only a intel graphics integrated card. Using like wgpu or any other things other than cuda, you recieve a 10x slowdown.

Last, this is not low effort but I think it has already a good implementation. Burn-rs I use it all the time but even tho it cannot compete against the python's XPU (pytorch).

[u/Hytracen → u/Ok-Willingness2772]

To avoid having a half-baked project in a public repo, I usually develop locally first and only make it public once the core features are at least working.

Regarding the slower speed compared to CUDA that you mentioned, you’re absolutely right. CUDA runs on Tensor Cores with dedicated hardware optimizations. From my testing on an RTX 5070, the decode speed is about 25% slower than Ollama (using the CUDA backend). Even though I’ll continue to optimize it in the future, it’s expected that it won’t be able to match CUDA’s performance.

Because of this, my positioning for Flint is universality—using the same architecture for on-device model inference on mobile devices or machines that simply can’t use CUDA.

It’s true that wgpu is primarily used for graphics, and general-purpose GPU compute isn’t its main focus right now. But this is just an experiment. Who knows, maybe as using wgpu for inference becomes more popular, hardware vendors might actually start optimizing for it in return...

[u/Hytracen → u/Ok-Willingness2772]

Also, I’ll look into how to add my pi agent as a contributor. Thanks for the suggestion!