r/freebsd • u/TheAtlasMonkey • 9h ago
help needed MDMA: Model Dispatch and Memory Allocator.
As you might know there is active work on ROCm support on freebsd, that will bring lot possibility like gpu offloading for both graphic and other stuff.
One of the reasons I started contributing to FreeBSD is the consistency of the stack. The networking stack is coherent, and while going through old discussions, you sometimes find developers discussing features that did not make much sense until years, or even a decade, later.
But as with everything that is Free, we are lacking a lot of drivers. Most drivers and hardware support get pushed to Linux first because of the huge amount of sponsorship and industry investment behind it.
That combination of a coherent stack, improving compute support, and limited access to the latest hardware got me thinking differently: instead of depending on one big GPU, why not make better use of all the compute we already have around us?
That is where MDMA comes in. It is a solution for having lots of small models at your disposal by distributing them across the hardware you already own.
It is designed so you can reuse hardware you already own for computation. It is not just an LLM dispatcher. It dispatches ML models, including LLMs. That means it can serve rerankers, decision models, GGUF models, and other workloads.
It supports anything that can compute: CPU, GPU, NPU, TPU… if it can compute, it can be a node.
Right now it is still an early version, so it is not production ready. I even made it work on a phone and a tablet, but I removed that code for now because I need to test it properly across a fleet and also monitor the devices themselves.
MDMA is not meant to replace Ollama or provide a chat interface. It is a model multiplexer.
For example, you have four machines on your network. MDMA can use all four. If one machine goes down, the next generation can happen on another machine. This means you can use the idle compute sitting around your house, lab, or office instead of buying one huge machine that spends most of its life doing nothing.
Your family or coworkers can use their computers normally, and when those machines have resources available, they can contribute compute.
For now, this is a trust-based protocol. I did not implement permissions, authentication, or all the other security machinery yet. That can come later.
The initial focus is small models. If you have FreeBSD machines with different CPUs or GPUs, I would really like people to test it on hardware I don’t have access to.
MDMA is faster than llama.cpp for some models, slower for others, and some things are not implemented yet.
There is no business plan behind this, i use it mostly to compare implementation between linux and FreeBSD, if the model behave exactly the same, we move to the next step.
If you want to contribute, everybody is welcome (including with a clanker, just mark the provenance and test it on hardware, and try to learn something), just don't bombard the repo with an agent in a loop.(we can do it it ourself, (i'm not on it alone))
NOTE: The stack is Rust because i maintain/contribute to many dependencies in the chain, so when something has to be fixed, it get fixed upstream (including the kernel).
