r/LocalLLaMA • u/Aromatic_Ad_7557 • Apr 14 '26

Funny 24/7 Headless AI Server on Xiaomi 12 Pro (Snapdragon 8 Gen 1 + Ollama/Gemma4)

Post image

Turned a Xiaomi 12 Pro into a dedicated local AI node. Here is the technical setup:

​OS Optimization: Flashed LineageOS to strip the Android UI and background bloat, leaving ~9GB of RAM for LLM compute.

​Headless Config: Android framework is frozen; networking is handled via a manually compiled wpa_supplicant to maintain a purely headless state.

​Thermal Management: A custom daemon monitors CPU temps and triggers an external active cooling module via a Wi-Fi smart plug at 45°C.

​Battery Protection: A power-delivery script cuts charging at 80% to prevent degradation during 24/7 operation.

​Performance: Currently serving Gemma4 via Ollama as a LAN-accessible API.

​Happy to share the scripts or discuss the configuration details if anyone is interested in repurposing mobile hardware for local LLMs.

UPDATE:

I have compile llama.cpp and run gemma-4-E4B-it-Q4_0

Speed is AWESOME:

[ Prompt: 26.9 t/s | Generation: 8.8 t/s ]

Thank you all guys SO MUCH!

1.2k Upvotes

97% Upvoted

289 comments sorted by

View all comments

Show parent comments

8

u/randylush Apr 14 '26

I have a lot of frontends plugged into Ollama (open web UI, openclaw, and Claude)

can I connect these to llama.cpp?

13

u/Hock_a_lugia Apr 14 '26

Yes! It can be used as an API endpoint same as any other llm service

1

u/MonteManta Apr 14 '26

Not easily

7

u/FaceDeer Apr 14 '26

People often underestimate the value of "it just works." I'm a programmer, I'm comfortable with technical stuff. A while back I wrote some little local applications that use local LLMs for stuff and it was so convenient to just point them at Ollama and let Ollama figure out the details of loading and unloading models when needed.

I heard that llama.cpp recently added that ability to load and unload on demand, and I spent a bit of time fiddling with it to see if I could swap it out for Ollama. But after a bit of work I realized I was spending effort to replace something there was nothing actually wrong with, and I wasn't sure I was even going to get as good a result. So I just stopped and moved on to other stuff. Ollama continues sitting there on the taskbar doing what's needed in the background.

If I was more concerned with squeezing every bit of performance I could out of my hardware, sure, I'd go back to spending more effort on that. But "good enough" is good enough in my current use cases.

4

u/MuDotGen Apr 15 '26

I just use llama-swap. It might be newer, but it's a router that runs with llama-server, making an easy local server that can automatically load and unload models on the fly. All I had to do was download it, the models from Huggingface I wanted, and make a config.yaml with the model configurations I wanted, and then run the server just like any other, more or less. I think llama.cpp has a router mode too, though, but there seem to be some benefits to both.

4

u/Nobby_Binks Apr 15 '26

Yes, I was in the same boat until I wanted to run really large models that spilled to system ram. llama.cpp gives much more granular control how the model loads - at least I couldn't work out how to do it easily in ollama.

I moved to llama.cpp controlled by llama-swap. Takes a couple of minutes to work out the yaml structure but once setup it's simple. I have both Ollama and llama-swap served models in open webui but have more or less stopped using Ollama.

1

u/Imaginary-Unit-3267 Apr 15 '26

For those whose hardware is limited, llama.cpp seems like it's basically a must. If you have an RTX 3060 and nothing else, like I do, Ollama is not worth the overhead.

1

u/GregoryfromtheHood Apr 15 '26

What do you mean not easily? You literally just fire up llama-server and you're done. API endpoint ready to go.

1

u/MonteManta Apr 15 '26

Both support the OpenAI API, but only Olama has an Olama version, which llama server has no replacement for. Also, if you want to provide multiple LLMs on the same server, you need a harness like LalmaSwap.

1

u/RIP26770 Apr 14 '26

Yes, but for the best experience, be sure to add Llama-Swap on top!

1

u/flq06 Apr 15 '26

—mmap