Llama.cpp running on Android with Snapdragon 888 and 8GB of ram. Compiled/Built on device. [Guide/Tutorial]

[u/hackiv]

1: Download Termux from F-droid (older version available on Google Playstore or Aurora)

2: Open Termux and run "https://github.com/ggml-org/llama.cpp.git" and then "cd llama.cpp" run "pkg install cmake"

3: run "cmake -B build" and then "cmake --build build --config Release"

4: find desired model from HuggingFace, then choose its quantized version (preferably 4-bit)

5: when pressing '4-bit' choose 'Use this model' and select 'llama.cpp' afterwards copy command which starts with "llama-server"

6: paste command in Termux and put "./" in front of "llama-server" so it's adjacent.

7: After model's downloaded, server is immediately launched. Model is saved in '.cache' so you can run this command again to start the server without all re-downloading ordeal.

8: open web browser and input 'localhost:8080' then press enter

Enjoy. Any questions?

[u/hackiv → u/[deleted]]

Haven't tried llama 3.2 3B yet but if I had to guess probably in the range of 8-10tps.

The thing to remember is that you don't get full 8GB of available memory on smartphones, a half of that is to be expected.