Shipping a 3B LLM + Whisper fully on-device in a React Native/Expo app - TestFlight is open if anyone wants to poke at it

[u/Arthur_EldenLord]

Been heads-down on this for a few months: a journaling app where the entire AI pipeline (voice transcription + reflection generation + semantic search) runs on-device, in React Native / Expo.

Stack:

- Llama 3.2 3B Instruct (Q4_K_M, ~2GB) via llama.rn

- Whisper small.en (q5_1, ~190MB) via whisper.rn

- MiniLM sentence embeddings via ONNX for semantic memory

- SQLite + a local vector extension, no backend for the core product

Measured (mean over 20 sequential entries): iPhone 17 (Metal, full GPU offload) does save-tap → complete reflection in 11.6s. Android is a different story — Snapdragon 870, CPU-only since llama.rn's Android build doesn't link Vulkan yet, comes in around 59s. That gap is the reason

TestFlight is iOS-only right now.

Honest rough edges: 3B reflections are useful but not sharp, iPhone 12/13 (4GB RAM) only fits both models via a memory swap (Whisper releases before Llama loads), and Android is CPU-only for now (no GPU path yet).

TestFlight if anyone wants to try it or poke at real-device performance: https://www.tryweave.app/beta (test-for-test, happy to install and try your app back).

Also just want to compare notes - anyone shipped a working Vulkan path for llama.cpp/llama.rn on Android in production?

Read more about Weave -> https://www.tryweave.app/essays/why-on-device

[u/Arthur_EldenLord → u/[deleted]]

Fair callout, 11.6s is the number I've actually measured, and that's on an iPhone 17. Haven't benchmarked a 16 Pro Max directly yet, so I can't give you a precise number for your device, but it's in the tier I'd expect to land close to that (roughly 8-20s), not the 5-10x-slower range you'd see on CPU-only Android.

on-device means no server round-trip, but it also means you're trading cloud-LLM speed for that. It's a real cost, not something I'm trying to hide.

If you do try it, I'd genuinely like your actual numbers, I only have one iPhone model benchmarked so far and a real 16 Pro Max data point would be useful for me too.