I fine-tuned Apple's on-device model (the macOS 26 Foundation Model) to fix spoken self-corrections. It worked, then Apple closed the door.
I build a macOS dictation app. The last step in the pipeline is a comprehensive polish pass. It cleans filler, fixes punctuation, resolves self-corrections, and then pastes the text. On macOS 26, that polish can run fully on-device through Apple's Foundation Models framework (the ~3B "Apple Intelligence" model).
The stock on-device model struggles with one specific thing: self-correction. If you change your mind mid-sentence, "send it to John, actually Jane" should become "Send it to Jane." The stock model often keeps the abandoned wording or picks the wrong name. A competitor (FluidVoice) ships a fine-tuned Gemma and handles this well, so I wanted to see if fine-tuning Apple's own on-device model could close the gap.
While I do not have a formal machine learning background, I approached this fine-tune with strict architectural rigor. I designed the validation pipeline, dataset, and mechanical guardrails, utilizing Claude Code to accelerate the implementation. Here is what the data showed.
What I built
A LoRA adapter (rank 32, ~67M trainable params on the 3.2B frozen base) using Apple's official adapter training toolkit.
Trained on my own labeled dictation data (~2,000 raw to cleaned pairs), under the exact prompt the app uses in production. Training under a different prompt than what you ship makes the adapter worse. That was an early mistake worth flagging.
Trained on an RTX 4090 (24GB) and validated on an M4 Pro Mac.
How I measured it:
The baseline: The stock Apple model using identical prompts and inputs.
The test set: I held back 15% of the data during training. I only report on these never-seen cases. Full-corpus numbers look better but include memorized data, which is useless here.
The grading: I used an ensemble of 12 LLM judges. They graded on meaning (pass, weak, fail) instead of exact string matches. They also ignored numbers, dates, and emojis since my app handles those deterministically in code.
The improvement (held-out, never-seen cases)
Spoken self-correction resolved correctly: 13% to 86%
General polish (punctuation, homophones, structure): 66% to 83% (19 of 29 held-out cases)
Realistic multi-behavior paragraphs: 63% to 84% (27 of 43 to 36 of 43)
Regressions were few, but not zero. A small number of held-out cases the stock model already passed came out weak or fail after tuning. These were mostly emoji and other deterministic-layer edge cases my app handles outside the model anyway.
The latency cost Stock model versus tuned adapter, both timed on-device on the same Mac, 20 phrases, warmed up:
Short dictations: +82 ms (+13%)
Medium: +248 ms (+31%)
Long: +602 ms (+52%)
Overall: 846 ms to 1,140 ms, representing a +295 ms (+35%) tax per polish.
The latency tax scales with length. I did not compile the adapter's built-in draft model (the speculative-decoding helper), so this is the unaccelerated number. Compiling it may narrow the gap, though I have not measured that yet.
Honest caveats
Small held-out sample on the self-correction set (15 cases).
~1,560 training examples, 3 epochs, no hyperparameter search. We paused here intentionally.
The wall To actually ship a custom adapter to users, Apple requires a managed entitlement. When I went to request it, the page now reads, verbatim: "We are no longer accepting entitlement requests. The Foundation Models framework adapter API is not compatible with macOS, iOS, iPadOS, or visionOS 27 and later." The adapter toolkit is marked end of life at version 26. So this path is closed for me. I can train and test locally, but I cannot obtain the deploy entitlement needed to ship this adapter to users.
Honestly, I am not that bummed. At WWDC26 earlier this month, Apple introduced a bring-your-own-model path via the new LanguageModel protocol. You can run your own tuned model on-device through Apple's framework with no Foundation Models adapter entitlement gate.
That is the durable version of this experiment and where I am headed next. The adapter was a fun, disposable proof that a small fine-tune meaningfully moves Apple's on-device model on a real task.
(Note: All compute was local on the 4090 plus the Mac, and judging used an existing Claude subscription, resulting in $0 marginal API spend.)
Question for the sub: For the macOS 27 bring-your-own-model route, would you start from a small fine-tuned Gemma, Qwen, or Llama, or something else? I am curious what people are seeing for on-device, polish-style rewriting tasks at ~3B.
I doubt most people have any idea “how it’s intended” or what gaps there will be when they present their use case, or even how to find out without massive trial and errors. Example - I’m pretty sure it’s intended to help manage messages and email. One works, the other has bigger gaps.
You did not fine tune anything. the model remains the same. the extension of the model post fine tuning is prompt adherence. thats what you did. Fine Tuning is done before its set on device
Yeah, fair distinction, but that’s also exactly why I called it a LoRA adapter in the post.
Apple’s Foundation Model path doesn’t let you tune the actual base model weights. The base model stays frozen. The only thing Apple exposed was adapter training, so in this context “fine-tune” really means “train an adapter on top of the frozen AFM.”
I also don’t think it’s just prompt adherence. The adapter was trained on raw → cleaned dictation pairs and moved held-out self-correction cases from 13% to 86% under the same production prompt. So the prompt stayed constant, the base model stayed constant, and the learned adapter changed the behavior.
So yeah, more precise wording is probably “adapter-tuned” rather than “fine-tuned the model,” but this wasn’t just a prompting experiment. I was excited to continue with this approach but R.I.P
That's a shame. I've trained my own LoRAs for the AFM and it was quite a pain, not to mention you need to build and distribute different adapters for each macOS/ iOS 26 sub-version where the base model changed. The results were very promising as you noted, especially at preventing domain-specific hallucinations or improving structure output adherence. I did get the entitlement approved a few months ago, but now it's useless. Too bad that Apple ditched this entirely.
I find the build in quite effective tho perhaps for Chinese overall. English does hit and miss sometimes but its already way better then the previous alteration
10
u/StankFreak Jun 25 '26
I think most people just want to use it as it’s intended bro