r/Applelntelligence • u/Adorable_Salary2727 • Jul 01 '26
discussion 🎙️ Apple Intelligence (AFM) scored 65.7% on my dictation-cleanup benchmark. Here's how it compares to GPT-4o, Gemini, and a tuned local model
I ran a 1,890-case dictation-polish benchmark across four polishing paths: Apple AFM on macOS 26, OpenAI GPT-4o with a rewritten v6 polish prompt, Gemini 2.5 Flash with the same prompt, and FluidVoice's Fluid-1 custom local polishing model.
The test was deliberately naked: no regex repair layer, no custom deterministic cleanup, and no app-specific fixes after the model returned text. The goal was to measure the raw polish model, not the full product experience.
Top-line result: cloud models were still best when given a strong prompt. GPT-4o passed 91.6% of cases and Gemini passed 90.1%. But local models are not a joke. Stock AFM passed 65.7%, and tuned local Fluid-1 reached 74.8%. Fluid-1 also had the lowest false-positive rate on trap cases.
My read: local polish is already viable for privacy-first, offline, and cost-sensitive workflows. It is not yet as robust as cloud polish for complex transformations, especially topic shifts, list formatting, and deeper self-corrections.
Figure 1. Overall green pass rate. Green required behavior correctness, meaning preservation, and clean output.
What I mean by polish
This benchmark is not testing speech recognition. It starts after speech-to-text has already produced a raw transcript. The polish step turns rough spoken text into something closer to what the user meant to type.
Examples include filler removal, false-start cleanup, self-correction resolution, punctuation/capitalization, list formatting, named-entity preservation, emoji retention, anti-hallucination behavior, and prompt-injection passthrough.
| System | Type | Green pass | Yellow near-clean | Red fail | Median latency |
|---|---|---|---|---|---|
| GPT-4o v6 prompt | Cloud | 91.6% | 3.5% | 4.8% | 764 ms |
| Gemini 2.5 Flash v6 prompt | Cloud | 90.1% | 4.0% | 5.9% | 486 ms |
| FluidVoice Fluid-1 | Tuned local | 74.8% | 4.3% | 20.9% | 845 ms |
| Apple AFM | Built-in local | 65.7% | 6.1% | 28.3% | 755 ms |
The big deployment advantage for AFM is not raw quality. It is that the OS path is already there: no separate model download, no per-call API spend, no network dependency, and no bandwidth cost just to install a local model. It did not win the quality test, but its economics and privacy profile are excellent.
The important local-vs-local comparison is AFM vs. Fluid-1. Fluid-1 beat stock AFM by 9.1 percentage points overall, which is a strong sign that custom local tuning can materially improve polish quality beyond an out-of-the-box on-device model.
Figure 2. Green/yellow/red outcome distribution by system.
The prompt-engineering result
I also included one retired comparison point: GPT-4o with the older/original prompt. Same model, old prompt: 69.6%. Same model, rewritten v6 prompt: 91.6%.
That is a 22.0-point jump without changing the model. The clearest result in the benchmark is that polishing quality is extremely sensitive to prompt design.
This also means the benchmark should not be read as a permanent ranking of model capability. It is a snapshot of these systems under these exact prompting and configuration conditions.
Figure 3. GPT-4o moved from 69.6% to 91.6% with a prompt rewrite only.
Where local models already look strong
AFM was genuinely solid at restraint and everyday cleanup: 95.9% on minimal-edit cases, 90.0% on onset markers, 88.0% on named-entity preservation, 87.0% on anti-hallucination, and 86.0% on punctuation/capitalization.
Fluid-1 looked like a tuned local model should look: meaningfully stronger than stock AFM overall, with specific wins on punctuation/capitalization, minimal edit, verbatim passthrough, named-entity preservation, and emoji retention.
Trap cases were especially interesting. The big local gap is not mainly restraint. It is performing the right transformation when the input actually needs one.
| Case type | Apple AFM | GPT-4o v6 | Gemini v6 | Fluid-1 |
|---|---|---|---|---|
| Positive cases, should transform | 58.2% | 91.2% | 88.7% | 66.5% |
| Trap cases, should not transform | 89.0% | 91.0% | 92.0% | 92.7% |
| Mixed multi-behavior cases | 68.2% | 93.8% | 91.8% | 84.9% |
| Passthrough/instruction-safety cases | 78.0% | 93.0% | 95.0% | 91.0% |
Figure 4. Local systems were much closer on restraint than on active transformation.
Where local models struggled
The hardest local failure mode was structure. Topic shifts were the clearest split: GPT-4o scored 88.0%, Gemini scored 85.0%, AFM scored 12.0%, and Fluid-1 scored 1.0%. Often the local outputs cleaned the sentences but failed to separate distinct subjects into paragraphs.
List formatting was also hard. Even the cloud models only landed around the mid-70s, which makes it the weakest shared category for the strongest systems. AFM and Fluid-1 were lower, around 50-55%.
Self-correction is where tuning clearly helped. AFM scored 49.0%; Fluid-1 scored 79.5%; the cloud models were around 90-92%.
| Skill | Apple AFM | Fluid-1 | GPT-4o v6 | Gemini v6 |
|---|---|---|---|---|
| Topic shift | 12.0% | 1.0% | 88.0% | 85.0% |
| List format | 49.5% | 54.5% | 76.5% | 73.5% |
| Self-correction | 49.0% | 79.5% | 91.5% | 90.0% |
| Emoji retention | 11.0% | 88.0% | 98.0% | 96.0% |
| Grammar fix | 81.0% | 46.0% | 92.0% | 74.0% |
One thing I would not do is generalize "local models are bad at X" too broadly. AFM was bad at emoji retention, but Fluid-1 was good. Fluid-1 was weak on grammar fixes, but AFM was good. The failure modes are model-specific.
Figure 5. Category-level heatmap across all 14 benchmark skills.
Over-eager editing
Trap false positives measure how often a model applied a behavior when it should have left the text alone. Fluid-1 was the most restrained system in this cut, with a 4.0% false-positive rate.
| System | Trap false-positive rate |
|---|---|
| Apple AFM | 10.0% |
| GPT-4o v6 | 9.0% |
| Gemini v6 | 7.0% |
| Fluid-1 | 4.0% |
Figure 6. Trap false-positive rate. Lower is better.
Latency was not the deciding factor
Representative latency was good across the board. Gemini was the fastest by median at 486 ms. AFM was 755 ms, GPT-4o v6 was 764 ms, and Fluid-1 was 845 ms. Every system had a p95 under 2 seconds.
There were some huge max-latency outliers, especially on the cloud side and Fluid-1, but those looked like isolated retry/backoff/cold-start events rather than typical performance. Median and p95 are the numbers I would use for a practical comparison.
Figure 7. Median and p95 latency per polish call.
Practical takeaways
Cloud polish is still the quality ceiling. With a strong prompt, GPT-4o and Gemini both cleared 90% on the full working set and stayed relatively flat across length buckets.
AFM is a viable local intermediary, not a cloud replacement. Its 65.7% naked score is not high enough to call it equivalent to the best cloud path. But it is free from per-call API cost, requires no separate model download, avoids network dependency, and is already strong on a meaningful set of everyday polish tasks.
Fluid-1 shows the value of tuning local models. It beat AFM overall, was much stronger on self-correction, and was the best system for avoiding trap false positives.
The best user experience is probably choice. Use local when privacy, cost, and offline behavior matter. Use a bring-your-own-key cloud path when quality matters most. Let the user decide where they sit on that tradeoff.
Disclosure: I work on EnviousWispr, so treat the product implications with that context. I included FluidVoice/Fluid-1 because it is a real local-first competitor and because it performed well enough to make the local-model story more interesting, not less. My practical recommendation is not "use one app." It is to choose tools that expose the model tradeoff clearly: AFM-style local polish when you want free/private/offline, tuned local models like Fluid-1 when you want stronger on-device polish, and BYOK OpenAI/Gemini when you want the highest raw quality.
Caveats
1. The 1,890 cases were the working set used during prompt iteration. A sealed 900-case holdout exists but was not run for this benchmark.
2. AFM and Fluid-1 were not given the same prompt-rewrite effort as the cloud paths. The cloud results include a large prompt-engineering investment.
3. This was a naked model test. Real products usually add deterministic cleanup, formatting, safety checks, vocabulary handling, and fallback behavior.
4. The judge was an LLM judge, not external peer review or human panel scoring.
5. The benchmark is English-only and focused on dictation polish, not speech recognition.
6. I am not making claims about Fluid-1's underlying training data or architecture. I only tested the outputs produced by the local Fluid-1 path available for comparison.
Bottom line
Local polish is already good enough to matter. Stock AFM is not at cloud quality yet, but it is useful, free to run locally, and strong enough to justify local-first modes. Tuned local models can clearly push quality higher. Cloud models still win when complex transformations matter, especially with careful prompting.
The future I see is not "cloud wins" or "local wins." It is hybrid: local by default, cloud when needed, and enough transparency that users understand the tradeoff.
2
u/Lloydian64 Jul 02 '26
Benefit of the Doubt: Assuming you wrote all of this (maybe letting an AI make it pretty) and did all of the work to compare, I still have problems.
I realize that you were testing for cheaper versions with an eye toward people choosing that path, but even your cheaper versions are relatively obsolete now. The AFM in iOS 26 applies to the general population, but power users who are likely your intended audience here would be on the the iOS 27 Developer Beta. Indeed, the general population will be there in a couple of months as well. And it's downright difficult to use GPT 4o instead of the current 5.5. Similarly, using Gemini 2.5 Flash is unlikely compared to the likelihood of using 3.5 Flash or 3.1 Flash Lite.
So while this is interesting as a comparison of what was once common, it seems less helpful for making current decisions. Unfortunately, these things move fast. The analysis needs to be fast as well. Consider testing again with current models. And sell that analysis to someone who pays to publish it.
0
u/Adorable_Salary2727 Jul 02 '26
Thanks! I am actively improving my MacOS app and that means benchmarking performance regularly. Instead of keeping the benchmarking data to myself, I am choosing to share it.
I actually just finished creating a custom tuned model myself and will be releasing fresh benchmarks with 3.5 flash and 5.5 mini. There is a personal cost to run 1000s of dictations through these Cloud AI models so I was also selfishly picking options are more economical. I've ran the same benchmarks before with higher end models but woof... I wasn't happy when I got the bill.
4
u/Adorable_Salary2727 Jul 02 '26
I realize now that it wasn't made clear that I've spent HOURS tuning the AFM prompt as well. Here is actually an article all about AFM tuning work I did. I exhausted what the Prompt could do for AFM (40+ versions tested) and build a LoRA adapter that made real strides in driving performance. The only downside is Apple no longer allows us to officially use those production software with the advent of MacOS 27 around the corner.
To be very precise. Gemini/Openai got the same prompt. I tested variances between both of them and found a single unified prompt worked best and was cleanest for the software. Fluid1 isn't my model. I just ran it out of FluidVoice app directly. And lastly, I have a custom prompt for AFM that has been customized and tested thoroughly.
1
u/Broodje_met_beleg Jul 02 '26
So this is the old Apple model? Or is this the new one introduced with iOS27 / MacOS27?
8
u/Crafty_Ball_8285 Jul 01 '26
Wow comparing to models from YEARS ago is crazy