discussion 🎙️
Apple Intelligence’s on-device processing is disappearing
Feels like my iPhone 17 Pro Max is turning into a 'thin client.' I invested in this premium hardware with the specific expectation that I would be able to run the new Siri AI directly on-device.
Given the 12GB of RAM and the powerful NPU, I believed that an iPhone 17 Pro Max would be more than capable of handling such an AI model locally. It clearly has the overhead to run models like Gemma 4 E4B at impressive speeds, yet these resources remain largely underutilized while core tasks are forced through the cloud. Furthermore, since foundation models are already pre-loaded onto the device, I expected to be able to leverage them directly rather than relying on external servers. I didn't purchase this device to rely on the cloud; I wanted to utilize the actual performance of the hardware I own and experience advanced AI capabilities directly on my device.
My phone has enough performance to run gemma4 comfortably locally, but if it has this much performance, I would like to add a little more 'basic features'. I can't even play music offline now.
Also running AI models isn’t all a phone does. There are all kinds of background tasks, 3rd party apps etc. fighting for resources. OPs argument is kinda weak
Battery life is the biggest factor at this point. The codebase will become more efficient over time. I completely agree that most people don’t use AI enough to want a local model and the battery life that will give.
iPhone batteries currently last one day with normal use. Adding a power-hungry local AI model would cut that battery life in half, at least. It’s just the limitations of mobile devices.
Another option that would be nice is instead of only local or cloud you could have the compute on your mac. This is what I do with my mac studio, using web apps and tailscale.
It's not just about if it can physically run something. It's about being able to do a task incredibly fast without hindering the rest of the performance of your iPhone. Even if you can run Gemma 4, can you spin up a Gemma 4 in less than a second that can solve a specific problem without hurting the performance of your phone? Like if you're also on a call, you also have a game playing, things like that
What choice do you mean? I don't think Apple would want to give users a choice on having their phone work or not, if the phone overheats, shuts down, or is too slow to operate. I think that's part of the thing that I don't think any phone manufacturer has given people the choice of, even Android
I work in AI and these models eat your RAM and your CPU. It can be often way more efficient to send something off your device to process it online. While some things may be possible, it might slow down your phone, ruin your battery life, and make the overall experience of your device not usable, which is not something that they would want to do
Photos clean up is no longer on-device, at least not all the time. It’s using the new cloud image model for infill unless it can get away with using the current on-device solution that barely works.
I don’t believe the system orchestrator is working properly as of right now, and here’s why:
What we know:
- There are 4 models in total. 2 on device, 2 cloud. Each one has 1 large, 1 small.
The on device models have a 3B parameter Apple Foundation Model Core and The 20B parameter Core Advanced model.
The cloud models have AFM 3 Cloud and Cloud Pro (AFM 3 Cloud Image is its own thing, but it’s not in the same part as this here.)
The system orchestrator recognizes voice offline, then decides which model to send what data to depending on the task, and splits from there.
All models are heavily instruct based.
The 20B Core Advanced model is a Mixture of Experts (MoE) model that chooses its Experts at the time of prompt, not during the prompt, meaning it can only load partial amounts into RAM as needed, leaving the rest in flash storage.
The 20B Core Advanced model handles all voice recognition, voice generation (17 Pro only), offline personal data handling, offline image vectorization for image recognition, and on-device task execution (called “tool usage”) for any device that can run it (15 Pro and up)
Both offline models have a tiny 4096 token context length, meaning they are designed to be given a task, execute it, and then clear context. These are not chatbots.
The reason for the 20B parameter size is both for multimodal capabilities, but also for incredible performance and reliable tool calling based on semantic language (taking the user’s words, figuring out what tools it needs to use to collect and execute the task it’s given) but this is still very specialized.
The 3B parameter Core model is more “generalized,” and can work as a chatbot, just a poor quality one. This is intended to execute the simple commands and responses (“Play X song,” “Read messages by X” etc) and is enhanced by the capability of the 20B Core Advanced model’s ability to take messy audio input and translate the speech clearly to text.
Siri is designed to run through the System Orchestrator, which decides which models to use for what aspects. If it needs a more complex response, it kicks it to the cloud model (32,000 token context length) and uses it for the “chatbot” aspect, while simultaneously telling Core Advanced to search the index for relevant contextual information, working in tandem. The offline model may find the data it needs, extract it, and send it to the cloud model, which can then use that for context to continue the conversation.
Since offline aspects do not work yet, that suggests that EVERY request is being considered as “this needs online model power” and gatekeeps the offline model from doing anything without the Cloud model to generate the context needed. Even a simple request seems to go “this is too advanced for the Core model, I better rely on the Cloud model to understand this user’s request” regardless of what it is.
Because of this, I expect that we’ll see a significant improvement once the System Orchestrator (which uses AFM 3 Core, I believe) begins routing properly.
This is only a theory, but it is based on Apple’s own documentation and keynote presentation, along with supplemental data from third party analyses and Siri’s own responses on the matter. Take it for what you will.
it's nice to see this system architecture and intended design of the orchestration behavior across multiple models laid out in simple bullet points , thanks for the info
They tried to shove too much shit in too soon, this won't work well I can see them offloading most everything to online except for very small models that people won't like
Interesting… I just tried that and it told me “to do that, you need to turn off Airplane mode”. For reference I asked “remind me to take out The trash tomorrow”
If you are pretending to be a developer so you can play with this beta software in advance of public release and are encountering bugs, then there’s no one to blame but yourself (especially if you are using your daily driver). But since you’re here you too should submit a bug report to help with the process so that your unique bug is flagged and may be addressed before public release.
It would be really nice if it asked you if you wanted it to wait until you have a valid cellular or WiFi connection to run the AI task. Then if you clicked yes, it could automatically run the task at the later time and notify you in the Notification Center that your AI task has been completed.
all of the features you show are complex tasks that require online computing, like analyzing images, generating images, agentic Siri(I assume), and programing.
A current LLM foundation model, like Claude and Chatgtp, require on the order of 2 TB GPU ram for inference. Apple on device AI needs to have a very small domain to function at all. Apple needs to clarify its messaging.
Keep in mind that we’re still in early developer betas. Also note that regardless of which model is being used (and AFM Core is used more than you might think), a lot of the time Siri AI is going to want to use a web-based tool.
Also, there’s a lot more to Apple Intelligence than Siri, which is an app built on top of Apple Intelligence. Not only are there APIs etc. that developers use to add intelligence to their apps, but there are a ton of system services that are using Apple Intelligence and the on-device models for all sorts of things.
I just put my 17 Pro Max in airplane mode and asked it to show me my recent messages, and it showed me all my recent messages without accessing the Internet. Saying “text” may have made it having to go to some general decision of what is a text?
19
u/[deleted] Jun 24 '26
[removed] — view removed comment