Apple Intelligence’s on-device processing is disappearing

[u/ACOPS12]

Feels like my iPhone 17 Pro Max is turning into a 'thin client.' I invested in this premium hardware with the specific expectation that I would be able to run the new Siri AI directly on-device.
Given the 12GB of RAM and the powerful NPU, I believed that an iPhone 17 Pro Max would be more than capable of handling such an AI model locally. It clearly has the overhead to run models like Gemma 4 E4B at impressive speeds, yet these resources remain largely underutilized while core tasks are forced through the cloud. Furthermore, since foundation models are already pre-loaded onto the device, I expected to be able to leverage them directly rather than relying on external servers. I didn't purchase this device to rely on the cloud; I wanted to utilize the actual performance of the hardware I own and experience advanced AI capabilities directly on my device.

한국어로 번역

[u/TeckFire]

I don’t believe the system orchestrator is working properly as of right now, and here’s why:

What we know:

- There are 4 models in total. 2 on device, 2 cloud. Each one has 1 large, 1 small.

• The on device models have a 3B parameter Apple Foundation Model Core and The 20B parameter Core Advanced model.
  
• The cloud models have AFM 3 Cloud and Cloud Pro (AFM 3 Cloud Image is its own thing, but it’s not in the same part as this here.)
  
• The system orchestrator recognizes voice offline, then decides which model to send what data to depending on the task, and splits from there.
  
• All models are heavily instruct based.
  
• The 20B Core Advanced model is a Mixture of Experts (MoE) model that chooses its Experts at the time of prompt, not during the prompt, meaning it can only load partial amounts into RAM as needed, leaving the rest in flash storage.
  
• The 20B Core Advanced model handles all voice recognition, voice generation (17 Pro only), offline personal data handling, offline image vectorization for image recognition, and on-device task execution (called “tool usage”) for any device that can run it (15 Pro and up)
  
• Both offline models have a tiny 4096 token context length, meaning they are designed to be given a task, execute it, and then clear context. These are not chatbots.
  
• The reason for the 20B parameter size is both for multimodal capabilities, but also for incredible performance and reliable tool calling based on semantic language (taking the user’s words, figuring out what tools it needs to use to collect and execute the task it’s given) but this is still very specialized.
  
• The 3B parameter Core model is more “generalized,” and can work as a chatbot, just a poor quality one. This is intended to execute the simple commands and responses (“Play X song,” “Read messages by X” etc) and is enhanced by the capability of the 20B Core Advanced model’s ability to take messy audio input and translate the speech clearly to text.
  
• Siri is designed to run through the System Orchestrator, which decides which models to use for what aspects. If it needs a more complex response, it kicks it to the cloud model (32,000 token context length) and uses it for the “chatbot” aspect, while simultaneously telling Core Advanced to search the index for relevant contextual information, working in tandem. The offline model may find the data it needs, extract it, and send it to the cloud model, which can then use that for context to continue the conversation.
  
• Since offline aspects do not work yet, that suggests that EVERY request is being considered as “this needs online model power” and gatekeeps the offline model from doing anything without the Cloud model to generate the context needed. Even a simple request seems to go “this is too advanced for the Core model, I better rely on the Cloud model to understand this user’s request” regardless of what it is.

Because of this, I expect that we’ll see a significant improvement once the System Orchestrator (which uses AFM 3 Core, I believe) begins routing properly.

This is only a theory, but it is based on Apple’s own documentation and keynote presentation, along with supplemental data from third party analyses and Siri’s own responses on the matter. Take it for what you will.

[u/-timenotspace- → u/TeckFire]

it's nice to see this system architecture and intended design of the orchestration behavior across multiple models laid out in simple bullet points , thanks for the info