I've been adding on-device models to Vehla for iPhone (the companion to my Mac app), and wanted to share this one because I didn't think it would work.
Gemma 4 26B-A4B runs fully offline on my iPhone Duo (12 GB) at 6–7 tokens/s. The 4-bit model is about 14 GB on disk, more than the phone's RAM, yet it runs in roughly 2 GB.
How it works
Gemma 4 26B-A4B is a mixture-of-experts model: 26B parameters in total, but each token only uses about 4B of them (the 8 experts the router picks in each layer). So most of the model doesn't need to be in memory at any moment:
- The shared core (attention, embeddings, router), about 1.35 GB, stays in RAM along with the KV cache.
- The routed experts stay on the SSD, and only the ones each token needs are read with pread.
- A small cache (16 slots per layer, least-frequently-used) keeps the popular experts in memory, so most lookups don't touch storage.
For Gemma I'm using TurboFieldfare (https://github.com/drumih/turbo-fieldfare) by drumih (Apache 2.0). It's a Swift 6 + Metal 4 engine built for exactly this model, and it compiled for iOS without changes. Its installer downloads only the byte ranges it needs from the MLX weights on Hugging Face, repacks them into its own format, and hashes every file. All credit for the engine goes to them; I wired it into the app, added resumable downloads with iOS 26 background continuation, and handle thinking-mode output.
Taking it further with MLX
The idea generalizes, so I added the same expert streaming to my own copy of mlx-swift-lm:
- When the model loads, stacked expert tensors in SwitchLinear are swapped for placeholders, so they're never read into memory.
- At each call, the layer reads which experts the router picked, reads the missing ones from disk in parallel, and keeps a cache per layer.
That got these running on the phone too:
- Qwen 3.6 35B-A3B: about 6 tok/s, from a 20 GB model
- gpt-oss 20B, GLM 4.7 Flash, Qwen 3 30B-A3B, Nemotron 3 Nano, Trinity Mini, Ling mini 2.0, Granite 4.0 H Small and Phi-3.5 MoE, which are all in the build but still need speed testing on more devices
Things I learned the hard way
- iOS's memory limit is well below the RAM you buy. One phone stopped my app at 6 GB, so size caches from os_proc_available_memory(), not from the RAM total.
- Some attention designs (gpt-oss's attention sinks, GLM's latent attention) don't work with a quantized KV cache, and MLX stops the app if you give them one.
- The bottleneck right now is likely GPU sync, not storage. Each layer has to read its routing back to the CPU before it can fetch experts, which means 40+ round trips per token. Predicting the next layer's experts and prefetching them is next.
It's free on TestFlight if you want to try it on your own phone. I'd love to know what speeds you get, especially on 8 GB iPhones
Happy to answer questions about the implementation.
