Posted by MehrdadKhnzd 6 days ago
so Claude via FM dies offline while Apple's on-device SystemLanguageModel (the ~3B one) keeps working. It isn't a hybrid really: the framework just has both implement the same LanguageModelSession protocol so "local 3B" and "remote frontier model" become a one-argument swap.
IMHO what's worth internalising is that the two share an API but nothing else: the on-device path runs on Apple's Neural Engine and costs battery (you can watch ANE power ramp while it works) while the cloud path costs API credits/tokens and does zero local compute. Same code, opposite cost model.
Apple's Foundation Models framework (shipping in iOS 27 / macOS 27 this fall) is the standard Swift API for on-device AI — the same API Apple uses for their own small model. This package makes Claude plug into that same API as a drop-in swap.
// Apple's on-device model
let session = LanguageModelSession(model: SystemLanguageModel.default)
// Claude — same API, just different model constructor
let session = LanguageModelSession(model: ClaudeLanguageModel(name: .sonnet4_6, auth: auth))
You don't call the Anthropic API directly. Apple's framework handles streaming, tool calling, and structured output (@Generable) — you just get Claude's capability through it.
Enough is enough. I’m seriously evaluating open models this week.