feat(ManifoldFoundation): wire AFM 3 image-input API (WWDC 2026)

[roryford]

Context

WWDC 2026 announced that AFM 3 (Apple's third-generation on-device model) is 'natively multimodal' — developer docs confirm: 'Multimodal prompts let you pass images alongside text so your app can reason about visual content, and Vision framework tools like OCR and barcode readers are available for your model to call directly, all on-device.'

FoundationBackend currently advertises supportsVision: false because the pre-WWDC FoundationModels SDK (Xcode 26.4, module 1.4.34) had no image-input surface. This issue tracks wiring the new API.

Ref: #20 (closed — FoundationBackend was the last unchecked checkbox)

Investigation required first (Phase 0)

A post-WWDC Xcode beta may have shipped with the image API. Before implementing, probe the installed SDK:

• Check for an image / attachment case on Transcript.Segment
• Check for a Data- or CGImage-accepting Prompt initializer
• Check for a PromptRepresentable conformance on image types
• Check PromptBuilder for new content cases

If the SDK doesn't yet expose image input, document the gap and revisit when the SDK ships.

Implementation (once SDK confirmed)

• Flip supportsVision: true in FoundationBackend.capabilities
• Map MessagePart.image(data:mimeType:) → SDK image type in generate() (follow ClaudeBackend / OpenAIBackend pattern)
• Handle Private Cloud Compute routing (surface hints if available; otherwise document behavior)
• Remove the 'no image API' note from the FoundationBackend doc comment; describe the wiring
• Integration tests: image-bearing session test, gated with FoundationBackend.probeIsReady()
• Docs: update quickstart with multimodal example

Cross-cutting

• UI: PhotoAttachmentButton / VisionInputButton already ship — no UI changes needed once vision is wired
• Open question: does image routing go on-device or to Private Cloud Compute? Impact on on-device positioning claim.

[roryford]

OQ1 Research findings: Foundation multimodal image API (June 9, 2026)

Image input API surface — CONFIRMED

Type: Attachment — used inside the respond result-builder:

let response = try await session.respond {
    "What animal is this?"
    Attachment(UIImage(named: "photo.jpg")!)
}

Accepted sources — arbitrary developer images, NOT restricted to PhotosPicker:

• UIImage (iOS/visionOS)
• NSImage (macOS)
• CGImage
• CIImage (Core Image)
• CVPixelBuffer
• File URL

UIImage does not directly conform to PromptRepresentable; it must be wrapped in Attachment(UIImage(...)). The Attachment type conforms to PromptRepresentable and works in the result-builder body.

CRITICAL: OS floor is iOS 26.4, NOT iOS 26.0

Vision capabilities are tied to AFM 3 Core Advanced (the new 20B sparse model), which ships as an iOS 26.4 update — not iOS 26.0. The WWDC26 session 241 transcript confirms image support is "from iOS 26.4" alongside token-counting and context-size APIs.

Device floor: iPhone 15 Pro+, iPad Pro (M3+), Apple Silicon Mac. Older devices run the text-only 3B model.

Implementation impact: The guard on is correct for the backend itself, but vision requires a nested guard around the branch and image-path code. Returning on 26.0–26.3 is correct.

PCC routing — on-device guaranteed for SystemLanguageModel

SystemLanguageModel is on-device only for image inputs. No automatic PCC escalation. The class is explicitly instantiated by the developer for cloud inference — it is NOT a routing mode on SystemLanguageModel.

No opt-out needed. The on-device claim holds. ManifoldFoundation's on-device positioning is intact.

Transcript.Segment image case

Not confirmed by name in any public documentation fetched. Recommend inspecting the in the Xcode 26.x SDK directly before assuming the case name. The Attachment type in the prompt builder is the stable public API surface.

Source: WWDC26 session 241 — 'What's new in the Foundation Models framework'

[roryford]

SDK probe result (2026-06-09)

Xcode version: Xcode 26.5 (Build 17F42)
FoundationModels module version: 1.5.2 (up from 1.4.34 noted in FoundationBackend.swift comment)
swiftinterface paths checked:

• /Applications/Xcode.app/.../MacOSX.sdk/System/Library/Frameworks/FoundationModels.framework/Versions/A/Modules/FoundationModels.swiftmodule/arm64e-apple-macos.swiftinterface
• /Applications/Xcode.app/.../iPhoneOS.sdk/System/Library/Frameworks/FoundationModels.framework/Modules/FoundationModels.swiftmodule/arm64e-apple-ios.swiftinterface

Image/Attachment API found: NO

Transcript.Segment still has exactly two cases in both the macOS and iOS interfaces:

public enum Segment : Swift.Sendable, Swift.Identifiable, Swift.Equatable {
  case text(FoundationModels.Transcript.TextSegment)
  case structure(FoundationModels.Transcript.StructuredSegment)
}

No CGImage, UIImage, NSImage, CVPixelBuffer, case image, case attachment, or case media found anywhere in either interface. The only 'attachment' hits in the entire file are logFeedbackAttachment — a telemetry helper, not model input. All 'vision' hits are visionOS availability annotations.

The FoundationBackend.swift doc comment (lines ~59–82) remains accurate despite the module version bump from 1.4.34 → 1.5.2: the public API surface still has no image ingress path.

P0 implementation remains blocked pending Apple releasing a post-WWDC Xcode beta with an updated FoundationModels module that exposes the multimodal surface.

Next step: monitor https://developer.apple.com/news/releases/ for an Xcode 26.x beta with a higher FoundationModels module version (> 1.5.2) and re-run this probe. The swiftinterface to check is at the path listed above.

[roryford]

Phase 0 SDK probe — RESOLVED (Xcode 27.0 beta, Swift 6.4, macOS 27 / iOS 27 SDK), 2026-06-13

Probed the FoundationModels module interface and type-checked throwaway spikes against the beta SDK (compile-only — this machine is macOS 26, so nothing was run; runtime behavior is untested, see caveats).

Image-input surface CONFIRMED — CGImage-based, @available(iOS 27.0, macOS 27.0, *), tvOS unavailable

Note this is 27.0, not the 26.4 the keynote read suggested, and CGImage-based, not UIImage.

• Transcript.Attachment enum → .image(Transcript.ImageAttachment)
• Transcript.ImageAttachment inits from CGImage / CIImage / CVPixelBuffer / imageURL: URL (+ optional CGImagePropertyOrientation)
• Result-builder Attachment<ImageAttachmentContent> (conforms PromptRepresentable/InstructionsRepresentable) with the same image inits and .label(_:)

Both construction forms (Transcript entry and result-builder Prompt { ...; Attachment(cgImage) }) type-check clean against the beta SDK.

Implementation is LOCALIZED to ManifoldFoundation — the contract/message model is already ready

• MessagePart.image(data:mimeType:) already exists in ManifoldContract with Codable round-trip. No contract/message-model gap.
• The cloud vision path (CloudMessageEncoder consuming MessagePart.image via StructuredHistoryReceiver) is the working reference to mirror.
• Gap: FoundationBackend is String-only today (its generate(prompt:) takes a flat string; it conforms only TokenizerVendor). To see image bytes it must adopt StructuredHistoryReceiver, then translate MessagePart.image → Transcript.ImageAttachment(cgImage:) (decode Data via CGImageSourceCreateWithData) in the streaming helpers.

Checklist refinements

• supportsVision must be flipped runtime-conditionally, not statically — the backend lives on macOS 26 but the image API is 27.0: supportsVision: { if #available(macOS 27.0, *) { return true }; return false }(). There's also a runtime query (SystemLanguageModel.default.capabilities.contains(.vision)) that's strictly better than a version guard.
• Adopt StructuredHistoryReceiver (the real work — flag flip alone is insufficient).
• PCC routing for image inputs still needs confirming before keeping isRemote: false for vision turns.

Runtime-untested until a macOS 27 machine exists

1. Whether AFM actually reasons over the attachment vs. ignoring/erroring.
2. PCC-vs-on-device routing for image inputs (affects isRemote).
3. The true availability floor — could be a 26.x point release, not 27.0.
4. CGImage/CVPixelBuffer orientation/colorspace round-trip; vision-specific error mapping.

Gated on the n-1 floor reaching macOS 27/iOS 27 (GA ~Sept 2026) before shipping unguarded. Spikes were throwaway (/tmp/mk-spikes/); repo untouched.