feat(manifold-tools): add llama and mlx local backend support#1763
Closed
roryford wants to merge 6 commits into
Closed
feat(manifold-tools): add llama and mlx local backend support#1763roryford wants to merge 6 commits into
roryford wants to merge 6 commits into
Conversation
Adds BackendDescriptor.swift with CloudProviderDescriptor, LocalModelDescriptor (+ sortOrder), and BackendDescriptorRegistry — a read-mostly OSAllocatedUnfairLock-guarded registry pre-seeded with all built-in provider and model-type metadata. Migrates six switch sites that were pure metadata lookups: - ModelLifecycleCoordinator.backendDisplayName(for:ModelType) - ModelLifecycleCoordinator.modelTypeLogLabel (duplicate of the above) - ModelLifecycleCoordinator.backendDisplayName(for:APIProvider) - CompiledBackends.orderedCloudProviders (now sortOrder-driven) - ModelInfo.backendLabel - ModelSelectionTabView.typeSortRank Dispatch-logic and protocol-behaviour switches (encoding strategy, auth policy, SSE parsing, backend construction) are left as exhaustive enum switches — the compiler must check those. Allowlists BackendDescriptor.swift in TrafficBoundaryAuditTest (same justification as APIProvider.swift: static default URL data). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… vars) - Replace deprecated String(cString:)/String(validatingUTF8:) with withUnsafeBytes + String(decoding:as:)/String(validating:as:) in LlamaModelLoader and LlamaTokenization - Replace deprecated MLX.GPU.set(cacheLimit:) with Memory.cacheLimit = in FluxDiffusionBackend and MLXDiffusionBackend - Add @sendable to completionHandler params in CompositeURLSessionDelegate and NetworkActivityTrackingDelegate - Mark StructuredOutputStrategy/Target @unchecked Sendable (metatypes carry no mutable state) - Restructure DefaultBackends switch to exhaustive per-case #if blocks, eliminating unreachable default - Add @discardableResult to GGUFSignedManifest.requireAcceptedSignature - Silence discarded withLock result in MemoryPressureBroadcaster - Drop unused handle binding in ConversationRuntimeScenario - Remove spurious await in InternalMCPTransport.readOutputLoop - var → let for uuid tuple in UUID+v5 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…Transcriber AVAudioNodeTap and SFSpeechRecognitionTask callbacks both fire on non-main threads. Closures defined inside @mainactor functions inherit @mainactor isolation (SE-0420), so both callbacks were crashing at runtime via dispatch_assert_queue_fail when the Swift 6 executor check ran on the realtime audio thread. Fix: extract both callbacks into nonisolated static factory methods (makeTapBlock / makeRecognitionHandler) so the returned closures are not @MainActor-bound. The latestTranscript write and onUpdate call inside makeRecognitionHandler still hop to @mainactor via Task { @mainactor in }. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add --backend llama and --backend mlx to the manifold-tools CLI. Both backends auto-discover models from ~/Documents/Models/, respect LLAMA_TEST_MODEL/MLX_TEST_MODEL env vars, and accept an explicit --model-path override. Trait-gated behind Llama/MLX compile defines; clear error message when built without the required trait. LlamaBackend teardown uses unloadAndWait() between scenarios. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…nd setup - `hint` for both llama and mlx cases incorrectly used `cli.modelPath ?? ...`, which caused a filesystem path string to be passed as `nameHint` to the discovery functions. Since `pathOverride` already receives `cli.modelPath` and short-circuits before `nameHint` is consulted, this was redundant but semantically wrong — a path is not a name hint. Fixed to compute hint as `model == scenario.backend.model ? nil : model` (a model name only). - Removed the unused `isDirectory: Bool = false` parameter from the private `searchModelsDirectory` helper. The parameter was never read; directory vs. file discrimination is handled entirely by the `isValid` closure. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Owner
Author
|
@copilot resolve the merge conflicts in this pull request |
Resolved. I merged |
Owner
Author
|
Closing — overtaken by the v0.48 packaging split (#1771). Core can no longer host family-backend (llama.cpp / MLX) dependencies without reintroducing the exact dependency cycle the split was built to remove, so The intent (exercising tool scenarios against real local models) re-homes to the companion packages (roryford/manifold-llama, roryford/manifold-mlx), where the family backends now live and can be depended on without a cycle. Note: the ManifoldVoice @mainactor isolation fix that shared this branch already landed on main by content, so nothing else here is lost. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this change?
Adds local backend support to
manifold-toolsforllama(GGUF) andmlx(safetensors snapshots), plus--model-pathfor explicit model selection. It also incorporates latestmainby resolving the merge conflict inPackage.swiftwithout changing the intended feature behavior.Motivation
manifold-toolsneeded parity with local backend workflows so scenarios can be exercised against GGUF and MLX models, not only Ollama/mock paths. This also keeps the PR mergeable after upstream trait/config changes landed onmain.Release Note
llamaandmlxbackend support with model auto-discovery and--model-pathoverride; preserve trait-gated behavior for optional backend builds.Testing
swift build --target manifold-tools --traits Llama,MLX,Ollamaswift build --target manifold-tools --disable-default-traitsMachO/Security/osmodules unavailable), so full local backend compile validation could not complete here.Sabotage evidence (regression-test PRs only)
Checklist
///doc commentsDX checklist
Sources/ManifoldKit/FeatureMatrix.swift(when present).CHANGELOG.mdordocs/.quickStart()path orMinimalExample? If yes, the example still compiles and runs end-to-end.LocalizedErrorwith a user-facingerrorDescription.