Add runanywhere-python: Python SDK (pybind11 over the rac_* C ABI)

[AmanSwar]

Summary

Adds sdk/runanywhere-python/ — a first-class, installable Python SDK for the RunAnywhere on-device AI engine, wrapping the rac_* C ABI via pybind11 (the desktop sibling of the Electron SDK: N-API → pybind11). Same capabilities as the other SDKs — LLM / VLM / embeddings / STT / TTS / VAD + secure store + a pure-Python catalog & downloader — with a Pythonic, snake_case, sync + async API. Business logic is a faithful port of the Electron SDK; only the syntax is Pythonic.

On top of the library, it ships a built-in OpenAI-compatible server (runanywhere serve) so the SDK doubles as a drop-in local OpenAI endpoint.

+12k / 63 files. CPU-only by default (CUDA + Hexagon/QHexRT are documented build-flag opt-ins).

Public API

from runanywhere import RunAnywhere

with RunAnywhere() as ra:
    llm = ra.load_llm("qwen2.5-0.5b")                 # id | local path | HF repo | URL; auto-downloads
    for tok in llm.generate("Capital of France?"):     # sync stream
        print(tok, end="", flush=True)
    print(llm.generate_text("Capital of France? One word."))   # -> "Paris"
    # async twins: `async for tok in llm.agenerate(...)`, `await llm.agenerate_text(...)`

    emb = ra.load_embedder("minilm").embed("on-device AI")     # np.float32
    txt = ra.load_stt("whisper-tiny").transcribe(pcm16)
    wav = ra.load_tts("piper-lessac").synthesize("hello")
    cap = ra.load_vlm("smolvlm-256m").caption_text("cat.jpg", "What is this?")

Instantiable client + context manager, process-global ref-counted native lifecycle. Higher-level features ported from Electron: Chat (multi-turn), VoiceAgent (STT→LLM→TTS), structured output + JSON-Schema→GBNF grammar, tool calling, streaming metrics, event bus, audio DSP helpers.

Built-in OpenAI-compatible server

pip install "runanywhere[server]" then runanywhere serve → point any OpenAI client (the openai lib, LangChain, LlamaIndex, curl) at http://localhost:8000/v1. Kept behind an optional extra so the base install stays numpy + stdlib. Modelled on how ollama / vllm / llama-server / mlx_lm.server ship theirs, and goes beyond the base LLM servers using our audio/vision/structured capabilities.

Endpoints

OpenAI endpoint	Backed by
POST /v1/chat/completions (stream + non-stream)	LLM / VLM (vision)
POST /v1/completions	LLM
POST /v1/embeddings (float + base64)	Embedder
POST /v1/audio/transcriptions (Whisper API)	STT
POST /v1/audio/speech	TTS
GET /v1/models · GET /v1/models/{id} · GET /health · GET /	catalog

chat/completions also supports vision (OpenAI image parts → VLM), structured output (response_format json_schema → grammar), and tool calling (tool_choice required/named → forced tool_calls; auto → model decides). CLI: runanywhere serve / runanywhere models / --version.

Architecture & packaging

• native/module.cpp — pybind11 _core, a port of the Electron addon; GIL-correct streaming (blocking native call on a worker thread, tokens over a bounded queue). Win32 (DPAPI) + POSIX platform adapters.
• Pure/native split — errors/events/options/results/grammar/structured/audio/catalog/download import no native code, so the bulk of the suite is hermetic. import runanywhere never loads _core or fastapi (lazy-load invariant, test-guarded).
• Wheels — scikit-build-core + pybind11; delvewheel (Windows) / auditwheel (Linux manylinux_2_35) / delocate (macOS) vendor the onnxruntime/sherpa sidecars. [project.scripts] runanywhere console entry point.

Testing & validation

• 387 hermetic tests / 6 gated-skips — pure-Python + a fake _core/model layer + FastAPI TestClient; run in CI on Windows and Linux (new python-windows + python-linux jobs: build → repair → validate release boundary → install → pytest).
• Real-model validation (separate from CI): generate → "Paris" on Windows; on an EC2 Linux box the server passed 18/18 end-to-end checks — every endpoint incl. tool-calling, vision, TTS→STT round-trip, and the real openai Python client driving it.

Production hardening (pre-merge self-review)

Three independent review passes over the serving system; every genuine finding fixed + regression-tested:

Security / correctness / concurrency fixes

• SSRF — server-side image_url fetch is opt-in (--allow-image-urls, off by default → the class is gone by default); when enabled, resolved-IP validation (incl. IPv4-mapped-IPv6) + redirects refused.
• DoS — data-URI capped before base64 decode; request-body 413 enforced on actual bytes (closes the chunked-transfer bypass).
• No info leak — catch-all + validation exception handlers return OpenAI-shaped bodies (generic 500, never a traceback/path); bytes-safe api-key compare.
• Model allowlist — clients load catalog ids only by default (--allow-arbitrary-models to opt into paths/HF repos).
• grammar.py 500-crash fixed (a boolean sub-schema; restores Electron parity).
• Concurrency — per-model load locks (a slow cold-load no longer serializes other models); lifespan start() inside try; a shielded stream-teardown join so a client disconnect can't leak the native generation guard.
• Input validation: empty messages/input → 400, out-of-range sampling params → 400.

Reviewer notes

• Behavioral source of truth is the Electron SDK; grammar/tool/download logic is a line-for-line port (tests mirror Electron's cases).
• Style divergence, on purpose: the FastAPI/pydantic files use Optional/Union (not the SDK's usual X | None) because those annotations are evaluated at runtime and PEP-604 string unions aren't safe on the declared floor (Python 3.9); rationale is in schemas.py.
• Deferred (documented, non-blocking): python-macos CI + macOS Keychain secure store (posix adapter branch is written; validate on a Mac); keep-alive/idle-unload; full DNS-rebind IP-pinning (opt-in fetch + redirect-refusal mitigates).
• Root CMake gains only option(RAC_BUILD_PYTHON_MODULE …) + a guarded add_subdirectory — default OFF, so no other build changes.

Commits are one-file-each for easy review.

Summary by CodeRabbit

• New Features
  • Introduced an Alpha Python SDK for on-device LLM/VLM, embeddings, speech (STT/TTS), VAD, structured output/tool calling, chat, voice agents, and RAG (with model download and secure storage).
  • Added token streaming with metrics and lifecycle/model events, plus “thinking”/answer separation.
  • Added an opt-in OpenAI-compatible local server and a Python CLI.
• Documentation
  • Updated README with Python installation, examples, and SDK/architecture details.
• Tests / Quality
  • Expanded hermetic CI to build/repair/test wheels across platforms, plus stronger published-artifact validation and broader test coverage.

[coderabbitai]

Actionable comments posted: 3

🧹 Nitpick comments (1)

sdk/runanywhere-python/runanywhere/rag.py (1)

276-304: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Missing async twin for blocking ingest/ingest_many.

rag_ingest is a blocking native call that computes embeddings (not token-streamed), yet only query/query_stream expose async twins. Add aingest/aingest_many via loop.run_in_executor(None, ...) to keep the async surface complete and avoid blocking the event loop during ingestion.

As per coding guidelines: "Blocking native calls that are not token-streamed, including transcription and synthesis, must provide an async twin using loop.run_in_executor(None, ...)."

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@sdk/runanywhere-python/runanywhere/rag.py` around lines 276 - 304, Add async
twins named aingest and aingest_many alongside Rag.ingest and Rag.ingest_many,
delegating each blocking operation through the current event loop’s
run_in_executor(None, ...) so the event loop is not blocked. Preserve the
existing ingestion arguments, ordering, and RagStatistics results, and have
aingest_many retain the empty-input behavior of ingest_many.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@sdk/runanywhere-python/native/module.cpp`:
- Around line 648-657: Add per-session lifetime leasing around the native handle
used by rag_query_stream and the operations at the referenced locations: reject
new leases after close or shutdown begins, increment active-operation counts
before releasing the map mutex, and decrement them on every exit path. Update
RagSession.close() and shutdown() to mark sessions closing and wait for active
leases to drain before destroying native handles, preventing concurrent
use-after-free.

In `@sdk/runanywhere-python/pyproject.toml`:
- Around line 32-39: Update the rag optional dependency in pyproject.toml from
protobuf>=5.29 to protobuf>=5.29,<7, preventing installation of protobuf 7.x
while preserving compatibility with supported 5.x and 6.x runtimes.

In `@sdk/runanywhere-python/runanywhere/_proto/rac_options_pb2.py`:
- Around line 1-6: Update the protobuf generation configuration used by
rac_options_pb2.py and rag_pb2.py so regenerated modules begin with a one-line
docstring followed by from __future__ import annotations; regenerate both
bindings rather than editing them manually. Also add from __future__ import
annotations after the docstring in
sdk/runanywhere-python/runanywhere/_proto/__init__.py (line 1). Ensure all three
modules retain the required preamble order.

---

Nitpick comments:
In `@sdk/runanywhere-python/runanywhere/rag.py`:
- Around line 276-304: Add async twins named aingest and aingest_many alongside
Rag.ingest and Rag.ingest_many, delegating each blocking operation through the
current event loop’s run_in_executor(None, ...) so the event loop is not
blocked. Preserve the existing ingestion arguments, ordering, and RagStatistics
results, and have aingest_many retain the empty-input behavior of ingest_many.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 88e549a8-17f3-4c27-aa94-8cf9880ed0be

📥 Commits

Reviewing files that changed from the base of the PR and between 9f19402 and 85760e5.

📒 Files selected for processing (25)

• .github/workflows/pr-build.yml
• idl/codegen/generate_python.sh
• sdk/runanywhere-python/PACKAGING.md
• sdk/runanywhere-python/README.md
• sdk/runanywhere-python/native/CMakeLists.txt
• sdk/runanywhere-python/native/module.cpp
• sdk/runanywhere-python/pyproject.toml
• sdk/runanywhere-python/runanywhere/__init__.py
• sdk/runanywhere-python/runanywhere/__main__.py
• sdk/runanywhere-python/runanywhere/_proto/__init__.py
• sdk/runanywhere-python/runanywhere/_proto/rac_options_pb2.py
• sdk/runanywhere-python/runanywhere/_proto/rag_pb2.py
• sdk/runanywhere-python/runanywhere/_thinking_splitter.py
• sdk/runanywhere-python/runanywhere/cli/__init__.py
• sdk/runanywhere-python/runanywhere/cli/handlers.py
• sdk/runanywhere-python/runanywhere/cli/output.py
• sdk/runanywhere-python/runanywhere/client.py
• sdk/runanywhere-python/runanywhere/options.py
• sdk/runanywhere-python/runanywhere/rag.py
• sdk/runanywhere-python/runanywhere/results.py
• sdk/runanywhere-python/runanywhere/stream_metrics.py
• sdk/runanywhere-python/scripts/validate_public_packages.py
• sdk/runanywhere-python/tests/test_cli.py
• sdk/runanywhere-python/tests/test_rag.py
• sdk/runanywhere-python/tests/test_thinking.py

🚧 Files skipped from review as they are similar to previous changes (10)

• sdk/runanywhere-python/tests/test_thinking.py
• sdk/runanywhere-python/runanywhere/options.py
• sdk/runanywhere-python/runanywhere/init.py
• sdk/runanywhere-python/runanywhere/main.py
• sdk/runanywhere-python/runanywhere/stream_metrics.py
• sdk/runanywhere-python/runanywhere/results.py
• sdk/runanywhere-python/native/CMakeLists.txt
• sdk/runanywhere-python/tests/test_cli.py
• sdk/runanywhere-python/scripts/validate_public_packages.py
• .github/workflows/pr-build.yml