feat(litert): add LiteRT-LM as second on-device inference engine

[dishit-wednesday]

Add LiteRT as a second on-device inference engine

Adds LiteRT (Google's on-device inference runtime) as a peer to the existing llama.cpp engine. Android-only at ship. The JS layer is built platform-agnostic so the iOS Swift LiteRT SDK can drop in when Google releases it with no code changes on this side.

Engine is decided by file extension — .gguf runs on llama.cpp as before, .litertlm runs on LiteRT. No engine toggle in the UI; it follows the model.

---

Why a second engine

LiteRT runs Gemma 4, Gemma 3n, and other LiteRT-LM packaged models with GPU acceleration paths that llama.cpp does not have on Android. CPU remains as the universal fallback.

Both engines stay in the app. Users keep their existing .gguf models and existing chats. Adding a .litertlm model is purely additive.

---

Architecture

Follows the same shape as the existing llama.cpp / ONNX split — a peer service, not a layer on top of llmService.

src/services/
  llm.ts            ──▶  llama.cpp via llama.rn (existing)
  litert.ts         ──▶  LiteRT via LiteRTModule.kt (new)
  engines.ts        ──▶  getActiveEngineService() router
  liteRTCompaction  ──▶  context summarization for LiteRT
activeModelService/loaders.ts

doLoadTextModel  ──┬─▶ doLoadLiteRTModel  (model.engine === 'litert')

└─▶ llama path         (model.engine === 'llama')
generationServiceHelpers.ts

generateResponseImpl

├─▶ runLiteRTResponseImpl

└─▶ llama path
android/.../litert/LiteRTModule.kt

ReactContextBaseJavaModule wrapping

com.google.ai.edge.litertlm:litertlm-android:0.11.0

Type model

DownloadedModel is a discriminated union over engine:

type DownloadedModel = LlamaDownloadedModel | LiteRTDownloadedModel;
interface LlamaDownloadedModel  { engine: 'llama';  mmProjPath?: string; ... }

interface LiteRTDownloadedModel { engine: 'litert'; liteRTVision: boolean; ... }

Engine-specific fields live on the engine-specific type. Consumers narrow with model.engine === 'llama' and let the compiler enforce correctness. isLlamaModel / isLiteRTModel type guards are exported for tests and selectors.

One downloadedModels collection, one activeModelId. The image-model pattern of separate collections is not used here because only one text model is loaded at a time and most call sites are engine-agnostic.

Settings model

LiteRT-specific settings are flat fields prefixed with liteRT:

Setting	Type	Default
liteRTBackend	'gpu' | 'cpu'	'gpu'
liteRTTemperature	number	0.7
liteRTTopP	number	0.9
liteRTMaxTokens	number	4096

---

Test plan

• npm run lint && npx tsc --noEmit && npm test clean
• Android cold start — llama model loads and generates
• Android cold start — LiteRT model loads and generates
• Switch from llama to LiteRT mid-session, switch back
• LiteRT GPU path: load Gemma 4 E2B, verify ~38 chars/sec decode
• LiteRT CPU fallback when GPU init refused
• Stop generation mid-stream, send another message — no FAILED_PRECONDITION
• Long conversation — watch auto-compact trigger near 65%
• Vision: Gemma 3n with image attached, model responds about the image
• Vision blocked: text-only model with image attached, clean error
• Tool calling: web_search invocation completes and returns to model
• Settings: change LiteRT temperature, verify next response reflects it
• Settings: change LiteRT max tokens, reload banner appears, reload works
• Memory: load LiteRT + image gen, watch memory check
• CI build succeeds from a fresh checkout (gesture-handler patch applies)
• iOS build succeeds, no LiteRT UI visible on iOS device

# Add LiteRT as a second on-device inference engine

Adds [LiteRT](https://ai.google.dev/edge/litert) (Google's on-device inference runtime) as a peer to the existing llama.cpp engine. Android-only at ship. The JS layer is built platform-agnostic so the iOS Swift LiteRT SDK can drop in when Google releases it with no code changes on this side.

Engine is decided by file extension — .gguf runs on llama.cpp as before, .litertlm runs on LiteRT. No engine toggle in the UI; it follows the model.

---

Why a second engine

LiteRT runs Gemma 4, Gemma 3n, and other LiteRT-LM packaged models with GPU acceleration paths that llama.cpp does not have on Android. CPU remains as the universal fallback.

Both engines stay in the app. Users keep their existing .gguf models and existing chats. Adding a .litertlm model is purely additive.

---

Architecture

Follows the same shape as the existing llama.cpp / ONNX split — a peer service, not a layer on top of llmService.

src/services/
  llm.ts            ──▶  llama.cpp via llama.rn (existing)
  litert.ts         ──▶  LiteRT via LiteRTModule.kt (new)
  engines.ts        ──▶  getActiveEngineService() router
  liteRTCompaction  ──▶  context summarization for LiteRT

activeModelService/loaders.ts
  doLoadTextModel  ──┬─▶ doLoadLiteRTModel  (model.engine === 'litert')
                     └─▶ llama path         (model.engine === 'llama')

generationServiceHelpers.ts
  generateResponseImpl
     ├─▶ runLiteRTResponseImpl
     └─▶ llama path

android/.../litert/LiteRTModule.kt
  ReactContextBaseJavaModule wrapping
  com.google.ai.edge.litertlm:litertlm-android:0.11.0

Type model

DownloadedModel is a discriminated union over engine:

type DownloadedModel = LlamaDownloadedModel | LiteRTDownloadedModel;

interface LlamaDownloadedModel  { engine: 'llama';  mmProjPath?: string; ... }
interface LiteRTDownloadedModel { engine: 'litert'; liteRTVision: boolean; ... }

Engine-specific fields live on the engine-specific type. Consumers narrow with model.engine === 'llama' and let the compiler enforce correctness. isLlamaModel / isLiteRTModel type guards are exported for tests and selectors.

One downloadedModels collection, one activeModelId. The image-model pattern of separate collections is not used here because only one text model is loaded at a time and most call sites are engine-agnostic.

Settings model

LiteRT-specific settings are flat fields prefixed with liteRT:

Setting	Type	Default
liteRTBackend	'gpu' | 'cpu'	'gpu'
liteRTTemperature	number	0.7
liteRTTopP	number	0.9
liteRTMaxTokens	number	4096

liteRTMaxTokens maps to LiteRT's EngineConfig.maxNumTokens — one budget covering system prompt, history, input, and output combined. Existing llama settings are untouched. Switching engines does not cross-contaminate preferences.

---

Core features

Text generation

liteRTService.sendMessage() streams tokens through four event types from the native module:

Native event	JS callback
litert_token	onToken — response stream
litert_thinking	onReasoning — extended-reasoning stream
litert_complete	onComplete with BenchmarkInfo stats
litert_error	onError
litert_tool_call	tool invocation request

The native Conversation object holds turn history internally. JS only sends the current user turn after the conversation is set up, avoiding re-prefilling the entire chat on every message.

prepareConversation only triggers a native resetConversation when activeConversationId, activeSystemPrompt, or activeToolsJson changes. Otherwise it reuses the live native session.

After load on GPU, the engine runs a one-token warmup against an empty prompt to prime the shader/kernel cache and avoid first-compile latency on the first real prompt.

Thinking

Gemma 4 models need a <|think|> token prepended to the system prompt to activate extended reasoning. applyGemma4ThinkToken handles this for both engines:

applyGemma4ThinkToken(prompt, isRemote, { isLiteRT, thinkingEnabled })

Thinking tokens stream into a separate channel (litert_thinking native / onReasoning JS). The chat UI renders them in a collapsible "Thought process" section above the response. Disabling the thinking toggle removes the prefix and the model produces direct answers only.

Context compaction

LiteRT models load with a fixed maxNumTokens budget. The service tracks cumulative tokens and auto-compacts at 65% of the budget:

Situation	Strategy
Active session loaded	Ask the model to summarize itself in 3-5 sentences, then reset with [summary, recent turns]
First load, no session	Slice the oldest history (cannot summarize what is not loaded)

Summarization runs inside the existing KV cache while ~35% headroom remains, then a single reset rebuilds the conversation with the compacted history. Recent-turn selection keeps the last 40% of context by char estimate, with a minimum of two turns. A 20-second timeout falls back to slice-only if summarization stalls.

Implementation: src/services/liteRTCompaction.ts — pure function, takes history + maxTokens + cumulativeTokens, returns the reset call to make.

Tools

Tool calling is handled natively by the LiteRT SDK. Tools are passed in as JSON at conversation reset time. When the model emits a tool call, the SDK fires litert_tool_call to JS with the tool name and arguments. JS executes the tool and calls respondToToolCall(id, result) back to the native side. The SDK feeds the result into the conversation and continues generation internally — JS never sees the second prefill.

Three text-based parsing fallbacks exist for models that do not use the SDK's tool format (Gemma 4 emits a non-standard <|tool_call> syntax). These parse the response text after generation and execute tools through the same path.

Vision

LiteRT supports multimodal models (Gemma 3n E2B / E4B variants) with a GPU-accelerated vision encoder. The liteRTVision boolean on the model record gates this:

• At import time, a dialog asks "Text Only / Vision" for any .litertlm file
• If liteRTVision === true, the engine loads with visionBackend = Backend.GPU()
• The image-attach button is shown only when the active model supports vision
• Attaching an image to a non-vision LiteRT model is blocked with a clear error rather than silently dropping the image

The dialog is needed because LiteRT does not expose introspection for vision support on a loaded model.

Backend selection and fallback

Backend	Notes
GPU (OpenCL via Adreno)	Works on most modern Android devices
CPU	Universal fallback

The native module does a two-tier fallback at load time with per-tier timeouts (GPU 20s, CPU 15s). The actually-loaded backend is reported back via getActiveBackend() so the UI can show "Requested GPU, running on CPU" if fallback occurs.

---

Model management

Import flow

Models screen → Import → pick a .litertlm file
             ↓
Vision support dialog (only for .litertlm)
             ↓
File copy to app documents
             ↓
DownloadedModel record created with engine: 'litert'

No LiteRT model catalog or HuggingFace download flow in this PR — users sideload .litertlm files. Curated downloads will come in a follow-up.

Recommended models for sideload:

• google/gemma-3n-E2B-it-litert-lm — vision-capable, smaller
• litert-community/gemma-4-E2B-it-litert-lm — text-only, larger

Settings UI

ModelSettingsScreen and GenerationSettingsModal render one of two sibling components based on the active model:

isLiteRT ? <LiteRTTextSettings /> : <LlamaTextSettings />

LiteRTTextSettings shows: Temperature, Max Tokens, Top P (advanced), Acceleration (GPU/CPU), Show Generation Details, Thinking toggle.

LlamaTextSettings shows: Temperature, Max Tokens, Context Length, Top P, Repeat Penalty, CPU Threads, Batch Size, Backend, Flash Attention, KV Cache Type, Model Loading Strategy.

LiteRT hides controls that do not map to its SDK (no flash attention, no GPU layer count, no manual thread count).

Settings that bake into the engine at load time (backend, maxTokens) trigger a "settings changed, reload required" banner. Settings the engine reads per-generation (temperature, top-P) take effect on the next reset.

Memory budget

The memory check at model load now sums RAM across both engines plus any loaded image model. Loading a LiteRT model while llama is resident unloads llama first, and vice versa. Loading a LiteRT model alongside an image model checks both budgets to prevent OOM.

---

Build infrastructure

Kotlin 2.2.0

The LiteRT SDK requires Kotlin 2.x. kotlinVersion is bumped to 2.2.0 in android/build.gradle along with the Kotlin Gradle plugin classpath. This unlocks the K2 compiler — most modules compile faster and produce slightly smaller bytecode.

react-native-gesture-handler patch

Kotlin 2.2's K2 compiler tightened smart-cast inference inside lambdas when the receiver is a property. react-native-gesture-handler@2.30.0 relies on pre-K2 behavior in findRootHelperForViewAncestor:

// Before (fails K2 — smart cast doesn't carry across && for property receivers)
it.rootView is ReactRootView && it.rootView.rootViewTag == rootViewTag

// After (patches/react-native-gesture-handler+2.30.0.patch)
it.rootView is ReactRootView && (it.rootView as ReactRootView).getRootViewTag() == rootViewTag

Same JVM bytecode, same runtime behavior. Applied via patch-package in postinstall. Will self-remove when gesture-handler ships a K2-compatible version upstream.

Native module

android/app/src/main/java/ai/offgridmobile/litert/LiteRTModule.kt registers as a standard ReactContextBaseJavaModule:

Method	What it does
loadModel	Two-tier backend fallback init
resetConversation	Close current, create new with system + history + tools
sendMessage	Send current turn, stream tokens via events
respondToToolCall	Feed tool result back to the SDK
stopGeneration	Cancel current generation, null active session
unloadModel	Close Conversation, then Engine, in that order
getMemoryInfo	Native RAM stats for DeviceStatsChip

OpenCL opted in via AndroidManifest:

<uses-native-library android:name="libOpenCL.so" android:required="false" />

Platform readiness

liteRTService detects availability by native module presence (!!NativeModules.LiteRTModule), not Platform.OS === 'android'. When the iOS Swift LiteRT SDK is released and registered under the same JS name, the JS service starts working on iOS with no code changes here.

---

Behavior summary

Action	Result
Tap a .gguf model	Loads on llama.cpp (existing)
Tap a .litertlm model	Loads on LiteRT, settings UI swaps
Change LiteRT backend while loaded	Reload banner appears
Change llama backend while LiteRT loaded	No banner (independent settings)
Stop mid-generation	Cancels, next turn safely restarts
Send an image to a non-vision LiteRT model	Clean error, no silent drop
Conversation context hits 65% of budget	Auto-summarizes and resets
App backgrounded with LiteRT loaded, then resumed	State syncs correctly
iOS user opens the app	Llama path unchanged, no LiteRT UI

---

Test plan

• npm run lint && npx tsc --noEmit && npm test clean
• Android cold start — llama model loads and generates
• Android cold start — LiteRT model loads and generates
• Switch from llama to LiteRT mid-session, switch back
• LiteRT GPU path: load Gemma 4 E2B, verify ~38 chars/sec decode
• LiteRT CPU fallback when GPU init refused
• Stop generation mid-stream, send another message — no FAILED_PRECONDITION
• Long conversation — watch auto-compact trigger near 65%
• Vision: Gemma 3n with image attached, model responds about the image
• Vision blocked: text-only model with image attached, clean error
• Tool calling: web_search invocation completes and returns to model
• Settings: change LiteRT temperature, verify next response reflects it
• Settings: change LiteRT max tokens, reload banner appears, reload works
• Memory: load LiteRT + image gen, watch memory check
• CI build succeeds from a fresh checkout (gesture-handler patch applies)
• iOS build succeeds, no LiteRT UI visible on iOS device

[greptile-apps]

Your free trial has ended. If you'd like to continue receiving code reviews, you can add a payment method here.

[gemini-code-assist]

Code Review

This pull request introduces LiteRT-LM on-device inference support for Android, featuring a new native module, JS bridge, and debug screen. It also migrates the build system to KSP and updates Kotlin and Gradle versions. Review feedback points out a hardcoded local Java path in gradle.properties that breaks CI, a resource leak in the native engine initialization fallback logic, and several instances where vision support is incorrectly hardcoded to true for all LiteRT models instead of respecting the specific model configuration.

[gemini-code-assist]

The org.gradle.java.home property is hardcoded to a local path on your machine. This will break the build for other developers and on CI environments. As noted in docs/LITERT_TODO.md, this should be removed before merging.

[gemini-code-assist]

There is a resource leak in the fallback chain. If eng.initialize() fails or times out, the local eng instance is never closed. The catch block calls engine?.close(), but engine (the class property) is still null at that point because the assignment at line 112 is never reached. You should declare eng outside the try block and ensure it is closed on failure.

        for (backend in chain) {
            val name = backendName(backend)
            Log.i(TAG, "initializeWithFallback — trying $name vision=$visionEnabled")
            var eng: Engine? = null
            try {
                val cfg = EngineConfig(
                    modelPath = modelPath,
                    backend = backend,
                    cacheDir = null,
                    visionBackend = if (visionEnabled) Backend.GPU() else null,
                )
                eng = Engine(cfg)
                val timeoutMs = when (backend) {
                    is Backend.NPU -> NPU_TIMEOUT_MS
                    is Backend.GPU -> GPU_TIMEOUT_MS
                    else           -> CPU_TIMEOUT_MS
                }
                withTimeout(timeoutMs) {
                    eng.initialize()
                }
                engine = eng
                Log.i(TAG, "initializeWithFallback — $name succeeded")
                return backend
            } catch (e: Exception) {
                Log.w(TAG, "initializeWithFallback — $name failed: ${e.message}")
                eng?.close()
                lastError = e
                if (backend == chain.last()) break
                Log.i(TAG, "initializeWithFallback — falling back to next tier")
            }
        }

References

1. When a process with an iteration limit exceeds that limit, treat it as a failure and invoke the same fallback logic used for other exceptions.
2. Avoid code duplication by refactoring common operations into a single call, especially when they appear in multiple exit paths of a function.

[gemini-code-assist]

For LiteRT models, supportsVision is hardcoded to true. This will enable vision UI elements (like the image attachment button) even for text-only LiteRT models. It should instead respect the liteRTVision flag stored in the model record.

Suggested change

	deps.setSupportsVision(activeModel.engine === 'litert' ? true : (multimodalSupport?.vision || false));
	deps.setSupportsVision(activeModel.engine === 'litert' ? !!activeModel.liteRTVision : (multimodalSupport?.vision || false));

References

1. Vision-language models should be specifically categorized with type 'vision' to ensure correct UI behavior.

[gemini-code-assist]

Similar to the previous comment, supportsVision should respect the liteRTVision flag from the model record for LiteRT models.

Suggested change

	deps.setSupportsVision(activeModel.engine === 'litert' ? true : (multimodalSupport?.vision || false));
	deps.setSupportsVision(activeModel.engine === 'litert' ? !!activeModel.liteRTVision : (multimodalSupport?.vision || false));

References

1. Vision-language models should be specifically categorized with type 'vision' to ensure correct UI behavior.

[gemini-code-assist]

Hardcoding supportsVision to true here will cause the UI to incorrectly show vision capabilities for text-only LiteRT models. Please use the liteRTVision property from the activeModel.

Suggested change

	deps.setSupportsVision(true);
	return;
	}
	dbg('log', `[LiteRT] ensureModelLoaded — model=${activeModel.name}, triggering load`);
	deps.setSupportsVision(true);
	deps.setSupportsVision(!!activeModel.liteRTVision);
	return;
	}
	dbg('log', `[LiteRT] ensureModelLoaded — model=${activeModel.name}, triggering load`);
	deps.setSupportsVision(!!activeModel.liteRTVision);

References

1. Vision-language models should be specifically categorized with type 'vision' to ensure correct UI behavior.

[gemini-code-assist]

Ensure supportsVision is set based on the model's actual capabilities rather than hardcoding it to true for all LiteRT models.

Suggested change

	deps.setSupportsVision(model.engine === 'litert' ? true : (multimodalSupport?.vision || false));
	deps.setSupportsVision(model.engine === 'litert' ? !!model.liteRTVision : (multimodalSupport?.vision || false));

References

1. Vision-language models should be specifically categorized with type 'vision' to ensure correct UI behavior.

[gemini-code-assist]

The supportsVision state should be derived from the liteRTVision flag in the model record to avoid misleading the user with vision UI on text-only models.

Suggested change

	setSupportsVision(true);
	setSupportsVision(!!activeModel.liteRTVision);

References

1. Vision-language models should be specifically categorized with type 'vision' to ensure correct UI behavior.

[sonarqubecloud]

Quality Gate failed

Failed conditions
1 Security Hotspot

See analysis details on SonarQube Cloud