You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Implement the structured llama.cpp/Skippy and Rust event pipeline described in the event-system specification.
The completed system should make normal runtime lifecycle, availability, progress, warning, resource, and serving state independent of parsing native llama.cpp log text. It must preserve the existing ownership boundary:
Native code emits bounded facts.
Rust validates, copies, correlates, buffers, reduces, and presents those facts.
Native callback observations never override authoritative ABI return values.
This is a draft implementation issue. Exact Rust module names, queue implementation, capacities, event enum names, and management API shapes remain implementation choices as long as the behavioral requirements below are preserved.
Current State
The current Skippy runtime event ABI exposes five model-open observations:
model open started;
model open progress;
backend device selected;
model open finished; and
handled model-open failure.
The current callback boundary already has useful properties: a C-compatible ABI, explicit event version and struct size, fixed-width fields, copied borrowed data, no callback after the model-open call returns, and authoritative success/failure through the normal return path.
The current limitations are:
callback consumer work runs inline on the native emitter thread;
one callback is directly coupled to one consumer;
there is no bounded event ingress, fan-out, replay, or reducer;
progress, lifecycle, and diagnostic observations have no separate backpressure policies;
structured native fields are flattened before other consumers can use them;
unknown future event kinds are discarded by the host projection;
model-open callbacks are optional and not used by every Skippy caller; and
session, prefill, decode, generation, KV, warning, unload, resource, and availability state still have coverage gaps and may depend on upstream log text.
Rust-owned telemetry is useful prior art for timing and lifecycle observations, but it is not the event system: telemetry may be disabled, is an optional export consumer, and does not provide authoritative local state.
validate pointers, ABI versions, readable struct sizes, and integer discriminants;
copy borrowed data before the callback returns;
attach operation-scoped Rust correlation when available;
submit owned facts through nonblocking ingress; and
return promptly.
They must not:
perform policy or readiness work;
block on queue capacity;
acquire application-wide locks;
perform formatting or I/O;
export telemetry;
call arbitrary application subscribers directly;
call back into Skippy; or
unwind through FFI.
The event system must keep three layers separate:
Native facts: direct observations from the native ABI.
Runtime domain events: typed Rust events with logical identities and backend-neutral meaning.
Consumer projections: reducer transitions, CLI/TUI/JSON output, management API records, telemetry, and optional diagnostic recordings.
Consumer projections must never be fed back as native facts.
Event Model
Every runtime domain event should support the relevant subset of:
schema version;
process-unique event ID;
event category and kind;
producer source and severity;
wall-clock and process-monotonic timestamps;
native timestamp and sequence when available;
operation, model, topology, stage, session, request, and device identities;
previous/current state for state transitions;
progress current, total, and unit;
outcome and stable machine-readable reason code;
duration;
bounded numeric summaries; and
a bounded human-readable summary.
Fields that do not apply must be absent rather than represented by ambiguous sentinel values.
Native pointer addresses must never become durable IDs. Correlation IDs must not be derived from prompt or completion contents. Unknown event kinds and newer append-only fields must be tolerated safely.
Terminal and warning events need stable reason codes covering cases such as invalid configuration, unsupported capability, missing artifact, I/O failure, model-load failure, backend failure, device unavailable, resource allocation failure, out of memory, context exhaustion, stage unavailable, timeout, cancellation, process crash, incompatible ABI, internal failure, and unknown failure.
Required Event Families
Native runtime lifecycle
Resolution, loading, ABI/feature compatibility, initialization, stopping, stopped, and crash events.
Model acquisition and preparation
Model queued, resolution, download progress, materialization/package preparation, cache reuse, completion, failure, and cancellation.
Model loading
Load requested/started, coarse phase changes, progress, backend/device selection, memory allocation or pressure, native completion, failure, and cancellation.
Normal events must remain coarse-grained. They must not emit individual tensor, tensor-name, layer, kernel, or graph-node events.
Model availability and readiness
Native model loaded, Rust backend initialization, available, degraded, unavailable, recovery, and capacity changes.
Native model-load completion must not imply serving availability. Rust readiness is emitted only after required serving surfaces and dependent stages are usable.
Model unloading
Unload requested/started/completed/failed, session draining, and forced unload, including active/draining session counts and cleanup reason where available.
Stage and topology lifecycle
Stage starting/loading/ready/degraded/unavailable/stopping/stopped/failed; topology assembling/ready/degraded/unavailable; and upstream/downstream connection established/lost/recovered.
These events must project into backend-neutral model availability.
Session lifecycle
Session requested/created/active/idle/reset/trimmed/restored/draining/closed/failed/abandoned, with bounded identity, lane, token-count, duration, and reason data.
Request and admission lifecycle
Request received/queued/admitted/rejected/started/completed/cancelled/timed out/failed, with request identity, model, serving mode, queue depth/wait, duration, outcome, reason, and attempt count.
No prompt, completion, tool argument, request body, or media content may be included.
Prompt processing and prefill
Prompt processing, tokenization, prefill start/progress/completion/cancellation/failure, media prefill, and prompt-cache restore outcomes.
Production prefill progress should be aggregated rather than emitted per chunk.
Decode and generation
Generation started, first token, aggregated progress, completed, cancelled, timed out, failed, and stop condition reached.
Normal generation events must expose bounded summaries such as token counts, time to first token, elapsed duration, coarse throughput, stop reason, speculative-mode status, and batching mode. Per-token text, logits, and token IDs belong only in an explicitly opt-in debug/evidence class.
KV and runtime state
KV/cache initialization, lookup outcomes, restore, record, trim/eviction, reset, pressure, context capacity, exhaustion, and state import/export outcomes.
Cache keys and prompt-derived hashes must not be exported in the normal event envelope.
Backend, device, and resource health
Backend/device initialization and readiness, selected device, degradation/unavailability/recovery, resource allocation, memory pressure, out-of-memory, fallback, compute failure, and device loss/reset.
Public projections must not expose native pointers, raw stable hardware identifiers, or absolute device paths.
Rust must never parse human-readable summaries to drive behavior.
Node serving availability
Reducer-owned node starting/accepting/degraded/unavailable/draining/stopped events, available model/stage set changes, request/lane/session capacity changes, and resource-pressure changes.
Internal runtime events must not be added to mesh gossip. Remote peers should receive only the existing protocol-compatible availability projection required for routing.
Event-system health
Ingress pressure, coalescing, sampling, drops by class, subscriber lag/disconnect, reducer errors, exporter degradation/recovery, schema incompatibility, and unknown native events.
Health reporting must be rate-limited and must not create recursive event storms.
Delivery, Backpressure, and Correctness
Events must be assigned at least four service classes:
State transitions: latest-state preservation, duplicate coalescing, bounded control-plane history, and independently queryable current state.
Progress: latest-value coalescing, duplicate/regression suppression, rate-limited presentation/export, and safe dropping of intermediate values.
Diagnostics: droppable, sampled or batched when high-rate, counted when lost, and never allowed to block inference or own readiness.
There is no required global order across independent models, stages, sessions, or requests. Per-operation order should be preserved where the producer supplies a sequence. Consumers must tolerate missing progress and diagnostics, unknown future events, and terminal Rust results without native terminal callbacks.
The reducer must maintain bounded current state for runtime health, model availability, topology readiness, request capacity, sessions, inflight requests, KV/cache pressure, device/resource health, and event-system health.
Readiness advances only through Rust-owned reducer transitions after required dependencies are ready.
Consumer Contracts
CLI, TUI, and JSON presentation must consume typed events or reducer transitions and preserve stable machine-readable event names rather than flattening everything into generic info/warning output.
The management API should expose current reduced state, bounded recent lifecycle/warning events, active progress, model/stage/topology availability, request/session capacity, and event pressure/drop summaries. It must not expose an unbounded internal stream or sensitive fields.
Telemetry is optional. Exporter failure must never fail startup or inference. Telemetry transformation occurs outside native callbacks, follows existing privacy rules, summarizes progress, and exposes local drop/export counters.
Diagnostics may record bounded structured traces, retain raw native logs as explicit debug artifacts, and subscribe to sampled observations. Removing diagnostics must not change state correctness.
Privacy and Cardinality
The normal event system must not collect or export:
prompt or completion text;
tool arguments or results;
request bodies or media;
prompt-derived cache keys or hashes;
token text, token IDs, or logits;
absolute paths;
credentials or signed URLs;
native pointers;
raw stable hardware identifiers;
unbounded arbitrary attribute maps; or
arbitrary upstream log lines as structured state.
Permitted summaries include token counts, durations, memory, capacity, queue depth, and progress.
ABI and Compatibility
Native event extensions must be append-only where possible.
The implementation must:
add feature bits for independently optional event families;
bump and synchronize Skippy ABI versions when the native event ABI changes;
use explicit integer discriminants;
validate ABI versions and struct sizes;
accept larger compatible structs by reading known fields;
copy borrowed strings during callbacks;
preserve legacy no-event entrypoints;
fall back to authoritative return values when event support is absent; and
avoid requiring unrelated optional event symbols.
Mixed-version mesh protocol behavior remains unchanged. Internal runtime events are not gossiped.
Native Log Parser Retirement
Parsed native logs may remain as opt-in debug artifacts, but must stop driving normal state and presentation only after structured coverage exists for:
model-load lifecycle and coarse progress;
backend/device selection;
model/context/cache allocation and pressure;
tensor loading/offload summaries;
KV/cache lifecycle;
tokenizer and auxiliary readiness;
recoverable warnings/fallbacks;
fatal native errors; and
authoritative Rust-side completion.
The migration must switch normal CLI/TUI/JSON, telemetry, and node state to structured events before removing parser-specific state and tests. Tests must prove normal operation with native log forwarding disabled.
Migration Plan
Phase 1: Rust event core
Add typed runtime domain events.
Add bounded ingress and dispatch.
Add service classes, coalescing, and drop counters.
Preserve the current native event envelope.
Route existing model-open callbacks through the ingress.
Reconcile terminal outcomes from ABI returns.
Preserve existing presentation through an adapter.
Phase 2: State reducer and consumer separation
Add backend-neutral model, stage, session, resource, and event-system state.
Move formatting out of callbacks.
Feed CLI/TUI/JSON and management API from events or reducer transitions.
Convert telemetry into an optional consumer.
Ensure no consumer can block native ingress.
Phase 3: Native model and resource coverage
Add model-load phases/progress.
Add backend/device/resource events.
Add KV/cache pressure.
Add warnings, fallback, fatal errors, and unload observations.
Compare structured coverage with parsed-log output.
Switch normal consumers to structured events.
Disable parsed logs in normal state/output paths.
Retain raw logs only for opt-in debugging.
Remove parser aggregation after compatibility evidence is complete.
Testing Requirements
Native ABI
Cover layout/discriminants, ABI and struct-size validation, single and multipart loading, real callback order, missing optional events, permitted callback threads, unknown kinds, larger structs, borrowed-data lifetime, no callback after return, feature probing, and legacy fallback.
At least one integration test must invoke the real patched native runtime event entrypoint.
Ingress and backpressure
Cover concurrent ingress, full progress/diagnostic queues, terminal capacity under pressure, progress coalescing, sampling/drop accounting, subscriber lag/failure, shutdown/draining, and no blocking on native producer threads.
Reconciliation
Cover callback/return success and failure combinations, missing terminal callbacks, process crash/worker termination, old runtimes without event support, and event loss with correct final state.
Reducer
Cover readiness gating, stage/topology degradation and recovery, session/capacity transitions, KV/resource pressure, invalid/duplicate transitions, unknown events, dropped progress, unload/forced cleanup, and node availability derived from multiple models/stages.
Performance
Compare events disabled/enabled for model load and inference, aggregated progress versus debug observations, slow/failed telemetry, full diagnostic queues, and simultaneous model/session producers. Measure callback latency, coalescing, and drops without materially regressing decode throughput or time to first token.
Acceptance Criteria
Native callback work is limited to validation, copying, correlation, and nonblocking ingress.
Arbitrary application closures are no longer invoked inline from native callbacks.
Typed Rust events preserve the native envelope.
Terminal operations are reconciled from authoritative return values.
Model, availability, unload, stage, session, prefill, generation, KV/cache, resource, warning, and node-availability families are represented.
State remains correct when progress and diagnostics are dropped.
CLI/TUI/JSON and normal node state no longer depend on parsed native logs.
Telemetry is optional and nonblocking.
Event pressure, coalescing, sampling, and drops are observable locally.
The generation-lifecycle work in PR #1149 is a useful focused producer-side integration point for the Phase 4 inference events. It should remain behaviorally authoritative for Skippy while exposing observations that can eventually be adapted into this pipeline.
Notes
The specification intentionally does not require a C++ event bus, native routing, native retry policy, an unbounded queue, detailed per-token production streams, detailed MTP accounting, or backend-specific public state.
Objective
Implement the structured llama.cpp/Skippy and Rust event pipeline described in the event-system specification.
The completed system should make normal runtime lifecycle, availability, progress, warning, resource, and serving state independent of parsing native llama.cpp log text. It must preserve the existing ownership boundary:
This is a draft implementation issue. Exact Rust module names, queue implementation, capacities, event enum names, and management API shapes remain implementation choices as long as the behavioral requirements below are preserved.
Current State
The current Skippy runtime event ABI exposes five model-open observations:
The current callback boundary already has useful properties: a C-compatible ABI, explicit event version and struct size, fixed-width fields, copied borrowed data, no callback after the model-open call returns, and authoritative success/failure through the normal return path.
The current limitations are:
Rust-owned telemetry is useful prior art for timing and lifecycle observations, but it is not the event system: telemetry may be disabled, is an optional export consumer, and does not provide authoritative local state.
Target Architecture
The intended flow is:
patched llama.cpp → synchronous operation-scoped callback → minimal Rust FFI trampoline → validated owned native fact → bounded nonblocking Rust ingress → typed dispatcher and state reducer → state, presentation, API, telemetry, and diagnostic projections.
The native callback and Rust trampoline must:
They must not:
The event system must keep three layers separate:
Consumer projections must never be fed back as native facts.
Event Model
Every runtime domain event should support the relevant subset of:
Fields that do not apply must be absent rather than represented by ambiguous sentinel values.
Native pointer addresses must never become durable IDs. Correlation IDs must not be derived from prompt or completion contents. Unknown event kinds and newer append-only fields must be tolerated safely.
Terminal and warning events need stable reason codes covering cases such as invalid configuration, unsupported capability, missing artifact, I/O failure, model-load failure, backend failure, device unavailable, resource allocation failure, out of memory, context exhaustion, stage unavailable, timeout, cancellation, process crash, incompatible ABI, internal failure, and unknown failure.
Required Event Families
Native runtime lifecycle
Resolution, loading, ABI/feature compatibility, initialization, stopping, stopped, and crash events.
Model acquisition and preparation
Model queued, resolution, download progress, materialization/package preparation, cache reuse, completion, failure, and cancellation.
Model loading
Load requested/started, coarse phase changes, progress, backend/device selection, memory allocation or pressure, native completion, failure, and cancellation.
Normal events must remain coarse-grained. They must not emit individual tensor, tensor-name, layer, kernel, or graph-node events.
Model availability and readiness
Native model loaded, Rust backend initialization, available, degraded, unavailable, recovery, and capacity changes.
Native model-load completion must not imply serving availability. Rust readiness is emitted only after required serving surfaces and dependent stages are usable.
Model unloading
Unload requested/started/completed/failed, session draining, and forced unload, including active/draining session counts and cleanup reason where available.
Stage and topology lifecycle
Stage starting/loading/ready/degraded/unavailable/stopping/stopped/failed; topology assembling/ready/degraded/unavailable; and upstream/downstream connection established/lost/recovered.
These events must project into backend-neutral model availability.
Session lifecycle
Session requested/created/active/idle/reset/trimmed/restored/draining/closed/failed/abandoned, with bounded identity, lane, token-count, duration, and reason data.
Request and admission lifecycle
Request received/queued/admitted/rejected/started/completed/cancelled/timed out/failed, with request identity, model, serving mode, queue depth/wait, duration, outcome, reason, and attempt count.
No prompt, completion, tool argument, request body, or media content may be included.
Prompt processing and prefill
Prompt processing, tokenization, prefill start/progress/completion/cancellation/failure, media prefill, and prompt-cache restore outcomes.
Production prefill progress should be aggregated rather than emitted per chunk.
Decode and generation
Generation started, first token, aggregated progress, completed, cancelled, timed out, failed, and stop condition reached.
Normal generation events must expose bounded summaries such as token counts, time to first token, elapsed duration, coarse throughput, stop reason, speculative-mode status, and batching mode. Per-token text, logits, and token IDs belong only in an explicitly opt-in debug/evidence class.
KV and runtime state
KV/cache initialization, lookup outcomes, restore, record, trim/eviction, reset, pressure, context capacity, exhaustion, and state import/export outcomes.
Cache keys and prompt-derived hashes must not be exported in the normal event envelope.
Backend, device, and resource health
Backend/device initialization and readiness, selected device, degradation/unavailability/recovery, resource allocation, memory pressure, out-of-memory, fallback, compute failure, and device loss/reset.
Public projections must not expose native pointers, raw stable hardware identifiers, or absolute device paths.
Warnings, recoveries, and errors
Structured warning raised/cleared, recoverable failure, fallback, degraded operation, fatal failure, invariant violation, stable code, severity, scope, recoverability, correlation, and resulting state.
Rust must never parse human-readable summaries to drive behavior.
Node serving availability
Reducer-owned node starting/accepting/degraded/unavailable/draining/stopped events, available model/stage set changes, request/lane/session capacity changes, and resource-pressure changes.
Internal runtime events must not be added to mesh gossip. Remote peers should receive only the existing protocol-compatible availability projection required for routing.
Event-system health
Ingress pressure, coalescing, sampling, drops by class, subscriber lag/disconnect, reducer errors, exporter degradation/recovery, schema incompatibility, and unknown native events.
Health reporting must be rate-limited and must not create recursive event storms.
Delivery, Backpressure, and Correctness
Events must be assigned at least four service classes:
There is no required global order across independent models, stages, sessions, or requests. Per-operation order should be preserved where the producer supplies a sequence. Consumers must tolerate missing progress and diagnostics, unknown future events, and terminal Rust results without native terminal callbacks.
The reducer must maintain bounded current state for runtime health, model availability, topology readiness, request capacity, sessions, inflight requests, KV/cache pressure, device/resource health, and event-system health.
Readiness advances only through Rust-owned reducer transitions after required dependencies are ready.
Consumer Contracts
CLI, TUI, and JSON presentation must consume typed events or reducer transitions and preserve stable machine-readable event names rather than flattening everything into generic info/warning output.
The management API should expose current reduced state, bounded recent lifecycle/warning events, active progress, model/stage/topology availability, request/session capacity, and event pressure/drop summaries. It must not expose an unbounded internal stream or sensitive fields.
Telemetry is optional. Exporter failure must never fail startup or inference. Telemetry transformation occurs outside native callbacks, follows existing privacy rules, summarizes progress, and exposes local drop/export counters.
Diagnostics may record bounded structured traces, retain raw native logs as explicit debug artifacts, and subscribe to sampled observations. Removing diagnostics must not change state correctness.
Privacy and Cardinality
The normal event system must not collect or export:
Permitted summaries include token counts, durations, memory, capacity, queue depth, and progress.
ABI and Compatibility
Native event extensions must be append-only where possible.
The implementation must:
Mixed-version mesh protocol behavior remains unchanged. Internal runtime events are not gossiped.
Native Log Parser Retirement
Parsed native logs may remain as opt-in debug artifacts, but must stop driving normal state and presentation only after structured coverage exists for:
The migration must switch normal CLI/TUI/JSON, telemetry, and node state to structured events before removing parser-specific state and tests. Tests must prove normal operation with native log forwarding disabled.
Migration Plan
Phase 1: Rust event core
Phase 2: State reducer and consumer separation
Phase 3: Native model and resource coverage
Phase 4: Inference lifecycle
Phase 5: Native log parser retirement
Testing Requirements
Native ABI
Cover layout/discriminants, ABI and struct-size validation, single and multipart loading, real callback order, missing optional events, permitted callback threads, unknown kinds, larger structs, borrowed-data lifetime, no callback after return, feature probing, and legacy fallback.
At least one integration test must invoke the real patched native runtime event entrypoint.
Ingress and backpressure
Cover concurrent ingress, full progress/diagnostic queues, terminal capacity under pressure, progress coalescing, sampling/drop accounting, subscriber lag/failure, shutdown/draining, and no blocking on native producer threads.
Reconciliation
Cover callback/return success and failure combinations, missing terminal callbacks, process crash/worker termination, old runtimes without event support, and event loss with correct final state.
Reducer
Cover readiness gating, stage/topology degradation and recovery, session/capacity transitions, KV/resource pressure, invalid/duplicate transitions, unknown events, dropped progress, unload/forced cleanup, and node availability derived from multiple models/stages.
Performance
Compare events disabled/enabled for model load and inference, aggregated progress versus debug observations, slow/failed telemetry, full diagnostic queues, and simultaneous model/session producers. Measure callback latency, coalescing, and drops without materially regressing decode throughput or time to first token.
Acceptance Criteria
Related Work
The generation-lifecycle work in PR #1149 is a useful focused producer-side integration point for the Phase 4 inference events. It should remain behaviorally authoritative for Skippy while exposing observations that can eventually be adapted into this pipeline.
Notes
The specification intentionally does not require a C++ event bus, native routing, native retry policy, an unbounded queue, detailed per-token production streams, detailed MTP accounting, or backend-specific public state.