Feature: recorded-stream API — stepWithStream cannot return the observed chunks (implementation available)

[arniwesth]

What problem does this solve?

We're building deterministic replay testing for an agent loop written in AILANG
(motoko_agent). To replay a run we have to record what
the model actually streamed, and std/ai.stepWithStream gives us no way to retain it.

Two properties must hold at the same time for the use case, and today we can have either but
never both:

1. Live visibility is production behaviour. Chunks must reach the UI as they arrive. Buffering
   them and emitting at the end is a real behaviour change, so a test-only "collect then flush"
   path is not equivalent to what ships.
2. The exact ordered chunks must land in an immutable trace, so a run can be replayed and its
   invariants asserted against the recording.

The blocker is the callback/result contract: the callback must return (), its effect row is
closed to {IO}, and the result type Result[StepResult, AIError] does not carry the observed
chunks. So chunks are observable only as a side effect that nothing can capture.

Expected: a caller can receive chunks immediately and obtain the identical ordered sequence
for recording.
Actual: chunks are visible only inside an IO-only, unit-returning callback; the result value
discards them.

Reproduction

Two minimal standalone modules. (A) the callback cannot thread an accumulator — it must return ():

module repro_a

import std/ai (stepWithStream, Message, StepResult, AIError, StreamChunk)
import std/result (Result)

func state_returning_callback(chunk: StreamChunk) -> [StreamChunk] ! {IO} {
  [chunk]
}

export func rejected(messages: [Message]) -> Result[StepResult, AIError] ! {AI} {
  stepWithStream("", messages, [], [], state_returning_callback)
}

$ ailang check repro_a.ail
Error: type error in repro_a (decl 1): type unification failed at
[function application at repro_a.ail:11:17]: failed to unify parameter 4:
cannot unify type constructor () with *types.TList

(B) the callback cannot tee to a side-channel — its effect row is closed and rejects SharedMem:

module repro_b

import std/ai (stepWithStream, Message, StepResult, AIError, StreamChunk)
import std/bytes (fromString)
import std/io (println)
import std/result (Result)
import std/sharedmem (put)

func capture(chunk: StreamChunk) -> () ! {IO, SharedMem} {
  let _ = put("stream-capture/probe", fromString(show(chunk)));
  println("captured")
}

export func probe(messages: [Message]) -> Result[StepResult, AIError] ! {AI, IO, SharedMem} {
  stepWithStream("", messages, [], [], capture)
}

$ ailang check repro_b.ail
Error: type error in repro_b (decl 1): type unification failed at
[function application at repro_b.ail:15:17]: failed to unify parameter 4:
failed to unify effect rows: incompatible closed rows: r1 has extra labels [],
r2 has extra labels [SharedMem]

("parameter 4" is the zero-indexed 5th argument — the on_chunk callback.)

Both errors are correct and clearly worded given the current signature. This is an API capability
gap, not a diagnostics problem — we are not asking you to change these messages.

Positive control — we can drive the current API correctly, so this is not user error. An
IO-only callback fires twice and a typed result comes back:

module probe_old_api

import std/ai (stepWithStream, Message, StreamChunk, ContentDelta, ThinkingDelta, Usage)
import std/io (println)

func render(chunk: StreamChunk) -> () ! {IO} {
  match chunk {
    ContentDelta(text) => println("LIVE content ${text}"),
    ThinkingDelta(_) => println("LIVE thinking"),
    Usage(_) => println("LIVE usage")
  }
}

export func main() -> () ! {AI, IO} {
  let messages: [Message] = [{
    role: "user", content: "old api regression probe",
    tool_calls: [], tool_call_id: "", images: []
  }];
  match stepWithStream("", messages, [], [], render) {
    Ok(result) => println("PASS old_api ${result.finish_reason}"),
    Err(error) => println("FAIL old_api ${error.code}")
  }
}

$ ailang run --caps AI,IO --ai-stub --entry main probe_old_api.ail
LIVE content {"kind":"Wait"}
LIVE usage
PASS old_api stop

(Drop images: [] on v0.26.0, where Message has four fields.) The only thing we cannot do is
retain those two chunks.

Proposed Solution

An additive entry point that keeps immediate callback delivery and additionally returns the
exact ordered observed chunks:

export func stepWithStreamRecorded(
  model: string,
  messages: [Message],
  tools: [ToolSchema],
  cache_breakpoints: [CacheBreakpoint],
  on_chunk: (StreamChunk) -> () ! {IO}
) -> { chunks: [StreamChunk], outcome: Result[StepResult, AIError] } ! {AI}

Three properties are load-bearing for us. If you reshape the surface, these are what we actually
need — the signature above is just one way to get them:

1. Both delivery and capture, not either. on_chunk must still fire as each chunk arrives.
   Returning the list instead of invoking the callback would silently remove live streaming,
   which is the reason we cannot simply buffer today.
2. Chunks on both outcomes. This is why the return is a record rather than
   Result[{result, chunks}, AIError] — that shape has nowhere to put chunks when the stream
   fails, and a partial stream followed by an error is precisely the run we most need a trace for.
3. Identity, not reconstruction. The returned chunks must be the values delivered to
   on_chunk, in the same order — not re-derived from the final message. Concatenating the
   ContentDelta payloads must still equal StepResult.message.content, per your own documented
   invariant. Our test asserts this by value identity, not equality.

Why additive. stepWithStreamRecorded breaks no existing caller; widening stepWithStream's
own return type would break every current user. We prefer the additive form for that reason and
because it is likelier to land — while acknowledging it adds a fourth step* variant, which you
may reasonably not want. If you would rather have one entry point with a richer result, we are
happy either way — the three properties matter, the spelling does not.

We have implemented and tested this against the v0.31.0 tag (details and diff in Additional
Context). We are happy to open a PR if you want it in this shape, or to drop it entirely if
you would rather design the surface yourselves — the evidence stands either way.

Alternatives Considered

Approach	Outcome
Return an accumulator from the callback	Rejected — callback must return () (repro A)
Tee each chunk to a SharedMem recorder scope	Rejected — closed effect row excludes SharedMem (repro B)
Read the chunk list off the result	Not available — result is Result[StepResult, AIError]
Project chunks to stdout and reconstruct	Works, but moves the oracle out of the driver onto parsed wire output — unacceptable for our trace contract
Buffer chunks and emit them after completion	Rejected by us, not by the compiler — it changes live streaming UX, which is production behaviour

We separately validated the recording design itself against a modelled provider — success,
partial-stream-then-error, scope isolation and 8-way process parallelism all pass — so the blocker
is the callback/result contract, not our recorder.

A fallback we are explicitly not asking for, listed only so the design space is complete:
widening the callback's effect row (row-polymorphic, or {IO, SharedMem}) would let a caller
install its own recorder. We do not want it — it puts SharedMem into an otherwise hermetic
capability profile, and a scoped recorder is more machinery than a returned list for the same
result. If you prefer it, we can work with it, but the record-returning shape is both smaller and
stricter.

Area

AI Integration

Additional Context

The implementation

Against v0.31.0 (1f6f7dd28dd22ce0e0d84d818a37fce874f1c351). Browsable diff:

arniwesth/ailang@v0.31.0...spike/motoko-009-prototype-v031

or git remote add arniwesth https://github.com/arniwesth/ailang.git && git fetch arniwesth spike/motoko-009-prototype-v031. Five files, +452 lines, no deletions, no provider changes:

File	Change
internal/effects/ai_step.go	new aiStepWithStreamRecorded returning the {chunks, outcome} record
internal/builtins/ai_step.go	registers _ai_step_with_stream_recorded and its type
std/ai.ail	exports stepWithStreamRecorded + the RecordedStream type
internal/effects/ai_step_with_stream_recorded_test.go	tests (below)

Recording happens where the AILANG closure is already wrapped into the Go onChunk callback — one
append beside the existing chunkCount++, reusing the value encodeStreamChunk already builds.
Provider.StepWithStream, its five implementations and SSE handling are untouched, so no provider
changes and the NO-OP fallback path is inherited unchanged.

$ go test ./internal/effects/ -run TestAIStepWithStream -v
--- PASS: TestAIStepWithStreamRecorded_ReturnsDeliveredChunksOnSuccess
--- PASS: TestAIStepWithStreamRecorded_ReturnsChunksOnErrorPath
--- PASS: TestAIStepWithStreamRecorded_ContentDeltaConcatEqualsMessageContent
--- PASS: TestAIStepWithStream_UnchangedByRecordedVariant
--- PASS: TestAIStepWithStream_FiresContentDeltaAndUsageChunks
--- PASS: TestAIStepWithStream_HandlerErrorReturnsErrResult
--- PASS: TestAIStepWithStream_NoFnCallerWiredReturnsTypedErr
--- PASS: TestAIStepWithStream_EncodesThinkingDeltaCorrectly
--- PASS: TestAIStepWithStream_PassesCacheBreakpointsToHandler
ok      github.com/sunholo-data/ailang/internal/effects

The last five are your existing tests, unchanged and still passing.
TestAIStepWithStream_UnchangedByRecordedVariant additionally pins that stepWithStream still
returns Result[StepResult, AIError] directly rather than a record, so the change is additive by
test and not only by inspection.

The error-path test needed a new fake handler that emits chunks then fails: the existing
fakeStepHandler returns before emitting anything when Step errors, so "partial stream, then
failure" — the case this request is about — was previously untestable in your suite.

We have driven this through a real consumer, not just a probe

Our original evidence was a standalone probe: does the API return the chunks. That is necessary but
weak — it does not show the shape is usable. We have since built the consumer.

We threaded a record/replay layer through our real session driver — the production loop, not a
harness around it — and ran the same program twice: once live through stepWithStreamRecorded
under --ai-stub, recording what came back, and once replaying that recording deterministically
with no provider call. The two agree on the emission log, the interaction log and the normalised
trace:

discovery_emissions = content:{"kind":"Wait"} | usage
replay_emissions    = content:{"kind":"Wait"} | usage
discovery_interactions = provider::msgs=2
replay_interactions    = provider::msgs=2

The check is non-vacuous — it asserts the recording is non-empty, since two empty logs would
compare equal — and load-bearing: seeding the replay with a deliberately wrong chunk fails it.

The part likely most useful to you: building the consumer showed the remaining work is mostly
on our side, not yours. Our own provider seam discarded the chunk list one layer above std/ai,
so landing this API alone would have changed nothing for us until we fixed that too. We mention it
because it means we are not asking you to speculate about whether we would use this — we have
already paid the consumer-side cost against a local build of the proposal, and the only remaining
gap is the API.

Version coverage

The gap is current at three points, each checked by compiling — not by reading docs:

Version	Evidence
v0.26.0 (our pinned floor)	Both repros above, commit 3b52a24d24431c372ed5605289ef039592209514
v0.30.0 (released)	Both repros reproduce identically. Release archive SHA-256 58561c11ca7be7710b3b4eca9ddfdf263f39bc4e36428969a1968175f10b84b6, compiler reports commit e37b370d1d7a9c4e7136b319e38bec4d5f2bd9a0. std/ai.ail:331-337 carries the same signature
v0.31.0 (current release, 1f6f7dd28dd22ce0e0d84d818a37fce874f1c351)	std/ai.ail:330-337 still carries on_chunk: (StreamChunk) -> () ! {IO} returning Result[StepResult, AIError]; grep -c stepWithStreamRecorded std/ai.ail is 0. This is the tag the proposal is implemented against

On v0.30.0 and later the positive control additionally needs images: [] in its Message literal
— that is the v0.30.0 vision-input widening, unrelated to streaming, mentioned only so the change
is not misread as a streaming regression.

Known gap: native-only

The variant is native-only. cmd/wasm/effects.go has its own WasmAIHandler.StepWithStream and a
documented ailangSetAIStepWithStreamHandler JS hook; we did not add a browser-side counterpart,
since the JS Response contract is yours to extend and we did not want to presume its shape. We
note #137/#138 covered the browser-side callback contract separately, so that seam looks like yours
to shape rather than ours to guess at.

Related issues

Checked before filing: #136 (generic bidirectional std/stream primitives, closed) is the nearest
neighbour and a different ask — it is about stream transport, not about retaining what a model
streamed through std/ai. #137/#138 (closed) are WASM browser-streaming issues, adjacent to the
native-only note above.

Unrelated docs-side observation

Offered separately, ignore if known: the stdlib snapshot served by the public docs MCP for 0.30.0
does not list stepWithStream under std/ai at all — its function list ends at stepWithCache,
and runTools is likewise absent — even though the same version's StreamChunk docstring still
describes it as "one event emitted by stepWithStream's on_chunk callback" and the guide "Browser
ai.step with BYO API key" documents its handler contract. Looks like a snapshot/indexing gap
rather than an API removal.

[sunholo-voight-kampff]

Thanks for the exceptionally complete filing — repro pair, duplicate check, a clean-applying patch, and the reference implementation at arniwesth#2.

Triaged + prioritized: this is queued as the mission loop's first pick when it resumes Monday 2026-08-03 07:00 (loops are paused for quota this weekend). The iteration will (1) reproduce at HEAD, (2) review the reference implementation with the two DST ADRs (Projects 007/009) as design context — the intent is to adopt your implementation rather than reinvent, with authorship credited, subject to the normal design-quorum pass on the std/ai surface change, (3) report back here with the verdict and any requested changes.

Also: your five upstream motoko PRs from us (#73/#76/#96/#97/#98) are all green as of today — #98's AILANG_REF guard mismatch was the last red and is fixed — whenever you have review bandwidth.

[sunholo-voight-kampff]

Triage verdict: REAL, not a ghost — and the design doc has landed. ADOPT recommended.

Reproduced first-party at HEAD 130ad1da2 on a freshly rebuilt binary, positive control beside the negative probe:

• an {IO} rendering callback type-checks — rc=0, ✓ No errors found!
• an {FS} callback appending each chunk to a file (the cheapest way to record while still streaming) fails — rc=1, failed to unify effect rows: incompatible closed rows: r1 has extra labels [IO], r2 has extra labels [FS]
• StepResult carries no chunks, and std/io has no file write — so there is no {IO}-legal side-channel either.

So live-streaming and chunk-recording genuinely are mutually exclusive. Project 009 ADR-001 line 134 records the same result independently against v0.30.0.

@arniwesth — your patch was reviewed, not reinvented: git apply --check clean, purely additive (+452/5 files), all four of your tests pass here, and go test ./internal/effects -skip TestNetHTTPRequestBytes_RoundTripSHA is rc=0 with 658 passing, so it breaks nothing existing. The {chunks, outcome} shape over Result[{result, chunks}, err] is judged correct for exactly the reason you gave — the rejected shape discards chunks observed before a mid-stream failure, which is the case replay needs most.

Design doc: design_docs/planned/v0_31_0/m-recorded-stream-api.md (commit d85934df4). Authorship will be credited to you in the adopting commit.

It is PARKED on one open question, and your view would help. A 3-reviewer quorum blocked twice; the design direction was never contested. The survivor: on the fail-loud path (chosen because silently skipping an unencodable chunk contradicts a "lossless" contract), the drain is unbounded — and internal/effects/ai.go:87 declares StepWithStream with no context.Context, so the provider call cannot be cancelled. There are 7 implementers across 6 files including cmd/wasm. Options: (a) land now with a documented caveat, (b) take the cancellation interface change as a blocking dependency, (c) bound the drain locally inside the recorded op with no interface change. Current lean is (c).

One thing your patch inherits rather than causes: std/ai.ail:324 and examples/runnable/ai_streaming.ail:40-42 both claim the callback's effect row "is open" — false at HEAD, and directly above the closed declaration. Fixing that is a milestone in the doc, since adopting the recorded sibling does not widen the row.

No date promised — keep Project 009 blocked rather than waiting on us.

🤖 Generated with Claude Code

[arniwesth]

Direct answer to the question you asked: (c), and I'd argue specifically against (b).
But my reason is a structural fact about the current code rather than a preference between the
three, so let me lead with that.

(Letters throughout refer to the park's options — land-with-caveat / cancellation-dependency /
bound-locally. The design doc's §Count invariant uses a second, unrelated (a)/(b)/(c); where I
mean that one I name it in words.)

The fail-loud path has no reachable trigger

ai.StreamChunk is a sealed interface — internal/ai/provider.go:160-162 closes it with an
unexported streamChunkMarker(), so only types declared in package internal/ai can satisfy it.
Repo-wide and tests included, the grep returns the interface, its doc line, and exactly three
implementers:

$ git grep -n 'streamChunkMarker' 130ad1da2 -- '*.go'
130ad1da2:internal/ai/provider.go:160:	// streamChunkMarker is intentionally unexported — closes the type
130ad1da2:internal/ai/provider.go:162:	streamChunkMarker()
130ad1da2:internal/ai/provider.go:172:func (StreamContentDelta) streamChunkMarker() {
130ad1da2:internal/ai/provider.go:198:func (StreamThinkingDelta) streamChunkMarker() {
130ad1da2:internal/ai/provider.go:213:func (StreamUsage) streamChunkMarker() {

encodeStreamChunk (internal/effects/ai_step.go:422) has a case for all three; its
default: return nil is at :447-448. The only other way to reach that branch is a nil
interface value, and the one call site that forwards a variable rather than a struct literal —
cmd/wasm/effects.go:244 — is guarded by if chunk != nil at :243. Every other invocation
constructs a concrete variant inline.

So the unencodable chunk that triggers fail-loud, that triggers the drain, that motivates the
cancellation dependency, cannot occur at 130ad1da2 or at dev. Identical result at both.

A corollary you'll hit in M1: the merge requirement for "unencodable first and middle chunks"
can only be written by having a fake handler call onChunk(nil). The seal stops internal/effects
from declaring a fourth variant, and no test does today. The guard's only constructible input is
one no provider can produce.

And where chunks are actually lost, the guard can't see it

This is the part I'd most like on the record, because it cuts against the completeness
guarantee rather than against the drain.

Unknown wire events never become an unrepresentable StreamChunk. They are dropped one layer
lower, in the provider adapters, before the effects layer observes anything:

• internal/ai/anthropic/streamstep.go:330 switches on ev.Delta.Type with no default — a
  future delta type is silently ignored, and onChunk is never called for it.
• Same file :336: input_json_delta is accumulated into the block but deliberately not
  emitted as a chunk. Tool-call stream content is already absent from any recorded log by
  design — that is the ToolCallDelta your provider.go:158 comment anticipates.
• cmd/wasm/effects.go:348-366: jsToStreamChunk returns nil for an unrecognised kind, and
  :243 filters it. A JS handler emitting an unknown kind produces no callback, no record, and
  no increment of your proposed provider_chunks — the loss is invisible to the effects
  layer entirely.

The fail-loud invariant sits at the encoder. The boundary where chunks are actually lost is the
adapter, one layer below it. So as specified, fail-loud adds a completeness guarantee exactly
where nothing can violate it, and does not add one where something can — including on the WASM target
you flagged as the expensive part of (b), and which any WASM counterpart to the recorded op
would inherit.

I don't think that argues against fail-loud, and I'm explicitly not proposing you move the
guard down — an unknown wire event has no faithful typed representation, so catching it at the
adapter needs the opaque-byte provider contract §Count invariant already rejected, and that is a
much larger change than the one I'm arguing against in (b).

The ask is smaller: scope the guarantee honestly in the LongDesc — the log is exact with
respect to emitted chunks, not with respect to the wire. For our replay use that distinction is
fine and we'd rely on it. It just needs to be stated, because "lossless" currently reads as the
stronger claim.

What that implies for (a)/(b)/(c)

(b) becomes hard to justify: a 7-implementer interface change reaching WASM, which forfeits
the purely-additive property your ADOPT verdict and the 1.x-stability claim rest on — to make
cancellable a path with no reachable trigger. If cancellation is worth doing, I'd suggest it's
worth doing on its own merits across the whole AI surface (deadlines on live streams are a real
user-facing need, independent of recording), not as a blocking dependency of this.

(c) is cheap insurance and I'd take it on those terms, with the caveat that it bounds a
hazard nothing currently reaches.

What actually pays is neither: the thing that would open that branch is someone adding a fourth
variant in internal/ai without updating encodeStreamChunk. Go will not complain — the switch
stays well-typed and the new variant falls to default. An exhaustiveness guard over the
sealed set, failing at the point a variant is introduced rather than at runtime in someone's
stream, closes it more cheaply than either bounding the drain or changing the interface, and it
protects the existing stepWithStream too. Happy to send that as a separate small patch if it's
wanted — it's independent of this issue.

One divergence worth making deliberate

The existing aiStepWithStream already silently skips an unencodable chunk — same guard,
shipped: internal/effects/ai_step.go:380-382. My patch inherits it verbatim and on purpose, so
the siblings differ only in return shape.

Adopting fail-loud in the recorded op alone means the two ops behave differently on identical
provider input: one skips and completes, the other returns typed Internal and drains. That may
be exactly what you want, but it should be a decision rather than a side effect — the
equivalence "the recorded sibling is the existing op plus a log" is what makes it safe for us to
swap in. Either apply the invariant to both, or state the divergence in the LongDesc.

On the no-silent-fallback argument

I've read §Count invariant, and I take the point that skip-and-continue was rejected on
no-silent-fallback grounds rather than overlooked. What I'd push back on is the silent premise,
not the principle.

For replay we don't need a guarantee that no chunk is ever dropped. We need a guarantee that we
can't mistake an incomplete log for a complete one. Your design already defines the mechanism
for that — provider_chunks vs delivered_chunks. The gap is that it puts them in trace
metadata, which is invisible at the AILANG level: a consumer holding the {chunks, outcome}
record cannot read them and so cannot tell. Surfacing the discrepancy on the record itself
satisfies no-silent-fallback without the stream having to terminate at all — fail loud at the
consumer, where the replay contract actually lives.

Worth having beside (a)/(b)/(c), and orthogonal to the drain either way.

Scheduling — nothing here gates us, so choose on merit

None of (a), (b) or (c) changes the {chunks, outcome} type, so none of them changes a line of
our consumer code. There is nothing for us to wait on between them.

And "keep Project 009 blocked rather than waiting on us" is the right call — we've already acted
on it. A spike through the real driver this week found that the change adjacent to this API on
our side (widening our own Ports.model_step to carry an emission log) has no upstream
dependency at all: the chunks were dying one layer above std/ai regardless, so no upstream
change alone would have fixed it. Doing that first shrinks eventual adoption of
stepWithStreamRecorded to a single closure. Please take the time the decision deserves.

Also noted, and appreciated: the two false "the callback's effect row is open" claims, and the
credit. I'll come back separately on the five motoko PRs.

[sunholo-voight-kampff]

@arniwesth thanks for the sealed-interface analysis — it settled the park. Ruled: option (c) (bound the drain locally, no interface change), with your structural argument recorded in the mission charter as the deciding evidence (the working iteration will re-verify it first-party as a formality). The adoption sprint is the loop's next pick; authorship credit in the adopting commit as promised. The exact-tool-surface work (#498 Lane B) was also just prioritized right behind it.