[Bug]: Codex CLI timeout recovery cannot resume the session observed by the failed Turn

[ObVious55]

Preflight

• I searched existing issues and discussions for this behavior.
• This report is not an unpatched security vulnerability.
• The report and attachments contain no credentials or private/internal material.

Issue origin

Observed or reproduced in a real environment

LoopX version or commit

8854ab6

Host or runtime surface

Codex CLI

LoopX area

Host or runtime integration

Problem

A Codex CLI Turn that starts a new session, exposes a valid session id, and then
times out cannot be resumed through the failed-Turn retry path.

The timeout path intentionally preserves the observed opaque session as
recovery metadata. However, retrying the failed Turn reloads the original
journaled plan, whose session action is still start_new.

The Codex CLI host then observes that a session binding now exists and rejects
the request before starting the host process:

Codex CLI session binding changed after planning

This strands a session that LoopX deliberately preserved for recovery. For
long-running work, the user must abandon the failed Turn or construct a fresh
Turn instead of using the advertised failed-Turn recovery path.

The behavior also conflicts with the LoopX Turn recovery model, where a timeout
after observing a session may preserve that session for a later side-effect-safe
resume.

Minimal reproduction

This can be reproduced by composing existing public test behavior:

1. Check out LoopX commit 8854ab68.

2. Build a loopx_turn_host_request_v0 whose session action is start_new.

3. Run the Codex CLI fake host so that it:

   • emits a thread.started event with a valid session id;
   • then exceeds the bounded host timeout.

4. Confirm that the session binding was preserved.

5. Retry the same failed Turn using its journaled plan, equivalent to
   --resume-turn-key together with --retry-failed-turn.

6. Observe that the journaled request still contains action=start_new.

7. Observe that run_codex_cli_host raises:

   Codex CLI session binding changed after planning

8. Confirm that the retry starts zero Codex host processes.

The existing
test_codex_cli_host_preserves_observed_session_after_timeout
fixture already provides steps 2-4. Extending that scenario with a
same-failed-Turn retry reproduces the missing composition.

As a control, a request using the same persisted session with
action=resume successfully starts one host process and returns
validated_progress.

Expected behavior

When a failed Codex CLI Turn is explicitly retried after a recoverable timeout:

• if the failed execution observed and preserved a compatible session, LoopX
  should resume that session through a freshly authorized recovery path;
• the retry should invoke the host exactly once using resume;
• the recovered result should still proceed through independent validation;
• durable writeback and quota spend should still occur at most once;
• failures that occur before any session is observed should remain
  non-resumable;
• unexpected session changes during an ordinary non-recovery start_new plan
  should continue to fail closed.

The implementation does not need to mutate every stale start_new request.

Any recovery-specific reconciliation should remain bounded to the failed Turn
and require matching goal, agent, Todo, and Turn lineage, a compatible persisted
session, an eligible failure class, and fresh retry authority.

Actual behavior

For the deterministic timeout fixture:

• Compatible session preserved after timeout: 1/1
• Failed-Turn recovery success: 0/1
• Host processes started by the retry: 0
• Retry result: host_failure
• Control request with action=resume: 1/1 successful
• Duplicate durable writeback: 0
• Duplicate quota spend: 0

The strict session-binding drift check executes before the Codex host process
starts.

It currently cannot distinguish between:

• an unexpected session appearing after an ordinary start_new plan; and
• the expected session created by the same Turn before its recoverable timeout.

As a result, the preserved recovery session causes the retried journaled
start_new plan to fail before Codex CLI is invoked.

Sanitized diagnostics

LoopX commit: 8854ab68
Host surface: Codex CLI
Affected path: start_new -> observed session -> timeout -> failed-Turn retry

Failed retry:
planned_action=start_new
saved_session_available=true
host_process_starts=0
recovery_success=false
error="Codex CLI session binding changed after planning"

Control:
planned_action=resume
saved_session_available=true
host_process_starts=1
result_kind=validated_progress
recovery_success=true

Focused existing recovery controls:
4 passed

Additional context

There is a second blocker on Windows in the same end-to-end recovery path.

The private Codex CLI session writer currently calls os.fchmod while
persisting the observed session. On Windows, this operation is unavailable.
The subsequent temporary-file cleanup may then mask the original failure with
a PermissionError.

Observed deterministic behavior on Windows/Python 3.12:

• Session persistence success: 0/1
• Session reload available: No
• Recovery eligible: No

This Windows persistence problem can be fixed in the same PR if maintainers
consider cross-platform session persistence part of the accepted scope.
Otherwise, it can be split into a linked follow-up issue so that the core
failed-Turn recovery fix remains reviewable.

GH-C70 already discusses provider-neutral host-loop parity for recoverable
timeout and termination. This report is narrower: it describes a concrete
failure in the built-in Codex CLI adapter combined with failed-Turn retry.

This issue does not request:

• a second scheduler or checkpoint store;
• blanket retry behavior for arbitrary exceptions;
• benchmark scoring changes;
• a live model or private benchmark run;
• treating session existence as proof of task progress.

The intended fix should preserve the existing fail-closed behavior for
unexpected non-recovery session drift while allowing a session created and
preserved by the same recoverable failed Turn to be resumed safely.

[huangruiteng]

Thanks for the detailed report — I checked it against the current code and it reproduces cleanly by inspection.

Root cause confirmation

The strict drift check in run_codex_cli_host (loopx/control_plane/turn_driver/codex_cli.py):

if planned_action == "start_new" and binding is not None:
    raise RuntimeError("Codex CLI session binding changed after planning")

This is exactly the line that fires on the retried journaled plan. The failed Turn preserved the observed session for recovery (which test_codex_cli_host_preserves_observed_session_after_timeout already covers), but the retry reloads the original journaled request whose session action is still start_new. The check cannot distinguish "a session appeared unexpectedly during an ordinary plan" from "the same Turn created this session and then timed out after observing it" — so the recovery session itself becomes the blocker.

Fix direction (agreed with the issue's expected behavior)

The reconciliation should be bounded to the explicit failed-Turn retry path:

• Only when retrying the same failed Turn (matching goal / agent / Todo / Turn lineage, resume_turn_key, eligible failure class, fresh retry authority), reconcile the journaled plan's session action from start_new to resume for that retry.
• Keep the existing fail-closed behavior for every other case: unexpected session drift on a non-recovery start_new plan, missing resume binding on resume, and failures before any session was observed must all still fail closed.
• The recovered run should go through the same independent validation, and durable writeback + quota spend stay at-most-once.

Good seams to build on: run_loopx_turn_once(..., retry_failed=True) in the turn executor (tests already exercise it in tests/test_loopx_turn_executor.py), and the host adapter's session-binding checks in codex_cli.py. A focused regression test extending test_codex_cli_host_preserves_observed_session_after_timeout with the same-failed-Turn retry composition would be the right first slice.

Windows secondary item

os.fchmod(descriptor, 0o600) in the same session-persistence writer is unavailable on Windows and the temp-file cleanup can mask the original failure. Wrapping it in a platform guard (or falling back to path-based chmod/best-effort mode setting) is small and can land in the same PR or a linked follow-up — either is fine, but keeping the core recovery fix reviewable is preferable.

You're welcome to attempt the fix. Suggested scope for a first PR: the bounded start_new -> resume reconciliation on eligible failed-Turn retry + the regression test, with the Windows fchmod guard either included or split into a follow-up issue. Maintainers will route it through the standard exact-head review process.

Thanks again for including the sanitized diagnostics and the minimal reproduction — that made this very easy to verify.

[huangruiteng]

ai给了个建议，可以按你的理解来修～

[huangruiteng]

loopx 的 turn 模式确实用的比较少。。。一开始设计了，但主力用的还是 codex app 这种伴生 agent 的方式

[huangruiteng]

但我感觉 turn 模式在某些场景上限更高，比如可以搞复杂的 dynamic state workflow