Graph network resilience: node retry + resumable failures

README.md

@@ -228,6 +228,8 @@ All live in [`examples/`](examples/):
 - **`basic_graph`** — a minimal typed state graph: `START`, nodes, edges, `END`.
 - **`complex_graph`** — conditional routing, fanout, and richer topology.
 - **`durable_graph`** — checkpoints, resume, and time-travel over supersteps.
+- **`resilient_graph`** — node-level retry over transient failures, plus a
+  resumable failure checkpoint that `retry` restarts after an outage clears.
 - **`agent_loop_tools`** — the agent ↔ tool loop the harness runs.
 - **`orchestrator_subagents`** — **recursion in action:** an orchestrator agent
   that calls sub-agents as tools, with depth tracking and rolled-up usage.

docs/modules/graph/fault-tolerance.md

@@ -1,42 +1,80 @@
 # Graph Error Handling And Fault Tolerance
 
-Default behavior:
-
-- node error fails the run
-- retry only if policy allows it
-- timeout fails the node
-- checkpoint remains at the last completed boundary
-- pending writes from completed tasks remain available when the checkpointer
-  supports them
-
-Target behavior:
-
-- route node errors to a node-specific or default error handler
-- expose handled errors in task events and checkpoint task metadata
-- retry with backoff and jitter
-- support optional nodes by explicit policy
-- support cooperative drain/shutdown with a drain reason
-- distinguish graph recursion errors from node failures
-
-Error types should distinguish:
-
-- missing start
-- duplicate node
-- missing node
-- missing edge target
-- duplicate route
-- missing route
-- invalid command target
-- invalid parent graph command
-- invalid send target
-- recursion limit
-- checkpoint required
-- checkpoint missing
-- interrupt resume mismatch
-- reducer conflict
-- invalid concurrent update
-- node timeout
-- node failure
-- graph drained
-- serialization failure
-- checkpoint backend failure
+The graph runtime survives transient network failures and makes a hard failure
+restartable, using two opt-in mechanisms plus the existing checkpoint/resume
+machinery.
+
+## Node retry (surviving network blips)
+
+`CompiledGraph::with_node_retry(RetryPolicy)` wraps every node handler. When a
+handler fails with a [retryable] error — `TinyAgentsError::Model` or
+`TinyAgentsError::Tool`, the transient class classified by
+`harness::retry::is_retryable` — the executor re-runs the node **from its start**
+up to the policy's `max_attempts`, emitting `GraphEvent::NodeRetryScheduled`
+before each retry. Because a node is never suspended mid-flight, a retry rebuilds
+the handler future and context from scratch, matching the durable-execution
+model.
+
+Backoff between attempts is computed from the `RetryPolicy` (exponential, with
+optional jitter) but only **slept on** when the policy opts in via
+`RetryPolicy::with_backoff_sleep(true)`. The default is sleep-free so unit tests
+stay deterministic; production callers enable it so retries wait a real, growing
+delay. The same opt-in switch governs the harness model-call retry loop and
+`RetryMiddleware`.
+
+Both the sequential and parallel runners drive handlers through one shared
+`run_node_with_retry` helper, which also applies the per-node timeout — so a
+retried attempt is still bounded by `node_timeout`.
+
+## Resumable failures (restarting / continuing after a hard failure)
+
+When a handler fails **beyond** the retry budget, or with a non-retryable error,
+the run is not lost. On a checkpointed thread the executor:
+
+1. folds the updates of the branches that completed *before* the failing one
+   into committed state (partial parallel progress is preserved, not discarded);
+2. persists a **failure-boundary checkpoint** whose `next_nodes` schedule the
+   failed node (and the not-yet-run tail of the step) for re-run, records the
+   successful nodes as `completed_tasks`, and stamps the rendered `error` and
+   `failed_node` into the checkpoint metadata;
+3. records a `Failed` run status carrying that checkpoint id;
+4. returns the error to the caller.
+
+Without a checkpointer/thread the run aborts immediately with the error, exactly
+as before — the failure checkpoint is a no-op.
+
+### Restarting and continuing
+
+Because `resume`/`resume_from` replay from a checkpoint's `next_nodes`, a failed
+run resumes exactly like an interrupted one:
+
+- `CompiledGraph::retry(thread)` — shorthand for `resume` with an empty command;
+  re-runs the failed node and the not-yet-run tail from the failure boundary.
+- **Continue on user feedback** — inspect committed state with `get_state`, edit
+  it with `update_state` (attributed through the reducers), then `retry`/`resume`.
+  The re-run sees the edited state.
+
+See the `resilient_graph` example (`cargo run --example resilient_graph`) for
+both mechanisms end to end.
+
+## Error taxonomy
+
+`TinyAgentsError` distinguishes structural/config errors (non-resumable) from
+node failures (resumable on a checkpointed thread):
+
+- structural / not retried, not resumable: missing start, missing node, missing
+  edge target, missing route, invalid command/parent/send target, recursion
+  limit, node-visit limit, sub-agent depth, checkpoint required/missing, resume
+  mismatch, reducer conflict, invalid concurrent update, serialization failure,
+  checkpoint backend failure;
+- transient / retryable and resumable: `Model`, `Tool` (and `Timeout` aborts the
+  node but leaves a resumable boundary like any other node failure).
+
+## Future work
+
+- Route node errors to a node-specific or default error-handler node.
+- Cooperative drain/shutdown with a drain reason.
+- Populate the checkpoint `pending_writes` list explicitly (today partial
+  progress is folded into committed state instead).
+
+[retryable]: ../../../src/harness/retry/mod.rs

examples/resilient_graph.rs

@@ -0,0 +1,135 @@
+//! Surviving network blips and restarting a failed run.
+//!
+//! Two resilience primitives, on one durable graph:
+//!
+//! 1. **Node-level retry.** [`CompiledGraph::with_node_retry`] re-runs a node
+//!    from its start when its handler fails with a transient
+//!    ([retryable][tinyagents::harness::retry::is_retryable]) error — a model or
+//!    tool/network error. A `fetch` node here fails its first two attempts (a
+//!    simulated connectivity blip) and succeeds on the third, so the run
+//!    completes without any operator involvement.
+//!
+//! 2. **Resumable failure + restart.** A `commit` node fails *harder* than the
+//!    retry budget allows. On a checkpointed thread the run does not vanish: it
+//!    persists a resumable failure-boundary checkpoint (the failed node is
+//!    scheduled to re-run, earlier progress is preserved) and returns the error.
+//!    The "outage" then clears and [`CompiledGraph::retry`] restarts the run
+//!    from that checkpoint to completion. A run could equally be *continued on
+//!    user feedback* by editing state with `update_state` before `retry`.
+//!
+//! Run with:
+//!
+//! ```text
+//! cargo run --example resilient_graph
+//! ```
+
+use std::sync::Arc;
+use std::sync::atomic::{AtomicUsize, Ordering};
+
+use tinyagents::harness::ids::ExecutionStatus;
+use tinyagents::harness::retry::RetryPolicy;
+use tinyagents::{
+    GraphBuilder, InMemoryCheckpointer, NodeContext, NodeResult, Result, TinyAgentsError,
+};
+
+/// A tiny pipeline state: how far we got, plus an audit log.
+#[derive(Clone, Debug, Default)]
+struct Pipeline {
+    fetched: bool,
+    committed: bool,
+    log: Vec<String>,
+}
+
+#[tokio::main]
+async fn main() -> Result<()> {
+    let checkpointer = Arc::new(InMemoryCheckpointer::<Pipeline>::new());
+
+    // A flaky `fetch`: fails (retryable model error) its first two attempts,
+    // then succeeds — a transient connectivity blip the node-retry policy
+    // absorbs on its own.
+    let fetch_attempts = Arc::new(AtomicUsize::new(0));
+    // A `commit` that stays down until the "outage" clears between runs. The
+    // outer restart, not the inner retry policy, is what recovers it.
+    let outage = Arc::new(AtomicUsize::new(0));
+
+    let fa = fetch_attempts.clone();
+    let og = outage.clone();
+    let graph = GraphBuilder::<Pipeline, Pipeline>::overwrite()
+        .add_node("fetch", move |mut state: Pipeline, _c: NodeContext| {
+            let fa = fa.clone();
+            async move {
+                let n = fa.fetch_add(1, Ordering::SeqCst);
+                if n < 2 {
+                    return Err(TinyAgentsError::Model(format!(
+                        "connection reset (fetch attempt {})",
+                        n + 1
+                    )));
+                }
+                state.fetched = true;
+                state.log.push(format!("fetched on attempt {}", n + 1));
+                Ok(NodeResult::Update(state))
+            }
+        })
+        .add_node("commit", move |mut state: Pipeline, _c: NodeContext| {
+            let og = og.clone();
+            async move {
+                // Down while the outage flag is 0; healthy once it is raised.
+                if og.load(Ordering::SeqCst) == 0 {
+                    return Err(TinyAgentsError::Tool(
+                        "commit endpoint unreachable (outage)".into(),
+                    ));
+                }
+                state.committed = true;
+                state.log.push("committed".into());
+                Ok(NodeResult::Update(state))
+            }
+        })
+        .set_entry("fetch")
+        .add_edge("fetch", "commit")
+        .set_finish("commit")
+        .compile()
+        .unwrap()
+        // 1 try + 3 retries per node, absorbing the fetch blips.
+        .with_node_retry(RetryPolicy::default().with_max_attempts(4))
+        .with_checkpointer(checkpointer.clone());
+
+    // ---- First run: fetch self-heals, but commit is fully down. ------------
+    let thread = "orders-4711";
+    let err = graph
+        .run_with_thread(thread, Pipeline::default())
+        .await
+        .expect_err("commit is down, so the first run fails");
+    println!("first run failed as expected: {err}");
+    println!(
+        "  fetch took {} attempts",
+        fetch_attempts.load(Ordering::SeqCst)
+    );
+
+    // The failure left a resumable checkpoint: `fetch`'s progress is committed
+    // and `commit` is scheduled to re-run.
+    let snapshot = graph
+        .get_state(thread, None)
+        .await?
+        .expect("a failure-boundary checkpoint exists");
+    println!(
+        "  durable state: fetched={}, committed={}",
+        snapshot.values.fetched, snapshot.values.committed
+    );
+    println!("  will re-run on resume: {:?}", snapshot.next_nodes);
+    println!("  checkpoints on thread: {}", checkpointer.count(thread));
+
+    // ---- The outage clears; restart the run from the checkpoint. -----------
+    outage.store(1, Ordering::SeqCst);
+    let resumed = graph.retry(thread).await?;
+    assert_eq!(resumed.status.status, ExecutionStatus::Completed);
+    println!("\nrestarted to completion:");
+    println!(
+        "  fetched={}, committed={}",
+        resumed.state.fetched, resumed.state.committed
+    );
+    for line in &resumed.state.log {
+        println!("  - {line}");
+    }
+
+    Ok(())
+}