Skip to content

Commit 0888719

Browse files
graycyrusgraycyrus
committed
Merge branch 'tf-wt-4'
2 parents d0c1a37 + 4df1ad2 commit 0888719

8 files changed

Lines changed: 359 additions & 0 deletions

File tree

‎wiki/.gitkeep‎

Whitespace-only changes.

‎wiki/Architecture.md‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
# Architecture
2+
3+
tinyflows takes a declarative workflow definition and runs it through a small,
4+
fixed pipeline.
5+
6+
```
7+
WorkflowGraph → validate → compile → engine::run
8+
(typed graph) (structural) (lowers) (drives to completion,
9+
lowered onto tinyagents)
10+
▲
11+
caps traits (LlmProvider / ToolInvoker / HttpClient /
12+
CodeRunner / StateStore) — host-injected, captured per run
13+
```
14+
15+
1. **`WorkflowGraph`** — the serializable source of truth: typed `Node`s and
16+
`Edge`s. JSON is the wire format.
17+
2. **`validate`** — structural checks (unique ids, exactly one trigger, edges
18+
reference existing nodes), run before compilation.
19+
3. **`compile`** — validates and produces a `CompiledWorkflow`.
20+
4. **`engine::run`** — lowers the compiled graph onto the
21+
[`tinyagents`](https://crates.io/crates/tinyagents) state-graph engine and
22+
drives it to completion, returning a `RunOutcome`.
23+
24+
## Per-run lowering
25+
26+
Lowering happens **per run**, inside `engine::run`. Each node becomes a
27+
`tinyagents` handler that captures that run's host `Capabilities`, so the graph
28+
built for one run carries exactly the caps handed to it. The engine wires:
29+
30+
- **Linear** paths (one successor per node).
31+
- **Conditional branching** (successors on distinct ports; the taken port is
32+
recorded into state and routed on).
33+
- **Parallel fan-out** (multiple successors sharing one port run concurrently via
34+
a `Command::goto`).
35+
- **Fan-in barrier** (a node with more than one predecessor is wired with waiting
36+
edges so it runs only once all predecessors finish — the `merge` barrier).
37+
38+
## State layout
39+
40+
Run state is a single `serde_json::Value`:
41+
42+
```json
43+
{
44+
"run": { "trigger": { /* trigger payload */ } },
45+
"nodes": { "<id>": { "items": [ /* Item… */ ], "port": "true" } }
46+
}
47+
```
48+
49+
Data flowing on a connection is an **array of items**, not a single value. Each
50+
`Item` is `{ json, binary?, paired_item? }`; a node maps its logic over its input
51+
items and emits output items. A merge reducer folds each node's partial
52+
`{ nodes: { id: { items } } }` update into the shared state — because every node
53+
writes under its own id, independent updates never collide, which keeps parallel
54+
fan-out correct. Field references use `=`-prefixed expressions.
55+
56+
## Host-agnostic seam
57+
58+
The crate never hard-codes an LLM, tool, HTTP, code, or persistence vendor.
59+
Anything touching the outside world goes through a **capability trait** bundled in
60+
`Capabilities` and injected by the host — see [Capability Traits](Capability-Traits).
61+
62+
## Deeper reading
63+
64+
- [`docs/01-architecture.md`](../blob/main/docs/01-architecture.md)
65+
- [`docs/04-execution-engine.md`](../blob/main/docs/04-execution-engine.md)

‎wiki/Capability-Traits.md‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
# Capability Traits
2+
3+
tinyflows is **host-agnostic**: the crate never hard-codes an LLM, tool, HTTP,
4+
code, or persistence vendor. Everything that touches the outside world is
5+
expressed as a trait the **embedding host implements**. This is the core design
6+
constraint — new outside-world effects go through a capability trait, not a direct
7+
dependency.
8+
9+
The host constructs these implementations and hands them to the engine per run.
10+
Deterministic, in-memory mocks ship behind the `mock` feature
11+
(`caps::mock::mock_capabilities()`) so workflows run end-to-end in tests and
12+
examples without any real backend.
13+
14+
## The traits
15+
16+
| Trait | Used by | Purpose |
17+
|-------|---------|---------|
18+
| `LlmProvider` | `agent`, `output_parser` | Runs a single LLM completion from a JSON request, returning JSON. |
19+
| `ToolInvoker` | `tool_call` | Invokes a named integration action (`slug` + `args`), returning its output. |
20+
| `HttpClient` | `http_request` | Issues an outbound HTTP request described by JSON, returning the response. |
21+
| `CodeRunner` | `code` | Executes sandboxed user code (`CodeLanguage::JavaScript` / `Python`) with a JSON input. |
22+
| `StateStore` | resumable / stateful workflows | Durable key/value state (`load` / `store`) for a run. |
23+
24+
## Connection references
25+
26+
`LlmProvider`, `ToolInvoker`, and `HttpClient` each take an optional `conn: Option<&str>`
27+
— an **opaque, host-managed connection reference** (e.g. a credential or Composio
28+
connection id) that names the account a call acts as. The crate never sees real
29+
secrets; the host resolves the reference to credentials inside its implementation.
30+
31+
## The `Capabilities` bundle
32+
33+
The engine receives a `Capabilities` struct — the per-run bundle of host
34+
implementations. It currently wires four: `llm`, `tools`, `http`, and `code`
35+
(each an `Arc<dyn Trait>`). `StateStore` is defined for durable, resumable state
36+
and is implemented by hosts that need it.
37+
38+
## Deeper reading
39+
40+
- [`docs/05-capability-traits.md`](../blob/main/docs/05-capability-traits.md)
41+
- [`docs/15-credentials-and-connections.md`](../blob/main/docs/15-credentials-and-connections.md)

‎wiki/Getting-Started.md‎

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
# Getting Started
2+
3+
## Install
4+
5+
tinyflows is a library crate. You need **Rust 1.85 or newer** (edition 2024);
6+
install it with [rustup](https://rustup.rs/).
7+
8+
Add the dependency to your `Cargo.toml`:
9+
10+
```toml
11+
[dependencies]
12+
tinyflows = "0.1"
13+
```
14+
15+
The crate is **host-agnostic**: outside-world effects go through capability
16+
traits you implement (see [Capability Traits](Capability-Traits)). For tests and
17+
examples, enable the `mock` feature to get deterministic, in-memory
18+
implementations via `caps::mock::mock_capabilities()`:
19+
20+
```toml
21+
[dev-dependencies]
22+
tinyflows = { version = "0.1", features = ["mock"] }
23+
```
24+
25+
## Quickstart
26+
27+
Build a two-node graph (`trigger → transform`), compile it, and run it against
28+
the mock capabilities. This mirrors the crate's `examples/hello_workflow.rs`:
29+
30+
```rust
31+
use serde_json::{Value, json};
32+
use tinyflows::caps::mock::mock_capabilities;
33+
use tinyflows::compiler::compile;
34+
use tinyflows::engine::run;
35+
use tinyflows::model::{Edge, Node, NodeKind, WorkflowGraph};
36+
37+
#[tokio::main(flavor = "current_thread")]
38+
async fn main() {
39+
let graph = WorkflowGraph {
40+
nodes: vec![
41+
Node {
42+
id: "t".into(),
43+
kind: NodeKind::Trigger,
44+
type_version: 1,
45+
name: "start".into(),
46+
config: Value::Null,
47+
ports: vec![],
48+
position: None,
49+
},
50+
Node {
51+
id: "greet".into(),
52+
kind: NodeKind::Transform,
53+
type_version: 1,
54+
name: "greet".into(),
55+
config: json!({ "set": { "greeting": "=item.name" } }),
56+
ports: vec![],
57+
position: None,
58+
},
59+
],
60+
edges: vec![Edge {
61+
from_node: "t".into(),
62+
from_port: "main".into(),
63+
to_node: "greet".into(),
64+
to_port: "main".into(),
65+
}],
66+
..Default::default()
67+
};
68+
69+
let compiled = compile(&graph).expect("compile");
70+
let outcome = run(&compiled, json!({ "name": "Ada" }), &mock_capabilities())
71+
.await
72+
.expect("run");
73+
println!("{}", serde_json::to_string_pretty(&outcome.output).unwrap());
74+
}
75+
```
76+
77+
Notes:
78+
79+
- The `config` value on a `transform` node uses `=`-prefixed expressions (an
80+
interim `=`-dotted-path form ships today).
81+
- `compile` runs structural validation and lowers the graph into a
82+
`CompiledWorkflow`; `run` drives it and returns a `RunOutcome` whose `output`
83+
is the final run state (`{ run, nodes: { id: { items } } }`).
84+
85+
## Run the example
86+
87+
The crate ships the same program as a runnable example:
88+
89+
```bash
90+
cargo run --example hello_workflow --features mock
91+
```
92+
93+
Next: read [Architecture](Architecture) for how the pipeline works, or the
94+
[Node Catalog](Node-Catalog) for the full node vocabulary.

‎wiki/Home.md‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
# tinyflows
2+
3+
**tinyflows** is a Rust-native, host-agnostic workflow automation engine, shipped
4+
as a library crate. A workflow is a directed graph of typed nodes
5+
(`WorkflowGraph`) that is validated, compiled, and lowered per-run onto the
6+
[`tinyagents`](https://crates.io/crates/tinyagents) state-graph engine, then
7+
driven to completion by `engine::run`.
8+
9+
Rust 2024 · MSRV 1.85 · `#![forbid(unsafe_code)]` · GPL-3.0-or-later.
10+
11+
## Highlights
12+
13+
- **Typed graph model.** Workflows are `WorkflowGraph`s of typed nodes and edges;
14+
JSON is the wire format, with `schema_version` / `type_version` axes and
15+
load-time migration.
16+
- **Host-agnostic capabilities.** Everything that touches the outside world —
17+
LLMs, integration tools, HTTP, code execution, durable state — is expressed as
18+
a capability trait the embedding host implements. Deterministic mocks ship
19+
behind the `mock` feature for tests and examples.
20+
- **11 node kinds + a trigger.** Native control flow (`condition`, `switch`,
21+
`merge`, `split_out`, `transform`) and capability-backed effects (`agent`,
22+
`tool_call`, `http_request`, `code`, `output_parser`, `sub_workflow`).
23+
- **Real routing.** Linear paths, conditional branching, parallel fan-out, and a
24+
fan-in merge barrier.
25+
- **Item-based data flow.** State is a `serde_json::Value` laid out as
26+
`{ run, nodes: { id: { items } } }`; data on a connection is an array of
27+
`Item { json, binary?, paired_item? }`, with `=`-prefixed expressions.
28+
- **Resilience & observability.** Per-node error handling (`on_error`
29+
stop/continue/route, `retry`, `error` port), `tracing` + `RunObserver`
30+
records, and human-in-the-loop approval gating (`requires_approval` →
31+
`RunOutcome::pending_approvals` + `engine::resume`).
32+
33+
## Pages
34+
35+
- [Getting Started](Getting-Started) — install and a runnable quickstart.
36+
- [Architecture](Architecture) — the compile-and-run pipeline and state layout.
37+
- [Node Catalog](Node-Catalog) — every node kind at a glance.
38+
- [Capability Traits](Capability-Traits) — the host-injected seam.
39+
- [Roadmap and Status](Roadmap-and-Status) — what's done and what's ahead.
40+
41+
## Going deeper
42+
43+
The full design specs live in the repository's
44+
[`docs/`](../blob/main/docs/README.md) directory (start with the index). For a
45+
quickstart and project layout, see the repo
46+
[`README.md`](../blob/main/README.md).

‎wiki/Node-Catalog.md‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
# Node Catalog
2+
3+
Every node has a `NodeKind`. Kind-specific settings live in the node's `config`
4+
(free-form JSON, validated per kind). Ports carry the item arrays between nodes;
5+
the default port is `main`.
6+
7+
## Trigger
8+
9+
Exactly one per workflow — the graph's entry node. Its firing mode is a
10+
`TriggerKind` in config (`manual`, `schedule`, `webhook`, `app_event`, `form`,
11+
`execute_by_workflow`, `chat_message`, `evaluation`, `system`). The host actually
12+
fires it; tinyflows injects the trigger payload as the initial run state.
13+
14+
| Node | Purpose | Ports / config gist |
15+
|------|---------|---------------------|
16+
| `trigger` | Entry node that starts the run | Out `main`; config `trigger_kind` |
17+
18+
## Control-flow nodes (native)
19+
20+
Native routing logic — no host capabilities required.
21+
22+
| Node | Purpose | Ports / config gist |
23+
|------|---------|---------------------|
24+
| `condition` | Two-way IF branch | Out `true` / `false`; config: boolean expression |
25+
| `switch` | Multi-way branch keyed by an expression | Out one port per case (+ optional `default`); config `expression`, `cases` |
26+
| `merge` | Fan-in barrier combining multiple inputs | Waits for all wired inputs; config `mode` (e.g. `append`) |
27+
| `split_out` | Fan-out: one item per element of a list | Downstream runs per item; config `path` |
28+
| `transform` | Pure, expression-based field mapping | Config `set` (field → `=`-expression map) |
29+
30+
## Capability-backed nodes
31+
32+
Reach the outside world through the host-injected [capability
33+
traits](Capability-Traits).
34+
35+
| Node | Purpose | Ports / config gist |
36+
|------|---------|---------------------|
37+
| `agent` | Runs an LLM agent turn | Sub-ports `chat_model` / `memory` / `tool` / `output_parser`; config `prompt`, `model`, … — via `LlmProvider` |
38+
| `tool_call` | Invokes one specific integration action | Config `slug`, `args` — via `ToolInvoker` |
39+
| `http_request` | Outbound HTTP request | Config `method`, `url`, `headers`, `query`, `body` — via `HttpClient` |
40+
| `code` | Runs sandboxed user code | Config `language` (`javascript`/`python`), `source` — via `CodeRunner` |
41+
| `output_parser` | Parses/validates an agent's output into a structured shape | May use `LlmProvider` for auto-fixing; can nest as a sub-agent |
42+
| `sub_workflow` | Runs another workflow as a nested sub-graph | Config `workflow_id`, `input` mapping |
43+
44+
All 11 node kinds plus the trigger are implemented and dispatched by the engine.
45+
Per-node error handling (`on_error` stop/continue/route, `retry`, an `error`
46+
port) and approval gating (`requires_approval`) are configured through the same
47+
free-form `config`.
48+
49+
See [`docs/03-node-catalog.md`](../blob/main/docs/03-node-catalog.md) for the full
50+
catalog, and [`docs/06-triggers.md`](../blob/main/docs/06-triggers.md) for trigger
51+
kinds.

‎wiki/Roadmap-and-Status.md‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
# Roadmap and Status
2+
3+
tinyflows has a **working runtime**. The public API runs end-to-end against the
4+
mock capabilities, and `cargo publish --dry-run` is clean.
5+
6+
Rust 2024 · MSRV 1.85 · `#![forbid(unsafe_code)]` · GPL-3.0-or-later.
7+
8+
## Implemented
9+
10+
- **Pipeline** — `WorkflowGraph → validate → compile → engine::run`, lowered
11+
per-run onto the `tinyagents` state-graph engine.
12+
- **All 11 node kinds + trigger** — control flow (`condition`, `switch`, `merge`,
13+
`split_out`, `transform`) and capability-backed (`agent`, `tool_call`,
14+
`http_request`, `code`, `output_parser`, `sub_workflow`).
15+
- **Routing** — linear paths, conditional branching, parallel fan-out, and a
16+
fan-in merge barrier.
17+
- **Item-based data flow** — state as `{ run, nodes: { id: { items } } }` with
18+
`Item { json, binary?, paired_item? }` and `=`-prefixed expressions.
19+
- **Per-node error handling** — `on_error` stop/continue/route, `retry`, and an
20+
`error` port.
21+
- **Observability** — `tracing` plus a `RunObserver` and `Run` / `ExecutionStep`
22+
records.
23+
- **Human-in-the-loop** — approval gating (`requires_approval` →
24+
`RunOutcome::pending_approvals`) with `engine::resume`.
25+
- **Credentials** — opaque, host-managed `connection_ref`s (the crate never sees
26+
secrets).
27+
- **Versioning** — `schema_version` / `type_version` axes plus load-time
28+
`migrate`.
29+
30+
## Not yet
31+
32+
Being honest about what's ahead:
33+
34+
- **Full expression engine** — a full jq/jaq (or templating) expression library.
35+
A minimal `=`-dotted-path interim ships today.
36+
- **Retry timing** — backoff and per-node timeouts (retries currently re-attempt
37+
without a delay, keeping the crate runtime-agnostic).
38+
- **Durable replay** — checkpointed super-step replay for resumable runs (resume
39+
is currently deterministic re-execution).
40+
- **Authoring surfaces** — visual canvas and agent-first chat authoring
41+
(host-side).
42+
- **OpenHuman host integration** — the first downstream host (a separate repo).
43+
- **crates.io publish** — the crate is not yet published.
44+
45+
## Deeper reading
46+
47+
- [`docs/08-roadmap.md`](../blob/main/docs/08-roadmap.md) — staged delivery
48+
(A0–A5 / B0–B5).
49+
- [`docs/19-feature-matrix.md`](../blob/main/docs/19-feature-matrix.md) — feature
50+
status matrix.

‎wiki/_Sidebar.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
### tinyflows
2+
3+
- [Home](Home)
4+
- [Getting Started](Getting-Started)
5+
- [Architecture](Architecture)
6+
- [Node Catalog](Node-Catalog)
7+
- [Capability Traits](Capability-Traits)
8+
- [Roadmap and Status](Roadmap-and-Status)
9+
10+
---
11+
12+
Deep design specs: [`docs/`](../blob/main/docs/README.md)

0 commit comments

Comments
 (0)