Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
87 changes: 87 additions & 0 deletions src/openhuman/flows/agents/workflow_builder/builder_prompt.rs
Original file line number Diff line number Diff line change
Expand Up @@ -753,6 +753,93 @@ mod tests {
);
}

/// Regression guard for the shipped prompt bug this test was added with:
/// the standing prompt used to claim an `agent` node "can also **read and
/// write the user's memory at run time**". Both halves were false. A plain
/// `agent` node is a single completion through `OpenHumanLlm::complete`
/// (`tinyflows/caps.rs`) — no tool loop, so it can neither read nor write
/// memory. Told otherwise, the builder authored a plain agent node
/// prompted to "recall the user's preference", and the model FABRICATED
/// one: the step silently invented context instead of failing, which is
/// strictly worse than not working. The banned strings below are the exact
/// wording that produced that, so it can never be reintroduced verbatim.
#[test]
fn standing_prompt_does_not_claim_plain_agent_nodes_reach_memory() {
const STANDING_PROMPT: &str = include_str!("prompt.md");

for banned in [
"read and write the user's\n memory at run time",
"wire\n an `agent` node that uses memory",
] {
assert!(
!STANDING_PROMPT.contains(banned),
"standing prompt must not tell the builder a plain `agent` node can \
reach memory ({banned:?}) — it has no tool loop, so the model \
fabricates the recalled value instead of looking it up"
);
}

assert!(
STANDING_PROMPT.contains("A plain `agent` node has NO\n memory access"),
"standing prompt must state outright that a plain agent node has no \
memory access, so the builder never authors a no-op recall step"
);
Comment on lines +782 to +786

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 The positive assertion embeds a literal (newline + 3 spaces of indentation), tying the test to the exact current line-wrap position inside the Markdown list. If this paragraph is later reflowed, re-indented, or pulled out of the numbered list, the assertion fails even though the semantic constraint is still satisfied. The negative banned-phrase checks share the same fragility. Every other positive assertion in this test module uses a short, single-line substring that is robust to reformatting. Extracting just the key phrase — without the accidental line break — keeps the guard meaningful while surviving common edits to prompt.md.

Suggested change
assert!(
STANDING_PROMPT.contains("A plain `agent` node has NO\n memory access"),
"standing prompt must state outright that a plain agent node has no \
memory access, so the builder never authors a no-op recall step"
);
assert!(
STANDING_PROMPT.contains("A plain `agent` node has NO"),
"standing prompt must state outright that a plain agent node has no \
memory access, so the builder never authors a no-op recall step"
);

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

}

/// The two mechanisms that DO reach memory from inside a running flow must
/// both be taught, with the correct binding path for the deterministic one.
/// A native `oh:` tool result is a `ToolResult` — `{ content: [{ type,
/// text }], is_error }` — so a downstream binding dereferences
/// `.item.json.content[0].text`, not the bare `.item.json.<field>` an
/// agent/`http_request` output would use. Getting that path wrong is the
/// same class of silent-null failure the `=`-binding rules exist to stop.
#[test]
fn standing_prompt_teaches_the_two_working_memory_read_paths() {
const STANDING_PROMPT: &str = include_str!("prompt.md");

for rule in [
"oh:memory_recall",
"oh:memory_hybrid_search",
"context_scout",
"=nodes.<id>.item.json.content[0].text",
] {
assert!(
STANDING_PROMPT.contains(rule),
"standing prompt must teach `{rule}` — it is one of the only two \
mechanisms that actually read memory at flow run time, or the \
binding path needed to consume one"
);
}
}

/// Flows run on trigger data a third party can influence (an inbound
/// email, a webhook payload), so writing that into the user's personal
/// memory is deliberately not offered. `agent_memory` is NOT an escape
/// hatch here despite being a registered, `read_only` builtin: its
/// `memory_tree` tool inherits the trait-default `PermissionLevel::ReadOnly`
/// while dispatching an `ingest_document` WRITE mode, so it survives the
/// read-only tool filter in `session/builder/factory.rs` (which consults
/// the argless `permission_level()`). Steering the builder there would
/// hand prompt-injected trigger content a memory-write foothold — exactly
/// the hole `context_scout`'s own agent.toml documents refusing.
#[test]
fn standing_prompt_states_flows_cannot_write_memory_and_avoids_agent_memory() {
const STANDING_PROMPT: &str = include_str!("prompt.md");

assert!(
STANDING_PROMPT.contains("can never WRITE the user's memory"),
"standing prompt must state plainly that a workflow cannot write the \
user's memory, so the builder stops authoring remember/store steps"
);
assert!(
!STANDING_PROMPT.contains("agent_memory"),
"standing prompt must not steer the builder to `agent_memory` as a \
flow agent_ref: its `memory_tree` tool declares ReadOnly but exposes \
an ingest_document write mode, so it would give prompt-injectable \
trigger data a memory-write path"
);
}

#[test]
fn repair_includes_run_id_error_and_failing_nodes() {
let mut r = req(BuildMode::Repair);
Expand Down
34 changes: 28 additions & 6 deletions src/openhuman/flows/agents/workflow_builder/prompt.md
Original file line number Diff line number Diff line change
Expand Up @@ -325,11 +325,32 @@ A `WorkflowGraph` is `{ name?, nodes: [...], edges: [...] }`.
what makes the output-parser coerce/validate it into a real boolean rather
than a string that merely looks like one.

An `agent` node inside a workflow can also **read and write the user's
memory at run time**. If a workflow genuinely needs the user's context
(recall a preference) or should remember a result/state across runs, wire
an `agent` node that uses memory instead of hardcoding context memory
already holds. Use sparingly — only when the workflow truly needs it.
**Reading the user's memory at run time.** A plain `agent` node has NO
memory access: it is a single completion, so it cannot look anything up
and it cannot decide to. Prompting one to "recall the user's preference"
does not read memory — the model simply INVENTS an answer, and the graph
still looks correct. Never author that. Two mechanisms actually work:
- **A `tool_call` node** with `config.slug` = `oh:memory_recall` (semantic
recall) or `oh:memory_hybrid_search` (keyword/lexical lookup). One
deterministic read at a fixed point in the graph. Its output is a native
tool result, so bind downstream off
`=nodes.<id>.item.json.content[0].text` — NOT `.item.json.<field>`.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Use a valid jq path for native tool output

This example includes an array index but still uses the bare nodes shorthand. The prompt's own expression rules say paths that continue into jq syntax like [...] must start from the jq root, so builders following this will author =nodes.<id>.item.json.content[0].text, which evaluates as invalid/null instead of reading the native tool result. For memory-recall flows this makes the newly recommended path fail to feed downstream nodes; use the jq form such as =.nodes["<id>"].item.json.content[0].text.

Useful? React with 👍 / 👎.

- **`config.agent_ref` = `context_scout`** when the step needs to DECIDE
what to look up, or to look up several things across multiple steps.
That runs a real read-only agent turn with memory recall, transcript
search, and thread reads, and returns a context bundle you feed into a
following `agent` node via `input_context`.

**A workflow can never WRITE the user's memory.** There is no
remember/store step, and no `agent_ref` that grants one — a flow runs on
trigger data that a third party can influence (an inbound email, a webhook
payload), so writing that into the user's memory is deliberately not
possible. If the user asks for a workflow that "remembers" something
across runs, say plainly that memory writes are not available to workflows
yet and build the rest of the flow without that step.
Comment on lines +344 to +350

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Enforce the no-memory-write guarantee

This now states that workflows can never write memory, but that guarantee is not enforced by the runtime: a saved/imported flow can still use a native tool_call with slug: "oh:memory_store", which goes through OpenHumanTools::invoke to runtime_node::execute_tool, and the runtime registry includes MemoryStoreTool. In supervised/full autonomy, trigger-derived args can therefore still be persisted to the user's memory while the prompt/tests assume the surface is impossible; either block memory-write native tools for flows or weaken this to builder guidance.

Useful? React with 👍 / 👎.


Use memory reads sparingly — only when the workflow genuinely needs the
user's context, rather than hardcoding what memory already holds.

**Picking a specialist via `agent_ref`.** A plain `agent` node (no
`agent_ref`) only has the default LLM plus whatever it's given in
Expand All @@ -347,7 +368,8 @@ A `WorkflowGraph` is `{ name?, nodes: [...], edges: [...] }`.
`config.agent_ref` — never hallucinate an id, exactly like grounding a
`tool_call` slug via `search_tool_catalog`. Examples: "generate an HTML
report from this data" → `code_executor`; "research our competitors" →
`researcher`.
`researcher`; "work out what this customer has asked us before" →
`context_scout` (see "Reading the user's memory at run time" above).
3. **`tool_call`** — an action. Two flavours by `config.slug`:
- **Composio app action** — `config.slug` = a real action slug (from
`search_tool_catalog`, e.g. `GMAIL_SEND_EMAIL`) + `config.connection_ref`
Expand Down
Loading