Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 23 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,16 +1,23 @@
# Context Tree Value Audit

`context-tree-value-audit` 0.3.0 is an explicit-only Skill for task-first,
`context-tree-value-audit` 0.4.0 is an explicit-only Skill for task-first,
evidence-first analysis of Context Tree decision value for the current First
Tree Runtime when its native historical evidence is supported. It reconstructs
complete Tasks from authorized Chats, records whether a Tree Read is observed
or unresolved, and judges one optional Effect without a minimum sample gate.
single-Agent-owned continuous Tasks from authorized Chats, records whether a
Tree Read is observed or unresolved, and judges one optional Effect without a
minimum sample gate.

The 0.2 series renamed the installable Skill from
`context-tree-insights` to `context-tree-value-audit`. Replace the old Skill
directory during upgrade; do not install both names because they represent one
explicit audit capability, not two independent workflows.

Version 0.4 replaces the 0.3 Task judgment schema v2 with continuous episode
schema v3. Do not reuse schema-v1 or schema-v2 `task-judgments.jsonl`:
reconstruct the episodes and their ownership/objective/outcome anchors, then
rerun the deterministic reporter. No migration or compatibility reader is
provided.

The audit core remains separate from First Tree core. Codex, Claude Code, and
Claude Code TUI use their existing native local transcripts. Cursor and Kimi
Code remain unsupported for historical value audits because their existing
Expand All @@ -29,10 +36,13 @@ Tree.
Chats visible to this one current Agent. The Skill trusts that explicit
scope and never broadens it or crosses to another Agent.
- `Chat UUID @ Agent UUID` remains the authorization and trace-mapping unit;
Task is the judgment and counting unit.
- Task means one complete objective-to-outcome work item. Planning,
implementation, review, QA, corrections, and continuations for the same
deliverable remain one Task.
a single-Agent-owned continuous work episode is the Task judgment and
counting unit.
- Short continuations, status prompts, context-dependent questions, repeated
review/fix requests, and phases of one delivery do not become separate Tasks.
- Clear Tasks carry source-backed ownership, objective, and outcome anchors;
work owned by another Agent remains context until the audited Agent visibly
receives, accepts, or takes over an objective.
- Local Runtime evidence is preflighted against authorized Chat and Agent IDs
before complete recorded output is scanned.
- Missing, cleaned, ambiguous, malformed, truncated, or unsupported traces are
Expand Down Expand Up @@ -182,10 +192,12 @@ The Skill orchestrates four stages:
grammar, reconstructs exact or read-only-composite evidence plus visible
choices, and distinguishes local default-branch matches from unverified
sources. Unsupported Runtime history produces unresolved Reads.
3. The Agent reconstructs complete Tasks and writes one observed/unresolved
Read plus at most one Effect in schema-v2 `task-judgments.jsonl`.
4. `report` validates source ownership, windows, cross-Chat linkage,
Read/choice timing, deduplication, and aggregate conservation, then creates
3. The Agent reconstructs continuous Task episodes with explicit
ownership/objective/outcome anchors and writes one observed/unresolved Read
plus at most one Effect in schema-v3 `task-judgments.jsonl`.
4. `report` validates episode ownership and anchors, weak fragment-only
objectives, source ownership, windows, cross-Chat linkage, Read/choice
timing, deduplication, and aggregate conservation, then creates
`evidence.jsonl` and `REPORT.md`. An optional hash-anchored reviewed
baseline is shown separately, so a current collector gap cannot erase
previously reviewed positive cases or silently import them into the rerun.
Expand Down
56 changes: 42 additions & 14 deletions evals/manual-behavior-checklist.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Use this checklist before admitting a `context-tree-value-audit` revision. Run i
in designated First Tree Agent workspaces against disposable or sanitized
records. Never commit generated artifacts.

For a 0.3.0 upgrade, confirm the installed payload exposes only
For a 0.4.0 upgrade, confirm the installed payload exposes only
`$context-tree-value-audit`; the superseded `$context-tree-insights` directory
must not remain as a second callable Skill. Exercise the README's minimal
move/copy/compare flow and confirm the old payload is outside both Skill
Expand Down Expand Up @@ -170,19 +170,42 @@ Pass when:
- malformed receipt is omitted with `context_decision_invalid`;
- no malformed receipt blocks the message, Chat export, or audit.

## 5. Complete Task reconstruction
## 5. Continuous Task reconstruction

Use one Chat containing one objective across planning, implementation, review,
QA, correction, and final delivery, followed by a genuinely independent
objective and outcome.
objective and outcome. Include short continuation/status prompts; polite or
deictic variants such as `please continue`, `status please`, `修一下吧`,
`please continue fixing it`, `please fix it`, `continue the work`,
`继续修一下`, `修复一下`, `修这个`, and `帮忙修下`; edge-decorated forms
such as `“please continue”`, `please continue 🙏`, and `请继续(谢谢)`;
one or more leading Agent mentions; and concrete objectives that contain
`continue` while naming an independent scope and deliverable.

Pass when:

- all phases and continuations for the first deliverable remain one Task;
- it produces two Task rows with separate source messages and windows;
- clear Tasks have objective, object scope, and outcome;
- a Task missing those boundaries is marked excluded;
- excluded Tasks contain no Read or Effect judgment;
- clear Tasks have audited-Agent ownership, objective, material object scope,
independent outcome, bounded source fragments, one primary deliverable, and
explicit ownership/objective/outcome anchors;
- weak prompt variants and weak normalized objectives are merged or excluded,
while concrete `continue` objectives remain eligible;
- a weak assignment cannot borrow another sender's concrete objective, and one
Agent message cannot simultaneously serve as objective and terminal outcome;
- an earlier weak objective anchor cannot admit a Read that precedes the
earliest ownership-compatible concrete objective anchor;
- an earlier ownership anchor from a sender incompatible with the ownership
kind cannot admit a Read before compatible ownership is established;
- every outcome anchor is a non-empty current-Agent message, so a later human
follow-up cannot extend the episode or become the bound Effect outcome;
- work owned by another Agent remains context until a visible assignment,
transfer, or acceptance;
- a candidate missing any clear-Task gate is excluded with the deterministic
structured exclusion kind;
- excluded candidates contain no episode, Read, or Effect judgment;
- Reads and choices outside the established episode are rejected;
- ownership/objective/outcome anchors copied across Tasks are rejected;
- one Read or choice copied into both Tasks is rejected.

## 6. Cross-Chat handoff merge
Expand All @@ -207,7 +230,8 @@ Pass when:
- the second is `unresolved` with a reason;
- unresolved Read has no Read IDs and a null Effect;
- no `not_observed`, `unused`, or negative-value state is emitted;
- Reads outside the Task window or source Chats are rejected;
- Reads outside the established episode, Task window, or source Chats are
rejected;
- unresolved Tasks appear in coverage counts, never an unused denominator.

## 8. Optional Effect
Expand All @@ -220,12 +244,14 @@ Pass when:
- unknown Effects, multiple Effects, confidence tiers, and numeric weights are
rejected;
- every Effect has observed Task Reads, later same-Agent choices, an outcome
anchor, and a summary;
anchor bound to the episode outcome, and a summary;
- the bound Effect outcome anchor is not earlier than any cited Read completion
or choice;
- post-choice Reads and out-of-window choices are rejected;
- a null Effect requires one short `effect_reason`;
- a decision receipt alone does not create an Effect;
- schema-v1 task types, sampling fields, `verified` / `probable`, rubrics, and
support fields are rejected;
- superseded task types, sampling fields, `verified` / `probable`, rubrics,
and support fields are rejected;
- there is at most one Effect per Task.

## 9. Sample handling
Expand All @@ -244,13 +270,15 @@ Pass when:

Pass when the report includes:

- clear and excluded Tasks;
- a complete clear Task inventory with ownership, objective, scope, primary
deliverable, outcome, Read, Effect, and evidence summary;
- every clear Task's ownership/objective/outcome anchors and boundary rationale;
- a complete excluded-candidate inventory with exclusion kind, observed scope,
and reason;
- observed and unresolved Read Tasks;
- Effect Tasks and observed Reads without an Effect;
- the four-Effect distribution;
- every clear Task's Read and Effect result;
- every excluded Task's reason;
- authorized Chat/message/trace coverage and gaps.
- authorized Chat/message/trace coverage and gaps;
- the four-class in-window Tree-read attempt conservation table.

Verify:
Expand Down
55 changes: 36 additions & 19 deletions skills/context-tree-value-audit/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,10 @@ description: Audit how Context Tree reads affected complete work Tasks when a hu

## Capability

Run a manual and read-only retrospective that reconstructs complete Tasks from
authorized Chats, determines whether a Tree Read is observed or unresolved,
and reports whether the Read reasonably confirmed, constrained, redirected, or
conflicted with the later choice.
Run a manual and read-only retrospective that reconstructs single-Agent-owned
continuous Tasks from authorized Chats, determines whether a Tree Read is
observed or unresolved, and reports whether the Read reasonably confirmed,
constrained, redirected, or conflicted with the later choice.

Keep three responsibilities separate:

Expand All @@ -20,9 +20,10 @@ Keep three responsibilities separate:
- the bundled script performs deterministic collection, reference validation,
deduplication, conservation checks, and reporting.

The collector establishes what records exist. The Agent performs semantic Task
reconstruction and passage-to-choice judgment. A Read or decision receipt is
evidence, not server-verified causality.
The collector establishes what records exist. The Agent reconstructs
single-Agent-owned continuous work episodes and performs passage-to-choice
judgment. A Read or decision receipt is evidence, not server-verified
causality.

## Gate the run

Expand Down Expand Up @@ -164,14 +165,26 @@ Read
[references/task-analysis-schema.md](references/task-analysis-schema.md), then
write exactly one `task-judgments.jsonl` row for every reconstructed Task.

A clear Task needs a concrete objective, object scope, outcome, Task window,
and source fragments. Treat the whole objective-to-outcome work item as one
Task: planning, implementation, review, QA, corrections, status questions, and
short continuations for the same deliverable stay together. Split only when a
new objective has a materially different scope and an independently judgeable
outcome. Otherwise mark the candidate excluded. Excluded Tasks carry no Read
A clear Task is one independently judgeable continuous work episode owned by
the audited Agent. It needs a concrete objective, material object scope,
independently judgeable outcome or terminal state, bounded source fragments
from objective through outcome, explicit ownership/objective/outcome anchors,
and one primary terminal deliverable. Otherwise mark the candidate excluded
with a structured exclusion kind. Excluded candidates carry no episode, Read,
or Effect judgment.

Short continuations, status prompts, context-dependent questions, merge
approval, repeated review/fix requests, and ordinary phase transitions are not
separate Tasks. Merge plan → implementation → review → QA → final delivery,
plus corrections to the same deliverable, into one episode. Split only when
there is a new objective, material scope or deliverable change, independent
outcome, and unambiguous source boundary.

For a single-Agent audit, another Agent's work is context until this Agent is
visibly assigned, transferred, or accepts an objective. A later independent
review, takeover, verification gate, or orchestration objective may form a new
owned episode only when it passes every clear-Task gate.

One Chat may contain multiple Tasks. Merge across Chats only for one PR/MR/
Issue, a visible handoff, or the same objective and primary delivery, and
record the explicit shared linkage. Do not copy one read or choice into
Expand Down Expand Up @@ -219,18 +232,22 @@ python3 "$CTVA_SKILL_DIR/scripts/context_tree_value_audit.py" report \
--report-output "$CTVA_ARTIFACT_DIR/REPORT.md"
```

Optionally supply a v2 `--reviewed-baseline` when an independently reviewed
Optionally supply a v3 `--reviewed-baseline` when an independently reviewed
earlier case set exists. The reporter keeps its hash-anchored Task and Effect
counts separate from the current rerun.

The deterministic reporter rejects v0.2 judgment fields, unauthorized source
messages, Task-window violations, unlinked cross-Chat merges, duplicated
Reads/choices, invalid Effects, missing outcome anchors, and non-conserving
aggregates.
The deterministic reporter rejects Task judgment schemas v1 and v2, weak
fragment-only
objectives, missing or invalid episode ownership and anchors, reused episode
identity anchors, Reads or choices outside the established episode,
unauthorized source messages, Task-window violations, unlinked cross-Chat
merges, duplicated Reads/choices, invalid Effects, unbound outcome anchors,
v0.2 judgment fields, and non-conserving aggregates.

The report must include:

- clear and excluded Tasks;
- a complete inventory of clear Tasks with boundary rationale and excluded
candidates with structured exclusion reasons;
- observed and unresolved Read Tasks;
- Effect Tasks and observed Reads without an Effect;
- the four-effect distribution;
Expand Down
2 changes: 1 addition & 1 deletion skills/context-tree-value-audit/VERSION
Original file line number Diff line number Diff line change
@@ -1 +1 @@
0.3.0
0.4.0
4 changes: 2 additions & 2 deletions skills/context-tree-value-audit/agents/openai.yaml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
interface:
display_name: "Context Tree Value Audit"
short_description: "Audit Task, Read, and Effect evidence"
default_prompt: "Use $context-tree-value-audit to audit how Context Tree reads affected complete Tasks for this Agent, using the available authorized sample."
short_description: "Audit continuous Task, Read, and Effect evidence"
default_prompt: "Use $context-tree-value-audit to audit how Context Tree reads affected single-Agent-owned continuous Tasks, using the available authorized sample."
policy:
allow_implicit_invocation: false
Loading
Loading