Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -12,3 +12,8 @@ __pycache__/
/candidates.jsonl
/chats.jsonl
/judgments.jsonl
/task-source.jsonl
/task-inventory-draft.jsonl
/task-inventory.jsonl
/read-attributions.jsonl
/effect-judgments.jsonl
54 changes: 31 additions & 23 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,22 +1,22 @@
# Context Tree Value Audit

`context-tree-value-audit` 0.4.0 is an explicit-only Skill for task-first,
`context-tree-value-audit` 0.5.0 is an explicit-only Skill for task-first,
evidence-first analysis of Context Tree decision value for the current First
Tree Runtime when its native historical evidence is supported. It reconstructs
single-Agent-owned continuous Tasks from authorized Chats, records whether a
Tree Read is observed or unresolved, and judges one optional Effect without a
minimum sample gate.
Tree Read is observed or unresolved, and judges zero or more independent
Effects without a minimum sample gate.

The 0.2 series renamed the installable Skill from
`context-tree-insights` to `context-tree-value-audit`. Replace the old Skill
directory during upgrade; do not install both names because they represent one
explicit audit capability, not two independent workflows.

Version 0.4 replaces the 0.3 Task judgment schema v2 with continuous episode
schema v3. Do not reuse schema-v1 or schema-v2 `task-judgments.jsonl`:
reconstruct the episodes and their ownership/objective/outcome anchors, then
rerun the deterministic reporter. No migration or compatibility reader is
provided.
Version 0.5 replaces the combined schema-v3 Task judgment with three ordered
schema-v4 artifacts: frozen Task inventory, Read attributions, and Effect
judgments. Do not reuse schema-v1/v2/v3 judgment files. Reconstruct Tasks from
the message-only projection, freeze the inventory digest, then rerun Read and
Effect analysis. No migration or compatibility reader is provided.

The audit core remains separate from First Tree core. Codex, Claude Code, and
Claude Code TUI use their existing native local transcripts. Cursor and Kimi
Expand All @@ -40,9 +40,14 @@ Tree.
counting unit.
- Short continuations, status prompts, context-dependent questions, repeated
review/fix requests, and phases of one delivery do not become separate Tasks.
- Clear Tasks carry source-backed ownership, objective, and outcome anchors;
work owned by another Agent remains context until the audited Agent visibly
receives, accepts, or takes over an objective.
- Clear Tasks carry source-backed objective and same-Agent outcome messages;
work owned by another Agent remains context until the audited Agent receives
or visibly accepts an objective. That acceptance is an analyst judgment from
work messages; the reporter does not infer the addressee of arbitrary prose.
- Task reconstruction uses a message-only projection that excludes
collector-derived Reads, passages, Tree-mention indexes, receipts, choice
projections, and Effect judgments; original work-message content remains
unchanged. Its normalized inventory is frozen before Read attribution.
- Local Runtime evidence is preflighted against authorized Chat and Agent IDs
before complete recorded output is scanned.
- Missing, cleaned, ambiguous, malformed, truncated, or unsupported traces are
Expand All @@ -59,8 +64,8 @@ Tree.
duplicate, pending, or out-of-window results stay unresolved.
- Read is only `observed` or `unresolved`; unresolved is never counted as
unused.
- Effect is optional and only `confirmed`, `constrained`, `redirected`, or
`conflicted`.
- Each Task has zero or more Effects; each is only `confirmed`, `constrained`,
`redirected`, or `conflicted`.
- A decision receipt may support an Effect but cannot create one by itself.
- There is no fixed Task quota, task-type gate, or saturation state.
- The output is a sampled evidence report, not causal proof, ROI, or a
Expand Down Expand Up @@ -184,23 +189,26 @@ tasks. Pin a reviewed commit or release when installing for another Agent.

## Pipeline

The Skill orchestrates four stages:
The Skill orchestrates six operational steps across three analysis stages:

1. `export-chats` resolves explicit authorization and exports visible records.
2. `collect` maps authorized Chats to supported native local evidence,
classifies every in-window Tree-read attempt into a conserving four-state
grammar, reconstructs exact or read-only-composite evidence plus visible
choices, and distinguishes local default-branch matches from unverified
sources. Unsupported Runtime history produces unresolved Reads.
3. The Agent reconstructs continuous Task episodes with explicit
ownership/objective/outcome anchors and writes one observed/unresolved Read
plus at most one Effect in schema-v3 `task-judgments.jsonl`.
4. `report` validates episode ownership and anchors, weak fragment-only
objectives, source ownership, windows, cross-Chat linkage, Read/choice
timing, deduplication, and aggregate conservation, then creates
`evidence.jsonl` and `REPORT.md`. An optional hash-anchored reviewed
baseline is shown separately, so a current collector gap cannot erase
previously reviewed positive cases or silently import them into the rerun.
3. `task-source` removes every collector-derived Tree/Read/Effect projection
and keeps only authorized work messages. It does not redact literal
Tree/Read/Effect discussion from the work itself.
4. The Agent reconstructs Tasks from that projection; `freeze-tasks` validates
and writes a digest-bound `task-inventory.jsonl`.
5. The Agent writes digest-bound Read attributions, then zero-or-more Effect
judgments for each clear Task without changing the frozen inventory.
6. `report` validates Task sources, weak fragment-only objectives, source
ownership, windows, cross-Chat linkage, Read/choice timing, inventory
digests, deduplication, multi-Effect conservation, and creates
`evidence.jsonl` and `REPORT.md`. An optional hash-anchored reviewed baseline
remains separate from the current rerun.

There is no default time window. `--days` is an optional data-acquisition
bound. Every available Task in the authorized bound is reported; sample size
Expand Down
77 changes: 43 additions & 34 deletions evals/manual-behavior-checklist.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Use this checklist before admitting a `context-tree-value-audit` revision. Run i
in designated First Tree Agent workspaces against disposable or sanitized
records. Never commit generated artifacts.

For a 0.4.0 upgrade, confirm the installed payload exposes only
For a 0.5.0 upgrade, confirm the installed payload exposes only
`$context-tree-value-audit`; the superseded `$context-tree-insights` directory
must not remain as a second callable Skill. Exercise the README's minimal
move/copy/compare flow and confirm the old payload is outside both Skill
Expand Down Expand Up @@ -170,7 +170,7 @@ Pass when:
- malformed receipt is omitted with `context_decision_invalid`;
- no malformed receipt blocks the message, Chat export, or audit.

## 5. Continuous Task reconstruction
## 5. Pure Task reconstruction and freeze

Use one Chat containing one objective across planning, implementation, review,
QA, correction, and final delivery, followed by a genuinely independent
Expand All @@ -184,29 +184,32 @@ one or more leading Agent mentions; and concrete objectives that contain

Pass when:

- `task-source.jsonl` contains authorized work messages but no
collector-derived Reads, passages, Tree-mention indexes, decision receipts,
choice projections, or Effect judgments;
- original work-message content remains byte-for-byte intact even when the
work itself discusses Tree, Read, or Effect, and those literal terms are not
treated as audit evidence during Task reconstruction;
- all phases and continuations for the first deliverable remain one Task;
- it produces two Task rows with separate source messages and windows;
- clear Tasks have audited-Agent ownership, objective, material object scope,
independent outcome, bounded source fragments, one primary deliverable, and
explicit ownership/objective/outcome anchors;
- clear Tasks have a concrete objective, independent same-Agent outcome,
bounded source fragments, and explicit objective/outcome sources;
- material scope and primary deliverable may clarify the boundary but are not
required for research, diagnosis, review, or decision Tasks;
- weak prompt variants and weak normalized objectives are merged or excluded,
while concrete `continue` objectives remain eligible;
- a weak assignment cannot borrow another sender's concrete objective, and one
Agent message cannot simultaneously serve as objective and terminal outcome;
- an earlier weak objective anchor cannot admit a Read that precedes the
earliest ownership-compatible concrete objective anchor;
- an earlier ownership anchor from a sender incompatible with the ownership
kind cannot admit a Read before compatible ownership is established;
- every outcome anchor is a non-empty current-Agent message, so a later human
- one Agent message cannot simultaneously serve as objective and terminal
outcome;
- every outcome source is a non-empty current-Agent message, so a later human
follow-up cannot extend the episode or become the bound Effect outcome;
- work owned by another Agent remains context until a visible assignment,
transfer, or acceptance;
- work owned by another Agent remains context until the audited Agent receives
or visibly accepts an objective;
- a candidate missing any clear-Task gate is excluded with the deterministic
structured exclusion kind;
- excluded candidates contain no episode, Read, or Effect judgment;
- Reads and choices outside the established episode are rejected;
- ownership/objective/outcome anchors copied across Tasks are rejected;
- one Read or choice copied into both Tasks is rejected.
- excluded candidates contain no clear-Task sources, Read, or Effect judgment;
- `freeze-tasks` writes one normalized inventory digest to every row;
- changing any frozen Task row invalidates the digest;
- Read/Effect fields in a Task inventory draft are rejected.

## 6. Cross-Chat handoff merge

Expand All @@ -228,31 +231,36 @@ Pass when:

- the first is `observed` with valid Read IDs;
- the second is `unresolved` with a reason;
- unresolved Read has no Read IDs and a null Effect;
- unresolved Read has no Read IDs and an empty Effect list;
- no `not_observed`, `unused`, or negative-value state is emitted;
- Reads outside the established episode, Task window, or source Chats are
- Reads outside the frozen Task window or source Chats are
rejected;
- one Read cannot be attributed to different Tasks;
- unresolved Tasks appear in coverage counts, never an unused denominator.

## 8. Optional Effect
## 8. Zero-or-more Effects

Prepare valid examples of `confirmed`, `constrained`, `redirected`, and
`conflicted`, plus observed-Read Tasks with no Effect.
`conflicted`, including one Task with two distinct Effects supported by the
same Read, plus observed-Read Tasks with no Effect.

Pass when:

- unknown Effects, multiple Effects, confidence tiers, and numeric weights are
rejected;
- unknown Effects, confidence tiers, and numeric weights are rejected;
- one Task can carry multiple Effects and the report separately counts Effect
Tasks and total Effects;
- every Effect has observed Task Reads, later same-Agent choices, an outcome
anchor bound to the episode outcome, and a summary;
- the bound Effect outcome anchor is not earlier than any cited Read completion
message inside the Task, and a summary;
- the Effect outcome message is not earlier than any cited Read completion
or choice;
- post-choice Reads and out-of-window choices are rejected;
- a null Effect requires one short `effect_reason`;
- an empty Effect list requires one short `effect_reason`;
- a decision receipt alone does not create an Effect;
- superseded task types, sampling fields, `verified` / `probable`, rubrics,
and support fields are rejected;
- there is at most one Effect per Task.
- the same Read may support multiple distinct choices;
- one choice cannot be reused across Effects;
- unresolved Reads cannot have Effects.

## 9. Sample handling

Expand All @@ -270,22 +278,23 @@ Pass when:

Pass when the report includes:

- a complete clear Task inventory with ownership, objective, scope, primary
deliverable, outcome, Read, Effect, and evidence summary;
- every clear Task's ownership/objective/outcome anchors and boundary rationale;
- a complete frozen Task inventory with objective, optional scope/deliverable,
outcome, source IDs, Read, zero-or-more Effects, and evidence summary;
- the Task inventory digest and every clear Task's objective/outcome sources;
- a complete excluded-candidate inventory with exclusion kind, observed scope,
and reason;
- observed and unresolved Read Tasks;
- Effect Tasks and observed Reads without an Effect;
- Effect Tasks, total Effects, and observed Reads without an Effect;
- the four-Effect distribution;
- authorized Chat/message/trace coverage and gaps;
- the four-class in-window Tree-read attempt conservation table.

Verify:

- observed + unresolved = clear Tasks;
- Effect + observed Read without Effect = observed Read Tasks;
- the four Effect counts sum to Effect Tasks;
- Effect Tasks + observed Read without Effect = observed Read Tasks;
- the four Effect counts sum to total Effects;
- total Effects is greater than or equal to Effect Tasks;
- the report does not output a global effectiveness rate;
- Read counts, receipts, and unresolved gaps are not represented as causal
value or non-value.
Expand Down
5 changes: 5 additions & 0 deletions scripts/validate_skill.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,11 @@
"chats.jsonl",
"judgments.jsonl",
"task-judgments.jsonl",
"task-source.jsonl",
"task-inventory-draft.jsonl",
"task-inventory.jsonl",
"read-attributions.jsonl",
"effect-judgments.jsonl",
}


Expand Down
Loading
Loading