Skip to content

feat(reward-memory): attribute post-outcome utility - #3280

Open
yuefengw wants to merge 2 commits into
huangruiteng:mainfrom
yuefengw:codex/c93-memory-utility-stage1
Open

feat(reward-memory): attribute post-outcome utility#3280
yuefengw wants to merge 2 commits into
huangruiteng:mainfrom
yuefengw:codex/c93-memory-utility-stage1

Conversation

@yuefengw

Copy link
Copy Markdown
Contributor

Summary

  • add the provider-neutral memory_utility_observation_v0 Stage 1 contract under the existing Reward Memory capability
  • bind complete Stage 3 application receipts, verified outcomes, scope, retrieval/policy snapshots, evaluator identity, and evidence into a stable observation identity
  • keep application disposition separate from post-outcome utility, including applied + success -> unknown when attribution evidence is insufficient
  • version Stage 5 dogfood receipts/batches from v0 to v1 so applied/not_applied/refuted no longer imply helpful/harmful utility
  • keep the optional evaluator default-off, proposal-only, fail-open, and unable to grant authority or perform memory/provider/external writes
  • reject exact provider references, local paths, malformed envelopes, duplicate application settlements, and non-canonical memory digests at both build and batch boundaries

Compatibility and identity

This intentionally changes the preview dogfood output schema from reward_memory_dogfood_receipt_v0 / reward_memory_dogfood_batch_v0 to v1 instead of silently redefining hit, miss, and refute.

The dogfood receipt_id identifies one application/outcome settlement and deliberately excludes evaluator delivery state, so evaluator absence, rejection, retry, or new evidence cannot duplicate disposition or cost metrics. Utility delivery has its own stable observation_id; a changed evaluator version or new evidence creates a new observation identity.

Validation

  • python -m pytest -q: 3228 passed, 2 skipped on a clean, non-concurrent rerun
  • focused utility attribution tests: 54 passed
  • Reward Memory related tests: 114 passed
  • all six Reward Memory public smokes passed, including the Stage 5 dogfood CLI path
  • loopx canary premerge --from-git-diff --no-progress: passed; 17/17 selected canaries/boundary checks passed, 0 warnings, 0 manual holds
  • capability placement, corpus registry, public-boundary, changed-path Ruff, compile, and diff checks passed
  • task contract scan returned ok=True; its only two warnings were the expected absence of the worktree-local .loopx/registry.json

The first full-suite run was executed while other validations were active and produced one unrelated timing failure in tests/extensions/test_extension_runtime.py::test_extension_run_terminates_provider_on_timeout (execution_failed vs timeout). That test passed immediately in isolation, and the subsequent non-concurrent full-suite run passed as reported above.

Out of scope

  • no Stage 2 reducer or read-only utility projection
  • no retrieval ranking or semantic candidate filtering change
  • no scheduler, quota, todo, memory lifecycle, or main-lane behavior change
  • no OpenViking/provider rank-prior integration or writeback
  • no new capability, global supervisor, or execution authority

Refs #3214

Signed-off-by: yuefengw <60574042+yuefengw@users.noreply.github.com>
Signed-off-by: yuefengw <60574042+yuefengw@users.noreply.github.com>

@huangruiteng huangruiteng left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approval conclusion (community PR; exact head reviewed)

详细中文评审:post-outcome 记忆效用归因 Stage 1(PR #3280

评审目标 head:5fee3c676f3072ccd35c45b6706a0b47dbd45498(base: main 0deff62a0,10 文件,+2074/-122;Refs #3214,实现 Stage 1 契约)

改动范围与结构

新增 loopx/capabilities/reward_memory/memory_utility.pymemory_utility_observation_v0 契约、校验与构建),并把 dogfood 输出从 v0 升级到 v1:application_disposition(applied/not_applied/refuted)与 utility_evaluation(accepted/rejected/not_requested)分离,applied + success 在归因证据不足时保持 unknown,不再暗示 helpful

  • application_receipt_id 绑定一次应用/结果结算,evaluator 缺席、拒绝、重试或新证据不会重复 disposition 与成本指标。
  • observation_id 是稳定身份字段(scope、application_receipt_id、memory digests、retrieval/policy snapshot、outcome_ref、attribution level/basis、evidence refs、evaluator ref/version)的规范哈希;evaluator 版本或证据变化会生成新的 observation 身份。
  • 边界强制为 False:grants_new_action_authorityprovider_write_performedexternal_writes_performedraw_content_captured;evaluator 默认不开启、fail-open、只能产生只读 observation。
  • 拒绝本地路径、provider 精确引用、畸形 envelope、重复 application settlement、非规范 memory digest;fixture 里包含 /Users/example/.../private/tmp/... 的负例(应被拒绝)。

安全与语义边界

  • 观察记录与 trusted facts(application receipt / verified outcome / attribution context)做绑定校验,归因对象必须落在 application 的 memory digests 内。
  • attribution_level(item/set/none)、evidence_basis(owner_correction / controlled_replay / deterministic_effect / evaluator_inference / insufficient)、confidencereason_codes 都是 typed 字段,弱推断不会自动升级为强结论。
  • 明确不在本 PR:Stage 2 reducer/只读投影、检索排序、scheduler/quota/todo/memory 生命周期改动、OpenViking/provider rank-prior 写回、新能力或执行权威。

验证

  • exact head 实跑:uv run --extra test python -m pytest -q tests/capabilities/test_reward_memory_agent_scoped_recall.py tests/capabilities/test_reward_memory_experiment.py tests/capabilities/test_reward_memory_ingestion.py tests/capabilities/test_reward_memory_utility_attribution.py98 passed(含 54 个 utility attribution 用例,覆盖 applied+success→unknown、harmful 保留、观察身份稳定性、重复 settlement 拒绝等)。
  • python examples/reward-memory-dogfood-smoke.py → ok。
  • git diff --check 无空白错误;变更文件未发现真实本地路径/凭据(负例 fixture 除外)。
  • GitHub 状态检查:5 项成功、0 失败、build/pytest 仍 pending;建议合入前确认两项跑绿。

非阻塞问题

  • memory_utility.py 约 740 行、dogfood.py 增加 458 行,属同一边界内的合理增长;若后续 Stage 2 继续膨胀,可把 schema/字段集合抽成独立契约模块。
  • dogfood v1 是预览输出 schema 的显式版本升级(非静默重定义),方向正确;任何外部消费方需跟随 v1 字段名。

我的整体评价

实现与 #3214 的 Stage 1 最小切片一致:disposition 与 utility 分离、稳定身份、fail-open、无权威/写回,且用 typed 字段与负例 fixture 把“applied 不等于 helpful”钉死。结论:APPROVE。合入按仓库策略执行;Stage 2(reducer + 只读投影)应复用同一 observation 契约。


English Verdict (PR #3280)

  • Exact head reviewed: 5fee3c676f3072ccd35c45b6706a0b47dbd45498
  • Verdict: APPROVE (Stage 1 of #3214; provider-neutral memory_utility_observation_v0; v1 dogfood receipts separate application disposition from utility evaluation)
  • Key findings: application settlement identity is stable and independent of evaluator delivery state; observation identity is a canonical hash so evaluator/evidence changes create new observations; authority/provider-write/raw-content flags are forced False; applied + success -> unknown when attribution evidence is insufficient. Focused validation on the exact head: 98 reward-memory tests passed plus the dogfood smoke; git diff --check clean. Non-blocking: GitHub build/pytest were still pending at review time; consider extracting the schema/field sets into a contract module if Stage 2 grows further.

@huangruiteng

Copy link
Copy Markdown
Owner

@yuefengw 你好,比较建议实际跑一下用用,在你的某些生产任务上体验一下 reward memory 的能力,也许有助于打磨这个 PR 的质量并合入

@huangruiteng huangruiteng left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

如上,建议实际用一用,有个脱敏的使用反馈~

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants