docs: propose multi-harness community benchmark plan for Legal Quants input - #7
Merged
ai-agent-on-behalf-of-john-hughes[bot] merged 24 commits intoJul 8, 2026
Conversation
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (6)
WalkthroughThis PR adds the multi-harness benchmark stack: canonical schemas and validation, adapter execution and conformance, deterministic run orchestration, community submission packaging and aggregation, publication surfaces, documentation, workflows, and tests. ChangesMulti-harness Stack Foundation
Estimated code review effort: 5 (Critical) | ~120 minutes Possibly related issues
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
Documentation-only PR adding a proposed plan for a community multi-harness benchmark that runs alongside the protected official LegalForecastBench path. The document is explicitly up for Legal Quants input and proposes no code changes.
Changes:
- Adds a 637-line plan file describing a new additive
legalforecast/multiharness/package, command-adapter protocol, host-owned sandbox policy, LAB CLI bridge, two-layer registry, and separate community publication surface. - Defines 17 sequenced work items, risks, open questions, and references.
- Avoids reintroducing the deprecated result-tier taxonomy per
.agents/AGENTS.md.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Contributor
End-to-end review of the multi-harness layer (tip
|
10 tasks
Contributor
Multi-harness layer: architecture summary and merge readinessThis branch now carries the full LegalQuants multi-harness implementation (previously only the plan doc was pushed; the remainder landed today — pushed via the documented break-glass path during the 2026-07-03 Cloudflare Durable Objects incident affecting the secure-gate approval plane). What's here:
Merge readiness: an end-to-end review ran against this exact commit (see review comment above): CLI, release-check smokes, adapter conformance, and the official/community boundary all verified — verdict ready-to-merge. Pre-merge housekeeping worth doing: the PR title/description still describe the original docs-only proposal and should be refreshed to reflect the implementation. Follow-ups before the first real multi-harness test are tracked in #10 (sandbox container execution is currently plan-only, env allowlist enforcement, real adapter/LAB pinning beyond fixtures, PyPI trusted-publisher setup for the bundled publish workflow, branding/mirror decisions). These are explicitly non-blocking for the official API benchmark, which continues in #11 (based on this branch; it will be rebased onto main once this merges). |
Comment on lines
+114
to
+115
| "<p>Use the linked run cards and methodology artifacts to inspect the " | ||
| "frozen cycle, model registry, scoring configuration, and release bundle.</p>", |
| @property | ||
| def manifest(self) -> AdapterManifest: | ||
| """Public adapter manifest.""" | ||
| ... |
|
|
||
| def capabilities(self, workspace: Path) -> AdapterCapabilities: | ||
| """Return adapter capabilities, writing private artifacts under workspace.""" | ||
| ... |
|
|
||
| def prepare(self, request: RunRequest, workspace: Path) -> AdapterPreparation: | ||
| """Validate and prepare a request before execution.""" | ||
| ... |
|
|
||
| def run(self, request: RunRequest, workspace: Path) -> RunResult: | ||
| """Run one request and return a validated canonical result.""" | ||
| ... |
| _CLI_PLAN_SCHEMA_VERSION = "legalforecast.multiharness.cli_plan.v1" | ||
| _SELECTION_MANIFEST_SCHEMA_VERSION = "legalforecast.multiharness.selection_manifest.v1" | ||
| _REPORT_SCHEMA_VERSION = "legalforecast.multiharness.report.v1" | ||
| _COMMUNITY_DEFERRED_MESSAGE = ( |
|
|
||
| lab_command: tuple[str, ...] | ||
| lab_root: Path | None = None | ||
| manifest: AdapterManifest = field(default_factory=lambda: harvey_lab_manifest()) |
| class LfbNativeAdapter: | ||
| """Run LFB packets through the repo's local no-network fixture harness.""" | ||
|
|
||
| manifest: AdapterManifest = field(default_factory=lambda: lfb_native_manifest()) |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit afd4a4c. Configure here.
| files: | | ||
| tmp/release-check/dist/*.whl | ||
| tmp/release-check/dist/*.tar.gz | ||
| tmp/release-check/dist/package-artifact-hashes.json |
There was a problem hiding this comment.
Manual publish creates branch releases
Medium Severity
The publish job runs softprops/action-gh-release whenever the workflow is a tag push or workflow_dispatch has publish enabled, but the job never checks for a version tag. On manual dispatch from a branch, the release step can attach wheels to a GitHub Release for that branch ref instead of a v* tag.
Reviewed by Cursor Bugbot for commit afd4a4c. Configure here.
| "../fixture_bridge.py", | ||
| "--profile", | ||
| "lq-ai" | ||
| ], |
There was a problem hiding this comment.
Fixture adapter omits Python interpreter
Medium Severity
First-class adapter manifests invoke ../fixture_bridge.py as the executable argv entry, while contributor docs show uv run python for the same bridge. CommandAdapter does not use a shell, so on Windows and on Unix without execute permission the capabilities/run phases can fail before any conformance or community example run.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit afd4a4c. Configure here.
There was a problem hiding this comment.
Actionable comments posted: 12
🧹 Nitpick comments (5)
legalforecast/multiharness/task_loaders.py (1)
155-155: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick winRedundant
git rev-parse HEADsubprocess call per task.
load_task_directorycallsself._lab_commit()for every task, andload_task_indexcallsload_task_directoryonce pertask.jsonfile found. For a suite with many tasks this spawns onegitsubprocess per task to compute the same commit hash forself.lab_rootrepeatedly, adding unnecessary process-spawn overhead to what should be a constant lookup.♻️ Cache the lab commit lookup
def __init__( self, lab_root: Path, *, suite_version: str = DEFAULT_LAB_SUITE_VERSION, ) -> None: if not suite_version.strip(): raise ValueError("suite_version must be non-empty") self.lab_root = lab_root self.suite_version = suite_version + self._cached_lab_commit: str | None | object = _UNSETThen in
_lab_commit, checkself._cached_lab_commitbefore shelling out, and cache the result (including the "no commit found" case) on first computation, e.g. viafunctools.cached_propertyinstead of a plain method.Also applies to: 164-233, 234-244
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@legalforecast/multiharness/task_loaders.py` at line 155, The task loading path is repeatedly shelling out for the same lab commit hash on every task. Update the commit lookup used by load_task_directory/load_task_index so _lab_commit on the task loader caches its result after the first call, including the “no commit found” case, and reuse that cached value for all task.json files instead of invoking git rev-parse HEAD each time. Consider using a cached_property or an equivalent _cached_lab_commit field to keep the lookup constant-time..github/workflows/community-multiharness-validation.yaml (1)
39-40: 🔒 Security & Privacy | 🔵 Trivial | ⚡ Quick winHarden checkout steps against credential persistence.
Neither
actions/checkout@v6step setspersist-credentials: false, so the GitHub token remains in the local git config for the rest of the job. Low risk here givenpermissions: contents: readand no broader artifact upload of the checkout dir, but it's cheap to harden per the flagged advisory.🔒 Proposed fix
- name: Check out repository uses: actions/checkout@v6 + with: + persist-credentials: falseApply the same change to both checkout steps (Line 40 and Line 121).
Also applies to: 120-121
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/community-multiharness-validation.yaml around lines 39 - 40, The checkout steps using actions/checkout@v6 should be hardened by disabling credential persistence so the GitHub token is not left in the local git config. Update both checkout invocations in the workflow to set persist-credentials to false on the existing checkout step definitions, including the Check out repository step and the other checkout step mentioned in the review.Source: Linters/SAST tools
tests/test_community_publication.py (1)
131-146: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winConsider asserting the output directory is clean after a guardrail failure.
This test confirms
PublicationGuardrailErroris raised on a leaked secret, but doesn't verify thattmp_path / "aggregate"contains no residual files (or the secret) afterward. Givenbuild_community_aggregatewrites several files before validating (see companion comment incommunity_aggregate.py), a regression test here would catch future leaks even if the exception is still raised correctly.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_community_publication.py` around lines 131 - 146, The guardrail test for build_community_aggregate only checks that PublicationGuardrailError is raised, but it should also verify the output_dir stays clean after failure. Update test_community_aggregate_rejects_public_secret_leak to assert the aggregate directory has no residual files or leaked secret content after build_community_aggregate aborts, using the existing tmp_path and CommunityAggregateConfig setup.legalforecast/multiharness/community.py (1)
879-884: 📐 Maintainability & Code Quality | 🔵 TrivialNarrow "private" path blocklist may diverge from actual guardrail enforcement.
_validate_public_artifact_pathonly rejects path segments literally starting with"private". Test fixtures elsewhere (tests/test_community_submission.py) exercise aPublicationGuardrailErrorfor a path like"source-documents/raw.json", which this local check would not catch — the real rejection apparently comes fromvalidate_public_record/enforce_publication_guardrailselsewhere. Having two divergent path-safety checks (one narrow, local; one presumably broader, external) risks false negatives if only this local check is relied upon in a code path that skips the other.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@legalforecast/multiharness/community.py` around lines 879 - 884, The local path validation in _validate_public_artifact_path is narrower than the real publication guardrails and can miss unsafe paths like source-documents/raw.json. Update this helper so it uses the same path-safety rules as validate_public_record/enforce_publication_guardrails, or delegate directly to that shared validation logic, and keep the MultiHarnessValidationError behavior consistent with the rest of the publication checks.legalforecast/publication/static_sites.py (1)
375-388: 📐 Maintainability & Code Quality | 🔵 Trivial
_file_sha256/_media_typeduplicated across files.Near-identical implementations exist in
legalforecast/multiharness/community.py(Lines 929-947) andexamples/adapters/fixture_bridge.py. Consider extracting a shared helper (e.g., inlegalforecast._json_ioor a new small hashing/media-type module).🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@legalforecast/publication/static_sites.py` around lines 375 - 388, The _file_sha256 and _media_type helpers are duplicated in static_sites.py and other modules, so consolidate them into a shared utility instead of keeping near-identical copies. Move the common logic into a reusable helper module (for example a small hashing/media-type helper such as legalforecast._json_io or a new shared module), then update static_sites.py and the matching helpers in community.py and fixture_bridge to call the shared functions. Keep the existing behavior and naming at the call sites where possible, but centralize the implementation to avoid further drift.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/publish-package.yaml:
- Around line 27-30: The checkout step in the release-check job is unnecessarily
persisting git credentials for the rest of the job. Update the actions/checkout
usage in the release-check workflow to set persist-credentials to false so
credentials are not left available while uv run scripts/release_check.py
installs dependencies and runs tests; use the existing Check out repository step
as the place to apply this change.
- Around line 29-32: Update the workflow action pins in publish-package by
bumping actions/checkout from v6 to the current major v7, and
actions/download-artifact from v6 to v8; also change astral-sh/setup-uv from the
moving v7 major pin to a specific immutable v8.x release such as v8.2.0, while
leaving actions/upload-artifact as-is since its current major is already
correct.
In `@legalforecast/multiharness/cli.py`:
- Around line 493-508: _cmd_community_validate_submission writes the plan/result
JSON with a passed status before the real validation happens, so a failing
validation can leave a false success artifact. Update this handler to run
validate_submission_file(submission) before calling write_json_object for the
non-dry-run path, or otherwise only emit passed after validation succeeds; keep
the dry_run branch emitting planned. Use _cmd_community_validate_submission,
validate_submission_file, and write_json_object to locate the change.
In `@legalforecast/multiharness/community.py`:
- Around line 652-703: The _submission_shards grouping currently uses family,
scoring_mode, adapter_id, adapter_version, and model_key, but it later reads
suite_version from shard_rows[0], so mixed suite versions can be merged into one
CommunitySubmissionShard. Update the grouping key in _submission_shards to
include suite_version, or add an explicit validation/assertion before building
the shard that all rows in the group share the same suite_version; use
_request_task_field and the CommunitySubmissionShard construction as the main
places to adjust.
In `@legalforecast/multiharness/harvey_lab_adapter.py`:
- Around line 112-138: prepare() is invoking the LAB CLI help probe twice
through capability checks, causing duplicate subprocess work and extra failure
risk. Reuse the result from self.capabilities(workspace) or have prepare() call
self.command_capabilities(workspace) once and derive both capabilities and
blockers from that single result. Update HarveyLabCliAdapter.prepare so the
command_capabilities() call is not repeated, while preserving the existing
validation and AdapterPreparation return values.
In `@legalforecast/multiharness/reporting.py`:
- Around line 135-147: Escape Markdown cell content before constructing the
report table in the row-building loop so values cannot break pipe-delimited
formatting. Update the table generation in the rows iteration to sanitize the
displayed fields from the row object, especially row_id, model_key, adapter_id,
adapter_version, and conformance_status, so any embedded pipes or newlines are
rendered safely. Keep the fix localized to the reporting table assembly logic in
the rows loop.
In `@legalforecast/multiharness/runner.py`:
- Line 76: `MultiHarnessRunConfig.max_parallelism` is validated and hashed but
never affects execution, so `_MultiHarnessRunner.run()` still processes
`row_plans` one by one. Update the runner to actually honor `max_parallelism` by
dispatching row execution through a bounded concurrent mechanism in
`_MultiHarnessRunner.run()` while keeping `_execute_row()` unchanged, and make
sure the config value controls the degree of parallelism instead of being
metadata only.
- Around line 557-576: The _artifact_index walk is too broad and can include
stale files from prior runs when output_dir is reused with resume=True. Update
_artifact_index to scope enumeration to the current run’s row workspaces derived
from rows/row_plans plus the fixed top-level artifacts written by
_write_run_outputs, instead of using root.rglob("*") over everything under
output_dir. Preserve the existing ArtifactRecord construction and filtering
logic, but only feed it files that belong to the current manifest/run selection.
- Around line 312-336: _execute_row() is writing request.json and
sandbox.plan.json before _resume_result() can validate the workspace, which
allows stale row state to bypass the hash check; move those writes until after
the resume decision or only write them for a fresh run. Also update
_artifact_index() so it only scans the current run’s outputs (or clears old
output artifacts first) to avoid publishing leftovers from prior runs, and make
run() actually honor max_parallelism by executing rows concurrently instead of
always processing them sequentially.
In `@legalforecast/multiharness/sandbox.py`:
- Around line 146-151: The _mount_arg helper builds a --mount string by
interpolating source and target directly, so commas in either value can break
the mount spec. Update _mount_arg to guard against or reject comma-containing
values before returning the bind mount string, and keep the existing readonly
handling intact so the mount argv cannot be silently split into extra fields.
In `@legalforecast/publication/community_aggregate.py`:
- Around line 76-121: build_community_aggregate() currently writes the registry,
reports, public submissions, and site output directly into config.output_dir
before enforce_publication_guardrails() runs, so a failed guardrail check can
leave behind a partially published bundle. Update build_community_aggregate() to
stage all outputs in a temporary working directory first, run
enforce_publication_guardrails() against that staged path, and only then promote
or move the approved bundle into config.output_dir; keep the existing flow
around _write_reports(), render_community_results_site(), and
_write_artifact_manifests() but make them operate on the staged location.
In `@legalforecast/publication/static_sites.py`:
- Around line 92-121: The artifact URLs are being generated from the source
artifact directories, but the site is written to a separate output tree, so the
links can point to paths that are not actually published. Update
render_official_results_site and the shared _artifact_links path-building logic
so hrefs are relative to the generated site location (or copy the referenced
artifacts into the site output before writing HTML), and ensure the links
produced by _link_list still resolve correctly from the published page.
---
Nitpick comments:
In @.github/workflows/community-multiharness-validation.yaml:
- Around line 39-40: The checkout steps using actions/checkout@v6 should be
hardened by disabling credential persistence so the GitHub token is not left in
the local git config. Update both checkout invocations in the workflow to set
persist-credentials to false on the existing checkout step definitions,
including the Check out repository step and the other checkout step mentioned in
the review.
In `@legalforecast/multiharness/community.py`:
- Around line 879-884: The local path validation in
_validate_public_artifact_path is narrower than the real publication guardrails
and can miss unsafe paths like source-documents/raw.json. Update this helper so
it uses the same path-safety rules as
validate_public_record/enforce_publication_guardrails, or delegate directly to
that shared validation logic, and keep the MultiHarnessValidationError behavior
consistent with the rest of the publication checks.
In `@legalforecast/multiharness/task_loaders.py`:
- Line 155: The task loading path is repeatedly shelling out for the same lab
commit hash on every task. Update the commit lookup used by
load_task_directory/load_task_index so _lab_commit on the task loader caches its
result after the first call, including the “no commit found” case, and reuse
that cached value for all task.json files instead of invoking git rev-parse HEAD
each time. Consider using a cached_property or an equivalent _cached_lab_commit
field to keep the lookup constant-time.
In `@legalforecast/publication/static_sites.py`:
- Around line 375-388: The _file_sha256 and _media_type helpers are duplicated
in static_sites.py and other modules, so consolidate them into a shared utility
instead of keeping near-identical copies. Move the common logic into a reusable
helper module (for example a small hashing/media-type helper such as
legalforecast._json_io or a new shared module), then update static_sites.py and
the matching helpers in community.py and fixture_bridge to call the shared
functions. Keep the existing behavior and naming at the call sites where
possible, but centralize the implementation to avoid further drift.
In `@tests/test_community_publication.py`:
- Around line 131-146: The guardrail test for build_community_aggregate only
checks that PublicationGuardrailError is raised, but it should also verify the
output_dir stays clean after failure. Update
test_community_aggregate_rejects_public_secret_leak to assert the aggregate
directory has no residual files or leaked secret content after
build_community_aggregate aborts, using the existing tmp_path and
CommunityAggregateConfig setup.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 8213d126-4fae-43e5-a546-a8d09e8643ce
📒 Files selected for processing (107)
.github/workflows/community-multiharness-validation.yaml.github/workflows/publish-package.yamlMODEL_RELEASE_DATES.mdREADME.mdcommunity/submissions/.gitkeepcommunity/submissions/2026/README.mdcommunity/submissions/2026/claude-agent-sdk-fixture-baseline/artifact-manifest.jsoncommunity/submissions/2026/claude-agent-sdk-fixture-baseline/canonical-runs.jsonlcommunity/submissions/2026/claude-agent-sdk-fixture-baseline/conformance-report.jsoncommunity/submissions/2026/claude-agent-sdk-fixture-baseline/hf-upload-plan.jsoncommunity/submissions/2026/claude-agent-sdk-fixture-baseline/public-summary.jsoncommunity/submissions/2026/claude-agent-sdk-fixture-baseline/row-results.jsonlcommunity/submissions/2026/claude-agent-sdk-fixture-baseline/run-manifest.jsoncommunity/submissions/2026/claude-agent-sdk-fixture-baseline/selection-manifest.jsoncommunity/submissions/2026/claude-agent-sdk-fixture-baseline/submission.jsoncommunity/submissions/2026/hermes-agent-fixture-bridge/artifact-manifest.jsoncommunity/submissions/2026/hermes-agent-fixture-bridge/canonical-runs.jsonlcommunity/submissions/2026/hermes-agent-fixture-bridge/conformance-report.jsoncommunity/submissions/2026/hermes-agent-fixture-bridge/hf-upload-plan.jsoncommunity/submissions/2026/hermes-agent-fixture-bridge/public-summary.jsoncommunity/submissions/2026/hermes-agent-fixture-bridge/row-results.jsonlcommunity/submissions/2026/hermes-agent-fixture-bridge/run-manifest.jsoncommunity/submissions/2026/hermes-agent-fixture-bridge/selection-manifest.jsoncommunity/submissions/2026/hermes-agent-fixture-bridge/submission.jsoncommunity/submissions/2026/lq-ai-fixture-bridge/artifact-manifest.jsoncommunity/submissions/2026/lq-ai-fixture-bridge/canonical-runs.jsonlcommunity/submissions/2026/lq-ai-fixture-bridge/conformance-report.jsoncommunity/submissions/2026/lq-ai-fixture-bridge/hf-upload-plan.jsoncommunity/submissions/2026/lq-ai-fixture-bridge/public-summary.jsoncommunity/submissions/2026/lq-ai-fixture-bridge/row-results.jsonlcommunity/submissions/2026/lq-ai-fixture-bridge/run-manifest.jsoncommunity/submissions/2026/lq-ai-fixture-bridge/selection-manifest.jsoncommunity/submissions/2026/lq-ai-fixture-bridge/submission.jsoncommunity/submissions/2026/openai-responses-fixture-baseline/artifact-manifest.jsoncommunity/submissions/2026/openai-responses-fixture-baseline/canonical-runs.jsonlcommunity/submissions/2026/openai-responses-fixture-baseline/conformance-report.jsoncommunity/submissions/2026/openai-responses-fixture-baseline/hf-upload-plan.jsoncommunity/submissions/2026/openai-responses-fixture-baseline/public-summary.jsoncommunity/submissions/2026/openai-responses-fixture-baseline/row-results.jsonlcommunity/submissions/2026/openai-responses-fixture-baseline/run-manifest.jsoncommunity/submissions/2026/openai-responses-fixture-baseline/selection-manifest.jsoncommunity/submissions/2026/openai-responses-fixture-baseline/submission.jsoncommunity/submissions/2026/openclaw-fixture-bridge/artifact-manifest.jsoncommunity/submissions/2026/openclaw-fixture-bridge/canonical-runs.jsonlcommunity/submissions/2026/openclaw-fixture-bridge/conformance-report.jsoncommunity/submissions/2026/openclaw-fixture-bridge/hf-upload-plan.jsoncommunity/submissions/2026/openclaw-fixture-bridge/public-summary.jsoncommunity/submissions/2026/openclaw-fixture-bridge/row-results.jsonlcommunity/submissions/2026/openclaw-fixture-bridge/run-manifest.jsoncommunity/submissions/2026/openclaw-fixture-bridge/selection-manifest.jsoncommunity/submissions/2026/openclaw-fixture-bridge/submission.jsondocs/adapters/hermes-agent.mddocs/adapters/lq-ai.mddocs/adapters/openclaw.mddocs/adapters/provider-baselines.mddocs/community-submissions.mddocs/multiharness-adapter-spec.mddocs/plans/multi-harness-community-benchmark-2026-05-29.mdexamples/adapters/claude-agent-sdk/adapter-manifest.jsonexamples/adapters/fixture_bridge.pyexamples/adapters/hermes-agent/adapter-manifest.jsonexamples/adapters/lq-ai/adapter-manifest.jsonexamples/adapters/openai-responses/adapter-manifest.jsonexamples/adapters/openclaw/adapter-manifest.jsonlegalforecast/cli.pylegalforecast/multiharness/__init__.pylegalforecast/multiharness/adapters.pylegalforecast/multiharness/artifacts.pylegalforecast/multiharness/cli.pylegalforecast/multiharness/command_adapter.pylegalforecast/multiharness/community.pylegalforecast/multiharness/conformance.pylegalforecast/multiharness/harvey_lab_adapter.pylegalforecast/multiharness/lfb_native.pylegalforecast/multiharness/reporting.pylegalforecast/multiharness/runner.pylegalforecast/multiharness/sandbox.pylegalforecast/multiharness/selection.pylegalforecast/multiharness/spec.pylegalforecast/multiharness/task_loaders.pylegalforecast/multiharness/validation.pylegalforecast/publication/__init__.pylegalforecast/publication/community_aggregate.pylegalforecast/publication/release_bundle.pylegalforecast/publication/static_sites.pylegalforecast/reporting/pilot_readiness.pyscripts/AGENTS.mdscripts/release_check.pytests/test_community_examples.pytests/test_community_multiharness_workflow.pytests/test_community_publication.pytests/test_community_submission.pytests/test_multiharness_cli.pytests/test_multiharness_command_adapter.pytests/test_multiharness_conformance.pytests/test_multiharness_external_adapters.pytests/test_multiharness_harvey_lab_adapter.pytests/test_multiharness_lfb_native.pytests/test_multiharness_runner.pytests/test_multiharness_sandbox.pytests/test_multiharness_selection.pytests/test_multiharness_spec.pytests/test_multiharness_task_loaders.pytests/test_publish_package_workflow.pytests/test_release_bundle.pytests/test_release_check.pytests/test_static_result_sites.py
✅ Files skipped from review due to trivial changes (47)
- examples/adapters/openai-responses/adapter-manifest.json
- examples/adapters/claude-agent-sdk/adapter-manifest.json
- examples/adapters/openclaw/adapter-manifest.json
- community/submissions/2026/claude-agent-sdk-fixture-baseline/public-summary.json
- community/submissions/2026/hermes-agent-fixture-bridge/run-manifest.json
- community/submissions/2026/openai-responses-fixture-baseline/selection-manifest.json
- community/submissions/2026/hermes-agent-fixture-bridge/hf-upload-plan.json
- community/submissions/2026/claude-agent-sdk-fixture-baseline/selection-manifest.json
- community/submissions/2026/openclaw-fixture-bridge/run-manifest.json
- community/submissions/2026/README.md
- community/submissions/2026/lq-ai-fixture-bridge/selection-manifest.json
- community/submissions/2026/openclaw-fixture-bridge/selection-manifest.json
- community/submissions/2026/openclaw-fixture-bridge/public-summary.json
- community/submissions/2026/openai-responses-fixture-baseline/artifact-manifest.json
- examples/adapters/lq-ai/adapter-manifest.json
- community/submissions/2026/hermes-agent-fixture-bridge/public-summary.json
- community/submissions/2026/openclaw-fixture-bridge/row-results.jsonl
- community/submissions/.gitkeep
- community/submissions/2026/hermes-agent-fixture-bridge/selection-manifest.json
- community/submissions/2026/hermes-agent-fixture-bridge/conformance-report.json
- community/submissions/2026/hermes-agent-fixture-bridge/canonical-runs.jsonl
- community/submissions/2026/lq-ai-fixture-bridge/run-manifest.json
- community/submissions/2026/lq-ai-fixture-bridge/row-results.jsonl
- community/submissions/2026/claude-agent-sdk-fixture-baseline/conformance-report.json
- community/submissions/2026/openai-responses-fixture-baseline/row-results.jsonl
- legalforecast/reporting/pilot_readiness.py
- docs/adapters/openclaw.md
- community/submissions/2026/claude-agent-sdk-fixture-baseline/hf-upload-plan.json
- community/submissions/2026/openai-responses-fixture-baseline/public-summary.json
- community/submissions/2026/claude-agent-sdk-fixture-baseline/run-manifest.json
- scripts/AGENTS.md
- community/submissions/2026/lq-ai-fixture-bridge/canonical-runs.jsonl
- docs/adapters/lq-ai.md
- community/submissions/2026/lq-ai-fixture-bridge/conformance-report.json
- community/submissions/2026/hermes-agent-fixture-bridge/submission.json
- docs/adapters/provider-baselines.md
- legalforecast/publication/init.py
- docs/adapters/hermes-agent.md
- community/submissions/2026/openai-responses-fixture-baseline/conformance-report.json
- examples/adapters/hermes-agent/adapter-manifest.json
- community/submissions/2026/lq-ai-fixture-bridge/public-summary.json
- community/submissions/2026/lq-ai-fixture-bridge/submission.json
- community/submissions/2026/openclaw-fixture-bridge/conformance-report.json
- README.md
- docs/multiharness-adapter-spec.md
- community/submissions/2026/openclaw-fixture-bridge/artifact-manifest.json
- docs/plans/multi-harness-community-benchmark-2026-05-29.md
Comment on lines
+27
to
+30
| steps: | ||
| - name: Check out repository | ||
| uses: actions/checkout@v6 | ||
|
|
There was a problem hiding this comment.
🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win
Persist git credentials disabled for release-check checkout.
release-check never pushes to the repo (job permission is contents: read), yet the checkout step keeps credentials persisted for the whole job while uv run scripts/release_check.py executes dependency installs/tests. Set persist-credentials: false to reduce the blast radius if that step is ever compromised.
🔒 Proposed fix
- name: Check out repository
uses: actions/checkout@v6
+ with:
+ persist-credentials: false📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| steps: | |
| - name: Check out repository | |
| uses: actions/checkout@v6 | |
| steps: | |
| - name: Check out repository | |
| uses: actions/checkout@v6 | |
| with: | |
| persist-credentials: false |
🧰 Tools
🪛 zizmor (1.26.1)
[warning] 28-29: credential persistence through GitHub Actions artifacts (artipacked): does not set persist-credentials: false
(artipacked)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/publish-package.yaml around lines 27 - 30, The checkout
step in the release-check job is unnecessarily persisting git credentials for
the rest of the job. Update the actions/checkout usage in the release-check
workflow to set persist-credentials to false so credentials are not left
available while uv run scripts/release_check.py installs dependencies and runs
tests; use the existing Check out repository step as the place to apply this
change.
Source: Linters/SAST tools
Comment on lines
+29
to
+32
| uses: actions/checkout@v6 | ||
|
|
||
| - name: Install uv | ||
| uses: astral-sh/setup-uv@v7 |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
What is the latest version of astral-sh/setup-uv GitHub Action?
💡 Result:
The latest version of the astral-sh/setup-uv GitHub Action is v8.2.0, released on June 3, 2026 [1][2]. It is recommended to use the immutable tag (e.g., astral-sh/setup-uv@v8.2.0) or the specific git hash when referencing the action in your workflows to enhance security [3]. By default, if no version is specified in the action configuration, the action will attempt to detect the required uv version from configuration files (such as uv.toml or pyproject.toml) or install the latest available version [1].
Citations:
- 1: https://github.com/astral-sh/setup-uv?tab=readme-ov-file
- 2: https://github.com/astral-sh/setup-uv/releases/tag/v8.2.0
- 3: https://github.com/astral-sh/setup-uv/releases/tag/v8.0.0
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Show the workflow file around the referenced lines.
nl -ba .github/workflows/publish-package.yaml | sed -n '1,120p'Repository: johnhughes3/LegalForecastBench
Length of output: 208
🌐 Web query:
What is the latest maintained major version of actions/checkout GitHub Action as of July 2026?
💡 Result:
As of July 3, 2026, the latest major version of the actions/checkout GitHub Action is v7 [1][2]. It was released on June 18, 2026 [1][3].
Citations:
- 1: https://github.com/actions/checkout/releases/tag/v7.0.0
- 2: https://github.com/actions/checkout?tab=readme-ov-file
- 3: https://github.blog/changelog/2026-06-18-safer-pull_request_target-defaults-for-github-actions-checkout/
🌐 Web query:
What are the latest maintained major versions of actions/upload-artifact and actions/download-artifact GitHub Actions as of July 2026?
💡 Result:
As of July 3, 2026, the latest major versions of the official GitHub Actions for artifacts are as follows: - actions/upload-artifact: v7 [1][2] - actions/download-artifact: v8 [3] For actions/upload-artifact, the latest stable release is v7.0.1, released on April 10, 2026 [4][1]. For actions/download-artifact, the latest stable release is v8.0.1, released on March 11, 2026 [3]. These versions run on Node.js 24 and require a minimum Actions Runner version of 2.327.1 [3][5]. Users of self-hosted runners are advised to ensure their runners are updated before upgrading to these versions [3][5].
Citations:
- 1: https://github.com/actions/upload-artifact/releases/tag/v7.0.1
- 2: https://github.com/actions/upload-artifact/releases/tag/v7.0.0
- 3: https://github.com/actions/download-artifact/releases
- 4: https://github.com/actions/upload-artifact/releases
- 5: https://github.com/actions/download-artifact/releases/tag/v7.0.0
🌐 Web query:
What is the latest maintained major version of astral-sh/setup-uv GitHub Action as of July 2026?
💡 Result:
As of July 3, 2026, the latest maintained major version of the astral-sh/setup-uv GitHub Action is v8 [1][2]. The latest specific release within this major version is v8.2.0, which was published on June 3, 2026 [1][2]. Note that starting with v8.0.0, released in March 2026, the project shifted to an immutable release model [3][4]. As a security measure to protect against supply chain attacks, the maintainers stopped publishing and supporting minor or major moving tags (such as @v8) [4]. Consequently, users are advised to pin their workflows to specific immutable versions (e.g., @v8.2.0) or full commit SHA hashes [1][4].
Citations:
- 1: https://github.com/astral-sh/setup-uv?tab=readme-ov-file
- 2: https://github.com/astral-sh/setup-uv/releases/tag/v8.2.0
- 3: https://pydevtools.com/handbook/how-to/how-to-upgrade-setup-uv-from-v7-to-v8/
- 4: https://github.com/astral-sh/setup-uv/releases/tag/v8.0.0
🏁 Script executed:
#!/bin/bash
set -euo pipefail
sed -n '1,120p' .github/workflows/publish-package.yaml | cat -nRepository: johnhughes3/LegalForecastBench
Length of output: 2685
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
p = Path('.github/workflows/publish-package.yaml')
for i, line in enumerate(p.read_text().splitlines(), 1):
if 1 <= i <= 120:
print(f"{i:4d} {line}")
PYRepository: johnhughes3/LegalForecastBench
Length of output: 2527
🏁 Script executed:
#!/bin/bash
set -euo pipefail
awk 'NR>=1 && NR<=120 { printf "%4d %s\n", NR, $0 }' .github/workflows/publish-package.yamlRepository: johnhughes3/LegalForecastBench
Length of output: 2527
Update the workflow action pins. .github/workflows/publish-package.yaml:29-32,60-61
actions/checkout@v6is behind the current major (v7).actions/download-artifact@v6is behind the current major (v8).astral-sh/setup-uv@v7should move to a specific immutablev8.xrelease (for examplev8.2.0);actions/upload-artifact@v7already matches the current major.
🧰 Tools
🪛 zizmor (1.26.1)
[error] 32-32: runtime artifacts potentially vulnerable to a cache poisoning attack (cache-poisoning): this step
(cache-poisoning)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/publish-package.yaml around lines 29 - 32, Update the
workflow action pins in publish-package by bumping actions/checkout from v6 to
the current major v7, and actions/download-artifact from v6 to v8; also change
astral-sh/setup-uv from the moving v7 major pin to a specific immutable v8.x
release such as v8.2.0, while leaving actions/upload-artifact as-is since its
current major is already correct.
Comment on lines
+493
to
+508
| def _cmd_community_validate_submission(args: argparse.Namespace) -> int: | ||
| submission = cast(Path, args.submission) | ||
| write_json_object( | ||
| cast(Path, args.output), | ||
| { | ||
| "schema_version": _CLI_PLAN_SCHEMA_VERSION, | ||
| "command": "community validate-submission", | ||
| "dry_run": cast(bool, args.dry_run), | ||
| "submission": submission.as_posix(), | ||
| "status": "planned" if cast(bool, args.dry_run) else "passed", | ||
| "checks": _community_validation_checks(), | ||
| }, | ||
| ) | ||
| if not cast(bool, args.dry_run): | ||
| validate_submission_file(submission) | ||
| return 0 |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Output written with "status": "passed" before validation actually runs.
In the non-dry-run branch, the plan/result JSON is written first with "status": "passed", and validate_submission_file(submission) is only called afterward. If validation raises, the artifact on disk still falsely claims "passed", even though the process will exit with an error. This is inconsistent with every other handler in this file (_cmd_adapters_inspect, _cmd_conformance, _cmd_run, _cmd_community_package), which all perform the real work before or in place of writing a success status.
🐛 Proposed fix: validate before writing status
def _cmd_community_validate_submission(args: argparse.Namespace) -> int:
submission = cast(Path, args.submission)
- write_json_object(
- cast(Path, args.output),
- {
- "schema_version": _CLI_PLAN_SCHEMA_VERSION,
- "command": "community validate-submission",
- "dry_run": cast(bool, args.dry_run),
- "submission": submission.as_posix(),
- "status": "planned" if cast(bool, args.dry_run) else "passed",
- "checks": _community_validation_checks(),
- },
- )
- if not cast(bool, args.dry_run):
- validate_submission_file(submission)
+ dry_run = cast(bool, args.dry_run)
+ status = "planned"
+ if not dry_run:
+ validate_submission_file(submission)
+ status = "passed"
+ write_json_object(
+ cast(Path, args.output),
+ {
+ "schema_version": _CLI_PLAN_SCHEMA_VERSION,
+ "command": "community validate-submission",
+ "dry_run": dry_run,
+ "submission": submission.as_posix(),
+ "status": status,
+ "checks": _community_validation_checks(),
+ },
+ )
return 0📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| def _cmd_community_validate_submission(args: argparse.Namespace) -> int: | |
| submission = cast(Path, args.submission) | |
| write_json_object( | |
| cast(Path, args.output), | |
| { | |
| "schema_version": _CLI_PLAN_SCHEMA_VERSION, | |
| "command": "community validate-submission", | |
| "dry_run": cast(bool, args.dry_run), | |
| "submission": submission.as_posix(), | |
| "status": "planned" if cast(bool, args.dry_run) else "passed", | |
| "checks": _community_validation_checks(), | |
| }, | |
| ) | |
| if not cast(bool, args.dry_run): | |
| validate_submission_file(submission) | |
| return 0 | |
| def _cmd_community_validate_submission(args: argparse.Namespace) -> int: | |
| submission = cast(Path, args.submission) | |
| dry_run = cast(bool, args.dry_run) | |
| status = "planned" | |
| if not dry_run: | |
| validate_submission_file(submission) | |
| status = "passed" | |
| write_json_object( | |
| cast(Path, args.output), | |
| { | |
| "schema_version": _CLI_PLAN_SCHEMA_VERSION, | |
| "command": "community validate-submission", | |
| "dry_run": dry_run, | |
| "submission": submission.as_posix(), | |
| "status": status, | |
| "checks": _community_validation_checks(), | |
| }, | |
| ) | |
| return 0 |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@legalforecast/multiharness/cli.py` around lines 493 - 508,
_cmd_community_validate_submission writes the plan/result JSON with a passed
status before the real validation happens, so a failing validation can leave a
false success artifact. Update this handler to run
validate_submission_file(submission) before calling write_json_object for the
non-dry-run path, or otherwise only emit passed after validation succeeds; keep
the dry_run branch emitting planned. Use _cmd_community_validate_submission,
validate_submission_file, and write_json_object to locate the change.
Comment on lines
+112
to
+138
| def prepare(self, request: RunRequest, workspace: Path) -> AdapterPreparation: | ||
| capabilities = self.capabilities(workspace) | ||
| if request.adapter.adapter_id != self.manifest.adapter_id: | ||
| raise HarveyLabCliAdapterError( | ||
| "run request adapter ID does not match manifest" | ||
| ) | ||
| if request.adapter.adapter_version != self.manifest.adapter_version: | ||
| raise HarveyLabCliAdapterError( | ||
| "run request adapter version does not match manifest" | ||
| ) | ||
| if request.task.family != "harvey_lab": | ||
| raise HarveyLabCliAdapterError( | ||
| "Harvey LAB adapter requires harvey_lab task" | ||
| ) | ||
| if request.task.scoring_mode != "lab_native": | ||
| raise HarveyLabCliAdapterError( | ||
| "Harvey LAB adapter requires lab_native mode" | ||
| ) | ||
| command_capabilities = self.command_capabilities(workspace) | ||
| if command_capabilities.blockers: | ||
| formatted = "; ".join(command_capabilities.blockers) | ||
| raise HarveyLabCliAdapterError(formatted) | ||
| return AdapterPreparation( | ||
| manifest=self.manifest, | ||
| capabilities=capabilities, | ||
| workspace=workspace, | ||
| ) |
There was a problem hiding this comment.
🚀 Performance & Scalability | 🟠 Major | ⚡ Quick win
prepare() runs the LAB CLI --help probe twice.
self.capabilities(workspace) (line 113) internally calls self.command_capabilities(workspace), which invokes the external LAB command with --help. prepare() then calls self.command_capabilities(workspace) again directly (line 130) to inspect blockers, triggering a second subprocess invocation of the same probe for every single run(). This doubles external-process latency and risk of subprocess failures/timeouts on a path that already has a timeout budget.
♻️ Proposed fix to avoid duplicate probing
def prepare(self, request: RunRequest, workspace: Path) -> AdapterPreparation:
- capabilities = self.capabilities(workspace)
+ command_capabilities = self.command_capabilities(workspace)
+ capabilities = AdapterCapabilities(
+ adapter_id=self.manifest.adapter_id,
+ adapter_version=self.manifest.adapter_version,
+ supported_families=("harvey_lab",),
+ supported_scoring_modes=("lab_native",),
+ supports_sandbox_policy=True,
+ capabilities_sha256=_record_sha256(command_capabilities.to_record()),
+ )
+ write_json_object(
+ workspace / "lab-command-capabilities.json",
+ command_capabilities.to_record(),
+ )
if request.adapter.adapter_id != self.manifest.adapter_id:
raise HarveyLabCliAdapterError(
"run request adapter ID does not match manifest"
)
if request.adapter.adapter_version != self.manifest.adapter_version:
raise HarveyLabCliAdapterError(
"run request adapter version does not match manifest"
)
if request.task.family != "harvey_lab":
raise HarveyLabCliAdapterError(
"Harvey LAB adapter requires harvey_lab task"
)
if request.task.scoring_mode != "lab_native":
raise HarveyLabCliAdapterError(
"Harvey LAB adapter requires lab_native mode"
)
- command_capabilities = self.command_capabilities(workspace)
if command_capabilities.blockers:
formatted = "; ".join(command_capabilities.blockers)
raise HarveyLabCliAdapterError(formatted)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| def prepare(self, request: RunRequest, workspace: Path) -> AdapterPreparation: | |
| capabilities = self.capabilities(workspace) | |
| if request.adapter.adapter_id != self.manifest.adapter_id: | |
| raise HarveyLabCliAdapterError( | |
| "run request adapter ID does not match manifest" | |
| ) | |
| if request.adapter.adapter_version != self.manifest.adapter_version: | |
| raise HarveyLabCliAdapterError( | |
| "run request adapter version does not match manifest" | |
| ) | |
| if request.task.family != "harvey_lab": | |
| raise HarveyLabCliAdapterError( | |
| "Harvey LAB adapter requires harvey_lab task" | |
| ) | |
| if request.task.scoring_mode != "lab_native": | |
| raise HarveyLabCliAdapterError( | |
| "Harvey LAB adapter requires lab_native mode" | |
| ) | |
| command_capabilities = self.command_capabilities(workspace) | |
| if command_capabilities.blockers: | |
| formatted = "; ".join(command_capabilities.blockers) | |
| raise HarveyLabCliAdapterError(formatted) | |
| return AdapterPreparation( | |
| manifest=self.manifest, | |
| capabilities=capabilities, | |
| workspace=workspace, | |
| ) | |
| def prepare(self, request: RunRequest, workspace: Path) -> AdapterPreparation: | |
| command_capabilities = self.command_capabilities(workspace) | |
| capabilities = AdapterCapabilities( | |
| adapter_id=self.manifest.adapter_id, | |
| adapter_version=self.manifest.adapter_version, | |
| supported_families=("harvey_lab",), | |
| supported_scoring_modes=("lab_native",), | |
| supports_sandbox_policy=True, | |
| capabilities_sha256=_record_sha256(command_capabilities.to_record()), | |
| ) | |
| write_json_object( | |
| workspace / "lab-command-capabilities.json", | |
| command_capabilities.to_record(), | |
| ) | |
| if request.adapter.adapter_id != self.manifest.adapter_id: | |
| raise HarveyLabCliAdapterError( | |
| "run request adapter ID does not match manifest" | |
| ) | |
| if request.adapter.adapter_version != self.manifest.adapter_version: | |
| raise HarveyLabCliAdapterError( | |
| "run request adapter version does not match manifest" | |
| ) | |
| if request.task.family != "harvey_lab": | |
| raise HarveyLabCliAdapterError( | |
| "Harvey LAB adapter requires harvey_lab task" | |
| ) | |
| if request.task.scoring_mode != "lab_native": | |
| raise HarveyLabCliAdapterError( | |
| "Harvey LAB adapter requires lab_native mode" | |
| ) | |
| if command_capabilities.blockers: | |
| formatted = "; ".join(command_capabilities.blockers) | |
| raise HarveyLabCliAdapterError(formatted) | |
| return AdapterPreparation( | |
| manifest=self.manifest, | |
| capabilities=capabilities, | |
| workspace=workspace, | |
| ) |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@legalforecast/multiharness/harvey_lab_adapter.py` around lines 112 - 138,
prepare() is invoking the LAB CLI help probe twice through capability checks,
causing duplicate subprocess work and extra failure risk. Reuse the result from
self.capabilities(workspace) or have prepare() call
self.command_capabilities(workspace) once and derive both capabilities and
blockers from that single result. Update HarveyLabCliAdapter.prepare so the
command_capabilities() call is not repeated, while preserving the existing
validation and AdapterPreparation return values.
Comment on lines
+312
to
+336
| def _execute_row(self, plan: _RowPlan) -> MultiHarnessRunRow: | ||
| plan.workspace.mkdir(parents=True, exist_ok=True) | ||
| private_logs = plan.workspace / "private-logs" | ||
| private_logs.mkdir(parents=True, exist_ok=True) | ||
| write_json_object(plan.workspace / "request.json", plan.request.to_record()) | ||
| write_json_object( | ||
| plan.workspace / "sandbox.plan.json", | ||
| build_container_plan(plan.request.sandbox_policy).to_record(), | ||
| ) | ||
|
|
||
| resumed = False | ||
| lfb_record: Mapping[str, Any] | None = None | ||
| try: | ||
| resumed_result = self._resume_result(plan) | ||
| if resumed_result is not None: | ||
| result, lfb_record = resumed_result | ||
| resumed = True | ||
| else: | ||
| result, lfb_record = self._run_adapter(plan) | ||
| except Exception as exc: | ||
| if self.config.incomplete_run_policy == "fail_fast": | ||
| raise | ||
| (private_logs / "error.txt").write_text(_plain_error(exc), encoding="utf-8") | ||
| result = _failure_result(plan, exc) | ||
| write_json_object(plan.workspace / "result.json", result.to_record()) |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
git ls-files legalforecast/multiharness/runner.py tests/test_multiharness_runner.pyRepository: johnhughes3/LegalForecastBench
Length of output: 239
🏁 Script executed:
sed -n '1,260p' legalforecast/multiharness/runner.py
printf '\n====\n'
sed -n '260,420p' legalforecast/multiharness/runner.py
printf '\n====\n'
sed -n '420,620p' legalforecast/multiharness/runner.py
printf '\n====\n'
sed -n '1,260p' tests/test_multiharness_runner.py
printf '\n====\n'
sed -n '260,520p' tests/test_multiharness_runner.pyRepository: johnhughes3/LegalForecastBench
Length of output: 36186
🏁 Script executed:
rg -n "max_parallelism" legalforecast/multiharness/runner.py tests/test_multiharness_runner.pyRepository: johnhughes3/LegalForecastBench
Length of output: 514
Don’t reuse stale row state or stale artifacts.
_execute_row()writesrequest.jsonbefore_resume_result()reads it back, so the hash check can no longer catch a mismatched workspace; move those writes behind the resume decision._artifact_index()walks the wholeoutput_dir, so leftovers from prior runs can be published in the currentartifact-index.json; scope it to the current run outputs or clear the directory first.max_parallelismis validated and serialized, butrun()still processes rows sequentially, so that setting has no effect.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@legalforecast/multiharness/runner.py` around lines 312 - 336, _execute_row()
is writing request.json and sandbox.plan.json before _resume_result() can
validate the workspace, which allows stale row state to bypass the hash check;
move those writes until after the resume decision or only write them for a fresh
run. Also update _artifact_index() so it only scans the current run’s outputs
(or clears old output artifacts first) to avoid publishing leftovers from prior
runs, and make run() actually honor max_parallelism by executing rows
concurrently instead of always processing them sequentially.
Comment on lines
+557
to
+576
| def _artifact_index(root: Path) -> list[dict[str, Any]]: | ||
| artifacts: list[dict[str, Any]] = [] | ||
| for path in sorted(item for item in root.rglob("*") if item.is_file()): | ||
| if path.name == "artifact-index.json": | ||
| continue | ||
| relative = path.relative_to(root).as_posix() | ||
| artifacts.append( | ||
| ArtifactRecord( | ||
| artifact_id=_artifact_id(relative), | ||
| path=relative, | ||
| sha256=_file_sha256(path), | ||
| media_type=_media_type(path), | ||
| public=( | ||
| not relative.startswith("rows/") or "/private-logs/" not in relative | ||
| ), | ||
| size_bytes=path.stat().st_size, | ||
| ).to_record() | ||
| ) | ||
| return artifacts | ||
|
|
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Artifact index is an unscoped directory walk; stale files from prior incompatible runs can leak into a resumed run's published index.
_artifact_index walks every file under output_dir via root.rglob("*") with no filtering by the current run's row_plans/manifest. If output_dir is reused across runs with resume=True but a different task selection (e.g., different --limit/families), leftover row directories and artifacts from the earlier, no-longer-selected tasks remain on disk and get included in the new artifact-index.json as if they belonged to the current run — undermining the accuracy of what's published for that run.
Consider scoping the walk to the current row workspaces plus known top-level output files (derived from rows/row_plans and the fixed artifact names written by _write_run_outputs), rather than every file under output_dir.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@legalforecast/multiharness/runner.py` around lines 557 - 576, The
_artifact_index walk is too broad and can include stale files from prior runs
when output_dir is reused with resume=True. Update _artifact_index to scope
enumeration to the current run’s row workspaces derived from rows/row_plans plus
the fixed top-level artifacts written by _write_run_outputs, instead of using
root.rglob("*") over everything under output_dir. Preserve the existing
ArtifactRecord construction and filtering logic, but only feed it files that
belong to the current manifest/run selection.
Comment on lines
+146
to
+151
| def _mount_arg(record: Mapping[str, str]) -> str: | ||
| source = record["source"] | ||
| target = record["target"] | ||
| mode = record["mode"] | ||
| readonly = ",readonly" if mode == "ro" else "" | ||
| return f"type=bind,src={source},dst={target}{readonly}" |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win
Unescaped commas in --mount value can corrupt the mount spec.
_mount_arg builds type=bind,src=...,dst=... by naive string interpolation. Docker/Podman treat , as the field separator for --mount; if source or target ever contains a comma, the resulting argv entry silently splits into unintended extra fields (potentially exposing/mounting an unintended path) instead of failing loudly.
🔧 Proposed guard
def _mount_arg(record: Mapping[str, str]) -> str:
source = record["source"]
target = record["target"]
mode = record["mode"]
+ if "," in source or "," in target:
+ raise ValueError("mount source/target must not contain ','")
readonly = ",readonly" if mode == "ro" else ""
return f"type=bind,src={source},dst={target}{readonly}"📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| def _mount_arg(record: Mapping[str, str]) -> str: | |
| source = record["source"] | |
| target = record["target"] | |
| mode = record["mode"] | |
| readonly = ",readonly" if mode == "ro" else "" | |
| return f"type=bind,src={source},dst={target}{readonly}" | |
| def _mount_arg(record: Mapping[str, str]) -> str: | |
| source = record["source"] | |
| target = record["target"] | |
| mode = record["mode"] | |
| if "," in source or "," in target: | |
| raise ValueError("mount source/target must not contain ','") | |
| readonly = ",readonly" if mode == "ro" else "" | |
| return f"type=bind,src={source},dst={target}{readonly}" |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@legalforecast/multiharness/sandbox.py` around lines 146 - 151, The _mount_arg
helper builds a --mount string by interpolating source and target directly, so
commas in either value can break the mount spec. Update _mount_arg to guard
against or reject comma-containing values before returning the bind mount
string, and keep the existing readonly handling intact so the mount argv cannot
be silently split into extra fields.
Comment on lines
+76
to
+121
| def build_community_aggregate( | ||
| config: CommunityAggregateConfig, | ||
| ) -> CommunityAggregateResult: | ||
| """Build the reviewed community registry and static comparison reports.""" | ||
|
|
||
| submissions = _load_submission_inputs(config.submissions_dir) | ||
| config.output_dir.mkdir(parents=True, exist_ok=True) | ||
| registry_dir = config.output_dir / "registry" | ||
| reports_dir = config.output_dir / "reports" | ||
| public_submissions_dir = config.output_dir / "submissions" | ||
| registry_dir.mkdir(parents=True, exist_ok=True) | ||
| reports_dir.mkdir(parents=True, exist_ok=True) | ||
| public_submissions_dir.mkdir(parents=True, exist_ok=True) | ||
|
|
||
| group_task_ids = _group_task_ids(submissions) | ||
| rows = _comparison_rows(submissions, group_task_ids) | ||
| registry_records = [_normalized_submission_record(item) for item in submissions] | ||
| coverage_records = _coverage_records(rows) | ||
| contributors = _contributor_index(submissions) | ||
| adapter_model_index = _adapter_model_index(rows) | ||
| shard_groups = _compatible_shard_groups(submissions, rows) | ||
|
|
||
| write_jsonl_objects(registry_dir / "submissions.jsonl", registry_records) | ||
| write_jsonl_objects(registry_dir / "task-coverage.jsonl", coverage_records) | ||
| write_json_object(registry_dir / "contributors.json", contributors) | ||
| write_json_object(registry_dir / "adapters-models.json", adapter_model_index) | ||
| write_json_object(registry_dir / "compatible-shard-groups.json", shard_groups) | ||
| write_json_object( | ||
| registry_dir / "site-summary.json", | ||
| _site_summary(submissions, rows, shard_groups), | ||
| ) | ||
| for item in submissions: | ||
| write_json_object( | ||
| public_submissions_dir / f"{item.manifest.submission_id}.json", | ||
| _normalized_submission_record(item), | ||
| ) | ||
|
|
||
| _write_reports(reports_dir, rows) | ||
| render_community_results_site( | ||
| community_aggregate_dir=config.output_dir, | ||
| output_dir=config.output_dir / "site", | ||
| ) | ||
| enforce_publication_guardrails( | ||
| PublicationGuardrailConfig(public_paths=(config.output_dir,)) | ||
| ) | ||
| _write_artifact_manifests(config.output_dir) |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== community_aggregate.py ==\n'
sed -n '1,220p' legalforecast/publication/community_aggregate.py
printf '\n== guardrails implementation ==\n'
rg -n "def enforce_publication_guardrails|class PublicationGuardrailConfig|PublicationGuardrailConfig" legalforecast -S
printf '\n== guardrail/tests references ==\n'
rg -n "test_community_aggregate_rejects_public_secret_leak|enforce_publication_guardrails|community_aggregate" tests legalforecast -SRepository: johnhughes3/LegalForecastBench
Length of output: 15139
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== publication_guardrails.py (relevant parts) ==\n'
sed -n '1,260p' legalforecast/publication/publication_guardrails.py
printf '\n== test_community_publication.py ==\n'
sed -n '1,240p' tests/test_community_publication.py
printf '\n== community multiharness workflow references ==\n'
rg -n "upload|artifact|output_dir|if: always|community-aggregate|publication" .github tests legalforecast/multiharness/cli.py -SRepository: johnhughes3/LegalForecastBench
Length of output: 50386
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== community aggregate CLI ==\n'
sed -n '500,540p' legalforecast/multiharness/cli.py
printf '\n== workflow references (community aggregate) ==\n'
rg -n "community-aggregate|community aggregate|output-dir|upload-artifact|if: always|always\(" .github/workflows tests -S
printf '\n== static_sites guardrail ordering ==\n'
sed -n '150,210p' legalforecast/publication/static_sites.pyRepository: johnhughes3/LegalForecastBench
Length of output: 8783
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== community-multiharness-validation workflow excerpt ==\n'
sed -n '96,150p' .github/workflows/community-multiharness-validation.yaml
printf '\n== other community aggregate call sites ==\n'
rg -n "build_community_aggregate\(|community aggregate" legalforecast tests .github/workflows -SRepository: johnhughes3/LegalForecastBench
Length of output: 3196
Stage community aggregate output before guardrails
build_community_aggregate() writes public registry/report/site files into config.output_dir before calling enforce_publication_guardrails(). If the guardrail check fails, the directory is left populated with the tainted bundle; writing to a temp dir and promoting it only after the scan would make the failure path safer.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@legalforecast/publication/community_aggregate.py` around lines 76 - 121,
build_community_aggregate() currently writes the registry, reports, public
submissions, and site output directly into config.output_dir before
enforce_publication_guardrails() runs, so a failed guardrail check can leave
behind a partially published bundle. Update build_community_aggregate() to stage
all outputs in a temporary working directory first, run
enforce_publication_guardrails() against that staged path, and only then promote
or move the approved bundle into config.output_dir; keep the existing flow
around _write_reports(), render_community_results_site(), and
_write_artifact_manifests() but make them operate on the staged location.
Comment on lines
+92
to
+121
| def render_official_results_site( | ||
| *, | ||
| official_artifacts_dir: Path, | ||
| output_dir: Path, | ||
| ) -> StaticSiteResult: | ||
| """Render an official-only static site from official aggregate artifacts.""" | ||
|
|
||
| output_dir.mkdir(parents=True, exist_ok=True) | ||
| rows = _official_rows(official_artifacts_dir) | ||
| artifact_links = _artifact_links(official_artifacts_dir) | ||
| body = [ | ||
| "<main>", | ||
| "<h1>LegalForecastBench Official Results</h1>", | ||
| ( | ||
| "<p class='lede'>Official benchmark results are produced only by the " | ||
| "protected LegalForecastBench evaluation workflow and official " | ||
| "aggregation artifacts.</p>" | ||
| ), | ||
| "<section><h2>Score Table</h2>", | ||
| _official_table(rows), | ||
| "</section>", | ||
| "<section><h2>Methodology and Run Cards</h2>", | ||
| "<p>Use the linked run cards and methodology artifacts to inspect the " | ||
| "frozen cycle, model registry, scoring configuration, and release bundle.</p>", | ||
| _link_list(artifact_links), | ||
| "</section>", | ||
| "</main>", | ||
| ] | ||
| _write_site(output_dir, "\n".join(body), OFFICIAL_RESULTS_SITE_SCHEMA_VERSION) | ||
| return _site_result(output_dir) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Check whether cli.py / release_bundle.py colocate source artifacts and site output_dir
rg -n 'render_official_results_site|render_community_results_site' --type=py -C5Repository: johnhughes3/LegalForecastBench
Length of output: 168
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== files ==\n'
git ls-files 'legalforecast/publication/*' 'legalforecast/*' | sed -n '1,200p'
printf '\n== static_sites outline ==\n'
ast-grep outline legalforecast/publication/static_sites.py --view expanded || true
printf '\n== renderer and link helpers ==\n'
python3 - <<'PY'
from pathlib import Path
p = Path("legalforecast/publication/static_sites.py")
text = p.read_text()
for start, end in [(1,220),(220,360)]:
print(f"\n--- {p}:{start}-{end} ---")
for i, line in enumerate(text.splitlines(), 1):
if start <= i <= end:
print(f"{i:4d}: {line}")
PY
printf '\n== search for output_dir / artifact copies ==\n'
rg -n "output_dir|artifact_links|copy_source|symlink|source-artifacts|official_artifacts_dir|community_aggregate_dir|render_official_results_site|render_community_results_site" legalforecast -C 3Repository: johnhughes3/LegalForecastBench
Length of output: 50387
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== call sites ==\n'
rg -n "render_official_results_site|render_community_results_site" legalforecast/cli.py legalforecast/publication -C 4
printf '\n== release bundle site layout ==\n'
python3 - <<'PY'
from pathlib import Path
p = Path("legalforecast/publication/release_bundle.py")
text = p.read_text().splitlines()
for start, end in [(1,220),(220,420)]:
print(f"\n--- {p}:{start}-{end} ---")
for i, line in enumerate(text, 1):
if start <= i <= end:
print(f"{i:4d}: {line}")
PYRepository: johnhughes3/LegalForecastBench
Length of output: 16920
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n "render_official_results_site" legalforecast -C 4Repository: johnhughes3/LegalForecastBench
Length of output: 746
Artifact links resolve against the source tree, not the published site.
_artifact_links() builds hrefs from official_artifacts_dir / community_aggregate_dir, but the HTML is written to a separate site directory. In render_community_results_site(...), that makes the links point at output_dir/site/... instead of the files in output_dir, so they 404 unless the artifacts are copied into the site tree first. The same pattern in render_official_results_site(...) has the same risk if it’s used with a distinct output dir.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@legalforecast/publication/static_sites.py` around lines 92 - 121, The
artifact URLs are being generated from the source artifact directories, but the
site is written to a separate output tree, so the links can point to paths that
are not actually published. Update render_official_results_site and the shared
_artifact_links path-building logic so hrefs are relative to the generated site
location (or copy the referenced artifacts into the site output before writing
HTML), and ensure the links produced by _link_list still resolve correctly from
the published page.
… input Adds docs/plans/multi-harness-community-benchmark-2026-05-29.md, a proposed plan for community review by Legal Quants and other contributors. The plan is intentionally additive: it preserves the protected official LegalForecastBench path (main-only workflows, official S3/OIDC boundaries, official aggregation guardrails) and introduces a separate community surface for multi-harness comparisons. Key features of the proposal: - New isolated package legalforecast/multiharness/ with canonical schemas for tasks, adapter capabilities, run requests/results, sandbox policies, conformance reports, and community submissions. Reuses existing evals/packet/prompt/parser/scoring/accounting code rather than duplicating benchmark logic. - Canonical task families with explicit scoring modes (legalforecast_mtd + lfb_brier, harvey_lab + lab_native). Community rows are grouped by (family, scoring_mode, selection_sha256); no single cross-suite winner is computed. - Language-agnostic command-adapter protocol (adapter.json manifest + `capabilities` / `run` CLI commands over JSON) as the public contribution surface, so contributors can ship a script/CLI without packaging Python into this repo. Subprocess execution never uses shell=True; stdout/stderr stay private unless summarized. - Host-owned sandbox policy planner that emits a sandbox.plan.json per run, with Docker/Podman backends, --network=none for tool containers by default, and provider egress confined to the host adapter process under declared env vars (no Docker-in-Docker, no LAB-in-LFB nesting). - Harvey LAB integration as a pinned CLI bridge rather than vendoring or importing unstable internals; capabilities are probed and recorded, native LAB scores.json is normalized into community artifacts, and LAB report.html/transcripts default to private. - First-class adapter tracks for LQ.AI, Hermes Agent, and OpenClaw, with OpenAI Responses/Codex-style and Claude Agent SDK adapters as provider/runtime baselines. Each track records harness-specific provenance (e.g. inference tier, terminal backend, runtimePlan). - Partial-run shards are first-class: each submission records selection_sha256, selectors, adapter/model/sandbox/run-config hashes, and shard/composite-group IDs. Composite rows only roll up when compatibility keys match and task IDs do not overlap, and every shard contributor is credited separately. - Two-layer registry: GitHub PRs under community/submissions/** are the reviewed registry of record (audit trail + attribution + moderation); Hugging Face Datasets or GitHub Releases mirror large artifacts by immutable URL + SHA-256. No custom upload service in v1; the "database" is versioned JSON/JSONL/Parquet rebuilt on every accepted change. - Two separate static publication surfaces, generated by Python renderers in v1: "LegalForecastBench Official Results" (consumes existing official aggregate artifacts) and "LegalForecastBench Community Harness Comparisons" (consumes community aggregate artifacts, with LAB and LFB rendered in separate sections and non-official disclaimers). Final naming is open for Legal Quants approval. - Provenance and presentation surfaces replace the deprecated result-tier taxonomy (result_tier, verified-community, community-unverified, alpha-non-canonical are explicitly banned in submission validation, consistent with .agents/AGENTS.md). - Adapter conformance suite (`legalforecast multiharness conformance`) runs without provider credentials by default and emits a plain-English report so non-maintainers can verify an adapter before submission. - Dedicated community validation workflow that uses contents:read only, no id-token:write, no AWS credentials, no provider secrets, and no official protected environment. - Attribution per row credits LegalForecastBench / John Hughes / Legal Quants (infrastructure), Harvey (task source where LAB tasks are used), adapter authors, submission authors/runners, and declared model/provider identity. Structure: 17 work items with explicit dependencies, sizing, key files, done-when criteria, and a recommended execution order that builds a testable core (schemas → loaders → selection → sandbox → adapter protocol → LFB native adapter → conformance → runner) before the LAB CLI bridge, CLI surface, community submission/aggregation/site, validation workflow, docs, release checks, and first-class external adapter tracks. Open questions flagged for community input: final public names for the two sites; exact installation/version pinning and adapter mode (command bridge vs in-process vs native plugin) for LQ.AI, Hermes, and OpenClaw; organization/repo names and retention policy for the Hugging Face artifact mirror; whether HF upload helpers ship in v1 or stay manual.
ai-agent-on-behalf-of-john-hughes
Bot
force-pushed
the
docs/multi-harness-community-benchmark-plan
branch
from
July 7, 2026 14:57
afd4a4c to
a60fb98
Compare
There was a problem hiding this comment.
Actionable comments posted: 4
♻️ Duplicate comments (2)
legalforecast/multiharness/reporting.py (1)
109-148: 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick winMarkdown table cells still unescaped.
row_id,model_key,adapter_id,adapter_version, andconformance_statuscan still contain|or newlines, breaking the table formatting, as previously flagged.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@legalforecast/multiharness/reporting.py` around lines 109 - 148, The Markdown table in render_community_comparison_markdown still writes raw row fields into cells, so values containing pipe characters or newlines can break formatting. Update the row rendering to escape or sanitize row_id, model_key, adapter_id, adapter_version, and conformance_status before concatenating them into the table, and normalize any embedded newlines so each cell stays single-line.legalforecast/multiharness/community.py (1)
652-703: 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick winShard homogeneity still not enforced for suite_version (and now also sandbox_policy_hash).
The grouping key
(family, scoring_mode, adapter_id, adapter_version, model_key)still omitssuite_version, so rows with different suite versions can land in the same shard whilesuite_versionis copied only fromshard_rows[0]. The same gap now also applies tosandbox_policy_hash, which is likewise derived only fromshard_rows[0]— a shard could silently misreport the sandbox policy for some of its rows.compatible_shard_group_id(built fromfamily/scoring_mode/selection_sha256only) inherits the same blind spot.🧩 Proposed fix: include suite_version in the grouping key and assert sandbox consistency
- groups: dict[tuple[str, str, str, str, str], list[Mapping[str, Any]]] = {} + groups: dict[tuple[str, str, str, str, str, str], list[Mapping[str, Any]]] = {} for row in rows: request = requests[_required_row_str(row, "row_id")] family = _required_row_str(row, "family") scoring_mode = _request_task_field(request, "scoring_mode") + suite_version = _request_task_field(request, "suite_version") adapter_id = _required_row_str(row, "adapter_id") adapter_version = _required_row_str(row, "adapter_version") model_key = _required_row_str(row, "model_key") groups.setdefault( - (family, scoring_mode, adapter_id, adapter_version, model_key), + (family, scoring_mode, suite_version, adapter_id, adapter_version, model_key), [], ).append(row)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@legalforecast/multiharness/community.py` around lines 652 - 703, The _submission_shards function still groups rows without considering suite_version, so mixed suite versions can be combined and suite_version is then taken from shard_rows[0]. Update the grouping key in _submission_shards to include suite_version, and ensure compatible_shard_group_id still matches the shard’s true grouping semantics. Also verify sandbox_policy_hash is consistent across all rows in each shard by checking the sandbox_policy / sandbox.plan.json-derived hash for every row, not just shard_rows[0], and reject or split any mismatched rows.
🧹 Nitpick comments (3)
legalforecast/multiharness/command_adapter.py (1)
94-138: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win
run()re-invokescapabilities()unnecessarily viaprepare().
run()callsprepare(), which always re-fetches capabilities from the adapter subprocess even when the caller (e.g.,conformance.py) already validated capabilities moments earlier. For real (non-fixture) external harnesses this means an extra, potentially expensive process launch on everyrun()call. Consider caching capabilities per adapter instance/workspace, or accepting already-validated capabilities as an optional parameter toprepare/run.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@legalforecast/multiharness/command_adapter.py` around lines 94 - 138, `CommandAdapter.run()` is triggering an extra `capabilities()` subprocess call through `prepare()` even when the adapter was just validated. Update `prepare()` and/or `run()` in `CommandAdapter` so previously fetched capabilities can be reused for the same adapter instance/workspace instead of always re-fetching them, and keep the existing validation checks against `request.adapter`, `request.task.family`, and `request.task.scoring_mode` using the cached `capabilities` value..github/workflows/community-multiharness-validation.yaml (1)
5-11: 🚀 Performance & Scalability | 🔵 Trivial | 💤 Low valueBroad
docs/**trigger runs full test/lint/type-check suite for any docs edit.Any change under
docs/**(including unrelated doc edits) triggers the entire validation job (sync, format, lint, pyright, full pytest run). Consider narrowing this to only the multi-harness/community docs paths that actually need code validation, to avoid unnecessary CI cost on pure documentation PRs.Also applies to: 15-22
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/community-multiharness-validation.yaml around lines 5 - 11, The workflow trigger in the community-multiharness validation job is too broad because `docs/**` causes the full validation suite to run for any documentation-only edit. Narrow the `paths` filter in the community validation workflow so it only matches the specific community/multiharness docs locations that should require code validation, using the existing workflow trigger block to keep unrelated docs changes from launching sync, lint, type-check, and full pytest runs.scripts/release_check.py (1)
543-608: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueConsider a triple-quoted template instead of a manually joined line list.
_fixture_adapter_script()builds ~65 lines of Python source as a list of individually quoted, comma-joined strings. It's syntactically correct but harder to read/maintain than an inline heredoc-style string.♻️ Proposed refactor using a triple-quoted template
def _fixture_adapter_script() -> str: - return "\n".join( - ( - "from __future__ import annotations", - ... - ) - ) + return textwrap.dedent( + """\ + from __future__ import annotations + import argparse + import json + import pathlib + import sys + ... + """ + )🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/release_check.py` around lines 543 - 608, The _fixture_adapter_script() helper is constructing a long Python script via a manually joined list of strings, which is hard to read and maintain. Refactor it to return a single triple-quoted template string instead, preserving the same behavior for capabilities(), run(), and the sys.argv phase dispatch while keeping the generated script identical in output.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/community-multiharness-validation.yaml:
- Around line 39-41: The workflow’s checkout steps are persisting Git
credentials longer than needed, which widens exposure during later steps that
run external code. Update both actions/checkout usages in the
community-multiharness-validation job to disable credential persistence by
setting persist-credentials to false, keeping the checkout behavior limited to
read-only repository access.
In `@legalforecast/multiharness/conformance.py`:
- Around line 120-142: The sandbox_negative_control check is marked as passed
without any real validation. Update the conformance flow in conformance.py
around the sandbox negative control artifact generation and the
checks["sandbox_negative_control"] assignment so it either performs an actual
assertion from the adapter/fixture run results or is removed/demoted if it is
only a recorded expectation; use the existing _artifact_for, write_json_object,
and _passed symbols as the anchor points when wiring in the real verification.
In `@legalforecast/multiharness/harvey_lab_adapter.py`:
- Around line 428-446: The `_run_subprocess` helper currently converts only
`subprocess.TimeoutExpired` into `HarveyLabCliAdapterError`, so missing or
misconfigured executables can still escape as raw `FileNotFoundError`/`OSError`.
Update `_run_subprocess` to catch those launch failures alongside the existing
timeout handling and re-raise them as `HarveyLabCliAdapterError` with an
actionable message that includes the command context from
`argv`/`self.lab_command`. Keep the existing timeout behavior intact so all
subprocess startup and execution failures follow the module’s domain-error
contract.
- Around line 411-421: The _lab_commit helper is calling subprocess.run for git
rev-parse without any timeout, so it can block the capabilities/prepare/run flow
indefinitely. Update _lab_commit in harvey_lab_adapter.py to use the same
bounded subprocess pattern as _run_subprocess, adding a timeout and handling
timeout/failure by returning "unknown" or equivalent fallback. Keep the change
localized to _lab_commit so the git HEAD lookup cannot hang the overall adapter.
---
Duplicate comments:
In `@legalforecast/multiharness/community.py`:
- Around line 652-703: The _submission_shards function still groups rows without
considering suite_version, so mixed suite versions can be combined and
suite_version is then taken from shard_rows[0]. Update the grouping key in
_submission_shards to include suite_version, and ensure
compatible_shard_group_id still matches the shard’s true grouping semantics.
Also verify sandbox_policy_hash is consistent across all rows in each shard by
checking the sandbox_policy / sandbox.plan.json-derived hash for every row, not
just shard_rows[0], and reject or split any mismatched rows.
In `@legalforecast/multiharness/reporting.py`:
- Around line 109-148: The Markdown table in
render_community_comparison_markdown still writes raw row fields into cells, so
values containing pipe characters or newlines can break formatting. Update the
row rendering to escape or sanitize row_id, model_key, adapter_id,
adapter_version, and conformance_status before concatenating them into the
table, and normalize any embedded newlines so each cell stays single-line.
---
Nitpick comments:
In @.github/workflows/community-multiharness-validation.yaml:
- Around line 5-11: The workflow trigger in the community-multiharness
validation job is too broad because `docs/**` causes the full validation suite
to run for any documentation-only edit. Narrow the `paths` filter in the
community validation workflow so it only matches the specific
community/multiharness docs locations that should require code validation, using
the existing workflow trigger block to keep unrelated docs changes from
launching sync, lint, type-check, and full pytest runs.
In `@legalforecast/multiharness/command_adapter.py`:
- Around line 94-138: `CommandAdapter.run()` is triggering an extra
`capabilities()` subprocess call through `prepare()` even when the adapter was
just validated. Update `prepare()` and/or `run()` in `CommandAdapter` so
previously fetched capabilities can be reused for the same adapter
instance/workspace instead of always re-fetching them, and keep the existing
validation checks against `request.adapter`, `request.task.family`, and
`request.task.scoring_mode` using the cached `capabilities` value.
In `@scripts/release_check.py`:
- Around line 543-608: The _fixture_adapter_script() helper is constructing a
long Python script via a manually joined list of strings, which is hard to read
and maintain. Refactor it to return a single triple-quoted template string
instead, preserving the same behavior for capabilities(), run(), and the
sys.argv phase dispatch while keeping the generated script identical in output.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 51bbb3aa-5f6b-4fd6-9d02-df4de016e4a0
📒 Files selected for processing (107)
.github/workflows/community-multiharness-validation.yaml.github/workflows/publish-package.yamlMODEL_RELEASE_DATES.mdREADME.mdcommunity/submissions/.gitkeepcommunity/submissions/2026/README.mdcommunity/submissions/2026/claude-agent-sdk-fixture-baseline/artifact-manifest.jsoncommunity/submissions/2026/claude-agent-sdk-fixture-baseline/canonical-runs.jsonlcommunity/submissions/2026/claude-agent-sdk-fixture-baseline/conformance-report.jsoncommunity/submissions/2026/claude-agent-sdk-fixture-baseline/hf-upload-plan.jsoncommunity/submissions/2026/claude-agent-sdk-fixture-baseline/public-summary.jsoncommunity/submissions/2026/claude-agent-sdk-fixture-baseline/row-results.jsonlcommunity/submissions/2026/claude-agent-sdk-fixture-baseline/run-manifest.jsoncommunity/submissions/2026/claude-agent-sdk-fixture-baseline/selection-manifest.jsoncommunity/submissions/2026/claude-agent-sdk-fixture-baseline/submission.jsoncommunity/submissions/2026/hermes-agent-fixture-bridge/artifact-manifest.jsoncommunity/submissions/2026/hermes-agent-fixture-bridge/canonical-runs.jsonlcommunity/submissions/2026/hermes-agent-fixture-bridge/conformance-report.jsoncommunity/submissions/2026/hermes-agent-fixture-bridge/hf-upload-plan.jsoncommunity/submissions/2026/hermes-agent-fixture-bridge/public-summary.jsoncommunity/submissions/2026/hermes-agent-fixture-bridge/row-results.jsonlcommunity/submissions/2026/hermes-agent-fixture-bridge/run-manifest.jsoncommunity/submissions/2026/hermes-agent-fixture-bridge/selection-manifest.jsoncommunity/submissions/2026/hermes-agent-fixture-bridge/submission.jsoncommunity/submissions/2026/lq-ai-fixture-bridge/artifact-manifest.jsoncommunity/submissions/2026/lq-ai-fixture-bridge/canonical-runs.jsonlcommunity/submissions/2026/lq-ai-fixture-bridge/conformance-report.jsoncommunity/submissions/2026/lq-ai-fixture-bridge/hf-upload-plan.jsoncommunity/submissions/2026/lq-ai-fixture-bridge/public-summary.jsoncommunity/submissions/2026/lq-ai-fixture-bridge/row-results.jsonlcommunity/submissions/2026/lq-ai-fixture-bridge/run-manifest.jsoncommunity/submissions/2026/lq-ai-fixture-bridge/selection-manifest.jsoncommunity/submissions/2026/lq-ai-fixture-bridge/submission.jsoncommunity/submissions/2026/openai-responses-fixture-baseline/artifact-manifest.jsoncommunity/submissions/2026/openai-responses-fixture-baseline/canonical-runs.jsonlcommunity/submissions/2026/openai-responses-fixture-baseline/conformance-report.jsoncommunity/submissions/2026/openai-responses-fixture-baseline/hf-upload-plan.jsoncommunity/submissions/2026/openai-responses-fixture-baseline/public-summary.jsoncommunity/submissions/2026/openai-responses-fixture-baseline/row-results.jsonlcommunity/submissions/2026/openai-responses-fixture-baseline/run-manifest.jsoncommunity/submissions/2026/openai-responses-fixture-baseline/selection-manifest.jsoncommunity/submissions/2026/openai-responses-fixture-baseline/submission.jsoncommunity/submissions/2026/openclaw-fixture-bridge/artifact-manifest.jsoncommunity/submissions/2026/openclaw-fixture-bridge/canonical-runs.jsonlcommunity/submissions/2026/openclaw-fixture-bridge/conformance-report.jsoncommunity/submissions/2026/openclaw-fixture-bridge/hf-upload-plan.jsoncommunity/submissions/2026/openclaw-fixture-bridge/public-summary.jsoncommunity/submissions/2026/openclaw-fixture-bridge/row-results.jsonlcommunity/submissions/2026/openclaw-fixture-bridge/run-manifest.jsoncommunity/submissions/2026/openclaw-fixture-bridge/selection-manifest.jsoncommunity/submissions/2026/openclaw-fixture-bridge/submission.jsondocs/adapters/hermes-agent.mddocs/adapters/lq-ai.mddocs/adapters/openclaw.mddocs/adapters/provider-baselines.mddocs/community-submissions.mddocs/multiharness-adapter-spec.mddocs/plans/multi-harness-community-benchmark-2026-05-29.mdexamples/adapters/claude-agent-sdk/adapter-manifest.jsonexamples/adapters/fixture_bridge.pyexamples/adapters/hermes-agent/adapter-manifest.jsonexamples/adapters/lq-ai/adapter-manifest.jsonexamples/adapters/openai-responses/adapter-manifest.jsonexamples/adapters/openclaw/adapter-manifest.jsonlegalforecast/cli.pylegalforecast/multiharness/__init__.pylegalforecast/multiharness/adapters.pylegalforecast/multiharness/artifacts.pylegalforecast/multiharness/cli.pylegalforecast/multiharness/command_adapter.pylegalforecast/multiharness/community.pylegalforecast/multiharness/conformance.pylegalforecast/multiharness/harvey_lab_adapter.pylegalforecast/multiharness/lfb_native.pylegalforecast/multiharness/reporting.pylegalforecast/multiharness/runner.pylegalforecast/multiharness/sandbox.pylegalforecast/multiharness/selection.pylegalforecast/multiharness/spec.pylegalforecast/multiharness/task_loaders.pylegalforecast/multiharness/validation.pylegalforecast/publication/__init__.pylegalforecast/publication/community_aggregate.pylegalforecast/publication/release_bundle.pylegalforecast/publication/static_sites.pylegalforecast/reporting/pilot_readiness.pyscripts/AGENTS.mdscripts/release_check.pytests/test_community_examples.pytests/test_community_multiharness_workflow.pytests/test_community_publication.pytests/test_community_submission.pytests/test_multiharness_cli.pytests/test_multiharness_command_adapter.pytests/test_multiharness_conformance.pytests/test_multiharness_external_adapters.pytests/test_multiharness_harvey_lab_adapter.pytests/test_multiharness_lfb_native.pytests/test_multiharness_runner.pytests/test_multiharness_sandbox.pytests/test_multiharness_selection.pytests/test_multiharness_spec.pytests/test_multiharness_task_loaders.pytests/test_publish_package_workflow.pytests/test_release_bundle.pytests/test_release_check.pytests/test_static_result_sites.py
✅ Files skipped from review due to trivial changes (44)
- community/submissions/2026/openclaw-fixture-bridge/row-results.jsonl
- community/submissions/2026/README.md
- community/submissions/2026/openclaw-fixture-bridge/canonical-runs.jsonl
- scripts/AGENTS.md
- community/submissions/2026/hermes-agent-fixture-bridge/run-manifest.json
- community/submissions/2026/lq-ai-fixture-bridge/hf-upload-plan.json
- community/submissions/2026/claude-agent-sdk-fixture-baseline/public-summary.json
- community/submissions/2026/openclaw-fixture-bridge/public-summary.json
- community/submissions/2026/lq-ai-fixture-bridge/row-results.jsonl
- community/submissions/2026/claude-agent-sdk-fixture-baseline/conformance-report.json
- community/submissions/2026/openclaw-fixture-bridge/artifact-manifest.json
- examples/adapters/lq-ai/adapter-manifest.json
- docs/adapters/openclaw.md
- community/submissions/2026/hermes-agent-fixture-bridge/selection-manifest.json
- examples/adapters/openai-responses/adapter-manifest.json
- community/submissions/2026/claude-agent-sdk-fixture-baseline/artifact-manifest.json
- docs/adapters/provider-baselines.md
- community/submissions/.gitkeep
- community/submissions/2026/hermes-agent-fixture-bridge/public-summary.json
- community/submissions/2026/openai-responses-fixture-baseline/selection-manifest.json
- community/submissions/2026/claude-agent-sdk-fixture-baseline/run-manifest.json
- community/submissions/2026/openai-responses-fixture-baseline/hf-upload-plan.json
- community/submissions/2026/lq-ai-fixture-bridge/canonical-runs.jsonl
- community/submissions/2026/lq-ai-fixture-bridge/artifact-manifest.json
- community/submissions/2026/openclaw-fixture-bridge/conformance-report.json
- docs/adapters/lq-ai.md
- community/submissions/2026/lq-ai-fixture-bridge/conformance-report.json
- community/submissions/2026/lq-ai-fixture-bridge/public-summary.json
- community/submissions/2026/openai-responses-fixture-baseline/run-manifest.json
- community/submissions/2026/openai-responses-fixture-baseline/public-summary.json
- community/submissions/2026/hermes-agent-fixture-bridge/row-results.jsonl
- community/submissions/2026/lq-ai-fixture-bridge/run-manifest.json
- community/submissions/2026/openai-responses-fixture-baseline/row-results.jsonl
- community/submissions/2026/hermes-agent-fixture-bridge/canonical-runs.jsonl
- community/submissions/2026/openclaw-fixture-bridge/submission.json
- community/submissions/2026/claude-agent-sdk-fixture-baseline/hf-upload-plan.json
- community/submissions/2026/openai-responses-fixture-baseline/submission.json
- legalforecast/reporting/pilot_readiness.py
- README.md
- community/submissions/2026/openai-responses-fixture-baseline/artifact-manifest.json
- community/submissions/2026/claude-agent-sdk-fixture-baseline/canonical-runs.jsonl
- docs/adapters/hermes-agent.md
- community/submissions/2026/hermes-agent-fixture-bridge/conformance-report.json
- docs/multiharness-adapter-spec.md
🚧 Files skipped from review as they are similar to previous changes (34)
- community/submissions/2026/lq-ai-fixture-bridge/selection-manifest.json
- community/submissions/2026/hermes-agent-fixture-bridge/submission.json
- community/submissions/2026/hermes-agent-fixture-bridge/artifact-manifest.json
- examples/adapters/hermes-agent/adapter-manifest.json
- legalforecast/publication/init.py
- community/submissions/2026/claude-agent-sdk-fixture-baseline/row-results.jsonl
- community/submissions/2026/openclaw-fixture-bridge/selection-manifest.json
- examples/adapters/claude-agent-sdk/adapter-manifest.json
- community/submissions/2026/openclaw-fixture-bridge/hf-upload-plan.json
- community/submissions/2026/openai-responses-fixture-baseline/conformance-report.json
- examples/adapters/openclaw/adapter-manifest.json
- community/submissions/2026/claude-agent-sdk-fixture-baseline/selection-manifest.json
- community/submissions/2026/openai-responses-fixture-baseline/canonical-runs.jsonl
- community/submissions/2026/hermes-agent-fixture-bridge/hf-upload-plan.json
- tests/test_release_bundle.py
- legalforecast/multiharness/adapters.py
- tests/test_community_examples.py
- legalforecast/cli.py
- tests/test_community_multiharness_workflow.py
- legalforecast/multiharness/init.py
- community/submissions/2026/openclaw-fixture-bridge/run-manifest.json
- tests/test_publish_package_workflow.py
- tests/test_multiharness_selection.py
- tests/test_multiharness_sandbox.py
- tests/test_multiharness_external_adapters.py
- legalforecast/multiharness/sandbox.py
- community/submissions/2026/claude-agent-sdk-fixture-baseline/submission.json
- legalforecast/publication/static_sites.py
- legalforecast/publication/release_bundle.py
- community/submissions/2026/lq-ai-fixture-bridge/submission.json
- legalforecast/multiharness/spec.py
- legalforecast/multiharness/validation.py
- legalforecast/publication/community_aggregate.py
- legalforecast/multiharness/cli.py
Comment on lines
+39
to
+41
| - name: Check out repository | ||
| uses: actions/checkout@v6 | ||
|
|
There was a problem hiding this comment.
🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win
Disable credential persistence on checkout.
Neither checkout step sets persist-credentials: false. The job only needs contents: read and never pushes, so leaving credentials persisted for the whole job (including uv run steps that execute installed/third-party code) needlessly widens the blast radius if that step is compromised.
🔒 Proposed fix
- name: Check out repository
uses: actions/checkout@v6
+ with:
+ persist-credentials: falseApply the same change to both occurrences (lines 39-41 and 120-122).
Also applies to: 120-122
🧰 Tools
🪛 zizmor (1.26.1)
[warning] 39-40: credential persistence through GitHub Actions artifacts (artipacked): does not set persist-credentials: false
(artipacked)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/community-multiharness-validation.yaml around lines 39 -
41, The workflow’s checkout steps are persisting Git credentials longer than
needed, which widens exposure during later steps that run external code. Update
both actions/checkout usages in the community-multiharness-validation job to
disable credential persistence by setting persist-credentials to false, keeping
the checkout behavior limited to read-only repository access.
Source: Linters/SAST tools
Contributor
|
[awt-judge] Opus decision Decision: block_for_fixes Codex action itemsAction Items for CodexCritical Blockers (must fix before merge)
Expected Outcome
Deferred to follow-up (not blocking this merge)
PR description drift (nice to fix while you're in there)
Codex follow-up guidanceFocus on the four critical blockers first; run |
Contributor
|
[awt-judge] Codex pushed after 45 minutes. Replying to: #7 (comment) |
| _WARNING = "warning" | ||
| _LFB_FIXTURE_REQUEST_ID = "conformance:lfb-fixture" | ||
| _LAB_FIXTURE_REQUEST_ID = "conformance:lab-fixture" | ||
| _ADAPTER_SPEC_PATH = "docs/multiharness-adapter-spec.md" |
Contributor
|
[awt-judge] Opus decision Decision: approve_with_issue Follow-up artifact (if needed)Title: Multi-harness benchmark: hardening and hygiene follow-ups Follow-up items deferred from PR #7 (all non-blocking; core blockers were fixed in 6d75a66). Workflow hardening
Reporting polish
Performance
Maintainability
Docs drift
Optional security review
|
ai-agent-on-behalf-of-john-hughes
Bot
merged commit Jul 8, 2026
536fbf9
into
main
9 of 10 checks passed
7 tasks
ai-agent-on-behalf-of-john-hughes Bot
pushed a commit
that referenced
this pull request
Jul 9, 2026
… labeling, inference wiring Replacement for #11, which GitHub closed automatically when its base branch docs/multi-harness-community-benchmark-plan was deleted after #7 merged. This keeps the official benchmark hardening stack visible and reviewable against main now that #7 has landed. Original PR: #11 <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **High Risk** > Changes touch the official GitHub Actions eval path (labels integrity gates, multi-ablation matrix, cost caps, S3 aggregation), which are security- and correctness-critical for published benchmark results. > > **Overview** > **Official benchmark workflow** is expanded so dispatch can run **multiple ablations**, require a **frozen labels file**, and **fail closed** if `labels_sha256` in the run-input manifest does not match the labels JSONL before the matrix fans out. Matrix build adds **projected model cost** caps (from registry pricing and packet token estimates), **repeat sampling** for selected cases, and stricter input validation. Per-case jobs now write to a **cycle-scoped S3 results root** and pass **repeat count**; a new **aggregate-results** job downloads artifacts and runs `legalforecast.publication.official_aggregate` (optional baselines, cycle-power metadata), then **syncs public bundles to S3** and uploads CI artifacts. > > **Release and community surfaces** add a **`v*` tag workflow** that reruns `scripts/release_check.py` and can publish wheels/sdist to PyPI plus GitHub release assets, and a **community multi-harness validation** workflow (tests, submission validation, publication guardrails, dry-run aggregate; on `main`, builds a static-site artifact). Checked-in **fixture community submission packages** and **adapter/contributor docs** document the non-official multi-harness path. > > **Project docs** update agent beads sync to a **centralized Dolt server** (no `bd dolt push/pull`), refresh **pilot model release anchors** and workflow defaults, add **labeling protocol** and README positioning (Brier skill vs baselines, contamination/snapshot caveats), and note deprecated preregistration/result-tier concepts in agent scope. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 5789966. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added repeat-run benchmarking, lawyer review/adjudication flow, packet render verification, and richer leaderboard/cadence reporting. * Expanded model registry and release metadata tracking for clearer release-date anchoring. * **Bug Fixes** * Tightened release-anchor checks, leakage screening, and validation for model/version consistency. * Improved handling of missing or ambiguous labeling details. * **Documentation** * Updated docs and release notes guidance for benchmark setup, labeling protocol, and prior-art positioning. * **Chores** * Refreshed workflow action pins and release-check automation. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
ai-agent-on-behalf-of-john-hughes Bot
pushed a commit
that referenced
this pull request
Jul 12, 2026
## Summary - fail closed before legacy Bedrock subprocess work when Claude Sonnet 5 is selected, including matching Bedrock IDs and ARNs - harden multiharness result secret hygiene for fail-fast cleanup, native writes, and resumed artifacts - pin the command-adapter environment allowlist regression against a planted `FAKE_SECRET` - refresh eligibility terminology and record the actual scope delivered by PR #7 - finish the remaining non-workflow #14 cleanup, including stable `textwrap.dedent` fixture rendering ## Validation - `uv run ruff format` - `uv run ruff check --fix` - `uv run pyright` - `uv run pytest -q` (928 passed, 2 skipped) Fixes #26 Fixes #31 Refs #10 Refs #14 <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > Touches live model routing and multiharness public artifact hygiene on eval/community paths; behavior is intentionally stricter (Sonnet 5 Bedrock blocked, resume/fail-fast secret checks) rather than broad refactors. > > **Overview** > Hardens **live eval** and **community multiharness** paths around misconfiguration and credential leakage, with small doc and release-smoke cleanup. > > **Claude Sonnet 5 + legacy Bedrock** now **fails closed** with `LiveModelConfigError` before any Bedrock `InvokeModel` subprocess when `LFB_ANTHROPIC_RUNTIME` selects Bedrock, including Bedrock-style model ID overrides and ARNs. Operators are told to unset the runtime env var and use the direct Anthropic API instead of attempting unsupported legacy transport. > > **Multiharness** runs **`validate_no_secret_values`** on resumed `result.json`, native LFB results before public write, and existing post-run checks. Under **`fail_fast`**, a rejected post-run result **deletes** `result.json` so a failed secret scan cannot leave a public artifact (native path included). Tests cover resume when secrets were planted in stale results and fail-fast cleanup for command and native rows. > > **Command-adapter** env allowlist tests use a planted **`FAKE_SECRET`** instead of a generically named undeclared var. **Eligibility** docstring for `decision_entered_on_or_after_model_deployment` now refers to **`series_release_timestamp`** (UTC date), not “deployment date.” **ADR 0001** records the real merged scope of PR #7’s community multiharness work and is linked from the docs index. **Release-check** fixture adapter script rendering uses **`textwrap.dedent`** with a **pinned length/SHA-256** stability test. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 8de0eaa. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Added clearer validation for unsupported legacy Bedrock configurations using Sonnet 5, preventing requests from reaching the runtime. * Improved run security by detecting secrets in resumed and native results. * Failed runs now remove publicly visible result files when validation rejects their output. * **Documentation** * Added an architectural decision record documenting the community multi-harness scope and its distinction from official benchmarks. * **Tests** * Added coverage for blocked model configurations, secret leakage prevention, fail-fast cleanup, and stable release-check output. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
This was referenced Jul 24, 2026
This was referenced Jul 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
docs/plans/multi-harness-community-benchmark-2026-05-29.md— a proposed plan (not yet approved or implemented) for a community-facing multi-harness benchmark that sits alongside, and remains strictly separate from, the protected official LegalForecastBench path.This PR is up for community input, especially from Legal Quants. Merging it accepts the plan as a starting point for discussion; the design is explicitly open to revision before any of the 17 work items are scheduled.
What the plan proposes
legalforecast/multiharness/that reuses existing evals/packet/prompt/parser/scoring/accounting code without rewriting the protected official path.legalforecast_mtd + lfb_brier,harvey_lab + lab_native). Community rows are grouped by(family, scoring_mode, selection_sha256); no cross-suite winner is computed.adapter.jsonmanifest +capabilities/runCLI over JSON) so contributors can ship a script/CLI rather than packaging Python.sandbox.plan.jsonper run; tool containers default to--network=nonewith provider egress confined to the host adapter process.community/submissions/**as the reviewed registry of record; Hugging Face Datasets / GitHub Releases as immutable large-artifact mirrors referenced by URL + SHA-256.result_tier,verified-community,community-unverified,alpha-non-canonicalare explicitly banned in submission validation, consistent with.agents/AGENTS.md).contents: readonly, no OIDC, no AWS credentials, no provider secrets, and no protected official environment.Requesting community input
Particularly from Legal Quants and other folks who would actually run this, feedback would be most valuable on:
1. Which adapters should be included as first-class
The plan currently proposes LQ.AI, Hermes Agent, and OpenClaw as first-class tracks, with OpenAI Responses/Codex-style and Claude Agent SDK as provider/runtime baselines. Open questions:
2. How folks would like to run them
The plan assumes a few defaults that are open for discussion:
--network=nonefor tool containers, provider egress only from the host adapter process. Is that the right default? Do contributors expect to run without containers (bare process), or with a different isolation backend?The adapter-selection and runtime-mode questions above are community-driven decisions that should be resolved by Legal Quants reviewers rather than implementation defaults.
3. Naming and branding
Working names are LegalForecastBench Official Results and LegalForecastBench Community Harness Comparisons. Final naming, attribution language, and any Legal Quants co-branding are open.
Scope of this PR
docs/plans/multi-harness-community-benchmark-2026-05-29.md(637 lines)..agents/AGENTS.mddecision to drop preregistration protocols and result-tier classification.Test plan
.agents/AGENTS.md:11-16(Item 11 banned fields list).Note
Medium Risk
Large additive surface (publication guardrails, community artifacts, tag-triggered PyPI publish) but official eval workflows stay separate; main risk is mis-published community content or a bad release artifact, mitigated by validation workflows and release_check gates.
Overview
This PR implements the multi-harness community benchmark (not just the plan doc): a new
legalforecast/multiharness/layer with canonical schemas, LFB/LAB task loading, deterministic selection, host-owned sandbox plans, command adapters, conformance, runner, community package/validate/aggregate, and alegalforecast multiharnessCLI wired into the mainlegalforecastentrypoint.CI and release: Adds Community Multi-Harness Validation (format/lint/pyright, multiharness + community tests, submission validation, guardrail scans, aggregate dry-run; on
main, rebuilds community site artifact) and Publish Python Package (v*tags runrelease_check, PyPI trusted publish, GitHub release assets).scripts/release_check.pygains no-network multiharness smokes (inspect, index, conformance, dry-run run, community aggregate).Contributor surfaces: New docs (
multiharness-adapter-spec,community-submissions, per-adapter guides),examples/adapters/no-networkfixture_bridge.pyprofiles (LQ.AI, Hermes, OpenClaw, OpenAI Responses, Claude SDK), and checked-incommunity/submissions/2026/example packages for those five tracks.Docs/product alignment: README documents community multi-harness vs official benchmark;
MODEL_RELEASE_DATES.mdand pilot model names shift to the checked-in pilot registry keys (Gemini 3 Flash Preview, GPT-5.4 mini, Claude Sonnet 4.6). The planning doc is updated with an implementation status section.Reviewed by Cursor Bugbot for commit afd4a4c. Bugbot is set up for automated code reviews on this repo. Configure here.
Summary by CodeRabbit