Skip to content

Build the generation-uplift benchmark (test pass-rate with vs without the Brain) #126

Description

@bejranonda

The retrieval benchmark harness now exists (PR #125), but the claim that actually backs the positioning — the Brain improves AI coding — is still only a design in docs/VALIDATION.md §2, not code.

Goal: run the same agent on a held-out set of coding tasks that ship with executable tests, twice (Brain-injected vs not), holding model/prompt/seed constant, and report test pass-rate with n + paired difference + CI. No LLM-as-judge.

Acceptance

  • Task suite authored independently of the corpus (small internal set, or SWE-bench-lite).
  • Agent harness that runs both arms deterministically.
  • Metric pre-registered; raw outputs committed.
  • Result written into docs/VALIDATION.md (or explicitly reported as null/negative if that's the truth).

See docs/VALIDATION.md "Proposed methodology §2".

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions