Skip to content

feat: add report contract linter for evaluation outputs #876

@luochen211

Description

@luochen211

Problem

career-ops now relies on evaluation reports as durable artifacts for downstream tooling: tracker links, pattern analysis, dashboard metadata, PDF decisions, and application follow-up. The report format has several implied contract fields, but there is no standalone validator for report files.

This means a report can silently miss key fields such as **URL:**, **Legitimacy:**, ## Machine Summary, score format, or PDF status, and later scripts only fail indirectly.

Proposed solution

Add a lightweight verify-reports.mjs script that scans reports/*.md and validates the report contract:

  • header includes **URL:**
  • header includes **Legitimacy:** {tier}
  • score is parseable as X.X/5 or a documented skip value
  • PDF status is present when the report came from auto-pipeline/pipeline
  • ## Machine Summary is present and parseable when available
  • machine summary fields are from the accepted schema used by analyze-patterns.mjs

Why this fits

This makes evaluation artifacts auditable before downstream scripts consume them. It also gives contributors a focused check to run after changing modes, Gemini output, or report generation.

Non-overlap audit

I checked open PRs before proposing this. This does not overlap with:

Suggested implementation scope

  • New verify-reports.mjs
  • Optional npm run verify:reports script
  • Add focused tests to test-all.mjs with temporary report fixtures
  • Document in docs/SCRIPTS.md

Test plan

  • node verify-reports.mjs --self-test
  • node test-all.mjs --quick
  • Fixture coverage for missing URL, missing legitimacy, malformed Machine Summary, and valid report

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions