Skip to content

fix(review): retry when an AI review response is missing its assessment - #7633

Merged
JSONbored merged 1 commit into
mainfrom
fix/review-summary-and-rerun-checkbox
Jul 21, 2026
Merged

fix(review): retry when an AI review response is missing its assessment#7633
JSONbored merged 1 commit into
mainfrom
fix/review-summary-and-rerun-checkbox

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

  • runWorkersOpinion accepted the first structurally-parseable model response even when the required assessment field was empty, so a review with blockers/nits but no narrative summary was published with a blank/missing summary instead of being retried.
  • Adds a missing_assessment diagnostic status and retries exactly like a parse failure, keeping the best incomplete response as a last-resort fallback instead of ever surfacing a null review when budget runs out.

Test plan

  • npx vitest run test/unit/ai-review.test.ts — 207/207 passing
  • Coverage scoped to src/services/ai-review.ts: 100% of changed lines and branches covered (verified via lcov DA/BRDA against the diff hunks)
  • npx tsc --noEmit clean

runWorkersOpinion accepted the first structurally-parseable response even when
the required assessment field was empty, so a model that returned valid JSON
with blockers/nits but no narrative summary was treated as a success and
published with no visible review text. Now a missing-assessment response is
retried like a parse failure, with the best incomplete response kept as a
last-resort fallback instead of surfacing a null review.
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@JSONbored JSONbored self-assigned this Jul 21, 2026
@loopover-orb loopover-orb Bot added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 21, 2026
@loopover-orb

loopover-orb Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Warning

⏸️ LoopOver review result - manual review recommended

Review updated: 2026-07-21 05:26:53 UTC

2 files · 1 AI reviewer · 2 blockers · CI green · clean

⏸️ Suggested Action - Manual Review

Review summary
This PR fixes a real defect in runWorkersOpinion: previously any structurally-parseable model response was accepted even if the required assessment field was empty, so blockers/nits could be published with a blank narrative summary. The fix adds a missing_assessment diagnostic, retries on empty assessment exactly like a parse failure, and only falls back to the last incomplete-but-usable review if every attempt across every model exhausts its budget — correctly returning null review is never made worse than today's behavior. Tests cover the retry-then-succeed path, the exhausted-fallback path, and confirm the INCOHERENT_DIFF_ASSESSMENT sentinel (a non-empty string) is not misclassified as missing_assessment.

Nits — 5 non-blocking
  • src/services/ai-review.ts: `bestIncompleteReview` is named as if it tracks the best incomplete response, but it's simply overwritten by the latest missing-assessment attempt (last, not best) — consider renaming to `lastIncompleteReview` for clarity.
  • src/services/ai-review.ts:1256 — the exhausted-fallback event is logged via `console.log` with `level: "error"` while the earlier missing-assessment warning uses `console.warn`; confirm this split matches existing logging conventions elsewhere in the file rather than being an inconsistency.
  • The inline comment block above `bestIncompleteReview` is quite long (7+ lines) for a single variable declaration — consider trimming to the essential rationale.
  • Consider renaming `bestIncompleteReview` to reflect that it holds the last (not best) incomplete candidate.
  • If there's an established severity convention for structured log events in this file, verify `console.log`/`console.warn` usage here matches it.

Concerns raised — review before merging

  • No linked issue detected — If this PR is intended to solve an issue, link it explicitly in the PR body.
  • Maintainer requires a linked issue — Link the relevant issue (for example Closes #123) before opening the PR.
📋 Copy for AI agents — paste into your coding agent
Fix the following blocker(s) from this PR review:

1. No linked issue detected — If this PR is intended to solve an issue, link it explicitly in the PR body.

2. Maintainer requires a linked issue — Link the relevant issue (for example `Closes #123`) before opening the PR.

Decision drivers

  • ❌ Code review — 2 blockers (1 reviewer)
  • ❌ Gate result — Blocking (Repo-configured hard blocker found.)
Context & advisory signals — never blocks the verdict
Signal Result Evidence
Linked issue ⚠️ Missing No linked issue or no-issue rationale found.
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (no linked issue context).
Validation posture ✅ 25/25 PR body includes validation/test evidence.
Contributor workload ✅ 10/10 Author activity: 23 registered-repo PR(s), 16 merged, 341 issue(s).
Contributor context ✅ Confirmed Gittensor contributor JSONbored; Gittensor profile; 23 PR(s), 341 issue(s).
Improvement ✅ Minor risk: clean · value: minor · LLM: moderate
Review context
  • Author: JSONbored
  • Role context: owner (maintainer lane)
  • Public audience mode: oss maintainer
  • Lane context: Repository is configured for direct PR review.
  • Public profile languages: Python, TypeScript, Ruby, Go, JavaScript, MDX, Shell, Solidity
  • Official Gittensor activity: 23 PR(s), 341 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Start here: Treat this as maintainer-lane context rather than normal contributor-lane activity.
  • Then work through the remaining 2 steps in the Signals table above.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.
🧪 Chat with LoopOver

Ask LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

  • @loopover ask <question> answers contribution-quality Q&A with source citations and freshness.
  • @loopover chat <question> answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
  • A plain-language @loopover mention with a real question is routed to the closest matching read-only command automatically — no exact syntax required.

Full command reference: https://loopover.ai/docs/loopover-commands

🧪 Experimental — new and may change.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.

  • Re-run LoopOver review

@loopover-orb loopover-orb Bot added the manual-review Gittensor contributor context label Jul 21, 2026
@JSONbored
JSONbored merged commit 22c2c4a into main Jul 21, 2026
15 checks passed
@JSONbored
JSONbored deleted the fix/review-summary-and-rerun-checkbox branch July 21, 2026 05:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. manual-review Gittensor contributor context

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant