CI: CodeRabbit's free-tier quota silently skips review on some PRs, and the merge gate cannot tell

[mlorentedev]

Observed on #900 during the ADR-032 session.

What happened

CodeRabbit posted this instead of a review:

Review limit reached — you've reached your PR review limit, so we couldn't start this review. Next review available in: 12 minutes. You've used all free OSS reviews for now.

The check still reported CodeRabbit=SUCCESS, and #900 merged. Nothing distinguished "reviewed, no findings" from "never reviewed". The same PR count that makes this repo productive is what exhausts the quota, so the skip lands preferentially on busy sessions — exactly when review is most valuable. For contrast, #901 was reviewed and produced ten findings, one of which corrected a factual claim in an accepted ADR.

Why this is a guard defect, not a vendor complaint

The repo treats an unexercised gate as a defect elsewhere — HARNESS-065 (#881) holds new enforcement layers until the first one is proven to fire, and HARNESS-061 (#852) is entirely about representations that look healthy at the moment they stop being true. A review check that reports SUCCESS when no review ran is that same shape: the representation is green and the behaviour is absent.

Options

• Make the skip visible: a job that greps the CodeRabbit comment for the limit banner and fails, or posts a "not reviewed" label, so a human merging knows what they are merging.
• Reduce consumption so the quota lasts: CodeRabbit's own suggestions are pausing incremental auto-reviews, label-based opt-in, or excluding docs-only/generated PRs. Docs-only PRs are a real slice of this repo's volume.
• Accept the skip explicitly for a declared class (docs-only), so the exhaustion falls where it costs least rather than wherever it happens to land.

The first is the one that matters: whatever the policy, it must be legible at merge time.

Acceptance

• A PR whose CodeRabbit review did not run is distinguishable from one that ran clean, without reading the comment thread.
• Quota consumption is deliberately allocated rather than first-come.

Refs

#900 (the instance), #881 and #852 (the same failure shape in other gates).

[mlorentedev]

Fired live again today on PR #918, with the check output visible this time:

CodeRabbit	pass	0		Review rate limited

Status pass, summary "Review rate limited". No review was performed, and the merge gate reports green — which is exactly the shape this issue describes.

Worth recording that it went both ways on the same PR within one hour: the first push got a real review (COMMENTED, three actionable findings, one of which caught a test that could silently degrade into a skip). A later push got the rate-limited no-op. So the failure is not per-PR, it is per-push, and a reviewer looking at the final check state cannot tell which of the two happened for the commit they are about to merge.

That makes this the third observed occurrence (two on 2026-08-09, one today) and, as far as the gate is concerned, all three were green.

Bearing on #786 (TOOL-013): the case for a second independent reviewer rests more on this redundancy than on review quality. A reviewer that silently no-ops is worse than one that is merely less good, because the first is indistinguishable from success.

[mlorentedev]

Live reproduction, 2026-08-13 — four PRs opened in parallel, three got a green check from a reviewer that never ran:

PR	CodeRabbit check	Actual state	Inline comments
#945	pass	Review completed	12 (several Major)
#948	pass	Review rate limited	0
#949	pass	Review rate limited	0
#950	pass	Review rate limited	0

Two things this adds to the issue as filed:

1. The failure is quota-starvation with a deterministic winner, not randomness. The first PR to land consumed the quota (docs: truth pass — README, ADR hygiene, architecture map, cli/README, opencode split #945, the largest diff), and every PR opened after it in the same window was skipped. So the failure mode is predictable from PR ordering, which also means it is reproducible on demand rather than intermittent.

2. Parallel work triggers it directly. Running N independent branches — the normal shape for a multi-session sprint — guarantees N-1 unreviewed PRs carrying a green check. The more the workflow parallelises, the less review coverage it gets, silently.

The distinguishing signal is available and unused: the check's own description string reads Review completed vs Review rate limited, while conclusion is success in both cases. A merge gate that read the description instead of only the conclusion could tell them apart today, without waiting for a CodeRabbit-side fix.

[mlorentedev]

Fresh instance, #973 (2026-08-15T00:41Z): every check green — lint, test,
test-windows, integration, spec-gate, goreleaser snapshot, GitGuardian — and
zero reviewer comments, because CodeRabbit posted:

Review limit reached — you've reached your PR review limit, so we couldn't start this
review. Next review available in: 40 minutes.

This is the failure mode this issue names, and it is worth recording what it looks like
from the outside: the PR is indistinguishable from a reviewed-and-clean one. Green checks,
no comments, mergeable. The only signal that no review happened is a bot comment whose
text has to be read — nothing in the check rollup, nothing in the mergeable state, and
nothing an agent babysitting "until CI is green or a comment arrives" would flag, because
a comment did arrive; it just wasn't a review.

That makes it the same family as #969 and the verification-fails-toward-unproven pattern:
the absence of review presents as a clean review. A babysit rule keyed on "comment arrived"
is satisfied by the quota notice itself.

Two things that would help, if this is picked up:

• The babysit/merge gate should treat "CodeRabbit posted a quota notice" as not
  reviewed, distinct from "reviewed, no findings" — matching on the notice is trivial and
  deterministic.
• dotf spec review's pool already gives this repo an independent reviewer that does not
  share CodeRabbit's quota. For a PR blocked on quota, running the pooled reviewer is a
  real substitute rather than a wait.

Cross-ref #963 (the babysit window that this defeats).