test(mq): one conformance suite for every Broker

[EricAndrechek]

Part of #613. This is PR D1 of the external-NATS workstream. It is based on main (#612, which it was stacked on, has merged).

What

• internal/mq/mqtest (new) is the conformance suite for mq.Broker. mqtest.Run(t, Harness{New, EndDelivery, Fill, Caps}) states the contract as behavior and uses the interfaces only, with no stream, subject or partition names. It covers:

  • round trips with names that need encoding;
  • that a topic without a tenant is refused;
  • trace context reaching Subscribe, and Subscribe seeing every tenant;
  • per-tenant order;
  • Nak and AckWait redelivery;
  • that DeadLetter keeps the topic and does not ack;
  • per-tenant, per-table dead-letter counts, every scope of a table counted under the table itself (refactor(keyenc): one key escaping, keep '-', DLQ counts per table #655), and an empty (never nil) Tables when nothing is parked;
  • replay bounds and isolation, ctx cancellation, and that a failed pull is an error;
  • exactly one failed report, and none after stop;
  • MaxBytes, Stats, ErrQueueFull, and PurgeAcked semantics.

  Each case runs as a parallel subtest on a fresh broker.

• mqtest.Caps flags the four places where the external backend legitimately differs:

  • PerTenantBudget
  • PurgesAcked
  • UnbudgetedNotFound (DLQ counts of a tenant never given a budget)
  • ConfiguresDurables (whether CreateConsumer applies AckWait or only finds an operator-made durable)

• The embedded broker passes the suite. Its run is internal/mq/mqtest/embedded_test.go.

• mq.go contract wording is updated as the design specifies:

  • The delivery unit is "a tenant's queue, or the partition that holds it".
  • ErrQueueFull is a byte limit, and the per-tenant no-queue case applies only to an implementation that opens queues per tenant.
  • DeadLetterCounts may return zero counts in place of ErrNoDeadLetterQueue.
  • CreateConsumer may find rather than create.
  • PurgeAcked may remove nothing.
  • Subscribe guarantees delivery only for events published after it returns.

• mq.ErrUnavailable (new) is mapped by the ingest handler to 503 + Retry-After: 5. Before, it would have been the 500 "publish failed". No backend returns it yet; D3's will. api.md says so.

• Two embedded bugs found by the suite are fixed:

  • A durable deleted on several tenants' queues could report on failed more than once. A CAS now allows one report, and it is pinned by a test that deletes the durable on real queues one after another.
  • ReplaySince read a pull that raced the connection closing as "caught up". It is now an error unless the connection is open.

Deviations from the design doc

• The embedded run lives in internal/mq/mqtest/embedded_test.go, not internal/mq/embedded_conformance_test.go. internal/mq's unit binary already takes about 10s of its 15s -race budget when the machine is idle, and 21–34s under heavy load on the base branch alone (measured). The suite in that binary pushed it over. In its own binary it takes about 2.5s. It also no longer needs an export_test.go hook into mq's internals.
• Harness.DeleteIngestDurable became Harness.EndDelivery, which ends delivery under a running consumer. The embedded harness closes the broker; D3 should delete the durable. The durable-deletion path for embedded is covered in internal/mq's own tests.
• Caps.NeverParkedNotFound became UnbudgetedNotFound. Embedded returns zero counts for a budgeted tenant that has parked nothing. Only a tenant with no budget gets ErrNoDeadLetterQueue.
• New cap ConfiguresDurables. The AckWait-redelivery case cannot pass against an operator-made durable with a 60s ack_wait.
• The suite uses a fixed mqtest.Durable = "buffer-consumer". It is the worker's name, so a backend that maps durable names has one to find.
• Run sets the global W3C propagator for its duration. The trace case needs it, so Run must not be called from a parallel test.

Left to later PRs

• D2: topology spec and verifier, manifests, S1.
• D3: ExternalNATS plus external_conformance_test.go, which runs mqtest.Run with its own Caps.
• D5: the Sharded cap and the shard-subset case. ConsumerConfig.Shards does not exist yet, so neither is here.
• D4: docs for the nats backend.

Evidence

• make ci: green on 6ebfc6f, after the merge of main with refactor(keyenc): one key escaping, keep '-', DLQ counts per table #655 (all coverage gates passed; Go total 94.4%).
• The suite passed 40/40 under -race -count=20 -cpu 1,4 (before the refactor(keyenc): one key escaping, keep '-', DLQ counts per table #655 merge).
• Pre-push reviewers: pre-push-reviewer and docs-reviewer both returned ship_it before the refactor(keyenc): one key escaping, keep '-', DLQ counts per table #655 merge, after 6 and 4 rounds. After it, pre-push-reviewer returned ship_it on 6ebfc6f (one round of review fixes: the nil-map assertion); the merge left the PR's own docs unchanged.

🤖 Generated with Claude Code

https://claude.ai/code/session_01EJr5tY4WQUy2sc4MbW67vL

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 29b1cd76-5907-4a1f-aa0e-903eec0e2708

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

📚 Docs preview is live → https://212d460e-wavehouse-docs.wave-rf.workers.dev

• Commit — ff67ab5: Merge remote-tracking branch 'origin/main' into feat/mq-conformance
• Author — @EricAndrechek, Claude Opus 5.5 (1M context)
• Committed — 2026-09-26 03:37 (UTC-04:00)
• Deployed — 2026-09-26 03:54 EDT

[github-code-quality]

Code Coverage Overview

Languages: Go

Go

The overall line coverage in commit ff67ab5 in the feat/mq-conformance branch remains at 94%, unchanged from commit 5b6efe0 in the main branch.

Show a line coverage summary of the most impacted files.

File	main 5b6efe0	feat/mq-conformance ff67ab5	+/-
internal/mq/embedded.go	89%	89%	0%
internal/api/ingest.go	96%	96%	0%
internal/mq/mqtest/cases.go	0%	94%	+94%
internal/mq/mqtest/mqtest.go	0%	95%	+95%

---

_{Updated September 26, 2026 07:54 UTC}

[EricAndrechek]

Heads-up from #655: it changes how dead letters are counted, and two cases in internal/mq/mqtest/cases.go pin the old behaviour, so whichever of this PR and #655 merges second goes red.

#655 counts every scope of a table under the table itself (internal/mq/deadletter.go, deadLetterTables), and a ?table= filter keeps all of that table's scopes. The old fold counted a scoped message under "table.scope", a name a dotted table could share (a.b unscoped vs a scope b). Scope is always empty today, so the API response doesn't change.

What has to flip here once both are in:

• deadLetterKeepsTheTopicAndDoesNotAck: a scoped topic counts under its table — map[string]uint64{"t": 1}, not {"t.s": 1}.
• deadLetterCounts: t1 and its scope t1.s sum under t1 (t1: 3), and the "t1" filter returns that sum rather than the unscoped messages alone.

Please don't restore the table.scope fold to make the suite pass: it isn't the Broker contract any more. #636 already carries this change on its copy of the suite, which can serve as the reference.