feat(app): process roles

[EricAndrechek]

Part of #613. This is PR C1 of the #613 core design (process roles). Stacked on #615 (B1), which is stacked on #618 (G1).

What

• roles / WH_ROLES (default api,ingest,sweeper) and instance_id / WH_INSTANCE_ID (default <hostname>-<8 hex>, with a fresh suffix at every boot) in internal/config/config.go, plus Config.Has(Role) and config.AllRoles(). Entries are trimmed at Load. An empty list, an empty entry, an unknown role, or a role named twice refuses boot (rule 1 for roles).
• Boot rules 2 and 5 in Validate (validateTopology):
  • Rule 2: any role set other than all three with mq.backend=embedded is refused. Until D ships, every split is refused, which is expected.
  • Rule 5: a process that runs exactly one of api and ingest with cache.backend=local is refused. See the deviation note below.
• Role guards in app.New:
  • Every process gets the settings registry, observability, the MQ, the coordinator, the reload triggers (SIGHUP and the watcher), and a listener on server.port.
  • api adds schema discovery, the dedupe stores, streaming (hub, hub bridge, keepalive), auth and the full router. These stay per API process.
  • ingest adds the ingest worker.
  • sweeper adds the sweeper. It stays lease-elected through the new a.elected(lease, fn) wrapper over coord.RunElected.
  • The ClickHouse pools and the cache come with api or ingest.
  • New refuses a Config with no roles. Only one built without config.Load can have none, so it follows G1's rule that the zero value is not the default. The three hand-built configs now set config.AllRoles().
• Ops-only router (api.NewOpsRouter, sharing newProbeRouter with NewRouter) for a process without api:
  • It serves /livez, /readyz (and their aliases), /version, the same-port metrics path, and POST /v1/ops/settings/reload. Every tenant route and the rest of /v1/ops answer 404.
  • The reload is gated by an operator-key-only Authenticator (wireOpsAuth) and RequireAdmin(nil), the same gate as /v1/ops/* over a nested directory. No token verifier or JWKS fetch runs without api, so an admin JWT gets 401 there.
  • /readyz pings the pools in an ingest process. A sweeper-only process is ready once it has booted.
• NeedsDataDir probes for Pebble only when the process runs api. The two shared-queue Warnings are skipped without api, because only api opens a cache it reads or a dedupe store.
• Docs: settings-directory.mdx (boot-config list), configuration.mdx (new "Process roles" section, the example file and env, and the data_dir probe step), deployment.md (new "One Deployment per role" section), architecture.md, AGENTS.md, CHANGELOG, and the root config.yaml.

Deviations and additions to the design

• Rule 5 keys on "exactly one of api/ingest", not "lacks api or lacks ingest". A sweeper-only process holds no cache, so refusing it would forbid a valid layout: an api,ingest replica with a local cache plus a sweeper process. api+sweeper and ingest+sweeper are still refused.
• The reload on the ops-only listener admits the operator key only. The design says "gated as today", but today's gate also admits a flat directory's admin JWT, and that needs the token verifiers, which the design keeps per API process. Running verifiers (and JWKS fetches) in workers only for this route did not seem worth it.
• roles entries are trimmed at Load, so WH_ROLES="api, ingest" works. The design's [unverified] note on cleanenv's , separator is now pinned by TestLoad_RolesFromEnv.

Ingest scaling (reconciliation.md)

C1 wires the ingest worker exactly as today: one consumer per ingest process on the shared durable, so N ingest processes are competing consumers (the MVP). Nothing here rules out shard claiming later. C5/D5 can wrap per-shard consumers in a.elected or TryAcquire("ingest/shard/<i>") inside wireIngestWorker without touching the role plumbing. The hub bridge stays per API process.

Open question: should the settings reload route be served on worker processes?

Will callers need to call POST /v1/ops/settings/reload on worker pods too, through the ops-only listener? This PR builds the route there, gated by the operator key as /v1/ops/* is, on the assumption that they will. If reload is only ever needed on API pods, the route can stay: it is harmless, and a worker still reloads on SIGHUP and, over a flat directory, through the watcher.

Left to later PRs (by design)

• The multi-process integration test (C2) and partitioned consumers (C5).
• The shared backends that make a split bootable: mq.backend=nats (D) and a shared cache.backend (E).
• The lease holder that records instance_id: B2 uses it. C1 only resolves it and logs it at boot (process roles line).
• Rules 3 and 4 (B2).

Tests

• internal/config/roles_test.go:
  • defaults: every role, and an instance_id that is the hostname plus 8 hex, new at every Load
  • WH_ROLES parsing: trimmed and in any order; one role; empty; YAML list
  • unboundEnv knows both variables
  • table tests for rule 1 and for rules 2 and 5 across the role sets on embedded and shared queues and caches
  • Warnings and NeedsDataDir without api
• internal/app/roles_test.go:
  • a role-to-component-set table test over a.components names
  • a Config with no roles is refused
  • ops-only router: probes and /version answer 200; reload is 200 with the operator key, 401 with an admin JWT that the full API admits, and 403 with no key or a wrong key; eight tenant and ops routes answer 404
  • ops-only readiness (the ingest process pings ClickHouse) and the inline metrics path
  • a sweeper-only process run over a real listener holds the sweeper lease
• internal/api/router_test.go: TestNewOpsRouter.

Verification

• make ci (queued, GOTOOLCHAIN=go1.26.6) passed at beab0fd and again at 5de4fd0. That covers unit, integration and e2e tests plus the coverage gates.

• pre-push-reviewer (opus): round 1 iterate. It found that instance_id claimed to name a lease holder, and that the YAML roles test could not tell the file's list from the env default. Both are fixed in 5de4fd0. Round 2 returned ship_it at 5de4fd0.

• docs-reviewer (opus): round 1 iterate. It found five problems:

  • architecture.md's config/ and router.go sections were not synced.
  • The "every other route 404" claim missed the probe aliases and the 403 that comes first under /v1/ops.
  • Sweeper exclusivity was not qualified by a shared coord.backend.
  • The instance_id wording claimed a lease holder.
  • settings-directory.mdx did not list roles.

  All are fixed in 5de4fd0. Round 2 returned ship_it.

• Known gate gap (fix(agents): pre-push review gate can attest to unreviewed code (marker inherited across commits; wrong worktree checked) #454): the reviewer markers land in the main checkout, so the verdicts are recorded here. No marker was hand-written.

🤖 Generated with Claude Code

https://claude.ai/code/session_01EJr5tY4WQUy2sc4MbW67vL

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Advanced

Run ID: ec04fa9f-bec4-4d4f-add3-7247a0143cdd

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 9ea43848-aaca-4c76-b800-86a87a7f2f5c

📥 Commits

Reviewing files that changed from the base of the PR and between a1774fb and 1c26d11.

📒 Files selected for processing (20)

• AGENTS.md
• CHANGELOG.md
• config.yaml
• docs/src/content/docs/architecture.md
• docs/src/content/docs/configuration.mdx
• docs/src/content/docs/deployment.md
• docs/src/content/docs/settings-directory.mdx
• internal/api/router.go
• internal/api/router_test.go
• internal/app/app.go
• internal/app/app_test.go
• internal/app/roles_test.go
• internal/app/wire.go
• internal/config/backends.go
• internal/config/backends_test.go
• internal/config/config.go
• internal/config/defaults_test.go
• internal/config/roles_test.go
• tests/integration/setup_test.go
• tests/integration/tenants_test.go

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

📜 Recent review details

⏰ Context from checks skipped due to timeout. (3)

• GitHub Check: Lint
• GitHub Check: PR title
• GitHub Check: Detect changes

🧰 Additional context used

📓 Path-based instructions (8)

See [AGENTS.md](AGENTS.md) for project conventions, architecture notes, and AI agent instructions.

📄 CodeRabbit inference engine (CLAUDE.md)

Files:

• AGENTS.md

Source excerpt: Register the route in `internal/api/router.go`.

📄 CodeRabbit inference engine (AGENTS.md)

Files:

• internal/api/router.go

Source excerpt: Create or modify a handler in `internal/api/` (follow existing patterns like `ingest.go`).

📄 CodeRabbit inference engine (AGENTS.md)

Files:

• internal/api/router_test.go
• internal/api/router.go

Source excerpt: Create `*_test.go` files in the same package as the code under test.

📄 CodeRabbit inference engine (AGENTS.md)

Files:

• tests/integration/tenants_test.go
• internal/app/app_test.go
• tests/integration/setup_test.go
• internal/config/backends_test.go
• internal/api/router_test.go
• internal/config/defaults_test.go
• internal/config/roles_test.go
• internal/app/roles_test.go

See [AGENTS.md](../AGENTS.md) for project conventions, architecture notes, and AI agent instructions.

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

Files:

• AGENTS.md

Source excerpt: Create the package under `internal/`.

📄 CodeRabbit inference engine (AGENTS.md)

Files:

• internal/app/app_test.go
• internal/config/backends_test.go
• internal/api/router_test.go
• internal/config/defaults_test.go
• internal/app/app.go
• internal/config/backends.go
• internal/config/config.go
• internal/app/wire.go
• internal/api/router.go
• internal/config/roles_test.go
• internal/app/roles_test.go

Source excerpt: **In MDX, leave a blank line between a JSX tag and a code fence.**

📄 CodeRabbit inference engine (AGENTS.md)

Files:

• docs/src/content/docs/settings-directory.mdx
• docs/src/content/docs/configuration.mdx

Source excerpt: Document in `docs/src/content/docs/architecture.md`.

📄 CodeRabbit inference engine (AGENTS.md)

Files:

• docs/src/content/docs/architecture.md

🧠 Learnings (1)

📚 Learning: 2026-06-26T12:23:22.696Z

Learnt from: EricAndrechek
Repo: Wave-RF/WaveHouse PR: 346
File: internal/stream/subscriber_test.go:9-28
Timestamp: 2026-06-26T12:23:22.696Z
Learning: In this Go repository, prefer table-driven tests (e.g., `[]struct{...}` with `t.Run(...)`) only for tests that cover multiple scenarios/inputs and can be cleanly enumerated. Do not artificially rewrite a clear single-scenario sequential behavioral-flow test into a table-driven form just to fit the pattern; if there’s only one meaningful scenario, keep the test as a straightforward linear flow (as in `TestSubscriber_SendDeliversThenDropsWhenFull`).

Applied to files:

• internal/config/roles_test.go

🪛 LanguageTool

CHANGELOG.md

[typographical] ~13-~13: Consider using an em dash in dialogues and enumerations.
Context: - **Process roles: the API and the backgr...

(DASH_RULE)

docs/src/content/docs/configuration.mdx

[style] ~59-~59: To elevate your writing, try using an alternative expression here.
Context: ...t, or a comma-separated variable. Order does not matter. An empty list, an empty entry, an unkn...

(MATTERS_RELEVANT)

---

[style] ~64-~64: Since ownership is already implied, this phrasing may be redundant.
Context: ...keepalive wheel. Every API process runs its own set of these, and each API process rece...

(PRP_OWN)

---

[style] ~64-~64: Since ownership is already implied, this phrasing may be redundant.
Context: ...ch API process receives every event for its own SSE clients. | | ingest | The ingest ...

(PRP_OWN)

---

[style] ~66-~66: Since ownership is already implied, this phrasing may be redundant.
Context: ... role; with local, each process holds its own lease. | Every process, whatever its r...

(PRP_OWN)

docs/src/content/docs/architecture.md

[typographical] ~80-~80: Consider using an em dash in dialogues and enumerations.
Context: - router.go — Route definitions. Publ...

(DASH_RULE)

---

[style] ~94-~94: This phrase is redundant. Consider writing “last”.
Context: ...nt. The SIGHUP registration is released last of all. Handler, Registry, and MQ expose...

(LAST_OF_ALL)

---

[style] ~94-~94: Since ownership is already implied, this phrasing may be redundant.
Context: ...ons.Listenerlets one serve the API on its own listener instead ofserver.port`. - **...

(PRP_OWN)

docs/src/content/docs/deployment.md

[style] ~340-~340: Since ownership is already implied, this phrasing may be redundant.
Context: ...istener | - API. Each API pod runs its own schema discovery, token verifiers, dedu...

(PRP_OWN)

---

[style] ~340-~340: Since ownership is already implied, this phrasing may be redundant.
Context: ...ceives every event so that it can serve its own SSE clients. Put your Service and ingre...

(PRP_OWN)

AGENTS.md

[typographical] ~27-~27: Consider using an em dash in dialogues and enumerations.
Context: - cmd/wavehouse/ — Standalone mode ...

(DASH_RULE)

---

[grammar] ~27-~27: Please add a punctuation mark at the end of paragraph.
Context: ...e same binary can be one Deployment per role Twenty internal packages under `intern...

(PUNCTUATION_PARAGRAPH_END)

🔇 Additional comments (20)

internal/config/config.go (1)

370-375: LGTM!

internal/config/backends.go (1)

121-140: LGTM!

internal/config/defaults_test.go (1)

301-313: LGTM!

internal/config/backends_test.go (1)

13-15: LGTM!

internal/config/roles_test.go (1)

1-152: LGTM!

AGENTS.md (1)

27-37: LGTM!

config.yaml (1)

11-18: LGTM!

docs/src/content/docs/architecture.md (1)

80-80: LGTM!

docs/src/content/docs/configuration.mdx (1)

53-74: LGTM!

docs/src/content/docs/settings-directory.mdx (1)

182-182: LGTM!

internal/api/router.go (1)

236-267: LGTM!

internal/app/wire.go (1)

961-979: LGTM!

internal/api/router_test.go (1)

1131-1165: LGTM!

internal/app/roles_test.go (1)

1-187: LGTM!

internal/app/app.go (1)

178-222: LGTM!

internal/app/app_test.go (1)

103-103: LGTM!

tests/integration/setup_test.go (1)

168-168: LGTM!

tests/integration/tenants_test.go (1)

69-69: LGTM!

CHANGELOG.md (1)

13-13: LGTM!

docs/src/content/docs/deployment.md (1)

330-349: LGTM!

---

📝 Summary

Summary by CodeRabbit

• New Features
  • Processes can be configured to run API, ingest, and sweeper roles, with all roles enabled by default.
  • Non-API processes provide health probes, version information, metrics, and operator-key-protected settings reload.
  • Optional instance IDs identify individual processes and can serve as lease holder names when shared coordination is configured.
  • Startup now rejects role and backend combinations that cannot support the requested deployment.
• Documentation
  • Updated configuration and deployment guidance explains role responsibilities, operational endpoints, and backend requirements.

Walkthrough

The change adds boot-configured API, ingest, and sweeper roles, role-aware component wiring, and instance IDs. Processes without the API role receive an ops router for probes, version, metrics, and operator-key settings reload. Configuration rejects role splits incompatible with embedded MQ or local cache.

Changes

Process roles and topology

Layer / File(s)	Summary
Role configuration and topology validation internal/config/*, config.yaml, docs/src/content/docs/configuration.mdx, docs/src/content/docs/settings-directory.mdx, docs/src/content/docs/architecture.md, AGENTS.md	Configuration adds roles and instance_id, defaults roles to API, ingest, and sweeper, and validates role names and backend topology. Loading trims configured values and generates an instance ID when none is supplied. Documentation and examples describe the settings and constraints.
Shared routes and ops listener internal/api/router.go, internal/api/router_test.go, internal/app/wire.go, internal/app/roles_test.go, docs/src/content/docs/architecture.md	Shared probe, version, metrics, middleware, and error handling move into common router setup. NewOpsRouter adds an ops-only route set, with settings reload protected by operator-key authentication when settings are configured.
Role-specific application wiring and runtime behavior internal/app/app.go, internal/app/wire.go, internal/app/app_test.go, internal/app/roles_test.go, tests/integration/*, docs/src/content/docs/deployment.md, docs/src/content/docs/architecture.md, CHANGELOG.md, AGENTS.md	app.New selects components according to configured roles. Processes without the API role use ops authentication and HTTP wiring. Tests cover role selection, readiness, metrics, and sweeper election; deployment documentation describes role-separated deployments and backend requirements.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant ConfigLoad
  participant appNew
  participant wireOpsHTTP
  participant NewOpsRouter
  participant MainServer
  ConfigLoad->>appNew: provide validated roles and instance ID
  appNew->>wireOpsHTTP: configure HTTP when the API role is absent
  wireOpsHTTP->>NewOpsRouter: provide health, version, settings, auth, and metrics
  wireOpsHTTP->>MainServer: register the ops handler
  MainServer->>NewOpsRouter: route probes, version, metrics, or settings reload
  NewOpsRouter-->>MainServer: return the route response

Loading

Merge Risk: ⚪ Minimal · up to 1c26d

No identified issue blocks merging; role validation and the configured metrics listener remain intact.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 70.59% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 34 functions across 13 files. (7 skipped:…	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely identifies the main change: adding process roles to the application.
Description check	✅ Passed	The description directly explains the process-role configuration, role-based wiring, ops-only router, validation rules, documentation, tests, and deferred work.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

Docstring coverage is 70.59% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 34 functions across 13 files. (7 skipped: 7 unsupported.)

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

✨ Simplify code

• Commit to this branch
• Create a new PR

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

📚 Docs preview is live → https://7cfe8491-wavehouse-docs.wave-rf.workers.dev

• Commit — ada4bdd: test(integration): name the roles in TestQueryErrors_ClickHouseDown
• Author — @EricAndrechek, Claude Opus 5.5 (1M context)
• Committed — 2026-09-26 03:08 (UTC-04:00)
• Deployed — 2026-09-26 03:27 EDT

[github-code-quality]

Code Coverage Overview

Languages: Go

Go

The overall line coverage in commit ada4bdd in the feat/process-roles branch remains at 94%, unchanged from commit d7420f8 in the main branch.

Show a line coverage summary of the most impacted files.

File	main d7420f8	feat/process-roles ada4bdd	+/-
internal/config/config.go	97%	97%	0%
internal/app/wire.go	93%	93%	0%
internal/api/router.go	98%	98%	0%
internal/config/backends.go	100%	100%	0%
internal/app/app.go	96%	97%	+1%

---

_{Updated September 26, 2026 07:27 UTC}