feat(ingest)!: require a declared Content-Type and honor it

AGENTS.md

@@ -51,7 +51,7 @@ The invariant index — what must stay true. Full narrative and rationale live i
 
 1. **Interface-first** — core behaviors are Go interfaces (`Cache`, `Deduplicator`, `Publisher`, `Subscriber`); standalone vs. future-clustered swap implementations.
 2. **Bring Your Own Schema** — users create ClickHouse tables; WaveHouse discovers them via `system.columns` and never auto-migrates.
-3. **Schema-driven ingest** — `POST /v1/ingest?table={table}` takes flat JSON, validated against the discovered schema (unknown fields rejected, types/nullability enforced). No envelope.
+3. **Schema-driven ingest** — `POST /v1/ingest?table={table}` takes flat JSON, validated against the discovered schema (unknown fields rejected, types/nullability enforced). No envelope. The **declared `Content-Type` chooses the format and the bytes never do** (arity within the JSON family is still the body's): no declaration, one whose **media type** is unsupported or unparseable, a comma-bearing value that, as a whole, does not parse as one media type, or repeated lines that **disagree**, is a `415` decided *before* the body is read. A malformed *parameter* on a comma-free line never costs the request (`; charset=a; charset=b` still reads as its media type), and repeated lines are accepted only when they all resolve to the same **supported** format — two agreeing `text/csv` lines are still a `415`. A body declared NDJSON stays NDJSON whatever its bytes, so a bad line is a per-record error rather than a silent re-framing; the reverse (NDJSON sent as `application/json`) is deliberately **not** caught — record one, `200`, the rest ignored ([#561](https://github.com/Wave-RF/WaveHouse/issues/561)). Fail-closed — preserve it when touching `internal/api`.
 4. **Async ingestion** — ingest returns 200 after optional dedup + MQ publish; ClickHouse writes happen later via `StartIngestWorker`. NATS full → 503 + Retry-After.
 5. **Per-table batching** — the worker groups events by table and bulk-INSERTs in schema column order; each table's batch is independent.
 6. **Dead Letter Queue** — failed batch inserts publish to `WAVEHOUSE_DLQ` (`dlq.<table>`), gated per table by `dlq.enabled` in the settings directory's `config.json` (hot-reloadable; off = leave the row unacked for redelivery). No silent data loss.

CHANGELOG.md

@@ -22,6 +22,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 
 - **Policy format v2: `policies.json` is role-first, and the two operations are separate permission types** (BREAKING; `internal/policy/policy.go`, `internal/policy/rowfilter.go`, `internal/settings/validate.go`, `internal/query/builder.go`, `internal/api/{ingest,structured_query}.go`, `internal/stream/hub.go`, `clients/ts/src/{types,index}.ts`, `deployments/compose/settings/policies.json`, `docs/src/content/docs/{access-control.mdx,settings-directory.mdx,architecture.md}`, `AGENTS.md`, `tests/e2e/sdk/`): a table entry was keyed `tables.<table>.select.<role>`; it is now keyed `tables.<table>.<role>.select`. A role appears once per table and its grant carries two optional blocks, so a role that could both read and write no longer has to be written out twice, and "this role has no insert grant" is a missing block rather than an absence you have to notice in a second map. The blocks are now distinct types rather than one struct whose halves were inert per operation: `select` takes `allow_columns`, `deny_columns`, `filter`, `allowed_aggregations`, `denied_aggregations` and the four `max_*` limits; `insert` takes `allow_columns`, `deny_columns`, `check`. Field names and semantics are unchanged — only the nesting moves — but a field on the wrong side that the old layout *accepted* — the four `max_*` limits and the two aggregation rules — is now a validation error instead of being silently ignored. `filter` under an `insert` grant and `check` under a `select` one are rejected too, but that is not new here — [#541](https://github.com/Wave-RF/WaveHouse/pull/541), also unreleased, added the runtime check; the split types now refuse them one layer earlier, as unknown keys at the strict decode. Upgrading from **0.1.0**, though, none of the eight were enforced: a 0.1.0 policy could carry an insert-side `filter` that resolved into a `WHERE` the insert path never read, as well as an ignored limit. Converting to the role-first layout drops both. Internally `ResolvedPermissions` splits the same way (`.Select` / `.Insert`), and `IsColumnAllowed` takes the side to consult, which is what stops the read allowlist from ever answering a write question or vice versa. **There is no automatic conversion** — the settings files are the source of truth and WaveHouse has no write path back to them — so `policies.json` must be converted by hand; run `wavehouse validate` before restarting. A document still in the old layout is reported as one clear finding naming the table and operation and pointing at [the migration note](https://wavehouse.dev/access-control#migrating-from-the-operation-first-layout), instead of the confusing strict-decode "unknown field" error it would otherwise produce (or, for an empty operation block, silently decoding as a role named `select` with no grants — which fails the undeclared-role check when `roles.json` does not declare a role named `select` — the usual case, since `checkRoleRefs` errors and moves on before reaching the "grant sets neither select nor insert" warning. If such a role *is* declared, you get that warning instead and the document adopts). One shape is refused differently: a grant keyed by a role named after the *other* operation (`tables.t.select.insert`) reads as a different grant under each layout — different role, different operation, or both — so it gets its own error asking you to rename the role rather than the migration pointer. A role named after its *own* operation (`tables.t.select.select`) means the same thing either way and is accepted.
 
+- **Ingest now requires a declared `Content-Type`, and it is authoritative** (BREAKING; `internal/api/record_reader.go`, `internal/api/ingest.go`, `clients/ts/src/table.ts`, `docs/src/content/docs/{api.md,architecture.md,sdk/queries.md}`): `POST /v1/ingest` used to sniff the body and treat the header as a hint — the first non-whitespace byte chose between a single object and an array, and an `application/x-ndjson` body that happened to start with `[` was silently re-read as a JSON array. A request that declares **no** `Content-Type`, or one whose media type is not in the accepted list, is now rejected with `415` before the body is parsed, naming every accepted type (`application/json`, `application/x-ndjson`, `application/ndjson`, `application/jsonl`, `application/jsonlines`) and quoting what was declared, bounded to four distinct header lines each capped at 128 bytes. The header is parsed per RFC 9110 §8.3 via `mime.ParseMediaType` rather than by hand, so the grammar's rules apply — parameters never affect the format (with the one exception below), and a comma inside a quoted value is data. Because `Content-Type` is a **singleton** field (§5.3 forbids repeating it), anything that is not exactly one readable media type is refused; the one accommodation is that repeated header lines are all resolved and accepted when they agree, since honoring just the first would let an NDJSON body be read as one JSON object and drop every record past it. A comma-joined value gets no such accommodation — §8.3 warns that taking a member of the pseudo-list is itself an interoperability and security hazard. **Four additional shapes 0.1.0 accepted now `415`** (beyond repeated header lines that disagree, which it also accepted): a present-but-empty header (a bare `Content-Type:` line, or one that is only whitespace); a value with a trailing or leading comma (`application/json,`); a comma-joined value that does not parse as a single media type (`application/json, application/json` — but a comma *inside a quoted parameter value* is legal data, so `application/json; a=", application/x-ndjson; b="` is one media type and is accepted); and a malformed parameter on a line that *also* carries a comma, which is refused rather than guessed at because the comma may be a second declaration joined on — so `application/json; profile="a,b"; charset` is a `415` while `application/json; profile="a,b"` and `application/json; charset` are each accepted ([#563](https://github.com/Wave-RF/WaveHouse/issues/563)). A repeated parameter name is *not* among them: the media type is re-parsed alone, so `; charset=a; charset=b` reads as `application/json` like every other malformed parameter. The declaration now decides the format outright: a body declared as NDJSON is read as NDJSON whatever its first byte, so a line that isn't a JSON object fails as a **per-record** error through the existing batch-result path instead of re-framing the whole request. The body still picks arity *within* the JSON family — `[` is an array, anything else a single object — because those are the same format at different lengths. Clients that relied on the sniffer must now send a header; the TS SDK already sent one on both paths (`application/json` for a single object, `application/x-ndjson` for arrays and `insertNDJSON`) and now states it at each call site rather than leaning on the request default, and every `curl` example in the docs already carried one. The format is modeled as an `IngestFormat` where the sniffing used to live, with the slot for CSV kept where the old comment marked it.
+
 - **`cors.allowed_origins: []` now denies every browser origin instead of allowing all of them** (`internal/api/router.go`, `internal/settings/{validate,store}.go`, `docs/src/content/docs/settings-directory.mdx`; closes [#515](https://github.com/Wave-RF/WaveHouse/issues/515)): `corsMiddleware` treated an empty or nil allowlist as `["*"]`, so the natural spelling for "no origins" in a hand-edited, hot-reloadable `config.json` silently opened the API to every origin — [#508](https://github.com/Wave-RF/WaveHouse/pull/508) could only warn about it. An empty list is now an empty allowlist: no `Access-Control-Allow-Origin` (or any other CORS header) is sent to any origin, preflights get a bare `204`, and `Vary: Origin` is still emitted so a shared cache can't replay the headerless reject to an origin a later reload allows. A nil getter or nil slice (no settings source wired) denies the same way, so a missing source fails closed rather than open. `["*"]` is the only allow-all spelling. The `[]` validator warning is gone since the spelling now means what it says; the seed and the compose settings still ship `["*"]`, so nothing changes for a directory written by `wavehouse bootstrap`.
 
 - **The landing page's live demo now reads from the stats deployment's new WaveHouse Cloud backend** (`docs/src/components/LiveDemo.astro`, `docs/scripts/screenshot.mjs`): the GitHub-activity dogfood deployment behind the hero panel (Wave-RF/WaveHouse-Stats) moved off its self-managed AWS infrastructure onto WaveHouse Cloud, so `BASE_URL` — the origin `@wavehouse/sdk` queries in the visitor's browser — points at `https://iefrrvavd5akvphk7pq3.wavehouse.app` instead of `https://stats.wavehouse.dev`, ahead of the AWS stack being torn down. The `PUBLIC_WAVEHOUSE_STATS_URL` build-time override is unchanged, so a fork or staging docs build still redirects the panel without a code edit. **`DEMO_HOST` deliberately stays `stats.wavehouse.dev`** — the demo *site* is still served there and is still what the panel's chrome label and "Full demo" link should show; the migration splits the site from the API origin behind it, and the two constants now carry comments saying so. Verified against the new deployment before the switch: all five pipes the panel reads (`gh_summary`, `gh_activity_recent`, `gh_events_per_minute`, and the pre-#19 `gh_stars_total` / `gh_forks_total` fallbacks) return `200` with the same row shapes, the structured-query backfill fallback (`POST /v1/query?table=gh_events`) matches its old-backend response byte for byte, `GET /v1/stream?table=gh_events` opens an SSE stream, and CORS is unchanged (`Access-Control-Allow-Origin: *`, `X-Cache` exposed) so the cross-origin browser reads keep working from the docs site. The new backend is already the live ingest target — it reported more recent events than the old one at cutover (5,175 vs 5,038 over 7d) — which is the other half of why the panel had to follow it. `screenshot.mjs`'s `networkidle` note is retargeted to "the stats demo backend" rather than naming a host it no longer connects to.

clients/ts/src/table.test.ts

@@ -75,6 +75,16 @@ describe("TableRef", () => {
     const [url, init] = fetchSpy.mock.calls[0];
     expect(url).toContain("/v1/ingest?table=clicks");
     expect(init.method).toBe("POST");
+    // Pins the header that reaches the server, NOT the explicit declaration at
+    // the call site. Those are indistinguishable here: http.ts defaults to
+    // `opts.contentType ?? "application/json"`, so deleting the declaration
+    // produces byte-identical requests and every test still passes (verified).
+    // What this does catch is the default drifting while the declaration is
+    // absent — the two together are what make the request wrong. The NDJSON
+    // paths differ from the default, so theirs are genuine pins; this one is
+    // not, and a real one would have to observe the options handed to
+    // request() rather than the fetch call.
+    expect(init.headers["Content-Type"]).toBe("application/json");
     expect(JSON.parse(init.body)).toEqual({ page: "/home", score: 42 });
   });
 

clients/ts/src/table.ts

@@ -14,6 +14,9 @@ import type {
 
 type CreateStreamFn<Row> = (table: string, opts?: StreamOptions) => StreamController<Row>;
 
+/** Content-Type for the JSON single-object ingest path. */
+const JSON_CONTENT_TYPE = "application/json";
+
 /** Content-Type for the NDJSON batch ingest path. */
 const NDJSON_CONTENT_TYPE = "application/x-ndjson";
 
@@ -119,6 +122,9 @@ export class TableRef<Row = Record<string, unknown>> {
       method: "POST",
       path: `/v1/ingest?table=${encodeURIComponent(this._table)}`,
       body: data,
+      // Ingest requires a declared format — an undeclared Content-Type is a
+      // 415 — so state it here rather than leaning on the request default.
+      contentType: JSON_CONTENT_TYPE,
       signal: opts?.signal,
     });