From 1a2dfe2bd27b3d8cba3e519231ce7c0fb6989d79 Mon Sep 17 00:00:00 2001 From: user Date: Tue, 18 Aug 2026 09:34:00 -0400 Subject: [PATCH 01/33] docs(specs): add web console distributed tracing spec (HYPERSHELL-27) Define the concrete OpenTelemetry wiring for the web console as a behavior contract that narrows WEB-OBS-01/02 and the domain-observability standard: - Browser spans derived from typed domain probes via a DomainProbeSink - Same-origin BFF telemetry ingest (OTLP/HTTP); no cross-origin export - Browser W3C traceparent injection on outbound /api/* calls - BFF validates/replaces inbound trace context and propagates upstream - BFF server span per proxied request, OTLP export - Configurable, optional OTLP endpoint + sampling; privacy/cardinality rules - 'trace' probe consumer in the catalog - bounded delivery + flush semantics Add a development Jaeger all-in-one to local-development.spec.md (gated by KIND_TRACING) and register the new spec in the index. Co-Authored-By: Claude Opus 4.8 --- specs/index.spec.md | 1 + specs/platform/local-development.spec.md | 40 +++++++ specs/web-console/tracing.spec.md | 130 +++++++++++++++++++++++ 3 files changed, 171 insertions(+) create mode 100644 specs/web-console/tracing.spec.md diff --git a/specs/index.spec.md b/specs/index.spec.md index a9b74065..b42927fd 100644 --- a/specs/index.spec.md +++ b/specs/index.spec.md @@ -39,6 +39,7 @@ Machine-readable index for autonomous reconciliation (`/reconcile` skill). | `platform/openshell-gateway-secret-rotation.spec.md` | platform | Secret rotation: DB password, KEK, TLS certificates | CP | openshell-gateway-database, openshell-gateway-credentials, openshell-gateway-tls | | `platform/openshell-gateway-keycloak.spec.md` | platform | Keycloak OIDC client provisioning, per-gateway OIDC role bridge | CP | openshell-gateway, openshell-gateway-oidc, rbac-enforcement | | `web-console/architecture.spec.md` | web-console | Web console, BFF, browser session, UI routes | WEB, SDK, API | data-model, security, UI standards | +| `web-console/tracing.spec.md` | web-console | Browser OTel trace sink, BFF W3C propagation, telemetry ingest, dev Jaeger | WEB, BFF | web-console/architecture, domain-observability, local-development | | `standards/platform/cross-cutting.spec.md` | standards | - | ALL | - | | `standards/control-plane/conventions.spec.md` | standards | - | CP | - | | `security/rbac-enforcement.spec.md` | security | User, Role, RoleBinding, RBAC middleware | API | data-model | diff --git a/specs/platform/local-development.spec.md b/specs/platform/local-development.spec.md index 410cc58a..fee2df21 100644 --- a/specs/platform/local-development.spec.md +++ b/specs/platform/local-development.spec.md @@ -291,6 +291,20 @@ Client Networking Gateway Gateway Pod | gRPC hostname pattern | `.gw.localhost` | `-.gw.` | | DNS resolution | CoreDNS container + OS resolver + pfctl/iptables port forwarding | Cluster DNS / external DNS | +### Development Tracing (Jaeger) + +The local environment SHALL deploy a Jaeger all-in-one instance so a developer can view distributed traces produced by the web console and the API server (see `web-console/tracing.spec.md` and HYPERSHELL-26). Jaeger all-in-one exposes an OTLP receiver and a query UI in one workload; a separate OpenTelemetry Collector is not required for local development. Trace storage uses in-memory storage, which is cleared on restart and is appropriate for development only. + +`make kind-up` SHALL deploy Jaeger into the target namespace with its OTLP receiver enabled, and SHALL create an HTTPRoute that exposes the Jaeger UI at `jaeger.hypershell.localhost`. The web console BFF SHALL be configured to export traces to the Jaeger OTLP endpoint through the `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable on the web-console Deployment. The browser exports through the same-origin BFF telemetry endpoint (see `web-console/tracing.spec.md`), so the browser does not reach Jaeger directly. + +| Setting | Value | +|---------|-------| +| Workload | `jaeger` Deployment (all-in-one, in-memory storage) | +| OTLP receiver (in-cluster) | `http://jaeger-collector..svc.cluster.local:4318` (OTLP/HTTP) | +| Query UI | `https://jaeger.hypershell.localhost` | + +Deployment of Jaeger SHALL be gated by the `KIND_TRACING` environment variable (default `true`). When `KIND_TRACING=false`, `make kind-up` SHALL skip the Jaeger workload and its route, and the BFF SHALL start with tracing disabled without failing readiness. + ## Requirements ### Requirement: Single-Command Environment Setup @@ -441,6 +455,7 @@ All services SHALL be accessible via `.localhost` hostnames routed through the n | HTTP API | `https://api.hypershell.localhost` | REST API access | | Web Console | `https://console.hypershell.localhost` | Browser UI | | Health | `https://health.hypershell.localhost` | Health check endpoint | +| Jaeger UI | `https://jaeger.hypershell.localhost` | Distributed tracing UI (when `KIND_TRACING` is enabled) | | gRPC | `https://.gw.localhost` | gRPC streaming (control plane, CLI) | The self-signed TLS certificate issued by cert-manager must be trusted by the developer's browser or CLI tool (e.g. `curl --cacert`). @@ -655,6 +670,29 @@ All `kind-*` targets operate on the namespace specified by `KIND_NAMESPACE` (def - THEN the API server SHALL be swapped only in the `hypershell-feature-add-auth` namespace - AND the default deployment SHALL remain unchanged +### Requirement: Development Tracing + +The system SHALL deploy a Jaeger all-in-one instance in the local environment and configure the web console BFF to export traces to it, so a developer can verify one end-to-end trace. Deployment SHALL be gated by `KIND_TRACING` (default `true`). + +#### Scenario: Jaeger Available After Setup +- GIVEN `KIND_TRACING` is unset or `true` +- WHEN a developer runs `make kind-up` +- THEN a Jaeger all-in-one workload SHALL be deployed into the target namespace +- AND an HTTPRoute SHALL expose the Jaeger UI at `jaeger.hypershell.localhost` +- AND the web console BFF SHALL be configured with `OTEL_EXPORTER_OTLP_ENDPOINT` pointing at the Jaeger OTLP receiver + +#### Scenario: End-to-End Trace Visible +- GIVEN the local environment is running with tracing enabled +- WHEN a developer completes a gateway workflow in the web console +- THEN one trace SHALL be visible in the Jaeger UI +- AND it SHALL contain the browser workflow span and the BFF server span joined by the same trace identifier + +#### Scenario: Tracing Disabled +- GIVEN `KIND_TRACING=false` +- WHEN a developer runs `make kind-up` +- THEN the Jaeger workload and its route SHALL NOT be deployed +- AND the web console BFF SHALL start with tracing disabled without failing readiness + ## Environment Variable Reference | Env Var | Default | Description | @@ -678,6 +716,7 @@ All `kind-*` targets operate on the namespace specified by `KIND_NAMESPACE` (def | `CERT_MANAGER_VERSION` | `v1.21.1` | cert-manager release version | | `KIND_DB_IMAGE` | `registry.access.redhat.com/hi/postgresql:18` | Database image for Gateway resource; override for OSS dev (unsupported) | | `KIND_NAMESPACE` | `hypershell-system` | Target namespace for all `kind-*` targets | +| `KIND_TRACING` | `true` | Deploy Jaeger all-in-one for distributed tracing; set to `false` to skip | ## Make Targets Summary @@ -710,6 +749,7 @@ All targets operate on `KIND_NAMESPACE` (default: `hypershell-system`). | Images loaded via tarball archive | Compatible with both Podman and Docker; avoids registry dependency | | Rebuild-and-replace on every swap call | Each `kind--up` rebuilds from the working tree and replaces the deployment, even if already swapped; developers iterate by re-running the same target | | Web console as first-class component | Node.js frontend (`components/web-console/`) deployed alongside API server and control plane; supports hot reload via `KIND_HOT_RELOAD` for rapid UI iteration | +| Jaeger all-in-one for local tracing | One workload provides both the OTLP receiver and the query UI with in-memory storage; avoids a separate OpenTelemetry Collector in development. The BFF exports over OTLP; the browser exports through the same-origin BFF endpoint. `KIND_TRACING=false` opts out | | Hot reload on by default | Swap targets for supported components (web console) mount host source and run a dev server in an interactive TTY by default; `KIND_HOT_RELOAD=false` opts out to rebuild-and-replace. Keeps the same `kind--up` entrypoint for both workflows | | Per-component targets require existing cluster | Avoids implicit full-stack deployment; keeps intent explicit | | Database provisioned by control plane | Gateway configured with HI postgresql image; control plane reconciler provisions the database via GatewayReconciler (`specs/platform/openshell-gateway-database.spec.md`, implemented in [#14](https://github.com/openshift-online/hypershell/pull/14)), exercising the same path as production. The API server's own database (`deploy/kind/postgres.yaml`) is always deployed directly by `kind-up` | diff --git a/specs/web-console/tracing.spec.md b/specs/web-console/tracing.spec.md new file mode 100644 index 00000000..f6349013 --- /dev/null +++ b/specs/web-console/tracing.spec.md @@ -0,0 +1,130 @@ +# Web Console Distributed Tracing + +**Status:** Active +**Applies to:** `components/web-console` browser application, the web-console BFF, `components/web-console/domain-probes`, the gateway management UI probe catalog, and the local development observability workflow +**Jira:** HYPERSHELL-27 + +## Purpose + +Give the web console distributed tracing that follows one user workflow from the browser, through the BFF, to the HyperShell REST API. Traces are derived from the typed domain probes already defined by `standards/ui/domain-observability.spec.md`; they are not a second, parallel telemetry path. This specification narrows the tracing requirements in `web-console/architecture.spec.md` (`WEB-OBS-01`, `WEB-OBS-02`) and the standard in `standards/ui/domain-observability.spec.md` into a concrete OpenTelemetry (OTel) wiring. It does not replace those documents; where they impose a stricter rule, that rule governs. + +The API server is instrumented separately (HYPERSHELL-26). This specification requires the browser and BFF to produce spans and to propagate W3C Trace Context so the API server can join the same trace once it extracts that context. + +## Requirements + +### Requirement: WEB-TRACE-01 -- Browser Spans Derived From Domain Probes + +The browser SHALL produce spans from the typed domain probes published through the `DomainProbePublisher` fan-out, not from ad hoc instrumentation. A trace sink SHALL implement the `DomainProbeSink` contract and register through the existing `additionalSinks` extension point of the gateway observability adapter. The sink SHALL map a workflow `started` probe and its matching terminal probe (`succeeded`, `failed`, `cancelled`, `denied`, or `conflicted`) to one span, and each dependency-attempt probe pair to one child span. Span status SHALL reflect the terminal outcome. The sink SHALL NOT create spans from React renders, pointer movement, or pure computation. + +Browser application, domain, and API-adapter code SHALL NOT import an OpenTelemetry package. Only the observability adapter and the composition root MAY import OTel, as enforced by `eslint.architecture.mjs`. Experimental browser auto-instrumentation SHALL NOT be a foundational dependency. + +**Verification:** Run each gateway use case with a recording publisher plus the trace sink; assert one workflow span per invocation, one child span per dependency attempt, and a span status that matches the terminal outcome. Confirm no span is emitted for a rerender. Run the architecture lint and confirm an OTel import outside the approved paths fails. + +#### Scenario: Workflow Produces One Span + +- GIVEN a user runs a gateway workflow (for example list, provision, rename, or delete) +- WHEN the workflow publishes its started probe and one terminal probe +- THEN the trace sink SHALL produce exactly one workflow span with a status derived from the terminal outcome +- AND each control-plane dependency attempt SHALL produce one child span + +### Requirement: WEB-TRACE-02 -- Same-Origin Browser Telemetry Export + +The browser SHALL export spans to a same-origin BFF telemetry endpoint using OTLP over HTTP. The browser SHALL NOT export directly to a collector origin and SHALL NOT require a Content Security Policy `connect-src` value other than `'self'`. The BFF SHALL expose the ingest endpoint, apply a request-body size limit, require the same session and CSRF protection as other state-changing browser requests, and forward accepted spans to the configured collector. The endpoint SHALL reject a body that is not valid OTLP. + +**Verification:** Load the built console behind the BFF with the enforcing CSP; confirm the browser exporter posts to the same-origin path and that no cross-origin telemetry request is attempted. Post an oversized and a malformed body and confirm rejection. Confirm the BFF forwards a well-formed body to the collector. + +#### Scenario: Browser Exports Through the BFF + +- GIVEN the console has produced one or more spans +- WHEN the browser exporter flushes +- THEN it SHALL post OTLP data to the same-origin BFF telemetry endpoint +- AND the BFF SHALL forward the accepted spans to the configured collector +- AND no telemetry request SHALL go to a cross-origin destination + +### Requirement: WEB-TRACE-03 -- Browser Outbound Trace Context + +The browser SHALL set the W3C `traceparent` header, and `tracestate` when present, on every outbound `/api/*` request at the single correlated-fetch chokepoint. The `traceparent` value SHALL reference the active workflow span so the BFF and API spans join the same trace. The browser SHALL keep sending the existing `x-hypershell-correlation-id` header. The trace context and the correlation identifier SHALL both appear in the `DomainProbeContext` so probe consumers can join a workflow to its trace. + +**Verification:** Run a representative browser-to-API workflow; capture the outbound request and assert a well-formed `traceparent` that references the workflow span, plus the correlation header. Confirm the probe context carries the same trace identifier. + +### Requirement: WEB-TRACE-04 -- BFF W3C Trace Context Propagation + +The BFF SHALL treat an inbound `traceparent`/`tracestate` pair as untrusted input. It SHALL extract valid context, continue that trace, and set a valid `traceparent` (and forward a valid `tracestate`) on the upstream API request. When the inbound context is absent or malformed, the BFF SHALL start a new trace and set a valid `traceparent`; it SHALL NOT forward a malformed value. The upstream request header allowlist SHALL include `traceparent` and `tracestate`. The BFF SHALL keep validating, echoing, and propagating `x-hypershell-correlation-id` as it does today. + +**Verification:** Send requests with valid, absent, and malformed inbound trace context; assert the upstream request always carries a valid `traceparent`, a malformed value is replaced rather than forwarded, and the correlation identifier is preserved. Confirm `traceparent` and `tracestate` are in the upstream allowlist. + +#### Scenario: BFF Continues an Inbound Trace + +- GIVEN a browser request arrives with a valid `traceparent` +- WHEN the BFF proxies the request to the API +- THEN the upstream request SHALL carry a `traceparent` that continues the same trace +- AND a valid inbound `tracestate` SHALL be forwarded + +#### Scenario: BFF Replaces Malformed Context + +- GIVEN a browser request arrives with a malformed `traceparent` +- WHEN the BFF proxies the request to the API +- THEN the BFF SHALL start a new trace and set a valid `traceparent` +- AND it SHALL NOT forward the malformed value + +### Requirement: WEB-TRACE-05 -- BFF Server Spans + +The BFF SHALL produce one server span per proxied `/api/*` request as a child of the extracted inbound context, and export it by OTLP to the configured collector. The BFF OTel SDK SHALL be initialized once in the server bootstrap path, which is exempt from the observability import ban. Product and route code SHALL NOT call the OTel API directly; only the observability adapter, the composition root, and the bootstrap MAY. Span attributes SHALL record the templated route, the request method, the upstream outcome class, and the correlation identifier. The BFF SHALL keep its existing structured request log. + +**Verification:** Proxy a representative request and confirm one BFF server span that is a child of the inbound context, carries the templated-route and outcome attributes, and reaches the collector. Confirm route and product code contain no direct OTel API calls. + +### Requirement: WEB-TRACE-06 -- Configurable Export and Sampling + +The collector endpoint and the trace sample rate SHALL be deployment configuration, validated at BFF startup through the existing configuration schema. Configuration SHALL be separate from code. When tracing configuration is absent, the BFF SHALL start normally with tracing disabled and SHALL NOT fail readiness. A browser-visible value SHALL be limited to the non-sensitive, allowlisted schema required by `WEB-DEPLOY-02`; a collector origin or secret SHALL NOT be embedded in the browser bundle. + +**Verification:** Start the BFF with valid, absent, and invalid tracing configuration; confirm valid configuration exports to the collector, absent configuration disables tracing without failing readiness, and invalid configuration fails startup with a clear message. Inspect the browser bundle for an embedded collector origin or secret. + +### Requirement: WEB-TRACE-07 -- Trace Privacy and Cardinality + +Spans and their attributes SHALL follow the privacy and cardinality rules of `standards/ui/domain-observability.spec.md` (`UI-OBS-07`). Tokens, cookies, credentials, secrets, raw headers, request or response bodies, user-entered content, and raw resource identifiers in span names SHALL be prohibited. Stable route templates, bounded enums, and error classes SHALL replace raw URLs and messages. A high-cardinality identifier SHALL NOT become a span name segment or propagated baggage. When an API failure returns an operation identifier, the matching workflow and dependency terminal spans SHALL retain it so support can join a user-visible failure to server evidence. + +**Verification:** Run redaction tests with seeded secrets and user data across the browser sink and the BFF span exporter. Confirm span names use route templates, attributes are bounded, no prohibited value appears, and an API operation identifier is retained on failure spans. + +### Requirement: WEB-TRACE-08 -- Trace Consumer in the Probe Catalog + +The gateway probe catalog SHALL declare `trace` as an allowed consumer for every probe the trace sink reads, and the observability import allowlist SHALL permit the trace sink to consume those probes. The catalog SHALL remain in agreement with the typed probe union and the sink mappings. A probe that no consumer reads and a duplicate probe name SHALL NOT be introduced. + +**Verification:** Compare the catalog with the probe union and the trace sink mapping; confirm every probe the sink reads lists `trace` in `allowedConsumers`, and that catalog and code agree in CI. + +### Requirement: WEB-TRACE-09 -- Bounded Delivery and Flush + +The browser trace sink and the BFF exporter SHALL each own a bounded buffer and an explicit flush policy; they SHALL NOT grow without limit. The browser SHALL flush on page hide and visibility change so spans are not lost on navigation or tab close. A tracing export failure SHALL be best-effort: it SHALL NOT change a workflow result and SHALL surface through the existing bounded delivery-failure diagnostic rather than a recursive publication. Fan-out SHALL still attempt every other sink when the trace sink fails. + +**Verification:** Inject a trace sink and an exporter that throw, block, and overflow; confirm other sinks still receive the probe, the workflow result is unchanged, memory stays bounded, and the failure is visible without recursion. Hide the page mid-workflow and confirm buffered spans flush. + +### Requirement: WEB-TRACE-10 -- End-to-End Trace in Development + +The local development environment SHALL provide a collector and a Jaeger instance so a developer can view one trace that joins the browser and the BFF (and the API server once it is instrumented). The development collector endpoint SHALL be supplied to the BFF by configuration. The details of the development deployment are defined in `platform/local-development.spec.md`. + +**Verification:** Run the local environment, complete a representative gateway workflow in the browser, and confirm one trace in Jaeger that contains the browser workflow span and the BFF server span joined by W3C Trace Context. + +#### Scenario: Developer Views the Trace + +- GIVEN the local development environment is running with the collector and Jaeger +- WHEN a developer completes a gateway workflow in the browser +- THEN one trace SHALL be visible in Jaeger +- AND it SHALL contain the browser workflow span and the BFF server span joined by the same trace identifier + +## Design Decisions + +| Decision | Rationale | +| --- | --- | +| Derive spans from domain probes, not auto-instrumentation | Keeps one authoritative telemetry model, satisfies the domain-observability standard, and avoids the bundle, privacy, and stability risk that `WEB-OBS-02` cautions against | +| Browser exports through a same-origin BFF endpoint | Keeps the collector origin out of the browser, works with the strict `connect-src 'self'` CSP, and reuses the BFF session and CSRF controls | +| BFF owns W3C Trace Context propagation | The BFF is the trust boundary; it validates or replaces untrusted inbound context and sets a valid value upstream so the API can join the trace | +| Tracing configuration is optional and validated at startup | Configuration is separate from code; a deployment without a collector starts normally with tracing disabled | +| Trace is a probe consumer in the catalog | The catalog stays the single source of truth for who consumes each probe, per `UI-OBS-10` | + +## Primary Basis + +- `standards/ui/domain-observability.spec.md` +- `web-console/architecture.spec.md` (`WEB-OBS-01`, `WEB-OBS-02`, `WEB-BFF-01`, `WEB-DEPLOY-02`, `WEB-SEC-01`) +- [OpenTelemetry JavaScript](https://opentelemetry.io/docs/languages/js/) +- [OpenTelemetry semantic conventions](https://opentelemetry.io/docs/concepts/semantic-conventions/) +- [W3C Trace Context](https://www.w3.org/TR/trace-context/) +- [OTLP specification](https://opentelemetry.io/docs/specs/otlp/) From 4b83ff89dd1d50e6bfd605df34479cf83cbe6ce8 Mon Sep 17 00:00:00 2001 From: user Date: Tue, 18 Aug 2026 09:51:31 -0400 Subject: [PATCH 02/33] feat(web-console): thread W3C trace id through gateway domain probes Add a required createTraceId() capability to GatewayWorkflowRuntime and publish the resulting W3C trace id in every gateway probe context, so a trace sink can adopt it as the span trace id and probe consumers can join a workflow to its trace (WEB-TRACE-03, UI-OBS-08). Declare `trace` as an allowed consumer for every gateway probe in the catalog (WEB-TRACE-08). The production runtime in the observability adapter generates a 16-byte random trace id rendered as 32 lowercase hex digits. No OpenTelemetry dependency is introduced by this change. Co-Authored-By: Claude Opus 4.8 --- .../gateway-observability.test.ts | 16 +++++++++++ .../observability/gateway-observability.ts | 16 +++++++++++ .../application/gateway-operations.test.ts | 14 ++++++++++ .../src/application/gateway-operations.ts | 9 ++++++ .../src/application/gateway-probes.ts | 28 ++++++++++++++++--- .../src/application/gateway-types.ts | 7 +++++ 6 files changed, 86 insertions(+), 4 deletions(-) diff --git a/components/web-console/app/adapters/observability/gateway-observability.test.ts b/components/web-console/app/adapters/observability/gateway-observability.test.ts index 8d4ae69e..c7493678 100644 --- a/components/web-console/app/adapters/observability/gateway-observability.test.ts +++ b/components/web-console/app/adapters/observability/gateway-observability.test.ts @@ -51,11 +51,27 @@ describe("gateway observability adapter", () => { it("provides deterministic workflow context through injected capabilities", () => { const observability = createGatewayObservability({ createCorrelationId: () => "correlation-1", + createTraceId: () => "0af7651916cd43dd8448eb211c80319c", now: () => "2026-08-06T18:00:00.000Z", performanceTarget: { clearMarks: vi.fn(), mark: vi.fn() }, }); expect(observability.runtime.createCorrelationId()).toBe("correlation-1"); + expect(observability.runtime.createTraceId()).toBe( + "0af7651916cd43dd8448eb211c80319c", + ); expect(observability.runtime.now()).toBe("2026-08-06T18:00:00.000Z"); }); + + it("generates a valid W3C trace identifier by default", () => { + const observability = createGatewayObservability({ + performanceTarget: { clearMarks: vi.fn(), mark: vi.fn() }, + }); + + const traceId = observability.runtime.createTraceId(); + + expect(traceId).toMatch(/^[0-9a-f]{32}$/); + expect(traceId).not.toBe("0".repeat(32)); + expect(observability.runtime.createTraceId()).not.toBe(traceId); + }); }); diff --git a/components/web-console/app/adapters/observability/gateway-observability.ts b/components/web-console/app/adapters/observability/gateway-observability.ts index 74576377..8845cb2c 100644 --- a/components/web-console/app/adapters/observability/gateway-observability.ts +++ b/components/web-console/app/adapters/observability/gateway-observability.ts @@ -13,6 +13,20 @@ import { const recentProbeLimit = 100; const recentFailureLimit = 20; +const traceIdByteLength = 16; + +/** + * Creates a W3C trace identifier: a 16-byte random value rendered as 32 + * lowercase hex digits. A random 16-byte value is never the all-zero value + * that the W3C Trace Context specification forbids. + */ +function createTraceId(): string { + const bytes = new Uint8Array(traceIdByteLength); + globalThis.crypto.getRandomValues(bytes); + return Array.from(bytes, (byte) => byte.toString(16).padStart(2, "0")).join( + "", + ); +} interface PerformanceProbeTarget { clearMarks(name?: string): void; @@ -30,6 +44,7 @@ export interface GatewayObservability { export interface GatewayObservabilityOptions { additionalSinks?: readonly DomainProbeSink[]; createCorrelationId?: () => string; + createTraceId?: () => string; now?: () => string; performanceTarget?: PerformanceProbeTarget; } @@ -88,6 +103,7 @@ export function createGatewayObservability( runtime: { createCorrelationId: options.createCorrelationId ?? (() => globalThis.crypto.randomUUID()), + createTraceId: options.createTraceId ?? createTraceId, now: options.now ?? (() => new Date().toISOString()), }, }; diff --git a/packages/gateway-management-ui/src/application/gateway-operations.test.ts b/packages/gateway-management-ui/src/application/gateway-operations.test.ts index 67de9393..e94d78b8 100644 --- a/packages/gateway-management-ui/src/application/gateway-operations.test.ts +++ b/packages/gateway-management-ui/src/application/gateway-operations.test.ts @@ -70,6 +70,9 @@ function setup() { correlation += 1; return `correlation-${String(correlation)}`; }, + createTraceId() { + return `trace-${String(correlation)}`; + }, now() { return "2026-08-06T18:00:00.000Z"; }, @@ -162,6 +165,9 @@ describe("gateway application operations", () => { ({ context }) => context.correlationId === "correlation-1", ), ).toBe(true); + expect(received.every(({ context }) => context.traceId === "trace-1")).toBe( + true, + ); }); it("publishes a conflicted terminal outcome and preserves the typed failure", async () => { @@ -209,4 +215,12 @@ describe("gateway application operations", () => { "gateway.dependency.completed", ]); }); + + it("declares trace as an allowed consumer for every gateway probe", () => { + expect( + gatewayProbeCatalog.every(({ allowedConsumers }) => + allowedConsumers.includes("trace"), + ), + ).toBe(true); + }); }); diff --git a/packages/gateway-management-ui/src/application/gateway-operations.ts b/packages/gateway-management-ui/src/application/gateway-operations.ts index 02790c10..9349b664 100644 --- a/packages/gateway-management-ui/src/application/gateway-operations.ts +++ b/packages/gateway-management-ui/src/application/gateway-operations.ts @@ -51,6 +51,7 @@ function failureOutcome(error: unknown): GatewayProbeOutcome { function probe( action: GatewayAction, correlationId: string, + traceId: string, failure: ReturnType | null, name: GatewayProbe["name"], occurredAt: string, @@ -61,6 +62,7 @@ function probe( return Object.freeze({ context: Object.freeze({ correlationId, + traceId, ...(operationId === undefined ? {} : { operationId }), ...(parentInvocationId === undefined ? {} : { parentInvocationId }), }), @@ -82,6 +84,7 @@ export function createGatewayOperations({ task: (context: GatewayInvocationContext) => Promise, ): Promise { const correlationId = runtime.createCorrelationId(); + const traceId = runtime.createTraceId(); const context: GatewayInvocationContext = { correlationId, ...(signal === undefined ? {} : { signal }), @@ -90,6 +93,7 @@ export function createGatewayOperations({ probe( action, correlationId, + traceId, null, "gateway.workflow.started", runtime.now(), @@ -100,6 +104,7 @@ export function createGatewayOperations({ probe( action, correlationId, + traceId, null, "gateway.dependency.attempted", runtime.now(), @@ -115,6 +120,7 @@ export function createGatewayOperations({ probe( action, correlationId, + traceId, null, "gateway.dependency.completed", runtime.now(), @@ -127,6 +133,7 @@ export function createGatewayOperations({ probe( action, correlationId, + traceId, null, "gateway.workflow.completed", runtime.now(), @@ -143,6 +150,7 @@ export function createGatewayOperations({ probe( action, correlationId, + traceId, kind, "gateway.dependency.completed", runtime.now(), @@ -155,6 +163,7 @@ export function createGatewayOperations({ probe( action, correlationId, + traceId, kind, "gateway.workflow.completed", runtime.now(), diff --git a/packages/gateway-management-ui/src/application/gateway-probes.ts b/packages/gateway-management-ui/src/application/gateway-probes.ts index 07c5147f..b8fe3494 100644 --- a/packages/gateway-management-ui/src/application/gateway-probes.ts +++ b/packages/gateway-management-ui/src/application/gateway-probes.ts @@ -36,7 +36,12 @@ export type GatewayProbePublisher = DomainProbePublisher; export const gatewayProbeCatalog = Object.freeze([ { - allowedConsumers: ["structured-log", "performance", "product-health"], + allowedConsumers: [ + "structured-log", + "performance", + "product-health", + "trace", + ], deliveryClass: "best-effort", fields: { action: "bounded operational enum", @@ -49,7 +54,12 @@ export const gatewayProbeCatalog = Object.freeze([ trigger: "A gateway application use case starts", }, { - allowedConsumers: ["structured-log", "performance", "product-health"], + allowedConsumers: [ + "structured-log", + "performance", + "product-health", + "trace", + ], deliveryClass: "best-effort", fields: { action: "bounded operational enum", @@ -62,7 +72,12 @@ export const gatewayProbeCatalog = Object.freeze([ trigger: "A gateway application use case reaches one terminal outcome", }, { - allowedConsumers: ["structured-log", "performance", "product-health"], + allowedConsumers: [ + "structured-log", + "performance", + "product-health", + "trace", + ], deliveryClass: "best-effort", fields: { action: "bounded operational enum", @@ -75,7 +90,12 @@ export const gatewayProbeCatalog = Object.freeze([ trigger: "A gateway control-plane dependency attempt starts", }, { - allowedConsumers: ["structured-log", "performance", "product-health"], + allowedConsumers: [ + "structured-log", + "performance", + "product-health", + "trace", + ], deliveryClass: "best-effort", fields: { action: "bounded operational enum", diff --git a/packages/gateway-management-ui/src/application/gateway-types.ts b/packages/gateway-management-ui/src/application/gateway-types.ts index efb4bd4a..b7a33024 100644 --- a/packages/gateway-management-ui/src/application/gateway-types.ts +++ b/packages/gateway-management-ui/src/application/gateway-types.ts @@ -157,5 +157,12 @@ export interface GatewayOperations { /** Application-owned port for nondeterministic workflow context. */ export interface GatewayWorkflowRuntime { createCorrelationId(): string; + /** + * Creates the W3C trace identifier (16-byte value as 32 lowercase hex + * digits) that identifies one workflow invocation across the browser, the + * BFF, and the API. A trace sink adopts this value as the span trace id, so + * probe consumers can join a workflow to its trace. + */ + createTraceId(): string; now(): string; } From 8200ddb0a147ed29f3ae967990cdb822bab7d868 Mon Sep 17 00:00:00 2001 From: user Date: Tue, 18 Aug 2026 10:04:03 -0400 Subject: [PATCH 03/33] feat(web-console): export browser workflow traces via OTel [HYPERSHELL-27] Add an OpenTelemetry trace sink behind the gateway domain-probe port. The sink projects gateway workflow and dependency probes onto spans, adopting the W3C trace id carried on each probe context so a workflow joins the same trace the browser propagates to the BFF and API. Spans batch and export over same-origin OTLP/HTTP, flushing on document hide. The API client gains an injected trace-context provider that stamps traceparent/tracestate onto outbound requests, keeping the API adapter free of any tracing vendor dependency. The composition root wires the tracer provider, registers the sink, and feeds the propagation reader to the client. Co-Authored-By: Claude Opus 4.8 --- .../app/adapters/api/api.client.test.ts | 42 +++ .../app/adapters/api/api.client.ts | 23 ++ .../observability/gateway-observability.ts | 2 - .../observability/gateway-trace-sink.test.ts | 208 +++++++++++++ .../observability/gateway-trace-sink.ts | 283 ++++++++++++++++++ .../app/composition/gateway-composition.ts | 21 +- components/web-console/package.json | 5 + pnpm-lock.yaml | 165 +++++++++- 8 files changed, 739 insertions(+), 10 deletions(-) create mode 100644 components/web-console/app/adapters/observability/gateway-trace-sink.test.ts create mode 100644 components/web-console/app/adapters/observability/gateway-trace-sink.ts diff --git a/components/web-console/app/adapters/api/api.client.test.ts b/components/web-console/app/adapters/api/api.client.test.ts index 014031f9..aa875604 100644 --- a/components/web-console/app/adapters/api/api.client.test.ts +++ b/components/web-console/app/adapters/api/api.client.test.ts @@ -76,6 +76,48 @@ describe("correlated API fetch", () => { expect(onReauth).not.toHaveBeenCalled(); expect(response.status).toBe(401); }); + + it("propagates the W3C trace context supplied by the provider", async () => { + const fetchImplementation = vi + .fn() + .mockResolvedValue(new Response(null, { status: 204 })); + const correlatedFetch = createCorrelatedFetch( + "44444444-4444-4444-8444-444444444444", + fetchImplementation, + undefined, + () => ({ + traceparent: + "00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01", + tracestate: "hypershell=1", + }), + ); + + await correlatedFetch("/api/hypershell/v1/gateways"); + + const headers = new Headers(fetchImplementation.mock.calls[0]?.[1]?.headers); + expect(headers.get("traceparent")).toBe( + "00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01", + ); + expect(headers.get("tracestate")).toBe("hypershell=1"); + }); + + it("omits trace headers when the provider reports no active span", async () => { + const fetchImplementation = vi + .fn() + .mockResolvedValue(new Response(null, { status: 204 })); + const correlatedFetch = createCorrelatedFetch( + "55555555-5555-4555-8555-555555555555", + fetchImplementation, + undefined, + () => undefined, + ); + + await correlatedFetch("/api/hypershell/v1/gateways"); + + const headers = new Headers(fetchImplementation.mock.calls[0]?.[1]?.headers); + expect(headers.has("traceparent")).toBe(false); + expect(headers.has("tracestate")).toBe(false); + }); }); describe("redirectToLogin", () => { diff --git a/components/web-console/app/adapters/api/api.client.ts b/components/web-console/app/adapters/api/api.client.ts index e6b646d5..bf54a2a6 100644 --- a/components/web-console/app/adapters/api/api.client.ts +++ b/components/web-console/app/adapters/api/api.client.ts @@ -2,6 +2,19 @@ import { SDKClient } from "@openshift-online/hypershell-sdk"; export const gatewayCorrelationHeader = "x-hypershell-correlation-id"; +/** W3C Trace Context headers a request carries to the BFF for propagation. */ +export interface RequestTraceContext { + traceparent: string; + tracestate?: string; +} + +/** + * Supplies the W3C trace context for the in-flight workflow, or `undefined` + * when tracing is disabled or no span is active. Injected so the API adapter + * stays free of any tracing vendor dependency. + */ +export type TraceContextProvider = () => RequestTraceContext | undefined; + /** The BFF's machine-readable request to restart authentication at the IdP. */ export interface ReauthSignal { loginUrl: string; @@ -71,10 +84,18 @@ export function createCorrelatedFetch( correlationId: string, fetchImplementation: typeof globalThis.fetch = globalThis.fetch, onReauthRequired?: ReauthHandler, + traceContext?: TraceContextProvider, ): typeof globalThis.fetch { return async (input, init) => { const headers = new Headers(init?.headers); headers.set(gatewayCorrelationHeader, correlationId); + const trace = traceContext?.(); + if (trace !== undefined) { + headers.set("traceparent", trace.traceparent); + if (trace.tracestate !== undefined && trace.tracestate !== "") { + headers.set("tracestate", trace.tracestate); + } + } const response = await fetchImplementation(input, { ...init, headers }); if (onReauthRequired) { const signal = await readReauthSignal(response); @@ -94,6 +115,7 @@ export function createCorrelatedFetch( export function createApiClient( correlationId: string, onReauthRequired: ReauthHandler = redirectToLogin, + traceContext?: TraceContextProvider, ): SDKClient { return new SDKClient({ baseUrl: "", @@ -102,6 +124,7 @@ export function createApiClient( correlationId, globalThis.fetch, onReauthRequired, + traceContext, ), }); } diff --git a/components/web-console/app/adapters/observability/gateway-observability.ts b/components/web-console/app/adapters/observability/gateway-observability.ts index 8845cb2c..478000c5 100644 --- a/components/web-console/app/adapters/observability/gateway-observability.ts +++ b/components/web-console/app/adapters/observability/gateway-observability.ts @@ -108,5 +108,3 @@ export function createGatewayObservability( }, }; } - -export const gatewayObservability = createGatewayObservability(); diff --git a/components/web-console/app/adapters/observability/gateway-trace-sink.test.ts b/components/web-console/app/adapters/observability/gateway-trace-sink.test.ts new file mode 100644 index 00000000..9f724c7e --- /dev/null +++ b/components/web-console/app/adapters/observability/gateway-trace-sink.test.ts @@ -0,0 +1,208 @@ +import type { + GatewayAction, + GatewayProbe, +} from "@openshift-online/hypershell-gateway-management-ui"; +import { SpanStatusCode } from "@opentelemetry/api"; +import { + AlwaysOnSampler, + BasicTracerProvider, + InMemorySpanExporter, + ParentBasedSampler, + SimpleSpanProcessor, + type ReadableSpan, +} from "@opentelemetry/sdk-trace-base"; +import { beforeEach, describe, expect, it } from "vitest"; + +import { createGatewayTraceSink } from "./gateway-trace-sink"; + +const traceId = "0af7651916cd43dd8448eb211c80319c"; +const parentSpanId = "aaaaaaaaaaaaaaaa"; +const correlationId = "correlation-1"; + +function probe( + name: GatewayProbe["name"], + overrides: Partial<{ + action: GatewayAction; + failureKind: GatewayProbe["fields"]["failureKind"]; + operationId: string; + outcome: GatewayProbe["fields"]["outcome"]; + traceId: string | undefined; + }> = {}, +): GatewayProbe { + const action = overrides.action ?? "list"; + return Object.freeze({ + context: Object.freeze({ + correlationId, + ...(overrides.operationId === undefined + ? {} + : { operationId: overrides.operationId }), + ...("traceId" in overrides + ? { traceId: overrides.traceId } + : { traceId }), + }), + fields: Object.freeze({ + action, + failureKind: overrides.failureKind ?? null, + outcome: overrides.outcome ?? "started", + }), + name, + occurredAt: "2026-08-06T18:00:00.000Z", + schemaVersion: 1, + }); +} + +function testTracer() { + const exporter = new InMemorySpanExporter(); + const provider = new BasicTracerProvider({ + sampler: new ParentBasedSampler({ root: new AlwaysOnSampler() }), + spanProcessors: [new SimpleSpanProcessor(exporter)], + }); + return { exporter, tracer: provider.getTracer("test") }; +} + +function byName(spans: readonly ReadableSpan[], name: string): ReadableSpan { + const span = spans.find((candidate) => candidate.name === name); + if (span === undefined) { + throw new Error(`no finished span named ${name}`); + } + return span; +} + +describe("gateway trace sink", () => { + let exporter: InMemorySpanExporter; + let sink: ReturnType; + + beforeEach(() => { + const harness = testTracer(); + exporter = harness.exporter; + sink = createGatewayTraceSink(harness.tracer, { + generateSpanId: () => parentSpanId, + }); + }); + + it("adopts the probe trace id and nests the dependency under the workflow", () => { + sink.sink.publish(probe("gateway.workflow.started")); + sink.sink.publish(probe("gateway.dependency.attempted")); + sink.sink.publish( + probe("gateway.dependency.completed", { outcome: "succeeded" }), + ); + sink.sink.publish( + probe("gateway.workflow.completed", { outcome: "succeeded" }), + ); + + const finished = exporter.getFinishedSpans(); + const workflow = byName(finished, "gateway.workflow.list"); + const dependency = byName(finished, "gateway.dependency.list"); + + expect(workflow.spanContext().traceId).toBe(traceId); + expect(dependency.spanContext().traceId).toBe(traceId); + // The workflow span descends from the manufactured remote parent, so the + // trace joins the id the caller propagates end to end. + expect(workflow.parentSpanContext?.spanId).toBe(parentSpanId); + expect(dependency.parentSpanContext?.spanId).toBe( + workflow.spanContext().spanId, + ); + expect(workflow.status.code).toBe(SpanStatusCode.OK); + expect(workflow.attributes["gateway.action"]).toBe("list"); + expect(workflow.attributes["gateway.outcome"]).toBe("succeeded"); + // The dependency span closes before the workflow span. + expect(finished.map((span) => span.name)).toEqual([ + "gateway.dependency.list", + "gateway.workflow.list", + ]); + }); + + it("marks a failed workflow and retains the operation identifier", () => { + sink.sink.publish(probe("gateway.workflow.started", { action: "rename" })); + sink.sink.publish( + probe("gateway.dependency.attempted", { action: "rename" }), + ); + sink.sink.publish( + probe("gateway.dependency.completed", { + action: "rename", + failureKind: "conflict", + operationId: "operation-1", + outcome: "conflicted", + }), + ); + sink.sink.publish( + probe("gateway.workflow.completed", { + action: "rename", + failureKind: "conflict", + operationId: "operation-1", + outcome: "conflicted", + }), + ); + + const finished = exporter.getFinishedSpans(); + for (const span of finished) { + expect(span.status.code).toBe(SpanStatusCode.ERROR); + expect(span.attributes["gateway.failure_kind"]).toBe("conflict"); + expect(span.attributes["gateway.outcome"]).toBe("conflicted"); + expect(span.attributes["hypershell.operation_id"]).toBe("operation-1"); + } + }); + + it("renders the active dependency span as a W3C traceparent", () => { + sink.sink.publish(probe("gateway.workflow.started")); + sink.sink.publish(probe("gateway.dependency.attempted")); + + const active = sink.traceParentFor(correlationId); + + expect(active?.traceparent).toMatch( + new RegExp(`^00-${traceId}-[0-9a-f]{16}-01$`), + ); + expect(active?.tracestate).toBeUndefined(); + }); + + it("reports no trace context once the workflow completes", () => { + sink.sink.publish(probe("gateway.workflow.started")); + sink.sink.publish(probe("gateway.dependency.attempted")); + sink.sink.publish( + probe("gateway.dependency.completed", { outcome: "succeeded" }), + ); + sink.sink.publish( + probe("gateway.workflow.completed", { outcome: "succeeded" }), + ); + + expect(sink.traceParentFor(correlationId)).toBeUndefined(); + }); + + it("ignores dependency and completion probes with no started workflow", () => { + expect(() => { + sink.sink.publish(probe("gateway.dependency.attempted")); + sink.sink.publish( + probe("gateway.dependency.completed", { outcome: "succeeded" }), + ); + sink.sink.publish( + probe("gateway.workflow.completed", { outcome: "succeeded" }), + ); + }).not.toThrow(); + expect(exporter.getFinishedSpans()).toHaveLength(0); + }); + + it("drops a trace when the sampler declines it", () => { + const harness = testTracer(); + const unsampled = createGatewayTraceSink(harness.tracer, { + generateSpanId: () => parentSpanId, + isSampled: () => false, + }); + + unsampled.sink.publish(probe("gateway.workflow.started")); + unsampled.sink.publish(probe("gateway.dependency.attempted")); + const active = unsampled.traceParentFor(correlationId); + unsampled.sink.publish( + probe("gateway.dependency.completed", { outcome: "succeeded" }), + ); + unsampled.sink.publish( + probe("gateway.workflow.completed", { outcome: "succeeded" }), + ); + + // A declined trace still propagates its ids, with the sampled flag cleared, + // so downstream services make the same decision. + expect(active?.traceparent).toMatch( + new RegExp(`^00-${traceId}-[0-9a-f]{16}-00$`), + ); + expect(harness.exporter.getFinishedSpans()).toHaveLength(0); + }); +}); diff --git a/components/web-console/app/adapters/observability/gateway-trace-sink.ts b/components/web-console/app/adapters/observability/gateway-trace-sink.ts new file mode 100644 index 00000000..1755cf8c --- /dev/null +++ b/components/web-console/app/adapters/observability/gateway-trace-sink.ts @@ -0,0 +1,283 @@ +import type { GatewayProbe } from "@openshift-online/hypershell-gateway-management-ui"; +import type { DomainProbeSink } from "@openshift-online/hypershell-domain-probes/fan-out"; +import { + ROOT_CONTEXT, + SpanKind, + SpanStatusCode, + TraceFlags, + context as otelContext, + isSpanContextValid, + trace as otelTrace, + type Context, + type Span, + type SpanContext, + type Tracer, +} from "@opentelemetry/api"; +import { + BasicTracerProvider, + BatchSpanProcessor, + ParentBasedSampler, + AlwaysOnSampler, + RandomIdGenerator, +} from "@opentelemetry/sdk-trace-base"; +import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http"; +import { resourceFromAttributes } from "@opentelemetry/resources"; +import { ATTR_SERVICE_NAME } from "@opentelemetry/semantic-conventions"; + +/** W3C `traceparent`/`tracestate` header pair for outbound propagation. */ +export interface GatewayTraceContext { + traceparent: string; + tracestate?: string; +} + +/** A gateway trace sink plus the propagation reader that feeds the API client. */ +export interface GatewayTraceSink { + /** + * Reads the active workflow (or in-flight dependency) span for one + * correlation identifier and renders its W3C context, or `undefined` when no + * span is active for that correlation identifier. + */ + traceParentFor: (correlationId: string) => GatewayTraceContext | undefined; + sink: DomainProbeSink; +} + +/** A gateway trace sink together with its provider lifecycle controls. */ +export interface GatewayTracing extends GatewayTraceSink { + forceFlush(): Promise; + shutdown(): Promise; +} + +export interface GatewayTraceSinkOptions { + /** + * Decides whether one trace is recorded, keyed by its trace id so the + * decision is stable for the whole trace. Defaults to always sampling. + */ + isSampled?: (traceId: string) => boolean; + /** Generates the 16-byte span id (32 hex) of the manufactured remote parent. */ + generateSpanId?: () => string; +} + +export interface GatewayTracingConfig { + serviceName: string; + /** Same-origin OTLP/HTTP traces path the browser exporter posts to. */ + tracesEndpoint: string; + /** Fraction of traces to record, 0..1. Defaults to 1 (record all). */ + sampleRatio?: number; +} + +const sinkId = "gateway-trace"; +const tracerName = "gateway-trace-sink"; + +interface SpanEntry { + workflow: Span; + dependency?: Span; +} + +/** + * Manufactures the remote parent context that makes a workflow span adopt the + * caller-chosen trace id. The parent is marked sampled only when `isSampled` + * accepts the trace id; a `ParentBasedSampler` then honours that decision for + * the whole trace, so the W3C sampled flag stays consistent end to end. + */ +function remoteParentContext( + traceId: string, + spanId: string, + sampled: boolean, +): Context { + const spanContext: SpanContext = { + isRemote: true, + spanId, + traceFlags: sampled ? TraceFlags.SAMPLED : TraceFlags.NONE, + traceId, + }; + return otelTrace.setSpanContext(ROOT_CONTEXT, spanContext); +} + +/** Terminal outcomes that mark a span failed rather than ok. */ +function isFailureOutcome(outcome: GatewayProbe["fields"]["outcome"]): boolean { + return outcome !== "started" && outcome !== "succeeded"; +} + +function applyTerminalOutcome(span: Span, probe: GatewayProbe): void { + const { failureKind, outcome } = probe.fields; + span.setAttribute("gateway.outcome", outcome); + if (failureKind !== null) { + span.setAttribute("gateway.failure_kind", failureKind); + } + // The operation identifier is present only on failing terminal probes and is + // the sole bridge from a failed workflow to its API-side operation record. + if (probe.context.operationId !== undefined) { + span.setAttribute("hypershell.operation_id", probe.context.operationId); + } + span.setStatus({ + code: isFailureOutcome(outcome) ? SpanStatusCode.ERROR : SpanStatusCode.OK, + }); +} + +/** + * Builds a domain probe sink that projects gateway workflow and dependency + * probes onto OpenTelemetry spans. The sink adopts the trace id carried on each + * probe context so a workflow span joins the same trace the browser propagates + * to the BFF and API. Span names are drawn from a bounded action template so + * cardinality stays fixed. + */ +export function createGatewayTraceSink( + tracer: Tracer, + options: GatewayTraceSinkOptions = {}, +): GatewayTraceSink { + const generateSpanId = + options.generateSpanId ?? (() => new RandomIdGenerator().generateSpanId()); + const isSampled = options.isSampled ?? (() => true); + const spansByCorrelation = new Map(); + + function startWorkflow(probe: GatewayProbe): void { + const { correlationId, traceId } = probe.context; + const parent = + traceId === undefined + ? otelContext.active() + : remoteParentContext(traceId, generateSpanId(), isSampled(traceId)); + const workflow = tracer.startSpan( + `gateway.workflow.${probe.fields.action}`, + { + attributes: { "gateway.action": probe.fields.action }, + kind: SpanKind.INTERNAL, + }, + parent, + ); + spansByCorrelation.set(correlationId, { workflow }); + } + + function startDependency(probe: GatewayProbe): void { + const entry = spansByCorrelation.get(probe.context.correlationId); + if (entry === undefined) { + return; + } + const parent = otelTrace.setSpan(otelContext.active(), entry.workflow); + entry.dependency = tracer.startSpan( + `gateway.dependency.${probe.fields.action}`, + { + attributes: { "gateway.action": probe.fields.action }, + kind: SpanKind.CLIENT, + }, + parent, + ); + } + + function completeDependency(probe: GatewayProbe): void { + const entry = spansByCorrelation.get(probe.context.correlationId); + if (entry?.dependency === undefined) { + return; + } + applyTerminalOutcome(entry.dependency, probe); + entry.dependency.end(); + entry.dependency = undefined; + } + + function completeWorkflow(probe: GatewayProbe): void { + const entry = spansByCorrelation.get(probe.context.correlationId); + if (entry === undefined) { + return; + } + // Defend against a dependency span left open by a dropped completion probe. + if (entry.dependency !== undefined) { + entry.dependency.end(); + } + applyTerminalOutcome(entry.workflow, probe); + entry.workflow.end(); + spansByCorrelation.delete(probe.context.correlationId); + } + + const sink: DomainProbeSink = { + id: sinkId, + publish(probe) { + switch (probe.name) { + case "gateway.workflow.started": + startWorkflow(probe); + return; + case "gateway.dependency.attempted": + startDependency(probe); + return; + case "gateway.dependency.completed": + completeDependency(probe); + return; + case "gateway.workflow.completed": + completeWorkflow(probe); + return; + } + }, + }; + + function traceParentFor( + correlationId: string, + ): GatewayTraceContext | undefined { + const entry = spansByCorrelation.get(correlationId); + const span = entry?.dependency ?? entry?.workflow; + if (span === undefined) { + return undefined; + } + const spanContext = span.spanContext(); + if (!isSpanContextValid(spanContext)) { + return undefined; + } + const flags = (spanContext.traceFlags & TraceFlags.SAMPLED) === 0 + ? "00" + : "01"; + const traceparent = `00-${spanContext.traceId}-${spanContext.spanId}-${flags}`; + const tracestate = spanContext.traceState?.serialize(); + return tracestate === undefined || tracestate === "" + ? { traceparent } + : { traceparent, tracestate }; + } + + return { sink, traceParentFor }; +} + +/** + * Wires a browser tracer provider that batches spans and exports them over + * same-origin OTLP/HTTP, and returns the gateway trace sink bound to it. The + * provider is not registered as the global tracer; the sink owns every span + * explicitly, keyed by correlation identifier, so no implicit context is + * needed. + * The batch processor flushes on document hide, so page unload does not lose + * buffered spans. + */ +export function createGatewayTracing( + config: GatewayTracingConfig, +): GatewayTracing { + const ratio = config.sampleRatio ?? 1; + const exporter = new OTLPTraceExporter({ url: config.tracesEndpoint }); + const provider = new BasicTracerProvider({ + resource: resourceFromAttributes({ + [ATTR_SERVICE_NAME]: config.serviceName, + }), + sampler: new ParentBasedSampler({ root: new AlwaysOnSampler() }), + spanProcessors: [new BatchSpanProcessor(exporter)], + }); + const tracer = provider.getTracer(tracerName); + const { sink, traceParentFor } = createGatewayTraceSink(tracer, { + isSampled: (traceId) => sampledByRatio(traceId, ratio), + }); + return { + forceFlush: () => provider.forceFlush(), + shutdown: () => provider.shutdown(), + sink, + traceParentFor, + }; +} + +/** + * Deterministic per-trace sampling decision. The leading 4 bytes of the trace + * id map to a value in [0, 1); a trace records when that value is below the + * configured ratio. Deterministic keying keeps the browser, BFF, and API in + * agreement without sharing a decision. + */ +function sampledByRatio(traceId: string, ratio: number): boolean { + if (ratio >= 1) { + return true; + } + if (ratio <= 0) { + return false; + } + const bucket = Number.parseInt(traceId.slice(0, 8), 16); + return bucket / 0x1_0000_0000 < ratio; +} diff --git a/components/web-console/app/composition/gateway-composition.ts b/components/web-console/app/composition/gateway-composition.ts index 799841b2..ab3b0ec4 100644 --- a/components/web-console/app/composition/gateway-composition.ts +++ b/components/web-console/app/composition/gateway-composition.ts @@ -2,10 +2,27 @@ import { createGatewayOperations } from "@openshift-online/hypershell-gateway-ma import { createApiClient } from "../adapters/api/api.client"; import { createGatewayControlPlaneAdapter } from "../adapters/api/gateway-operations"; -import { gatewayObservability } from "../adapters/observability/gateway-observability"; +import { createGatewayObservability } from "../adapters/observability/gateway-observability"; +import { createGatewayTracing } from "../adapters/observability/gateway-trace-sink"; + +// Same-origin OTLP/HTTP traces path the BFF exposes and forwards to the +// collector. Keeping it same-origin means the browser never sees a collector +// address and no cross-origin telemetry endpoint is exposed. +const browserTracesEndpoint = "/telemetry/v1/traces"; + +const tracing = createGatewayTracing({ + serviceName: "hypershell-web-console", + tracesEndpoint: browserTracesEndpoint, +}); + +const gatewayObservability = createGatewayObservability({ + additionalSinks: [tracing.sink], +}); const gatewayControlPlane = createGatewayControlPlaneAdapter((correlationId) => - createApiClient(correlationId), + createApiClient(correlationId, undefined, () => + tracing.traceParentFor(correlationId), + ), ); export const gatewayOperations = createGatewayOperations({ diff --git a/components/web-console/package.json b/components/web-console/package.json index c645afa4..fa04b3ec 100644 --- a/components/web-console/package.json +++ b/components/web-console/package.json @@ -24,6 +24,11 @@ "@openshift-online/hypershell-domain-probes": "workspace:0.0.0", "@openshift-online/hypershell-gateway-management-ui": "workspace:0.0.0", "@openshift-online/hypershell-sdk": "workspace:0.0.0", + "@opentelemetry/api": "1.9.1", + "@opentelemetry/exporter-trace-otlp-http": "0.221.0", + "@opentelemetry/resources": "2.10.0", + "@opentelemetry/sdk-trace-base": "2.10.0", + "@opentelemetry/semantic-conventions": "1.43.0", "@patternfly/react-core": "6.6.0", "@patternfly/react-icons": "6.6.0", "@patternfly/react-table": "6.6.0", diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 104a6a65..148715ee 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -26,6 +26,21 @@ importers: '@openshift-online/hypershell-sdk': specifier: workspace:0.0.0 version: link:../sdk-typescript + '@opentelemetry/api': + specifier: 1.9.1 + version: 1.9.1 + '@opentelemetry/exporter-trace-otlp-http': + specifier: 0.221.0 + version: 0.221.0(@opentelemetry/api@1.9.1) + '@opentelemetry/resources': + specifier: 2.10.0 + version: 2.10.0(@opentelemetry/api@1.9.1) + '@opentelemetry/sdk-trace-base': + specifier: 2.10.0 + version: 2.10.0(@opentelemetry/api@1.9.1) + '@opentelemetry/semantic-conventions': + specifier: 1.43.0 + version: 1.43.0 '@patternfly/react-core': specifier: 6.6.0 version: 6.6.0(react-dom@19.2.8(react@19.2.8))(react@19.2.8) @@ -149,7 +164,7 @@ importers: version: 8.1.5(@types/node@24.13.3)(esbuild@0.28.1)(tsx@4.23.1) vitest: specifier: 4.1.10 - version: 4.1.10(@types/node@24.13.3)(@vitest/coverage-v8@4.1.10)(jsdom@27.4.0(supports-color@7.2.0))(msw@2.15.0(@types/node@24.13.3)(typescript@6.0.3))(vite@8.1.5(@types/node@24.13.3)(esbuild@0.28.1)(tsx@4.23.1)) + version: 4.1.10(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(@vitest/coverage-v8@4.1.10)(jsdom@27.4.0(supports-color@7.2.0))(msw@2.15.0(@types/node@24.13.3)(typescript@6.0.3))(vite@8.1.5(@types/node@24.13.3)(esbuild@0.28.1)(tsx@4.23.1)) components/web-console/bff: dependencies: @@ -201,7 +216,7 @@ importers: version: 8.65.0(eslint@9.39.5(supports-color@7.2.0))(supports-color@7.2.0)(typescript@6.0.3) vitest: specifier: 4.1.10 - version: 4.1.10(@types/node@24.13.3)(@vitest/coverage-v8@4.1.10)(jsdom@27.4.0(supports-color@7.2.0))(msw@2.15.0(@types/node@24.13.3)(typescript@6.0.3))(vite@8.1.5(@types/node@24.13.3)(esbuild@0.28.1)(tsx@4.23.1)) + version: 4.1.10(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(@vitest/coverage-v8@4.1.10)(jsdom@27.4.0(supports-color@7.2.0))(msw@2.15.0(@types/node@24.13.3)(typescript@6.0.3))(vite@8.1.5(@types/node@24.13.3)(esbuild@0.28.1)(tsx@4.23.1)) components/web-console/domain-probes: devDependencies: @@ -228,7 +243,7 @@ importers: version: 8.65.0(eslint@9.39.5(supports-color@7.2.0))(supports-color@7.2.0)(typescript@6.0.3) vitest: specifier: 4.1.10 - version: 4.1.10(@types/node@24.13.3)(@vitest/coverage-v8@4.1.10)(jsdom@27.4.0(supports-color@7.2.0))(msw@2.15.0(@types/node@24.13.3)(typescript@6.0.3))(vite@8.1.5(@types/node@24.13.3)(esbuild@0.28.1)(tsx@4.23.1)) + version: 4.1.10(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(@vitest/coverage-v8@4.1.10)(jsdom@27.4.0(supports-color@7.2.0))(msw@2.15.0(@types/node@24.13.3)(typescript@6.0.3))(vite@8.1.5(@types/node@24.13.3)(esbuild@0.28.1)(tsx@4.23.1)) packages/gateway-management-ui: dependencies: @@ -322,7 +337,7 @@ importers: version: 8.1.5(@types/node@24.13.3)(esbuild@0.28.1)(tsx@4.23.1) vitest: specifier: 4.1.10 - version: 4.1.10(@types/node@24.13.3)(@vitest/coverage-v8@4.1.10)(jsdom@27.4.0(supports-color@7.2.0))(msw@2.15.0(@types/node@24.13.3)(typescript@6.0.3))(vite@8.1.5(@types/node@24.13.3)(esbuild@0.28.1)(tsx@4.23.1)) + version: 4.1.10(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(@vitest/coverage-v8@4.1.10)(jsdom@27.4.0(supports-color@7.2.0))(msw@2.15.0(@types/node@24.13.3)(typescript@6.0.3))(vite@8.1.5(@types/node@24.13.3)(esbuild@0.28.1)(tsx@4.23.1)) zod: specifier: 4.4.3 version: 4.4.3 @@ -1063,6 +1078,72 @@ packages: '@open-draft/until@2.1.0': resolution: {integrity: sha512-U69T3ItWHvLwGg5eJ0n3I62nWuE6ilHlmz7zM0npLBRvPRd7e6NYmg54vvRtP5mZG7kZqZCFVdsTWo7BPtBujg==} + '@opentelemetry/api-logs@0.221.0': + resolution: {integrity: sha512-OlanaW1vv7ufTqQ3/fPLI4arGt5ZoM+P8abOMki6uEYnpRazepSWDwDnnw+la7kE26SHVC18//SMccrDvLKOXQ==} + engines: {node: '>=8.0.0'} + + '@opentelemetry/api@1.9.1': + resolution: {integrity: sha512-gLyJlPHPZYdAk1JENA9LeHejZe1Ti77/pTeFm/nMXmQH/HFZlcS/O2XJB+L8fkbrNSqhdtlvjBVjxwUYanNH5Q==} + engines: {node: '>=8.0.0'} + + '@opentelemetry/core@2.10.0': + resolution: {integrity: sha512-/wNZ8twnEQQA4HoHu22+vcsdru6pWPWxW+7w+FlxT6Id7PE/WIbZmVKkte+PF72e0F2dnImFeHD2syyE1Mw6MQ==} + engines: {node: ^18.19.0 || >=20.6.0} + peerDependencies: + '@opentelemetry/api': '>=1.0.0 <1.10.0' + + '@opentelemetry/exporter-trace-otlp-http@0.221.0': + resolution: {integrity: sha512-AySXiKoC+meiWm6zdVj5T2LnPDZuatveBby1cMOeQteIWsYXAUxs8Sru13G2pVSPrUXz6vF+og7QVBX6GdC/oQ==} + engines: {node: ^18.19.0 || >=20.6.0} + peerDependencies: + '@opentelemetry/api': ^1.3.0 + + '@opentelemetry/otlp-exporter-base@0.221.0': + resolution: {integrity: sha512-UFPIq80OH3Ns/oPFHRj14d4DTOxUo+MUFU8hUiCq5jTqFhdeJnfVSANHT+xp92409cA+oxzvlZCe6NM1wvCuBA==} + engines: {node: ^18.19.0 || >=20.6.0} + peerDependencies: + '@opentelemetry/api': ^1.3.0 + + '@opentelemetry/otlp-transformer@0.221.0': + resolution: {integrity: sha512-lg6lkOU08Az23jVcn/0Els9HP+V8PnR4Km6p0KgpTggS0n/WuhnmY64rSh83Of9iR9nD+dpWr6adlcX8KzAwjg==} + engines: {node: ^18.19.0 || >=20.6.0} + peerDependencies: + '@opentelemetry/api': ^1.3.0 + + '@opentelemetry/resources@2.10.0': + resolution: {integrity: sha512-q6MMm2zhggzsHVNbabYwut+a6nbuQQe3URUoxaojM/8K1IBfwwPzvxIjNi2/lI1TFe+fMHMW9MWhrtDLEXEnkA==} + engines: {node: ^18.19.0 || >=20.6.0} + peerDependencies: + '@opentelemetry/api': '>=1.3.0 <1.10.0' + + '@opentelemetry/sdk-logs@0.221.0': + resolution: {integrity: sha512-FaDcazjyMp7TZZZAsqbo4IkovP0UegoCu0EBkiNt+qCqvUf7FPAsfcrZ3+ZEkKgXZ/jHafop+JoGPDk3A0SmLg==} + engines: {node: ^18.19.0 || >=20.6.0} + peerDependencies: + '@opentelemetry/api': '>=1.4.0 <1.10.0' + + '@opentelemetry/sdk-metrics@2.10.0': + resolution: {integrity: sha512-t6r1VSvXNtSDnPXU1FbZeetJb7yyovHmgu0wRSoftxtE0g2rSNhQZQUy69sRUCL+iioJpX8SN/S6wq6ZtvLySQ==} + engines: {node: ^18.19.0 || >=20.6.0} + peerDependencies: + '@opentelemetry/api': '>=1.9.0 <1.10.0' + + '@opentelemetry/sdk-trace-base@2.10.0': + resolution: {integrity: sha512-GuYQQT7QD2EeO8lcZLRQzcbOyhqAzL+6WWTKTU9mSUBYBazkEDl+VrQcXQhbB08OWM9anD1aHleVadzulpOaUQ==} + engines: {node: ^18.19.0 || >=20.6.0} + peerDependencies: + '@opentelemetry/api': '>=1.3.0 <1.10.0' + + '@opentelemetry/sdk-trace@2.10.0': + resolution: {integrity: sha512-MfQGq3GRmTh5fM/y+OjaO0vj6+luCB1XO2gfXCalKCfgKw0eHL++sm75DNweC6ohlp+aFvACqeE0fYayqdRaoQ==} + engines: {node: ^18.19.0 || >=20.6.0} + peerDependencies: + '@opentelemetry/api': '>=1.3.0 <1.10.0' + + '@opentelemetry/semantic-conventions@1.43.0': + resolution: {integrity: sha512-eSYWTm620tTk45EKSedaUL8MFYI8hW164hIXsgIHyxu3VobUB3fFCu5t0hQby6OoWRPsG1KkKUG2M5UadiLiVg==} + engines: {node: '>=14'} + '@oxc-parser/binding-android-arm-eabi@0.127.0': resolution: {integrity: sha512-0LC7ye4hvqbIKxAzThzvswgHLFu2AURKzYLeSVvLdu2TBOYWQDmHnTqPLeA597BcUCxiLqLsS4CJ5uoI5WYWCQ==} engines: {node: ^20.19.0 || >=22.12.0} @@ -4699,6 +4780,77 @@ snapshots: '@open-draft/until@2.1.0': {} + '@opentelemetry/api-logs@0.221.0': + dependencies: + '@opentelemetry/api': 1.9.1 + + '@opentelemetry/api@1.9.1': {} + + '@opentelemetry/core@2.10.0(@opentelemetry/api@1.9.1)': + dependencies: + '@opentelemetry/api': 1.9.1 + '@opentelemetry/semantic-conventions': 1.43.0 + + '@opentelemetry/exporter-trace-otlp-http@0.221.0(@opentelemetry/api@1.9.1)': + dependencies: + '@opentelemetry/api': 1.9.1 + '@opentelemetry/otlp-exporter-base': 0.221.0(@opentelemetry/api@1.9.1) + '@opentelemetry/otlp-transformer': 0.221.0(@opentelemetry/api@1.9.1) + '@opentelemetry/sdk-trace': 2.10.0(@opentelemetry/api@1.9.1) + + '@opentelemetry/otlp-exporter-base@0.221.0(@opentelemetry/api@1.9.1)': + dependencies: + '@opentelemetry/api': 1.9.1 + '@opentelemetry/core': 2.10.0(@opentelemetry/api@1.9.1) + '@opentelemetry/otlp-transformer': 0.221.0(@opentelemetry/api@1.9.1) + + '@opentelemetry/otlp-transformer@0.221.0(@opentelemetry/api@1.9.1)': + dependencies: + '@opentelemetry/api': 1.9.1 + '@opentelemetry/api-logs': 0.221.0 + '@opentelemetry/core': 2.10.0(@opentelemetry/api@1.9.1) + '@opentelemetry/resources': 2.10.0(@opentelemetry/api@1.9.1) + '@opentelemetry/sdk-logs': 0.221.0(@opentelemetry/api@1.9.1) + '@opentelemetry/sdk-metrics': 2.10.0(@opentelemetry/api@1.9.1) + '@opentelemetry/sdk-trace': 2.10.0(@opentelemetry/api@1.9.1) + + '@opentelemetry/resources@2.10.0(@opentelemetry/api@1.9.1)': + dependencies: + '@opentelemetry/api': 1.9.1 + '@opentelemetry/core': 2.10.0(@opentelemetry/api@1.9.1) + '@opentelemetry/semantic-conventions': 1.43.0 + + '@opentelemetry/sdk-logs@0.221.0(@opentelemetry/api@1.9.1)': + dependencies: + '@opentelemetry/api': 1.9.1 + '@opentelemetry/api-logs': 0.221.0 + '@opentelemetry/core': 2.10.0(@opentelemetry/api@1.9.1) + '@opentelemetry/resources': 2.10.0(@opentelemetry/api@1.9.1) + '@opentelemetry/semantic-conventions': 1.43.0 + + '@opentelemetry/sdk-metrics@2.10.0(@opentelemetry/api@1.9.1)': + dependencies: + '@opentelemetry/api': 1.9.1 + '@opentelemetry/core': 2.10.0(@opentelemetry/api@1.9.1) + '@opentelemetry/resources': 2.10.0(@opentelemetry/api@1.9.1) + + '@opentelemetry/sdk-trace-base@2.10.0(@opentelemetry/api@1.9.1)': + dependencies: + '@opentelemetry/api': 1.9.1 + '@opentelemetry/core': 2.10.0(@opentelemetry/api@1.9.1) + '@opentelemetry/resources': 2.10.0(@opentelemetry/api@1.9.1) + '@opentelemetry/sdk-trace': 2.10.0(@opentelemetry/api@1.9.1) + '@opentelemetry/semantic-conventions': 1.43.0 + + '@opentelemetry/sdk-trace@2.10.0(@opentelemetry/api@1.9.1)': + dependencies: + '@opentelemetry/api': 1.9.1 + '@opentelemetry/core': 2.10.0(@opentelemetry/api@1.9.1) + '@opentelemetry/resources': 2.10.0(@opentelemetry/api@1.9.1) + '@opentelemetry/semantic-conventions': 1.43.0 + + '@opentelemetry/semantic-conventions@1.43.0': {} + '@oxc-parser/binding-android-arm-eabi@0.127.0': optional: true @@ -5327,7 +5479,7 @@ snapshots: obug: 2.1.4 std-env: 4.2.0 tinyrainbow: 3.1.0 - vitest: 4.1.10(@types/node@24.13.3)(@vitest/coverage-v8@4.1.10)(jsdom@27.4.0(supports-color@7.2.0))(msw@2.15.0(@types/node@24.13.3)(typescript@6.0.3))(vite@8.1.5(@types/node@24.13.3)(esbuild@0.28.1)(tsx@4.23.1)) + vitest: 4.1.10(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(@vitest/coverage-v8@4.1.10)(jsdom@27.4.0(supports-color@7.2.0))(msw@2.15.0(@types/node@24.13.3)(typescript@6.0.3))(vite@8.1.5(@types/node@24.13.3)(esbuild@0.28.1)(tsx@4.23.1)) '@vitest/expect@3.2.4': dependencies: @@ -7652,7 +7804,7 @@ snapshots: fsevents: 2.3.3 tsx: 4.23.1 - vitest@4.1.10(@types/node@24.13.3)(@vitest/coverage-v8@4.1.10)(jsdom@27.4.0(supports-color@7.2.0))(msw@2.15.0(@types/node@24.13.3)(typescript@6.0.3))(vite@8.1.5(@types/node@24.13.3)(esbuild@0.28.1)(tsx@4.23.1)): + vitest@4.1.10(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(@vitest/coverage-v8@4.1.10)(jsdom@27.4.0(supports-color@7.2.0))(msw@2.15.0(@types/node@24.13.3)(typescript@6.0.3))(vite@8.1.5(@types/node@24.13.3)(esbuild@0.28.1)(tsx@4.23.1)): dependencies: '@vitest/expect': 4.1.10 '@vitest/mocker': 4.1.10(msw@2.15.0(@types/node@24.13.3)(typescript@6.0.3))(vite@8.1.5(@types/node@24.13.3)(esbuild@0.28.1)(tsx@4.23.1)) @@ -7675,6 +7827,7 @@ snapshots: vite: 8.1.5(@types/node@24.13.3)(esbuild@0.28.1)(tsx@4.23.1) why-is-node-running: 2.3.0 optionalDependencies: + '@opentelemetry/api': 1.9.1 '@types/node': 24.13.3 '@vitest/coverage-v8': 4.1.10(vitest@4.1.10) jsdom: 27.4.0(supports-color@7.2.0) From 717d1c9e244cff2d3fd593e2a9d60ff4fab1a953 Mon Sep 17 00:00:00 2001 From: user Date: Tue, 18 Aug 2026 10:17:21 -0400 Subject: [PATCH 04/33] feat(web-console-bff): add OTel tracing, W3C propagation, telemetry ingest Add distributed tracing to the web-console BFF behind a hexagonal tracing port. The proxy starts a SERVER span per upstream request, continues a valid inbound W3C traceparent (else starts a new trace), and forwards the span-derived traceparent/tracestate to the API server. A same-origin /telemetry/v1/traces route relays browser OTLP spans to the collector best-effort. Tracing stays disabled unless OTEL_EXPORTER_OTLP_ENDPOINT is set, so a deployment without a collector starts normally. The OTel SDK lives only in the observability adapter and bootstrap; app.ts depends on the plain BffTracing port. Refs HYPERSHELL-27 Co-Authored-By: Claude Opus 4.8 --- components/web-console/bff/package.json | 5 + .../adapters/observability/otel-tracing.ts | 192 +++++++++++ components/web-console/bff/src/app.ts | 302 +++++++++++------- components/web-console/bff/src/config.ts | 33 ++ components/web-console/bff/src/index.ts | 7 +- components/web-console/bff/src/tracing.ts | 70 ++++ .../web-console/bff/test/app-tracing.test.ts | 176 ++++++++++ .../web-console/bff/test/config.test.ts | 45 +++ .../web-console/bff/test/otel-tracing.test.ts | 129 ++++++++ pnpm-lock.yaml | 15 + 10 files changed, 864 insertions(+), 110 deletions(-) create mode 100644 components/web-console/bff/src/adapters/observability/otel-tracing.ts create mode 100644 components/web-console/bff/src/tracing.ts create mode 100644 components/web-console/bff/test/app-tracing.test.ts create mode 100644 components/web-console/bff/test/otel-tracing.test.ts diff --git a/components/web-console/bff/package.json b/components/web-console/bff/package.json index 07e7fac5..55d4035e 100644 --- a/components/web-console/bff/package.json +++ b/components/web-console/bff/package.json @@ -22,6 +22,11 @@ "@fastify/helmet": "13.1.0", "@fastify/secure-session": "8.3.0", "@fastify/static": "10.1.2", + "@opentelemetry/api": "1.9.1", + "@opentelemetry/exporter-trace-otlp-http": "0.221.0", + "@opentelemetry/resources": "2.10.0", + "@opentelemetry/sdk-trace-base": "2.10.0", + "@opentelemetry/semantic-conventions": "1.43.0", "fastify": "5.10.0", "openid-client": "6.8.4", "zod": "4.4.3" diff --git a/components/web-console/bff/src/adapters/observability/otel-tracing.ts b/components/web-console/bff/src/adapters/observability/otel-tracing.ts new file mode 100644 index 00000000..b9fee1c5 --- /dev/null +++ b/components/web-console/bff/src/adapters/observability/otel-tracing.ts @@ -0,0 +1,192 @@ +import { + ROOT_CONTEXT, + SpanKind, + SpanStatusCode, + TraceFlags, + isSpanContextValid, + trace as otelTrace, + type Span, + type SpanContext, + type Tracer, +} from "@opentelemetry/api"; +import { + BasicTracerProvider, + BatchSpanProcessor, + ParentBasedSampler, + TraceIdRatioBasedSampler, +} from "@opentelemetry/sdk-trace-base"; +import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http"; +import { resourceFromAttributes } from "@opentelemetry/resources"; +import { ATTR_SERVICE_NAME } from "@opentelemetry/semantic-conventions"; + +import type { TracingConfig } from "../../config.js"; +import { + disabledTracing, + type BffTracing, + type ProxyOutcome, + type ProxySpan, + type StartProxySpanInput, + type TelemetryIngestResult, + type UpstreamTraceContext, +} from "../../tracing.js"; + +const traceparentPattern = + /^00-(?[0-9a-f]{32})-(?[0-9a-f]{16})-(?[0-9a-f]{2})$/u; +// A permissive W3C `tracestate`: comma-separated members, bounded length. The +// value is untrusted and only forwarded, never parsed, so a light structural +// check is enough to drop obvious garbage before propagation. +const tracestatePattern = /^[ \t]*[!-~]+=[ -~]*(,[ \t]*[!-~]+=[ -~]*){0,31}$/u; +const zeroTraceId = "0".repeat(32); +const zeroSpanId = "0".repeat(16); +const ingestTimeoutMs = 5_000; + +/** Parses a W3C `traceparent`, returning a remote span context or `undefined`. */ +function parseTraceparent(value: string): SpanContext | undefined { + const groups = traceparentPattern.exec(value)?.groups; + if ( + groups?.flags === undefined || + groups.spanId === undefined || + groups.traceId === undefined + ) { + return undefined; + } + const { flags, spanId, traceId } = groups; + if (traceId === zeroTraceId || spanId === zeroSpanId) { + return undefined; + } + return { + isRemote: true, + spanId, + traceFlags: Number.parseInt(flags, 16), + traceId, + }; +} + +function isValidTracestate(value: string | undefined): value is string { + return ( + value !== undefined && value.length <= 512 && tracestatePattern.test(value) + ); +} + +function upstreamContext( + span: Span, + forwardedTracestate: string | undefined, +): UpstreamTraceContext | undefined { + const spanContext = span.spanContext(); + if (!isSpanContextValid(spanContext)) { + return undefined; + } + const flags = + (spanContext.traceFlags & TraceFlags.SAMPLED) === 0 ? "00" : "01"; + const traceparent = `00-${spanContext.traceId}-${spanContext.spanId}-${flags}`; + return forwardedTracestate === undefined + ? { traceparent } + : { traceparent, tracestate: forwardedTracestate }; +} + +function spanStatusFor(outcome: ProxyOutcome): SpanStatusCode { + return outcome === "server_error" || outcome === "timeout" + ? SpanStatusCode.ERROR + : SpanStatusCode.OK; +} + +function isOtlpTracePayload(payload: unknown): boolean { + return ( + typeof payload === "object" && + payload !== null && + Array.isArray((payload as { resourceSpans?: unknown }).resourceSpans) + ); +} + +/** + * Builds the BFF tracing adapter. Spans are managed explicitly per request and + * exported over OTLP/HTTP; no global tracer or context manager is registered. + * Returns the disabled port when no collector is configured, so a deployment + * without tracing starts normally. + */ +export function createBffTracing( + config: TracingConfig | undefined, +): BffTracing { + if (config === undefined) { + return disabledTracing; + } + const tracing = config; + + const exporter = new OTLPTraceExporter({ url: tracing.tracesEndpoint }); + const provider = new BasicTracerProvider({ + resource: resourceFromAttributes({ + [ATTR_SERVICE_NAME]: tracing.serviceName, + }), + sampler: new ParentBasedSampler({ + root: new TraceIdRatioBasedSampler(tracing.sampleRatio), + }), + spanProcessors: [new BatchSpanProcessor(exporter)], + }); + const tracer: Tracer = provider.getTracer("hypershell-web-console-bff"); + + function startProxySpan(input: StartProxySpanInput): ProxySpan { + const parent = + input.traceparent === undefined + ? undefined + : parseTraceparent(input.traceparent); + const continued = parent !== undefined && isSpanContextValid(parent); + const parentContext = continued + ? otelTrace.setSpanContext(ROOT_CONTEXT, parent) + : ROOT_CONTEXT; + // A `tracestate` is forwarded only alongside a valid inbound `traceparent`; + // a state without a parent trace has nothing to continue. + const forwardedTracestate = + continued && isValidTracestate(input.tracestate) + ? input.tracestate + : undefined; + const span = tracer.startSpan( + input.routeTemplate, + { + attributes: { + "http.request.method": input.method, + "http.route": input.routeTemplate, + "hypershell.correlation_id": input.correlationId, + }, + kind: SpanKind.SERVER, + }, + parentContext, + ); + + return { + end(outcome, statusCode) { + span.setAttribute("http.response.status_code", statusCode); + span.setAttribute("hypershell.outcome", outcome); + span.setStatus({ code: spanStatusFor(outcome) }); + span.end(); + }, + upstream: () => upstreamContext(span, forwardedTracestate), + }; + } + + async function ingestTraces( + payload: unknown, + ): Promise { + if (!isOtlpTracePayload(payload)) { + return "rejected"; + } + try { + const response = await fetch(tracing.tracesEndpoint, { + body: JSON.stringify(payload), + headers: { "content-type": "application/json" }, + method: "POST", + signal: AbortSignal.timeout(ingestTimeoutMs), + }); + return response.ok ? "accepted" : "unavailable"; + } catch { + // Best-effort: an unreachable collector never fails the browser request. + return "unavailable"; + } + } + + return { + enabled: true, + ingestTraces, + shutdown: () => provider.shutdown(), + startProxySpan, + }; +} diff --git a/components/web-console/bff/src/app.ts b/components/web-console/bff/src/app.ts index d91c28dc..46c121c5 100644 --- a/components/web-console/bff/src/app.ts +++ b/components/web-console/bff/src/app.ts @@ -10,6 +10,11 @@ import Fastify, { type FastifyInstance, LogController } from "fastify"; import { clearSession, persistTokenSet, registerAuth } from "./auth.js"; import type { ServerConfig } from "./config.js"; import { tokenExpired } from "./tokens.js"; +import { + disabledTracing, + type BffTracing, + type ProxyOutcome, +} from "./tracing.js"; const correlationHeader = "x-hypershell-correlation-id"; const validCorrelationId = @@ -71,7 +76,26 @@ function inlineScriptHashes(document: string): string[] { return [...hashes]; } -export async function buildApp(config: ServerConfig): Promise { +function singleHeaderValue( + value: string | string[] | undefined, +): string | undefined { + return typeof value === "string" ? value : undefined; +} + +function outcomeForStatus(statusCode: number): ProxyOutcome { + if (statusCode >= 500) { + return "server_error"; + } + if (statusCode >= 400) { + return "client_error"; + } + return "success"; +} + +export async function buildApp( + config: ServerConfig, + tracing: BffTracing = disabledTracing, +): Promise { const indexPath = path.join(config.staticRoot, "index.html"); const indexDocument = await readFile(indexPath, "utf8"); const scriptHashes = inlineScriptHashes(indexDocument); @@ -252,128 +276,188 @@ export async function buildApp(config: ServerConfig): Promise { }; }; - app.all("/api/*", async (request, reply) => { - const incoming = new URL(request.url, "http://bff.invalid"); - const target = new URL( - `${incoming.pathname}${incoming.search}`, - config.apiOrigin, - ); - const headers = new Headers(); - for (const header of forwardedRequestHeaders) { - const value = request.headers[header]; - if (typeof value === "string") { - headers.set(header, value); - } + // Same-origin browser telemetry ingest. The browser exporter posts OTLP/HTTP + // JSON here; the BFF validates it and relays it to the configured collector, + // keeping the collector origin out of the browser and reusing the session and + // CSRF controls the other state-changing routes rely on (WEB-TRACE-02). The + // global 1 MiB body limit bounds the payload. + app.post("/telemetry/v1/traces", async (request, reply) => { + reply.header("Cache-Control", "no-store"); + if (config.oidcIssuer && !request.session.get("accessToken")) { + reply.code(401); + return { error: "Unauthorized", statusCode: 401 }; } - headers.set(correlationHeader, request.correlationId); - - const refreshToken = config.oidcIssuer - ? request.tokenSession.get("refreshToken") - : undefined; - let refreshed = false; - - // Ensure a valid access token before forwarding (proactive refresh). - if (config.oidcIssuer) { - let accessToken = request.session.get("accessToken"); - const expiresAt = request.session.get("expiresAt"); - if (!accessToken || tokenExpired(expiresAt)) { - if (!refreshToken || !app.refreshAccessToken) { - clearSession(request); - return respondReauth(reply); - } - try { - const tokens = await app.refreshAccessToken(refreshToken); - persistTokenSet(request, tokens); - accessToken = tokens.accessToken; - refreshed = true; - } catch { - clearSession(request); - return respondReauth(reply); - } - } - headers.set("authorization", `Bearer ${accessToken}`); + const result = await tracing.ingestTraces(request.body); + if (result === "rejected") { + reply.code(400); + return { error: "Bad Request", statusCode: 400 }; } + // "accepted" and "unavailable" are both success from the browser's view: + // telemetry is best-effort and must never surface a collector outage. + reply.code(202); + return { status: "accepted" }; + }); - // Perform the upstream request with its own timeout and downstream-abort - // wiring, so it can be retried once after a reactive refresh. - const runUpstream = async (): Promise => { - const controller = new AbortController(); - const timeoutReason = new Error("Upstream API request timed out"); - const timeout = setTimeout(() => { - controller.abort(timeoutReason); - }, config.apiTimeoutMs); - const abortDownstream = () => { - controller.abort(); - }; - request.raw.once("aborted", abortDownstream); - try { - return await fetch(target, { - body: proxyBody(request.method, request.body), - headers, - method: request.method, - redirect: "manual", - signal: controller.signal, - }); - } catch (error) { - if (error === timeoutReason) { - const timeout504 = new Error("Upstream API request timed out"); - timeout504.name = "UpstreamTimeout"; - throw timeout504; - } - throw error; - } finally { - clearTimeout(timeout); - request.raw.off("aborted", abortDownstream); - } + app.all("/api/*", async (request, reply) => { + // Start one BFF server span per proxied request. It continues a valid + // inbound W3C context and yields the validated upstream context to set on + // the API request, so browser, BFF, and API join one trace (WEB-TRACE-04, + // WEB-TRACE-05). A malformed inbound value is never forwarded: only the + // span-derived context reaches upstream. + const span = tracing.startProxySpan({ + correlationId: request.correlationId, + method: request.method, + routeTemplate: request.routeOptions.url ?? "/api/*", + traceparent: singleHeaderValue(request.headers.traceparent), + tracestate: singleHeaderValue(request.headers.tracestate), + }); + // The span ends in the finally block with the outcome recorded here, so a + // tracing failure never changes the proxy result (WEB-TRACE-09). + let spanOutcome: ProxyOutcome = "server_error"; + let spanStatusCode = 500; + const recordOutcome = (statusCode: number) => { + spanOutcome = outcomeForStatus(statusCode); + spanStatusCode = statusCode; }; try { - let upstream = await runUpstream(); - - // Reactive refresh: a token we believed valid was rejected upstream - // (clock skew, revocation, or key rotation). Refresh once and retry. - if ( - config.oidcIssuer && - upstream.status === 401 && - !refreshed && - refreshToken && - app.refreshAccessToken - ) { - try { - const tokens = await app.refreshAccessToken(refreshToken); - persistTokenSet(request, tokens); - headers.set("authorization", `Bearer ${tokens.accessToken}`); - upstream = await runUpstream(); - } catch { - // Fall through to the re-authentication response below. + const incoming = new URL(request.url, "http://bff.invalid"); + const target = new URL( + `${incoming.pathname}${incoming.search}`, + config.apiOrigin, + ); + const headers = new Headers(); + for (const header of forwardedRequestHeaders) { + const value = request.headers[header]; + if (typeof value === "string") { + headers.set(header, value); } } - - if (config.oidcIssuer && upstream.status === 401) { - clearSession(request); - return respondReauth(reply); + headers.set(correlationHeader, request.correlationId); + const upstreamTrace = span.upstream(); + if (upstreamTrace) { + headers.set("traceparent", upstreamTrace.traceparent); + if (upstreamTrace.tracestate) { + headers.set("tracestate", upstreamTrace.tracestate); + } } - for (const header of forwardedResponseHeaders) { - const value = upstream.headers.get(header); - if (value !== null) { - reply.header(header, value); + const refreshToken = config.oidcIssuer + ? request.tokenSession.get("refreshToken") + : undefined; + let refreshed = false; + + // Ensure a valid access token before forwarding (proactive refresh). + if (config.oidcIssuer) { + let accessToken = request.session.get("accessToken"); + const expiresAt = request.session.get("expiresAt"); + if (!accessToken || tokenExpired(expiresAt)) { + if (!refreshToken || !app.refreshAccessToken) { + clearSession(request); + recordOutcome(401); + return respondReauth(reply); + } + try { + const tokens = await app.refreshAccessToken(refreshToken); + persistTokenSet(request, tokens); + accessToken = tokens.accessToken; + refreshed = true; + } catch { + clearSession(request); + recordOutcome(401); + return respondReauth(reply); + } } + headers.set("authorization", `Bearer ${accessToken}`); } - reply.code(upstream.status); - if (request.method === "HEAD" || upstream.status === 204) { - return await reply.send(); - } - return await reply.send(Buffer.from(await upstream.arrayBuffer())); - } catch (error) { - if (error instanceof Error && error.name === "UpstreamTimeout") { - reply.code(504); - return { - error: "Gateway Timeout", - statusCode: 504, + + // Perform the upstream request with its own timeout and downstream-abort + // wiring, so it can be retried once after a reactive refresh. + const runUpstream = async (): Promise => { + const controller = new AbortController(); + const timeoutReason = new Error("Upstream API request timed out"); + const timeout = setTimeout(() => { + controller.abort(timeoutReason); + }, config.apiTimeoutMs); + const abortDownstream = () => { + controller.abort(); }; + request.raw.once("aborted", abortDownstream); + try { + return await fetch(target, { + body: proxyBody(request.method, request.body), + headers, + method: request.method, + redirect: "manual", + signal: controller.signal, + }); + } catch (error) { + if (error === timeoutReason) { + const timeout504 = new Error("Upstream API request timed out"); + timeout504.name = "UpstreamTimeout"; + throw timeout504; + } + throw error; + } finally { + clearTimeout(timeout); + request.raw.off("aborted", abortDownstream); + } + }; + + try { + let upstream = await runUpstream(); + + // Reactive refresh: a token we believed valid was rejected upstream + // (clock skew, revocation, or key rotation). Refresh once and retry. + if ( + config.oidcIssuer && + upstream.status === 401 && + !refreshed && + refreshToken && + app.refreshAccessToken + ) { + try { + const tokens = await app.refreshAccessToken(refreshToken); + persistTokenSet(request, tokens); + headers.set("authorization", `Bearer ${tokens.accessToken}`); + upstream = await runUpstream(); + } catch { + // Fall through to the re-authentication response below. + } + } + + if (config.oidcIssuer && upstream.status === 401) { + clearSession(request); + recordOutcome(401); + return respondReauth(reply); + } + + for (const header of forwardedResponseHeaders) { + const value = upstream.headers.get(header); + if (value !== null) { + reply.header(header, value); + } + } + reply.code(upstream.status); + recordOutcome(upstream.status); + if (request.method === "HEAD" || upstream.status === 204) { + return await reply.send(); + } + return await reply.send(Buffer.from(await upstream.arrayBuffer())); + } catch (error) { + if (error instanceof Error && error.name === "UpstreamTimeout") { + reply.code(504); + spanOutcome = "timeout"; + spanStatusCode = 504; + return { + error: "Gateway Timeout", + statusCode: 504, + }; + } + throw error; } - throw error; + } finally { + span.end(spanOutcome, spanStatusCode); } }); diff --git a/components/web-console/bff/src/config.ts b/components/web-console/bff/src/config.ts index 07709f6b..e70c8b29 100644 --- a/components/web-console/bff/src/config.ts +++ b/components/web-console/bff/src/config.ts @@ -43,6 +43,15 @@ const configSchema = z.object({ .enum(["fatal", "error", "warn", "info", "debug", "trace", "silent"]) .default("info"), NODE_ENV: z.enum(["development", "test", "production"]).default("production"), + // Tracing is optional: when the collector endpoint is absent the BFF starts + // normally with tracing disabled and readiness unaffected (WEB-TRACE-06). + OTEL_EXPORTER_OTLP_ENDPOINT: httpUrl.optional(), + OTEL_SERVICE_NAME: z + .string() + .trim() + .min(1) + .default("hypershell-web-console-bff"), + OTEL_TRACES_SAMPLE_RATIO: z.coerce.number().min(0).max(1).default(1), OIDC_CLIENT_ID: z.string().trim().min(1).optional(), OIDC_ISSUER: httpUrl.optional(), OIDC_POST_LOGOUT_REDIRECT_URI: httpUrl.optional(), @@ -65,6 +74,14 @@ const configSchema = z.object({ .default(path.resolve(process.cwd(), "../build/client")), }); +/** Resolved tracing configuration, present only when a collector is configured. */ +export interface TracingConfig { + collectorEndpoint: string; + sampleRatio: number; + serviceName: string; + tracesEndpoint: string; +} + export interface ServerConfig { apiOrigin: string; apiTimeoutMs: number; @@ -79,6 +96,12 @@ export interface ServerConfig { sessionSecret?: Buffer; sessionTtlSeconds: number; staticRoot: string; + tracing?: TracingConfig; +} + +/** Derives the OTLP/HTTP traces URL from a collector base endpoint. */ +function tracesEndpointFor(collectorEndpoint: string): string { + return `${collectorEndpoint.replace(/\/+$/u, "")}/v1/traces`; } export function loadConfig( @@ -125,5 +148,15 @@ export function loadConfig( : undefined, sessionTtlSeconds: result.data.SESSION_TTL_SECONDS, staticRoot: path.resolve(result.data.STATIC_ROOT), + tracing: result.data.OTEL_EXPORTER_OTLP_ENDPOINT + ? { + collectorEndpoint: result.data.OTEL_EXPORTER_OTLP_ENDPOINT, + sampleRatio: result.data.OTEL_TRACES_SAMPLE_RATIO, + serviceName: result.data.OTEL_SERVICE_NAME, + tracesEndpoint: tracesEndpointFor( + result.data.OTEL_EXPORTER_OTLP_ENDPOINT, + ), + } + : undefined, }; } diff --git a/components/web-console/bff/src/index.ts b/components/web-console/bff/src/index.ts index 42bb0ea9..28a04ca6 100644 --- a/components/web-console/bff/src/index.ts +++ b/components/web-console/bff/src/index.ts @@ -1,13 +1,18 @@ +import { createBffTracing } from "./adapters/observability/otel-tracing.js"; import { buildApp } from "./app.js"; import { loadConfig } from "./config.js"; const config = loadConfig(); -const app = await buildApp(config); +// The bootstrap is the one server path exempt from the telemetry import ban; it +// owns the OTel SDK lifecycle and injects the tracing port into the app. +const tracing = createBffTracing(config.tracing); +const app = await buildApp(config, tracing); const shutdown = async (signal: NodeJS.Signals): Promise => { app.log.info({ signal }, "shutting down"); try { await app.close(); + await tracing.shutdown(); process.exitCode = 0; } catch (error) { app.log.error({ err: error }, "graceful shutdown failed"); diff --git a/components/web-console/bff/src/tracing.ts b/components/web-console/bff/src/tracing.ts new file mode 100644 index 00000000..12208a76 --- /dev/null +++ b/components/web-console/bff/src/tracing.ts @@ -0,0 +1,70 @@ +/** + * Application-owned tracing port. The Fastify app depends on this contract + * only; the concrete OpenTelemetry adapter lives in + * `adapters/observability/otel-tracing.ts`, and the bootstrap wires the two. + * Keeping the port free of any tracing vendor lets route code stay inside the + * telemetry import ban that `eslint.config.mjs` enforces. + */ + +/** Bounded outcome class recorded on a proxy span; never a raw status code. */ +export type ProxyOutcome = + "client_error" | "server_error" | "success" | "timeout"; + +/** W3C trace context a proxied request carries to the upstream API. */ +export interface UpstreamTraceContext { + traceparent: string; + tracestate?: string; +} + +/** One in-flight BFF server span for a proxied request. */ +export interface ProxySpan { + /** + * The valid upstream trace context that references this server span, or + * `undefined` when tracing is disabled. The value never echoes a malformed + * inbound header; it is derived from the span this BFF started. + */ + upstream(): UpstreamTraceContext | undefined; + end(outcome: ProxyOutcome, statusCode: number): void; +} + +/** Result of relaying a browser OTLP payload to the collector. */ +export type TelemetryIngestResult = "accepted" | "rejected" | "unavailable"; + +export interface StartProxySpanInput { + correlationId: string; + method: string; + routeTemplate: string; + traceparent?: string; + tracestate?: string; +} + +/** Application-owned port for BFF request tracing and browser telemetry relay. */ +export interface BffTracing { + readonly enabled: boolean; + /** + * Starts a server span for a proxied request. A valid inbound W3C context is + * continued; an absent or malformed inbound context starts a new trace. + */ + startProxySpan(input: StartProxySpanInput): ProxySpan; + /** + * Relays a browser OTLP/HTTP trace payload to the configured collector. + * Rejects a payload that is not well-formed OTLP; a collector that is + * unreachable is reported as unavailable rather than surfaced as an error. + */ + ingestTraces(payload: unknown): Promise; + /** Flushes buffered spans and shuts the exporter down on server shutdown. */ + shutdown(): Promise; +} + +const disabledProxySpan: ProxySpan = { + end: () => undefined, + upstream: () => undefined, +}; + +/** A tracing port with tracing turned off: no spans, no context, no relay. */ +export const disabledTracing: BffTracing = { + enabled: false, + ingestTraces: () => Promise.resolve("unavailable"), + shutdown: () => Promise.resolve(), + startProxySpan: () => disabledProxySpan, +}; diff --git a/components/web-console/bff/test/app-tracing.test.ts b/components/web-console/bff/test/app-tracing.test.ts new file mode 100644 index 00000000..3409bfa6 --- /dev/null +++ b/components/web-console/bff/test/app-tracing.test.ts @@ -0,0 +1,176 @@ +import { createServer, type Server } from "node:http"; +import { mkdtemp, mkdir, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import path from "node:path"; + +import type { FastifyInstance } from "fastify"; + +import { buildApp } from "../src/app.js"; +import type { ServerConfig } from "../src/config.js"; +import type { + BffTracing, + ProxyOutcome, + StartProxySpanInput, +} from "../src/tracing.js"; + +const upstreamTraceparent = + "00-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-bbbbbbbbbbbbbbbb-01"; + +function stubTracing() { + const started: StartProxySpanInput[] = []; + const ended: { outcome: ProxyOutcome; statusCode: number }[] = []; + const ingested: unknown[] = []; + const tracing: BffTracing = { + enabled: true, + ingestTraces: (payload) => { + ingested.push(payload); + return Promise.resolve( + Array.isArray((payload as { resourceSpans?: unknown }).resourceSpans) + ? "accepted" + : "rejected", + ); + }, + shutdown: () => Promise.resolve(), + startProxySpan: (input) => { + started.push(input); + return { + end: (outcome, statusCode) => { + ended.push({ outcome, statusCode }); + }, + upstream: () => ({ + traceparent: upstreamTraceparent, + tracestate: "vendor=1", + }), + }; + }, + }; + return { ended, ingested, started, tracing }; +} + +describe("web-console BFF tracing wiring", () => { + let app: FastifyInstance; + let apiServer: Server; + let staticRoot: string; + let received: { headers: Record }[]; + let trace: ReturnType; + let upstreamStatus: number; + + beforeEach(async () => { + received = []; + upstreamStatus = 200; + apiServer = createServer((request, response) => { + request.on("data", () => undefined); + request.on("end", () => { + received.push({ headers: request.headers }); + response.statusCode = upstreamStatus; + response.setHeader("content-type", "application/json"); + response.end('{"items":[]}'); + }); + }); + await new Promise((resolve) => { + apiServer.listen(0, "127.0.0.1", resolve); + }); + const address = apiServer.address(); + if (address === null || typeof address === "string") { + throw new Error("Expected the test API server to use a TCP address"); + } + + staticRoot = await mkdtemp(path.join(tmpdir(), "hypershell-bff-trace-")); + await mkdir(path.join(staticRoot, "assets")); + await writeFile( + path.join(staticRoot, "index.html"), + "
Hello
", + ); + + const config: ServerConfig = { + apiOrigin: `http://127.0.0.1:${String(address.port)}`, + apiTimeoutMs: 5_000, + host: "127.0.0.1", + logLevel: "silent", + nodeEnv: "test", + port: 8080, + sessionTtlSeconds: 28_800, + staticRoot, + }; + trace = stubTracing(); + app = await buildApp(config, trace.tracing); + }); + + afterEach(async () => { + await app.close(); + await new Promise((resolve, reject) => { + apiServer.close((error) => { + if (error) { + reject(error); + } else { + resolve(); + } + }); + }); + await rm(staticRoot, { force: true, recursive: true }); + }); + + it("propagates the span-derived trace context to the upstream API", async () => { + const response = await app.inject({ + headers: { traceparent: "00-inbound-ignored" }, + method: "GET", + url: "/api/hypershell/v1/gateways", + }); + + expect(response.statusCode).toBe(200); + expect(received).toHaveLength(1); + expect(received[0]?.headers.traceparent).toBe(upstreamTraceparent); + expect(received[0]?.headers.tracestate).toBe("vendor=1"); + }); + + it("starts a span per proxied request and ends it with the outcome", async () => { + await app.inject({ + headers: { traceparent: "00-inbound-value" }, + method: "GET", + url: "/api/hypershell/v1/gateways", + }); + + expect(trace.started).toHaveLength(1); + expect(trace.started[0]).toMatchObject({ + method: "GET", + routeTemplate: "/api/*", + traceparent: "00-inbound-value", + }); + expect(trace.started[0]?.correlationId).toBeTruthy(); + expect(trace.ended).toEqual([{ outcome: "success", statusCode: 200 }]); + }); + + it("records a server-error outcome when the upstream fails", async () => { + upstreamStatus = 502; + + const response = await app.inject({ + method: "GET", + url: "/api/hypershell/v1/gateways", + }); + + expect(response.statusCode).toBe(502); + expect(trace.ended).toEqual([{ outcome: "server_error", statusCode: 502 }]); + }); + + it("accepts a well-formed OTLP payload at the ingest endpoint", async () => { + const response = await app.inject({ + method: "POST", + payload: { resourceSpans: [{ scopeSpans: [] }] }, + url: "/telemetry/v1/traces", + }); + + expect(response.statusCode).toBe(202); + expect(response.headers["cache-control"]).toBe("no-store"); + expect(trace.ingested).toHaveLength(1); + }); + + it("rejects a malformed telemetry payload", async () => { + const response = await app.inject({ + method: "POST", + payload: { notOtlp: true }, + url: "/telemetry/v1/traces", + }); + + expect(response.statusCode).toBe(400); + }); +}); diff --git a/components/web-console/bff/test/config.test.ts b/components/web-console/bff/test/config.test.ts index 51288637..d41773e7 100644 --- a/components/web-console/bff/test/config.test.ts +++ b/components/web-console/bff/test/config.test.ts @@ -33,6 +33,51 @@ describe("loadConfig", () => { loadConfig({ HYPERSHELL_API_ORIGIN: "https://api.example.test/v1" }), ).toThrow(/HYPERSHELL_API_ORIGIN/u); }); + + it("leaves tracing disabled when no collector endpoint is set", () => { + const config = loadConfig({ STATIC_ROOT: "./public" }); + + expect(config.tracing).toBeUndefined(); + }); + + it("derives the OTLP traces endpoint and sampling from configuration", () => { + const config = loadConfig({ + OTEL_EXPORTER_OTLP_ENDPOINT: "http://collector.example.test:4318/", + OTEL_SERVICE_NAME: "web-console-bff", + OTEL_TRACES_SAMPLE_RATIO: "0.25", + STATIC_ROOT: "./public", + }); + + expect(config.tracing).toEqual({ + collectorEndpoint: "http://collector.example.test:4318/", + sampleRatio: 0.25, + serviceName: "web-console-bff", + tracesEndpoint: "http://collector.example.test:4318/v1/traces", + }); + }); + + it("defaults the service name and sample ratio when only the endpoint is set", () => { + const config = loadConfig({ + OTEL_EXPORTER_OTLP_ENDPOINT: "http://collector.example.test:4318", + STATIC_ROOT: "./public", + }); + + expect(config.tracing?.serviceName).toBe("hypershell-web-console-bff"); + expect(config.tracing?.sampleRatio).toBe(1); + expect(config.tracing?.tracesEndpoint).toBe( + "http://collector.example.test:4318/v1/traces", + ); + }); + + it("rejects an out-of-range trace sample ratio", () => { + expect(() => + loadConfig({ + OTEL_EXPORTER_OTLP_ENDPOINT: "http://collector.example.test:4318", + OTEL_TRACES_SAMPLE_RATIO: "5", + STATIC_ROOT: "./public", + }), + ).toThrow(/OTEL_TRACES_SAMPLE_RATIO/u); + }); }); function pathIsAbsolute(value: string): boolean { diff --git a/components/web-console/bff/test/otel-tracing.test.ts b/components/web-console/bff/test/otel-tracing.test.ts new file mode 100644 index 00000000..7c94c31a --- /dev/null +++ b/components/web-console/bff/test/otel-tracing.test.ts @@ -0,0 +1,129 @@ +import { afterEach, describe, expect, it, vi } from "vitest"; + +import type { TracingConfig } from "../src/config.js"; +import { createBffTracing } from "../src/adapters/observability/otel-tracing.js"; + +const config: TracingConfig = { + collectorEndpoint: "http://collector.test:4318", + sampleRatio: 1, + serviceName: "web-console-bff", + tracesEndpoint: "http://collector.test:4318/v1/traces", +}; + +const validTraceparent = + "00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01"; +const inboundTraceId = "0af7651916cd43dd8448eb211c80319c"; + +function proxyInput( + overrides: Partial< + Parameters["startProxySpan"]>[0] + > = {}, +) { + return { + correlationId: "11111111-1111-4111-8111-111111111111", + method: "GET", + routeTemplate: "/api/*", + ...overrides, + }; +} + +describe("BFF tracing adapter", () => { + afterEach(() => { + vi.restoreAllMocks(); + }); + + it("returns a disabled port when no collector is configured", async () => { + const tracing = createBffTracing(undefined); + + expect(tracing.enabled).toBe(false); + expect(tracing.startProxySpan(proxyInput()).upstream()).toBeUndefined(); + await expect(tracing.ingestTraces({ resourceSpans: [] })).resolves.toBe( + "unavailable", + ); + }); + + it("continues a valid inbound trace and references its own span upstream", () => { + const tracing = createBffTracing(config); + + const span = tracing.startProxySpan( + proxyInput({ traceparent: validTraceparent }), + ); + const upstream = span.upstream(); + + expect(upstream?.traceparent).toMatch( + new RegExp(`^00-${inboundTraceId}-[0-9a-f]{16}-01$`), + ); + // The upstream span id references the BFF span, not the inbound one. + expect(upstream?.traceparent).not.toContain("b7ad6b7169203331"); + }); + + it("starts a new trace when the inbound traceparent is malformed", () => { + const tracing = createBffTracing(config); + + const upstream = tracing + .startProxySpan(proxyInput({ traceparent: "not-a-traceparent" })) + .upstream(); + + expect(upstream?.traceparent).toMatch(/^00-[0-9a-f]{32}-[0-9a-f]{16}-01$/); + expect(upstream?.traceparent).not.toContain(inboundTraceId); + expect(upstream?.tracestate).toBeUndefined(); + }); + + it("forwards a valid tracestate only alongside a valid parent", () => { + const tracing = createBffTracing(config); + + const continued = tracing + .startProxySpan( + proxyInput({ traceparent: validTraceparent, tracestate: "vendor=1" }), + ) + .upstream(); + const orphaned = tracing + .startProxySpan( + proxyInput({ traceparent: "bad", tracestate: "vendor=1" }), + ) + .upstream(); + + expect(continued?.tracestate).toBe("vendor=1"); + expect(orphaned?.tracestate).toBeUndefined(); + }); + + it("rejects a payload that is not well-formed OTLP", async () => { + const fetchSpy = vi.spyOn(globalThis, "fetch"); + const tracing = createBffTracing(config); + + await expect(tracing.ingestTraces({ notOtlp: true })).resolves.toBe( + "rejected", + ); + expect(fetchSpy).not.toHaveBeenCalled(); + }); + + it("relays a well-formed OTLP payload to the collector", async () => { + const fetchSpy = vi + .spyOn(globalThis, "fetch") + .mockResolvedValue(new Response(null, { status: 200 })); + const tracing = createBffTracing(config); + + await expect( + tracing.ingestTraces({ resourceSpans: [{ scopeSpans: [] }] }), + ).resolves.toBe("accepted"); + + const [url, init] = fetchSpy.mock.calls[0] ?? []; + expect(url).toBe(config.tracesEndpoint); + expect(init?.method).toBe("POST"); + expect(new Headers(init?.headers).get("content-type")).toBe( + "application/json", + ); + const body = init?.body; + expect(typeof body).toBe("string"); + expect(body as string).toContain("resourceSpans"); + }); + + it("reports the collector as unavailable rather than failing", async () => { + vi.spyOn(globalThis, "fetch").mockRejectedValue(new Error("unreachable")); + const tracing = createBffTracing(config); + + await expect(tracing.ingestTraces({ resourceSpans: [] })).resolves.toBe( + "unavailable", + ); + }); +}); diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 148715ee..753120f5 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -180,6 +180,21 @@ importers: '@fastify/static': specifier: 10.1.2 version: 10.1.2 + '@opentelemetry/api': + specifier: 1.9.1 + version: 1.9.1 + '@opentelemetry/exporter-trace-otlp-http': + specifier: 0.221.0 + version: 0.221.0(@opentelemetry/api@1.9.1) + '@opentelemetry/resources': + specifier: 2.10.0 + version: 2.10.0(@opentelemetry/api@1.9.1) + '@opentelemetry/sdk-trace-base': + specifier: 2.10.0 + version: 2.10.0(@opentelemetry/api@1.9.1) + '@opentelemetry/semantic-conventions': + specifier: 1.43.0 + version: 1.43.0 fastify: specifier: 5.10.0 version: 5.10.0 From ac3fa4109d947a3137d3650a379f8a4360e29bc2 Mon Sep 17 00:00:00 2001 From: user Date: Tue, 18 Aug 2026 10:52:52 -0400 Subject: [PATCH 05/33] feat(kind): add Jaeger dev tracing infra for the web console Deploy Jaeger all-in-one in the Kind environment, gated on KIND_JAEGER, so a developer can view distributed traces from the web console. The manifest mirrors the API server observability work (HYPERSHELL-26) and additionally exposes the OTLP/HTTP receiver on 4318, which the browser and BFF require because browsers cannot speak OTLP gRPC (4317). When KIND_JAEGER=true, make kind-up applies the manifest, patches the web-console Deployment with OTEL_EXPORTER_OTLP_ENDPOINT pointing at the Jaeger OTLP/HTTP endpoint, and prints the Jaeger UI URL. Otherwise the endpoint stays unset and the BFF starts with tracing disabled. Reconcile the local-development spec to the KIND_JAEGER flag, the single jaeger Service, and the dual OTLP receivers. Refs HYPERSHELL-27 Co-Authored-By: Claude Opus 4.8 --- deploy/kind/jaeger.yaml | 82 ++++++++++++++++++++++++ scripts/kind/up.sh | 25 ++++++++ specs/platform/local-development.spec.md | 23 ++++--- 3 files changed, 120 insertions(+), 10 deletions(-) create mode 100644 deploy/kind/jaeger.yaml diff --git a/deploy/kind/jaeger.yaml b/deploy/kind/jaeger.yaml new file mode 100644 index 00000000..ee18fe2c --- /dev/null +++ b/deploy/kind/jaeger.yaml @@ -0,0 +1,82 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: jaeger + namespace: hypershell-system + labels: + app: jaeger +spec: + replicas: 1 + selector: + matchLabels: + app: jaeger + template: + metadata: + labels: + app: jaeger + spec: + containers: + - name: jaeger + image: jaegertracing/all-in-one:latest + ports: + - containerPort: 4317 + name: otlp-grpc + # OTLP/HTTP receiver for the web console browser + BFF, which cannot + # speak OTLP gRPC. Enabled by COLLECTOR_OTLP_ENABLED alongside 4317. + - containerPort: 4318 + name: otlp-http + - containerPort: 16686 + name: query-ui + env: + - name: COLLECTOR_OTLP_ENABLED + value: "true" + securityContext: + runAsNonRoot: true + allowPrivilegeEscalation: false + capabilities: + drop: ["ALL"] + seccompProfile: + type: RuntimeDefault + readinessProbe: + httpGet: + path: / + port: 16686 + initialDelaySeconds: 5 + periodSeconds: 5 +--- +apiVersion: v1 +kind: Service +metadata: + name: jaeger + namespace: hypershell-system + labels: + app: jaeger +spec: + selector: + app: jaeger + ports: + - name: otlp-grpc + port: 4317 + targetPort: 4317 + - name: otlp-http + port: 4318 + targetPort: 4318 + - name: query-ui + port: 16686 + targetPort: 16686 +--- +apiVersion: gateway.networking.k8s.io/v1 +kind: HTTPRoute +metadata: + name: jaeger + namespace: hypershell-system +spec: + parentRefs: + - name: hypershell-gw + namespace: hypershell-system + hostnames: + - jaeger.hypershell.localhost + rules: + - backendRefs: + - name: jaeger + port: 16686 diff --git a/scripts/kind/up.sh b/scripts/kind/up.sh index 8314690a..31914c9e 100755 --- a/scripts/kind/up.sh +++ b/scripts/kind/up.sh @@ -253,6 +253,23 @@ if [[ -z "${KIND_KEYCLOAK_URL:-}" ]]; then success "Keycloak ready" fi +# --- Jaeger (optional, for OTel trace inspection) --- +# Deploys the same all-in-one Jaeger as the API server observability work +# (HYPERSHELL-26). The web console browser and BFF export over OTLP/HTTP (4318) +# because browsers cannot speak OTLP gRPC; the API server uses gRPC (4317). +if [[ "${KIND_JAEGER:-}" == "true" ]]; then + header "Jaeger" + info "Deploying Jaeger..." + kube apply -f deploy/kind/jaeger.yaml + info "Patching web console BFF with OTEL_EXPORTER_OTLP_ENDPOINT..." + kube set env deployment/hypershell-web-console -c web-console -n "${KIND_NAMESPACE}" \ + OTEL_EXPORTER_OTLP_ENDPOINT="http://jaeger.${KIND_NAMESPACE}.svc.cluster.local:4318" + info "Waiting for Jaeger..." + kube wait --for=condition=available deployment/jaeger -n "${KIND_NAMESPACE}" --timeout=120s + success "Jaeger ready" + echo "" +fi + # --- Gateway trusted CA (self-signed CA for OIDC over HTTPS) --- # The gateway pod validates OIDC tokens against the canonical HTTPS issuer # (https://keycloak.hypershell.localhost). That endpoint is served by the @@ -701,6 +718,10 @@ if [[ "${CPK_RUNNING}" == "true" ]]; then info "Keycloak: ${KIND_KEYCLOAK_URL}" fi + if [[ "${KIND_JAEGER:-}" == "true" ]]; then + info "Jaeger UI: https://jaeger.hypershell.localhost${PORT_SUFFIX}" + fi + info "Login: https://${CONSOLE_HOSTNAME}${PORT_SUFFIX}/auth/login" info "Test users: admin/admin (admins + users), developer/developer (users only)" else @@ -714,6 +735,10 @@ else info "Keycloak: ${KIND_KEYCLOAK_URL}" fi + if [[ "${KIND_JAEGER:-}" == "true" ]]; then + info "Jaeger UI: http://localhost:16686" + fi + info "Login: http://localhost:3000/auth/login" info "Test users: admin/admin (admins + users), developer/developer (users only)" diff --git a/specs/platform/local-development.spec.md b/specs/platform/local-development.spec.md index fee2df21..de2c21f1 100644 --- a/specs/platform/local-development.spec.md +++ b/specs/platform/local-development.spec.md @@ -295,15 +295,16 @@ Client Networking Gateway Gateway Pod The local environment SHALL deploy a Jaeger all-in-one instance so a developer can view distributed traces produced by the web console and the API server (see `web-console/tracing.spec.md` and HYPERSHELL-26). Jaeger all-in-one exposes an OTLP receiver and a query UI in one workload; a separate OpenTelemetry Collector is not required for local development. Trace storage uses in-memory storage, which is cleared on restart and is appropriate for development only. -`make kind-up` SHALL deploy Jaeger into the target namespace with its OTLP receiver enabled, and SHALL create an HTTPRoute that exposes the Jaeger UI at `jaeger.hypershell.localhost`. The web console BFF SHALL be configured to export traces to the Jaeger OTLP endpoint through the `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable on the web-console Deployment. The browser exports through the same-origin BFF telemetry endpoint (see `web-console/tracing.spec.md`), so the browser does not reach Jaeger directly. +`make kind-up` SHALL deploy Jaeger into the target namespace with its OTLP receiver enabled, and SHALL create an HTTPRoute that exposes the Jaeger UI at `jaeger.hypershell.localhost`. The Jaeger workload SHALL expose both the OTLP/gRPC receiver (`4317`) used by the API server and the OTLP/HTTP receiver (`4318`) used by the web console, because browsers cannot speak OTLP gRPC. The web console BFF SHALL be configured to export traces to the Jaeger OTLP/HTTP endpoint through the `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable on the web-console Deployment. The browser exports through the same-origin BFF telemetry endpoint (see `web-console/tracing.spec.md`), so the browser does not reach Jaeger directly. | Setting | Value | |---------|-------| | Workload | `jaeger` Deployment (all-in-one, in-memory storage) | -| OTLP receiver (in-cluster) | `http://jaeger-collector..svc.cluster.local:4318` (OTLP/HTTP) | +| OTLP receiver (API server) | `jaeger..svc.cluster.local:4317` (OTLP/gRPC) | +| OTLP receiver (web console BFF) | `http://jaeger..svc.cluster.local:4318` (OTLP/HTTP) | | Query UI | `https://jaeger.hypershell.localhost` | -Deployment of Jaeger SHALL be gated by the `KIND_TRACING` environment variable (default `true`). When `KIND_TRACING=false`, `make kind-up` SHALL skip the Jaeger workload and its route, and the BFF SHALL start with tracing disabled without failing readiness. +Deployment of Jaeger SHALL be gated by the `KIND_JAEGER` environment variable (opt-in; unset or any value other than `true` skips it). When Jaeger is not deployed, `make kind-up` SHALL skip the Jaeger workload and its route and SHALL leave `OTEL_EXPORTER_OTLP_ENDPOINT` unset on the web-console Deployment, so the BFF starts with tracing disabled without failing readiness. ## Requirements @@ -455,7 +456,7 @@ All services SHALL be accessible via `.localhost` hostnames routed through the n | HTTP API | `https://api.hypershell.localhost` | REST API access | | Web Console | `https://console.hypershell.localhost` | Browser UI | | Health | `https://health.hypershell.localhost` | Health check endpoint | -| Jaeger UI | `https://jaeger.hypershell.localhost` | Distributed tracing UI (when `KIND_TRACING` is enabled) | +| Jaeger UI | `https://jaeger.hypershell.localhost` | Distributed tracing UI (when `KIND_JAEGER=true`) | | gRPC | `https://.gw.localhost` | gRPC streaming (control plane, CLI) | The self-signed TLS certificate issued by cert-manager must be trusted by the developer's browser or CLI tool (e.g. `curl --cacert`). @@ -672,14 +673,15 @@ All `kind-*` targets operate on the namespace specified by `KIND_NAMESPACE` (def ### Requirement: Development Tracing -The system SHALL deploy a Jaeger all-in-one instance in the local environment and configure the web console BFF to export traces to it, so a developer can verify one end-to-end trace. Deployment SHALL be gated by `KIND_TRACING` (default `true`). +The system SHALL deploy a Jaeger all-in-one instance in the local environment and configure the web console BFF to export traces to it, so a developer can verify one end-to-end trace. Deployment SHALL be gated by `KIND_JAEGER` (opt-in; deployed only when set to `true`). #### Scenario: Jaeger Available After Setup -- GIVEN `KIND_TRACING` is unset or `true` +- GIVEN `KIND_JAEGER` is `true` - WHEN a developer runs `make kind-up` - THEN a Jaeger all-in-one workload SHALL be deployed into the target namespace +- AND the workload SHALL expose the OTLP/gRPC receiver on `4317` and the OTLP/HTTP receiver on `4318` - AND an HTTPRoute SHALL expose the Jaeger UI at `jaeger.hypershell.localhost` -- AND the web console BFF SHALL be configured with `OTEL_EXPORTER_OTLP_ENDPOINT` pointing at the Jaeger OTLP receiver +- AND the web console BFF SHALL be configured with `OTEL_EXPORTER_OTLP_ENDPOINT` pointing at the Jaeger OTLP/HTTP receiver #### Scenario: End-to-End Trace Visible - GIVEN the local environment is running with tracing enabled @@ -688,9 +690,10 @@ The system SHALL deploy a Jaeger all-in-one instance in the local environment an - AND it SHALL contain the browser workflow span and the BFF server span joined by the same trace identifier #### Scenario: Tracing Disabled -- GIVEN `KIND_TRACING=false` +- GIVEN `KIND_JAEGER` is unset (or any value other than `true`) - WHEN a developer runs `make kind-up` - THEN the Jaeger workload and its route SHALL NOT be deployed +- AND `OTEL_EXPORTER_OTLP_ENDPOINT` SHALL be left unset on the web-console Deployment - AND the web console BFF SHALL start with tracing disabled without failing readiness ## Environment Variable Reference @@ -716,7 +719,7 @@ The system SHALL deploy a Jaeger all-in-one instance in the local environment an | `CERT_MANAGER_VERSION` | `v1.21.1` | cert-manager release version | | `KIND_DB_IMAGE` | `registry.access.redhat.com/hi/postgresql:18` | Database image for Gateway resource; override for OSS dev (unsupported) | | `KIND_NAMESPACE` | `hypershell-system` | Target namespace for all `kind-*` targets | -| `KIND_TRACING` | `true` | Deploy Jaeger all-in-one for distributed tracing; set to `false` to skip | +| `KIND_JAEGER` | (unset) | Set to `true` to deploy Jaeger all-in-one for distributed tracing (OTLP gRPC `4317` + HTTP `4318`) and export web console traces to it | ## Make Targets Summary @@ -749,7 +752,7 @@ All targets operate on `KIND_NAMESPACE` (default: `hypershell-system`). | Images loaded via tarball archive | Compatible with both Podman and Docker; avoids registry dependency | | Rebuild-and-replace on every swap call | Each `kind--up` rebuilds from the working tree and replaces the deployment, even if already swapped; developers iterate by re-running the same target | | Web console as first-class component | Node.js frontend (`components/web-console/`) deployed alongside API server and control plane; supports hot reload via `KIND_HOT_RELOAD` for rapid UI iteration | -| Jaeger all-in-one for local tracing | One workload provides both the OTLP receiver and the query UI with in-memory storage; avoids a separate OpenTelemetry Collector in development. The BFF exports over OTLP; the browser exports through the same-origin BFF endpoint. `KIND_TRACING=false` opts out | +| Jaeger all-in-one for local tracing | One workload provides both OTLP receivers (gRPC `4317` for the API server, HTTP `4318` for the web console) and the query UI with in-memory storage; avoids a separate OpenTelemetry Collector in development. The BFF exports over OTLP/HTTP; the browser exports through the same-origin BFF endpoint. `KIND_JAEGER=true` opts in | | Hot reload on by default | Swap targets for supported components (web console) mount host source and run a dev server in an interactive TTY by default; `KIND_HOT_RELOAD=false` opts out to rebuild-and-replace. Keeps the same `kind--up` entrypoint for both workflows | | Per-component targets require existing cluster | Avoids implicit full-stack deployment; keeps intent explicit | | Database provisioned by control plane | Gateway configured with HI postgresql image; control plane reconciler provisions the database via GatewayReconciler (`specs/platform/openshell-gateway-database.spec.md`, implemented in [#14](https://github.com/openshift-online/hypershell/pull/14)), exercising the same path as production. The API server's own database (`deploy/kind/postgres.yaml`) is always deployed directly by `kind-up` | From 1dabb5b8bdb2ee1383e1340d5337803f571543bc Mon Sep 17 00:00:00 2001 From: user Date: Tue, 18 Aug 2026 10:54:57 -0400 Subject: [PATCH 06/33] style(web-console): apply prettier to browser trace sink and client test Reformat two files from the browser tracing wave that were committed before prettier ran to completion, so pnpm run format:check passes. Refs HYPERSHELL-27 Co-Authored-By: Claude Opus 4.8 --- .../web-console/app/adapters/api/api.client.test.ts | 11 +++++++---- .../app/adapters/observability/gateway-trace-sink.ts | 5 ++--- 2 files changed, 9 insertions(+), 7 deletions(-) diff --git a/components/web-console/app/adapters/api/api.client.test.ts b/components/web-console/app/adapters/api/api.client.test.ts index aa875604..b2dc6399 100644 --- a/components/web-console/app/adapters/api/api.client.test.ts +++ b/components/web-console/app/adapters/api/api.client.test.ts @@ -86,15 +86,16 @@ describe("correlated API fetch", () => { fetchImplementation, undefined, () => ({ - traceparent: - "00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01", + traceparent: "00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01", tracestate: "hypershell=1", }), ); await correlatedFetch("/api/hypershell/v1/gateways"); - const headers = new Headers(fetchImplementation.mock.calls[0]?.[1]?.headers); + const headers = new Headers( + fetchImplementation.mock.calls[0]?.[1]?.headers, + ); expect(headers.get("traceparent")).toBe( "00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01", ); @@ -114,7 +115,9 @@ describe("correlated API fetch", () => { await correlatedFetch("/api/hypershell/v1/gateways"); - const headers = new Headers(fetchImplementation.mock.calls[0]?.[1]?.headers); + const headers = new Headers( + fetchImplementation.mock.calls[0]?.[1]?.headers, + ); expect(headers.has("traceparent")).toBe(false); expect(headers.has("tracestate")).toBe(false); }); diff --git a/components/web-console/app/adapters/observability/gateway-trace-sink.ts b/components/web-console/app/adapters/observability/gateway-trace-sink.ts index 1755cf8c..0fcc9176 100644 --- a/components/web-console/app/adapters/observability/gateway-trace-sink.ts +++ b/components/web-console/app/adapters/observability/gateway-trace-sink.ts @@ -219,9 +219,8 @@ export function createGatewayTraceSink( if (!isSpanContextValid(spanContext)) { return undefined; } - const flags = (spanContext.traceFlags & TraceFlags.SAMPLED) === 0 - ? "00" - : "01"; + const flags = + (spanContext.traceFlags & TraceFlags.SAMPLED) === 0 ? "00" : "01"; const traceparent = `00-${spanContext.traceId}-${spanContext.spanId}-${flags}`; const tracestate = spanContext.traceState?.serialize(); return tracestate === undefined || tracestate === "" From d0c3b6a47d58b66b432b583115b87ef2cce8af5e Mon Sep 17 00:00:00 2001 From: user Date: Tue, 18 Aug 2026 11:12:37 -0400 Subject: [PATCH 07/33] fix(web-console): flush browser spans on page hide BatchSpanProcessor exports on a timer, which a browser can discard when a tab is closed or navigated away, losing the tail of a workflow trace. The forceFlush handle existed but was never wired to any lifecycle event, so the last spans of a gateway workflow were dropped on unload. Register forceFlush on visibilitychange (to hidden) and pagehide, the last reliable hooks before unload, and remove the listeners on shutdown. Correct the docstring that falsely claimed the batch processor already flushed on hide. Add jsdom tests covering flush on hide, no flush while visible, and listener removal after shutdown. Closes WEB-TRACE-09. Co-Authored-By: Claude Opus 4.8 --- .../observability/gateway-trace-sink.test.ts | 67 ++++++++++++++++++- .../observability/gateway-trace-sink.ts | 31 ++++++++- 2 files changed, 93 insertions(+), 5 deletions(-) diff --git a/components/web-console/app/adapters/observability/gateway-trace-sink.test.ts b/components/web-console/app/adapters/observability/gateway-trace-sink.test.ts index 9f724c7e..eb3d4026 100644 --- a/components/web-console/app/adapters/observability/gateway-trace-sink.test.ts +++ b/components/web-console/app/adapters/observability/gateway-trace-sink.test.ts @@ -11,9 +11,12 @@ import { SimpleSpanProcessor, type ReadableSpan, } from "@opentelemetry/sdk-trace-base"; -import { beforeEach, describe, expect, it } from "vitest"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; -import { createGatewayTraceSink } from "./gateway-trace-sink"; +import { + createGatewayTraceSink, + createGatewayTracing, +} from "./gateway-trace-sink"; const traceId = "0af7651916cd43dd8448eb211c80319c"; const parentSpanId = "aaaaaaaaaaaaaaaa"; @@ -206,3 +209,63 @@ describe("gateway trace sink", () => { expect(harness.exporter.getFinishedSpans()).toHaveLength(0); }); }); + +describe("createGatewayTracing flush on page hide", () => { + const config = { + serviceName: "hypershell-web-console", + tracesEndpoint: "http://localhost/telemetry/v1/traces", + }; + + function setVisibility(state: "hidden" | "visible"): void { + Object.defineProperty(document, "visibilityState", { + configurable: true, + get: () => state, + }); + } + + afterEach(() => { + vi.restoreAllMocks(); + setVisibility("visible"); + }); + + it("flushes buffered spans on hidden visibilitychange and on pagehide", async () => { + const flush = vi + .spyOn(BasicTracerProvider.prototype, "forceFlush") + .mockResolvedValue(); + const tracing = createGatewayTracing(config); + + // A visible transition must not flush; only a hide is a last-chance export. + setVisibility("visible"); + document.dispatchEvent(new Event("visibilitychange")); + expect(flush).not.toHaveBeenCalled(); + + setVisibility("hidden"); + document.dispatchEvent(new Event("visibilitychange")); + expect(flush).toHaveBeenCalledTimes(1); + + window.dispatchEvent(new Event("pagehide")); + expect(flush).toHaveBeenCalledTimes(2); + + vi.spyOn(BasicTracerProvider.prototype, "shutdown").mockResolvedValue(); + await tracing.shutdown(); + }); + + it("stops flushing once shutdown removes the listeners", async () => { + const flush = vi + .spyOn(BasicTracerProvider.prototype, "forceFlush") + .mockResolvedValue(); + const shutdown = vi + .spyOn(BasicTracerProvider.prototype, "shutdown") + .mockResolvedValue(); + const tracing = createGatewayTracing(config); + + await tracing.shutdown(); + expect(shutdown).toHaveBeenCalledTimes(1); + flush.mockClear(); + + setVisibility("hidden"); + document.dispatchEvent(new Event("visibilitychange")); + window.dispatchEvent(new Event("pagehide")); + expect(flush).not.toHaveBeenCalled(); + }); +}); diff --git a/components/web-console/app/adapters/observability/gateway-trace-sink.ts b/components/web-console/app/adapters/observability/gateway-trace-sink.ts index 0fcc9176..d31bc486 100644 --- a/components/web-console/app/adapters/observability/gateway-trace-sink.ts +++ b/components/web-console/app/adapters/observability/gateway-trace-sink.ts @@ -237,8 +237,11 @@ export function createGatewayTraceSink( * provider is not registered as the global tracer; the sink owns every span * explicitly, keyed by correlation identifier, so no implicit context is * needed. - * The batch processor flushes on document hide, so page unload does not lose - * buffered spans. + * + * `BatchSpanProcessor` exports on a timer, which a browser can discard when a + * tab is closed or navigated away, losing the tail of a workflow. The provider + * therefore forces a flush on `visibilitychange` to hidden and on `pagehide`, + * the last reliable hooks before unload. `shutdown` removes those listeners. */ export function createGatewayTracing( config: GatewayTracingConfig, @@ -256,9 +259,31 @@ export function createGatewayTracing( const { sink, traceParentFor } = createGatewayTraceSink(tracer, { isSampled: (traceId) => sampledByRatio(traceId, ratio), }); + + const flushBufferedSpans = (): void => { + void provider.forceFlush(); + }; + const flushWhenHidden = (): void => { + if (document.visibilityState === "hidden") { + flushBufferedSpans(); + } + }; + let stopFlushOnHide = (): void => undefined; + if (typeof document !== "undefined" && typeof window !== "undefined") { + document.addEventListener("visibilitychange", flushWhenHidden); + window.addEventListener("pagehide", flushBufferedSpans); + stopFlushOnHide = () => { + document.removeEventListener("visibilitychange", flushWhenHidden); + window.removeEventListener("pagehide", flushBufferedSpans); + }; + } + return { forceFlush: () => provider.forceFlush(), - shutdown: () => provider.shutdown(), + shutdown: async () => { + stopFlushOnHide(); + await provider.shutdown(); + }, sink, traceParentFor, }; From 0bcc5d369d710a3bb5aaefe75f8068b53e1435b9 Mon Sep 17 00:00:00 2001 From: user Date: Tue, 18 Aug 2026 11:13:18 -0400 Subject: [PATCH 08/33] test(web-console): assert tracing redacts secrets and high-cardinality data The browser sink and BFF only ever attach an allowlisted, bounded set of span attributes and bounded span names, so privacy held by construction but had no test guarding it against regression. Add redaction tests that seed a secret-shaped correlation identifier and a URL query string carrying a fake token, then assert: - the browser sink emits only allowlisted attribute keys and bounded span names, and the seeded secret appears in no span name or attribute value; - the BFF records only the bounded route template, never the raw URL or query string, so request secrets never reach a span. Closes WEB-TRACE-07. Co-Authored-By: Claude Opus 4.8 --- .../observability/gateway-trace-sink.test.ts | 51 +++++++++++++++++++ .../web-console/bff/test/app-tracing.test.ts | 16 ++++++ 2 files changed, 67 insertions(+) diff --git a/components/web-console/app/adapters/observability/gateway-trace-sink.test.ts b/components/web-console/app/adapters/observability/gateway-trace-sink.test.ts index eb3d4026..53ab3d63 100644 --- a/components/web-console/app/adapters/observability/gateway-trace-sink.test.ts +++ b/components/web-console/app/adapters/observability/gateway-trace-sink.test.ts @@ -208,6 +208,57 @@ describe("gateway trace sink", () => { ); expect(harness.exporter.getFinishedSpans()).toHaveLength(0); }); + + it("keeps high-cardinality and sensitive values out of every exported span", () => { + // A correlation identifier is opaque and high-cardinality; treat it as a + // stand-in for any secret that must never reach the collector. + const secret = "corr-Bearer-eyJhbGciOi-SEEDED-SECRET"; + const withSecret = (base: GatewayProbe): GatewayProbe => + Object.freeze({ + ...base, + context: Object.freeze({ ...base.context, correlationId: secret }), + }); + + sink.sink.publish(withSecret(probe("gateway.workflow.started"))); + sink.sink.publish(withSecret(probe("gateway.dependency.attempted"))); + sink.sink.publish( + withSecret( + probe("gateway.dependency.completed", { outcome: "succeeded" }), + ), + ); + sink.sink.publish( + withSecret( + probe("gateway.workflow.completed", { + operationId: "operation-1", + outcome: "succeeded", + }), + ), + ); + + const allowedKeys = new Set([ + "gateway.action", + "gateway.outcome", + "gateway.failure_kind", + "hypershell.operation_id", + ]); + const finished = exporter.getFinishedSpans(); + expect(finished).toHaveLength(2); + for (const span of finished) { + // Span names come from a bounded template, never a raw identifier. + expect(span.name).toMatch(/^gateway\.(workflow|dependency)\.[a-z]+$/); + for (const key of Object.keys(span.attributes)) { + expect(allowedKeys.has(key)).toBe(true); + } + // The secret appears in no span name or attribute value. + const serialized = JSON.stringify({ + attributes: span.attributes, + name: span.name, + }); + expect(serialized).not.toContain(secret); + expect(serialized).not.toContain("SEEDED-SECRET"); + expect(serialized).not.toContain("Bearer"); + } + }); }); describe("createGatewayTracing flush on page hide", () => { diff --git a/components/web-console/bff/test/app-tracing.test.ts b/components/web-console/bff/test/app-tracing.test.ts index 3409bfa6..91cb18b3 100644 --- a/components/web-console/bff/test/app-tracing.test.ts +++ b/components/web-console/bff/test/app-tracing.test.ts @@ -152,6 +152,22 @@ describe("web-console BFF tracing wiring", () => { expect(trace.ended).toEqual([{ outcome: "server_error", statusCode: 502 }]); }); + it("keeps request secrets out of the span, recording only the bounded route", async () => { + const secret = "SEED-Bearer-token-do-not-log"; + + await app.inject({ + method: "GET", + url: `/api/hypershell/v1/gateways?access_token=${secret}&user=alice`, + }); + + expect(trace.started).toHaveLength(1); + // The span records the bounded route template, never the raw URL or query. + expect(trace.started[0]?.routeTemplate).toBe("/api/*"); + // No field fed to the span carries the secret or the raw query string. + expect(JSON.stringify(trace.started[0])).not.toContain(secret); + expect(JSON.stringify(trace.started[0])).not.toContain("access_token"); + }); + it("accepts a well-formed OTLP payload at the ingest endpoint", async () => { const response = await app.inject({ method: "POST", From 18aba9e30ffb170b85ab5d4b10a57b0148737669 Mon Sep 17 00:00:00 2001 From: user Date: Tue, 18 Aug 2026 13:53:40 -0400 Subject: [PATCH 09/33] chore(kind): upgrade dev Jaeger to v2 (2.20.0) Switch the local Kind Jaeger from the v1 all-in-one image to the v2 image (jaegertracing/jaeger:2.20.0), which is built on the OpenTelemetry Collector. The v2 all-in-one profile enables the OTLP receivers on 4317/4318 by default, so the v1 COLLECTOR_OTLP_ENABLED env is no longer needed. Verified live: browser -> BFF -> Jaeger cross-service traces land under v2 with OTLP/HTTP ingest on 4318. Co-Authored-By: Claude Opus 4.8 --- deploy/kind/jaeger.yaml | 11 ++++++----- scripts/kind/up.sh | 7 ++++--- 2 files changed, 10 insertions(+), 8 deletions(-) diff --git a/deploy/kind/jaeger.yaml b/deploy/kind/jaeger.yaml index ee18fe2c..b2110915 100644 --- a/deploy/kind/jaeger.yaml +++ b/deploy/kind/jaeger.yaml @@ -17,19 +17,20 @@ spec: spec: containers: - name: jaeger - image: jaegertracing/all-in-one:latest + # Jaeger v2 (built on the OpenTelemetry Collector). Running the image + # with no arguments starts the all-in-one profile with in-memory + # storage, and the OTLP receivers on 4317/4318 are enabled by default, + # so the v1 COLLECTOR_OTLP_ENABLED env is no longer needed. + image: jaegertracing/jaeger:2.20.0 ports: - containerPort: 4317 name: otlp-grpc # OTLP/HTTP receiver for the web console browser + BFF, which cannot - # speak OTLP gRPC. Enabled by COLLECTOR_OTLP_ENABLED alongside 4317. + # speak OTLP gRPC. - containerPort: 4318 name: otlp-http - containerPort: 16686 name: query-ui - env: - - name: COLLECTOR_OTLP_ENABLED - value: "true" securityContext: runAsNonRoot: true allowPrivilegeEscalation: false diff --git a/scripts/kind/up.sh b/scripts/kind/up.sh index 31914c9e..130d6c14 100755 --- a/scripts/kind/up.sh +++ b/scripts/kind/up.sh @@ -254,9 +254,10 @@ if [[ -z "${KIND_KEYCLOAK_URL:-}" ]]; then fi # --- Jaeger (optional, for OTel trace inspection) --- -# Deploys the same all-in-one Jaeger as the API server observability work -# (HYPERSHELL-26). The web console browser and BFF export over OTLP/HTTP (4318) -# because browsers cannot speak OTLP gRPC; the API server uses gRPC (4317). +# Deploys an all-in-one Jaeger v2 for local trace inspection alongside the API +# server observability work (HYPERSHELL-26). The web console browser and BFF +# export over OTLP/HTTP (4318) because browsers cannot speak OTLP gRPC; the API +# server uses gRPC (4317). if [[ "${KIND_JAEGER:-}" == "true" ]]; then header "Jaeger" info "Deploying Jaeger..." From c14a724c0a256fa290f2ab171b8af15d15959e26 Mon Sep 17 00:00:00 2001 From: user Date: Tue, 18 Aug 2026 14:15:06 -0400 Subject: [PATCH 10/33] docs(specs): require automated e2e trace verification (WEB-TRACE-11) Add WEB-TRACE-11 to the web-console tracing spec: the end-to-end trace of WEB-TRACE-10 must be verified automatically, not only by manual inspection. Add a matching "Web Console Distributed Trace Verification" requirement to the e2e-testing spec (KIND_JAEGER=true for e2e, a browser-driven cross-service trace assertion against Jaeger, and trace failure diagnostics), plus the E2E_CONSOLE_URL/E2E_JAEGER_URL variables. Co-Authored-By: Claude Opus 4.8 --- specs/platform/e2e-testing.spec.md | 26 ++++++++++++++++++++++++++ specs/web-console/tracing.spec.md | 14 ++++++++++++++ 2 files changed, 40 insertions(+) diff --git a/specs/platform/e2e-testing.spec.md b/specs/platform/e2e-testing.spec.md index 0dc8200f..0441f523 100644 --- a/specs/platform/e2e-testing.spec.md +++ b/specs/platform/e2e-testing.spec.md @@ -479,6 +479,8 @@ deploy/ | `E2E_DEV_PASSWORD` | `developer` | Password for the developer OIDC user (local dev only) | | `OPENSHELL_BIN` | `openshell` | Path to the openshell CLI binary | | `SSL_CERT_FILE` | (set by the suite) | Path to the extracted cluster CA so the openshell CLI trusts the gateway's TLS cert (replaces the removed `OPENSHELL_GATEWAY_INSECURE` bypass) | +| `E2E_CONSOLE_URL` | `https://console.hypershell.localhost` | Base URL of the deployed web console for the browser trace verification | +| `E2E_JAEGER_URL` | `https://jaeger.hypershell.localhost` | Base URL of the Jaeger query API queried by the trace verification | ### Requirement: OIDC Authentication in E2E Tests @@ -524,6 +526,30 @@ The test suite SHALL verify OIDC integration as part of its standard flow: - WHEN the Kind cluster is created - THEN `make kind-up` SHALL be invoked with `KIND_ENABLE_OIDC=true` +### Requirement: Web Console Distributed Trace Verification + +The CI e2e workflow SHALL verify web console distributed tracing end to end, satisfying `web-console/tracing.spec.md` (`WEB-TRACE-11`). The Kind cluster SHALL be created with tracing enabled (`KIND_JAEGER=true`) so Jaeger is deployed and the web-console BFF exports to it. After the bash suite runs, the workflow SHALL drive a representative gateway workflow through a real browser against the deployed console and assert that Jaeger holds one trace joining the browser and the BFF. The check SHALL use the same Node and Chromium setup as the web-console lint job and SHALL run from the deployed console, not a mocked dev server. The trace check SHALL fail the workflow if no cross-service trace appears within a bounded polling window, and failure diagnostics SHALL include Jaeger workload status and logs and the web-console tracing configuration. + +#### Scenario: Tracing Enabled for E2E + +- GIVEN the CI e2e workflow +- WHEN the Kind cluster is created +- THEN `make kind-up` SHALL be invoked with `KIND_JAEGER=true` +- AND Jaeger SHALL be deployed and the web-console BFF SHALL be configured to export to it + +#### Scenario: Cross-Service Trace Asserted + +- GIVEN the cluster is running with tracing enabled and the console is reachable +- WHEN the trace verification drives a gateway workflow in a real browser and queries Jaeger +- THEN it SHALL find one trace whose spans include a bounded browser workflow span and the BFF server span joined by the same trace identifier +- AND the workflow SHALL fail if no such trace appears within the polling window + +#### Scenario: Trace Failure Diagnostics + +- GIVEN the trace verification fails +- WHEN the workflow reaches its post-test phase +- THEN it SHALL collect Jaeger workload status and logs and the web-console tracing configuration alongside the existing diagnostics + ## Design Decisions | Decision | Rationale | diff --git a/specs/web-console/tracing.spec.md b/specs/web-console/tracing.spec.md index f6349013..62547763 100644 --- a/specs/web-console/tracing.spec.md +++ b/specs/web-console/tracing.spec.md @@ -110,6 +110,20 @@ The local development environment SHALL provide a collector and a Jaeger instanc - THEN one trace SHALL be visible in Jaeger - AND it SHALL contain the browser workflow span and the BFF server span joined by the same trace identifier +### Requirement: WEB-TRACE-11 -- Automated End-to-End Trace Verification + +The end-to-end trace defined by `WEB-TRACE-10` SHALL be verified automatically, not only by manual inspection. The CI end-to-end workflow SHALL bring the cluster up with tracing enabled, drive a representative gateway workflow through the deployed browser console, and assert that Jaeger holds one trace joining the browser and the BFF. The check SHALL confirm the browser spans use the bounded workflow and dependency span names and share a single trace identifier with the BFF server span. A missing trace, a browser-only trace, or a BFF-only trace SHALL fail the workflow. The verification details and its place in the e2e suite are defined in `platform/e2e-testing.spec.md`. + +**Verification:** Run the e2e workflow against a cluster with tracing enabled; confirm the trace check drives a browser workflow, finds a cross-service trace in Jaeger, and fails when no such trace is present. + +#### Scenario: CI Verifies the Cross-Service Trace + +- GIVEN the e2e cluster is running with tracing enabled and the console is reachable +- WHEN the trace verification drives a gateway workflow in a real browser +- THEN it SHALL find one trace in Jaeger whose spans include a bounded browser workflow span and the BFF server span +- AND those spans SHALL share the same trace identifier +- AND the workflow SHALL fail if no such cross-service trace appears within a bounded polling window + ## Design Decisions | Decision | Rationale | From c70a4858661f9a88d71308ef5407241aeaf74808 Mon Sep 17 00:00:00 2001 From: user Date: Tue, 18 Aug 2026 14:17:12 -0400 Subject: [PATCH 11/33] test(e2e): verify cross-service traces reach Jaeger in Kind (WEB-TRACE-11) Add a browser-driven trace check to the platform e2e. The e2e-kind job now brings the cluster up with KIND_JAEGER=true, installs Node + Chromium, and runs a live Playwright spec that logs into the deployed console through Keycloak, drives a gateway list workflow, then polls Jaeger and asserts one trace joins the browser (bounded gateway.* span names) and the BFF server span under a single trace id. A missing, browser-only, or BFF-only trace fails the job. - playwright.live.config.ts + e2e-live/tracing.live.spec.ts: live-cluster config and the cross-service trace assertion. - package.json: test:e2e:live script; tsconfig.test.json: type-check the e2e-live sources and both playwright configs. - Makefile: e2e-tracing target for the same check locally. - e2e.yml: enable Jaeger, add Node/pnpm/Chromium setup, run the trace check, and collect Jaeger diagnostics on failure; widen the job timeout. Co-Authored-By: Claude Opus 4.8 --- .github/workflows/e2e.yml | 35 ++++- Makefile | 11 ++ .../web-console/e2e-live/tracing.live.spec.ts | 134 ++++++++++++++++++ components/web-console/package.json | 1 + .../web-console/playwright.live.config.ts | 28 ++++ components/web-console/tsconfig.test.json | 2 + 6 files changed, 210 insertions(+), 1 deletion(-) create mode 100644 components/web-console/e2e-live/tracing.live.spec.ts create mode 100644 components/web-console/playwright.live.config.ts diff --git a/.github/workflows/e2e.yml b/.github/workflows/e2e.yml index aacd0f96..d1eb2c41 100644 --- a/.github/workflows/e2e.yml +++ b/.github/workflows/e2e.yml @@ -236,7 +236,10 @@ jobs: needs: resolve-images if: needs.resolve-images.outputs.should_run == 'true' runs-on: ubuntu-24.04 - timeout-minutes: 20 + # The browser-driven trace verification adds a Node install, a Chromium + # download, and a Playwright run on top of the bash suite, so allow a wider + # ceiling than the 20 minutes the bash-only suite needed. + timeout-minutes: 25 steps: - name: Check out repository uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 @@ -263,6 +266,9 @@ jobs: CONTROL_PLANE_IMAGE: ${{ needs.resolve-images.outputs.control_plane_image }} WEB_CONSOLE_IMAGE: ${{ needs.resolve-images.outputs.web_console_image }} KIND_ENABLE_OIDC: "true" + # Deploy Jaeger and point the web-console BFF at it so the browser + # trace verification below has a collector to export to (WEB-TRACE-10). + KIND_JAEGER: "true" run: make kind-up - name: Run e2e tests @@ -277,6 +283,23 @@ jobs: NO_COLOR: "1" run: bash tests/e2e/e2e-openshell.sh + # Browser-driven distributed-trace verification (WEB-TRACE-10): drive a + # real gateway workflow in the deployed console, then assert one trace in + # Jaeger joins the browser and the BFF. Reuses the same Node + Chromium + # setup the web-console lint job uses. + - name: Set up Node.js + uses: actions/setup-node@249970729cb0ef3589644e2896645e5dc5ba9c38 # v6.5.0 + with: + node-version-file: .node-version + - name: Install pinned pnpm + run: bash scripts/bootstrap_pnpm.sh + - name: Install dependencies + run: pnpm install --frozen-lockfile + - name: Install Chromium + run: pnpm --filter @openshift-online/hypershell-web-console exec playwright install --with-deps chromium + - name: Verify end-to-end traces reach Jaeger + run: pnpm --filter @openshift-online/hypershell-web-console test:e2e:live + - name: Collect diagnostics if: failure() run: | @@ -302,6 +325,16 @@ jobs: kubectl logs --all-containers --prefix --tail=100 -l app=keycloak -n keycloak 2>&1 | tee e2e-diagnostics/keycloak-logs.txt || true echo "::endgroup::" + echo "::group::Tracing (Jaeger) diagnostics" + { + kubectl get pods,svc,httproutes -l app=jaeger -n hypershell-system -o wide 2>&1 || true + kubectl logs --all-containers --prefix --tail=100 -l app=jaeger -n hypershell-system 2>&1 || true + echo "--- web-console OTEL env ---" + kubectl get deployment/hypershell-web-console -n hypershell-system \ + -o jsonpath='{range .spec.template.spec.containers[*].env[*]}{.name}={.value}{"\n"}{end}' 2>&1 | grep -i otel || true + } | tee e2e-diagnostics/tracing.txt + echo "::endgroup::" + echo "::group::Events (hypershell-system)" kubectl get events --sort-by=.lastTimestamp -n hypershell-system 2>&1 | tee e2e-diagnostics/events.txt || true echo "::endgroup::" diff --git a/Makefile b/Makefile index ebcd21a3..f5dba71e 100644 --- a/Makefile +++ b/Makefile @@ -407,3 +407,14 @@ e2e: E2E_PROVISION_TIMEOUT=300 \ E2E_SANDBOX_TIMEOUT=180 \ bash tests/e2e/e2e-openshell.sh + +# Browser-driven end-to-end trace verification (WEB-TRACE-10). Requires a Kind +# cluster brought up with tracing enabled (KIND_JAEGER=true make kind-up), so +# Jaeger is deployed and the web-console BFF exports to it. +.PHONY: e2e-tracing +e2e-tracing: + @echo "" + @echo "==> Verifying end-to-end traces reach Jaeger (Kind)" + @echo " (requires: KIND_JAEGER=true make kind-up)" + @echo "" + @pnpm --filter @openshift-online/hypershell-web-console test:e2e:live diff --git a/components/web-console/e2e-live/tracing.live.spec.ts b/components/web-console/e2e-live/tracing.live.spec.ts new file mode 100644 index 00000000..e279b160 --- /dev/null +++ b/components/web-console/e2e-live/tracing.live.spec.ts @@ -0,0 +1,134 @@ +import { expect, request as playwrightRequest, test } from "@playwright/test"; + +// End-to-end trace verification against a live cluster (WEB-TRACE-10). Drives a +// real gateway workflow in the browser, then queries Jaeger and asserts one +// trace joins the browser and the BFF by the same trace id -- proving the +// browser OTel SDK, the same-origin telemetry ingest, the BFF server span, and +// W3C context propagation all work together in a deployed environment. + +const jaegerUrl = + process.env.E2E_JAEGER_URL ?? "https://jaeger.hypershell.localhost"; +const oidcUser = process.env.E2E_OIDC_USERNAME ?? "admin"; +const oidcPassword = process.env.E2E_OIDC_PASSWORD ?? "admin"; + +const browserService = "hypershell-web-console"; +const bffService = "hypershell-web-console-bff"; +// Span names must come from the bounded workflow/dependency templates, never a +// raw identifier (WEB-TRACE-07). +const boundedSpanName = /^gateway\.(workflow|dependency)\.[a-z-]+$/u; + +interface JaegerSpan { + readonly traceID: string; + readonly spanID: string; + readonly operationName: string; + readonly processID: string; +} + +interface JaegerProcess { + readonly serviceName: string; +} + +interface JaegerTrace { + readonly traceID: string; + readonly spans: readonly JaegerSpan[]; + readonly processes: Readonly>; +} + +interface JaegerTracesResponse { + readonly data: readonly JaegerTrace[]; +} + +function serviceOf(trace: JaegerTrace, span: JaegerSpan): string | undefined { + return trace.processes[span.processID]?.serviceName; +} + +// A cross-service trace carries at least one bounded browser span and at least +// one BFF span under the same trace id -- the join that proves propagation. +function isCrossServiceTrace(trace: JaegerTrace): boolean { + let hasBrowserWorkflow = false; + let hasBffSpan = false; + for (const span of trace.spans) { + const service = serviceOf(trace, span); + if ( + service === browserService && + boundedSpanName.test(span.operationName) + ) { + hasBrowserWorkflow = true; + } + if (service === bffService) { + hasBffSpan = true; + } + } + return hasBrowserWorkflow && hasBffSpan; +} + +test("browser and BFF spans join one trace in Jaeger", async ({ page }) => { + // 1. Log in through Keycloak and land on the gateway list. Loading the list + // is a gateway "list" workflow, which drives an /api proxy call through the + // BFF and produces both a browser workflow span and a BFF server span. + await page.goto("/", { waitUntil: "domcontentloaded" }); + await page.fill("#username", oidcUser); + await page.fill("#password", oidcPassword); + await Promise.all([ + page.waitForURL(/console\.hypershell\.localhost/u), + page.click("#kc-login, input[type=submit], button[type=submit]"), + ]); + await expect(page.locator("h1").first()).toBeVisible(); + + // 2. Exercise an explicit refresh to emit another list workflow, then let the + // batch span processor timer flush; closing the page fires the pagehide + // flush as a belt-and-braces last-chance export (WEB-TRACE-09). + const refresh = page.getByRole("button", { name: /refresh gateways/iu }); + if (await refresh.count()) { + await refresh.first().click(); + } + await page.waitForTimeout(6_000); + await page.close(); + + // 3. Poll Jaeger for a trace that spans both services. Export is best-effort + // and asynchronous, so retry within a bounded window before failing. + const api = await playwrightRequest.newContext({ ignoreHTTPSErrors: true }); + let crossServiceTrace: JaegerTrace | undefined; + await expect + .poll( + async () => { + const response = await api.get( + `${jaegerUrl}/api/traces?service=${bffService}&lookback=1h&limit=20`, + ); + if (!response.ok()) { + return 0; + } + const body = (await response.json()) as JaegerTracesResponse; + crossServiceTrace = body.data.find(isCrossServiceTrace); + return crossServiceTrace === undefined ? 0 : 1; + }, + { + message: `no cross-service trace (${browserService} + ${bffService}) reached Jaeger`, + timeout: 60_000, + intervals: [2_000], + }, + ) + .toBe(1); + await api.dispose(); + + // 4. Confirm the join concretely: a bounded browser span and a BFF span share + // one trace id. + const trace = crossServiceTrace; + expect(trace).toBeDefined(); + if (trace === undefined) { + return; + } + const browserSpans = trace.spans.filter( + (span) => + serviceOf(trace, span) === browserService && + boundedSpanName.test(span.operationName), + ); + const bffSpans = trace.spans.filter( + (span) => serviceOf(trace, span) === bffService, + ); + expect(browserSpans.length).toBeGreaterThan(0); + expect(bffSpans.length).toBeGreaterThan(0); + for (const span of [...browserSpans, ...bffSpans]) { + expect(span.traceID).toBe(trace.traceID); + } +}); diff --git a/components/web-console/package.json b/components/web-console/package.json index fa04b3ec..80788a99 100644 --- a/components/web-console/package.json +++ b/components/web-console/package.json @@ -17,6 +17,7 @@ "test": "vitest", "test:e2e": "playwright test", "test:e2e:chromium": "playwright test --project=chromium", + "test:e2e:live": "playwright test --config playwright.live.config.ts", "test:run": "vitest run --coverage", "typecheck": "react-router typegen && tsc --project tsconfig.app.json --noEmit && tsc --project tsconfig.test.json --noEmit" }, diff --git a/components/web-console/playwright.live.config.ts b/components/web-console/playwright.live.config.ts new file mode 100644 index 00000000..6652ecbf --- /dev/null +++ b/components/web-console/playwright.live.config.ts @@ -0,0 +1,28 @@ +import { defineConfig, devices } from "@playwright/test"; + +// Live-cluster e2e configuration. Unlike playwright.config.ts (which serves a +// mocked dev build), this suite runs against a deployed HyperShell environment +// -- a Kind cluster in CI or a developer's local cluster -- and asserts real +// distributed traces reach Jaeger. There is no webServer: the console is the +// deployed BFF, reached over its self-signed gateway TLS. +const consoleUrl = + process.env.E2E_CONSOLE_URL ?? "https://console.hypershell.localhost"; + +export default defineConfig({ + testDir: "./e2e-live", + fullyParallel: false, + forbidOnly: Boolean(process.env.CI), + retries: process.env.CI ? 2 : 0, + reporter: process.env.CI ? "github" : "list", + // Trace export is asynchronous (batch span processor timer plus flush on page + // hide), so a single test polls Jaeger and needs a generous ceiling. + timeout: 120_000, + use: { + baseURL: consoleUrl, + // The gateway serves a Kind self-signed certificate for + // *.hypershell.localhost; trust it the same way the bash e2e curl -k does. + ignoreHTTPSErrors: true, + trace: "on-first-retry", + }, + projects: [{ name: "chromium", use: { ...devices["Desktop Chrome"] } }], +}); diff --git a/components/web-console/tsconfig.test.json b/components/web-console/tsconfig.test.json index f36017f2..379c9654 100644 --- a/components/web-console/tsconfig.test.json +++ b/components/web-console/tsconfig.test.json @@ -8,7 +8,9 @@ "app/**/*", ".storybook/**/*", "e2e/**/*", + "e2e-live/**/*", "playwright.config.ts", + "playwright.live.config.ts", "vitest.config.ts", "vitest.setup.ts" ], From 3a5f6b13c5074240e406737a00d52522c6b886fc Mon Sep 17 00:00:00 2001 From: user Date: Tue, 18 Aug 2026 14:37:36 -0400 Subject: [PATCH 12/33] fix(web-console): make the browser workflow span a true trace root The gateway trace sink adopted the app-chosen trace id by manufacturing a synthetic remote parent (a random span id that no service ever exports). Jaeger then showed the workflow span as a child of a missing parent, so every browser trace was decapitated and emitted spurious clock-skew warnings. Replace the manufactured parent with a RootTraceIdGenerator on the provider: the sink primes the next root trace id and starts the workflow span with no parent, so it is a genuine trace root that still owns the chosen id and joins the trace propagated to the BFF and API. Sampling moves to a TraceIdRatioBasedSampler at the root (matching the BFF's sampler), replacing the custom per-call decision; child dependency spans inherit it. Update WEB-TRACE-01 to require the workflow span be a true root with no synthetic remote parent. Co-Authored-By: Claude Opus 4.8 --- .../observability/gateway-trace-sink.test.ts | 30 +++-- .../observability/gateway-trace-sink.ts | 122 +++++++++--------- specs/web-console/tracing.spec.md | 2 + 3 files changed, 84 insertions(+), 70 deletions(-) diff --git a/components/web-console/app/adapters/observability/gateway-trace-sink.test.ts b/components/web-console/app/adapters/observability/gateway-trace-sink.test.ts index 53ab3d63..53cb2d5d 100644 --- a/components/web-console/app/adapters/observability/gateway-trace-sink.test.ts +++ b/components/web-console/app/adapters/observability/gateway-trace-sink.test.ts @@ -4,22 +4,24 @@ import type { } from "@openshift-online/hypershell-gateway-management-ui"; import { SpanStatusCode } from "@opentelemetry/api"; import { + AlwaysOffSampler, AlwaysOnSampler, BasicTracerProvider, InMemorySpanExporter, ParentBasedSampler, SimpleSpanProcessor, type ReadableSpan, + type Sampler, } from "@opentelemetry/sdk-trace-base"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import { + RootTraceIdGenerator, createGatewayTraceSink, createGatewayTracing, } from "./gateway-trace-sink"; const traceId = "0af7651916cd43dd8448eb211c80319c"; -const parentSpanId = "aaaaaaaaaaaaaaaa"; const correlationId = "correlation-1"; function probe( @@ -54,13 +56,15 @@ function probe( }); } -function testTracer() { +function testTracer(rootSampler: Sampler = new AlwaysOnSampler()) { const exporter = new InMemorySpanExporter(); + const idGenerator = new RootTraceIdGenerator(); const provider = new BasicTracerProvider({ - sampler: new ParentBasedSampler({ root: new AlwaysOnSampler() }), + idGenerator, + sampler: new ParentBasedSampler({ root: rootSampler }), spanProcessors: [new SimpleSpanProcessor(exporter)], }); - return { exporter, tracer: provider.getTracer("test") }; + return { exporter, idGenerator, tracer: provider.getTracer("test") }; } function byName(spans: readonly ReadableSpan[], name: string): ReadableSpan { @@ -79,7 +83,9 @@ describe("gateway trace sink", () => { const harness = testTracer(); exporter = harness.exporter; sink = createGatewayTraceSink(harness.tracer, { - generateSpanId: () => parentSpanId, + beginTrace: (id) => { + harness.idGenerator.primeTraceId(id); + }, }); }); @@ -99,9 +105,10 @@ describe("gateway trace sink", () => { expect(workflow.spanContext().traceId).toBe(traceId); expect(dependency.spanContext().traceId).toBe(traceId); - // The workflow span descends from the manufactured remote parent, so the - // trace joins the id the caller propagates end to end. - expect(workflow.parentSpanContext?.spanId).toBe(parentSpanId); + // The workflow span is a true trace root: it adopts the chosen trace id yet + // has no parent, so the trace is never decapitated by a synthetic remote + // parent that no service exports. The dependency nests under the workflow. + expect(workflow.parentSpanContext).toBeUndefined(); expect(dependency.parentSpanContext?.spanId).toBe( workflow.spanContext().spanId, ); @@ -185,10 +192,11 @@ describe("gateway trace sink", () => { }); it("drops a trace when the sampler declines it", () => { - const harness = testTracer(); + const harness = testTracer(new AlwaysOffSampler()); const unsampled = createGatewayTraceSink(harness.tracer, { - generateSpanId: () => parentSpanId, - isSampled: () => false, + beginTrace: (id) => { + harness.idGenerator.primeTraceId(id); + }, }); unsampled.sink.publish(probe("gateway.workflow.started")); diff --git a/components/web-console/app/adapters/observability/gateway-trace-sink.ts b/components/web-console/app/adapters/observability/gateway-trace-sink.ts index d31bc486..bcc72dee 100644 --- a/components/web-console/app/adapters/observability/gateway-trace-sink.ts +++ b/components/web-console/app/adapters/observability/gateway-trace-sink.ts @@ -8,17 +8,16 @@ import { context as otelContext, isSpanContextValid, trace as otelTrace, - type Context, type Span, - type SpanContext, type Tracer, } from "@opentelemetry/api"; import { BasicTracerProvider, BatchSpanProcessor, ParentBasedSampler, - AlwaysOnSampler, RandomIdGenerator, + TraceIdRatioBasedSampler, + type IdGenerator, } from "@opentelemetry/sdk-trace-base"; import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http"; import { resourceFromAttributes } from "@opentelemetry/resources"; @@ -49,12 +48,13 @@ export interface GatewayTracing extends GatewayTraceSink { export interface GatewayTraceSinkOptions { /** - * Decides whether one trace is recorded, keyed by its trace id so the - * decision is stable for the whole trace. Defaults to always sampling. + * Primes the trace id that the next root workflow span adopts, so the + * workflow span is a true root that owns the app-chosen trace id rather than + * descending from a synthetic remote parent. Wired to the provider's + * {@link RootTraceIdGenerator}. When omitted, a workflow keeps its probe + * trace id only for propagation and the exported root uses a generated id. */ - isSampled?: (traceId: string) => boolean; - /** Generates the 16-byte span id (32 hex) of the manufactured remote parent. */ - generateSpanId?: () => string; + beginTrace?: (traceId: string) => void; } export interface GatewayTracingConfig { @@ -68,31 +68,39 @@ export interface GatewayTracingConfig { const sinkId = "gateway-trace"; const tracerName = "gateway-trace-sink"; +/** + * Id generator that lets the caller choose the trace id of the next root span + * while keeping every span id random. A workflow span is the origin of the + * distributed trace, so it must be a true root; priming the trace id here lets + * that root still adopt the app-chosen id, joining the trace the browser + * propagates to the BFF and API without a synthetic remote parent (which would + * leave the trace decapitated by a parent span that no service ever exports). + */ +export class RootTraceIdGenerator implements IdGenerator { + private nextTraceId: string | undefined; + private readonly random = new RandomIdGenerator(); + + /** Sets the trace id the next generated root span adopts. */ + primeTraceId(traceId: string): void { + this.nextTraceId = traceId; + } + + generateTraceId(): string { + const chosen = this.nextTraceId; + this.nextTraceId = undefined; + return chosen ?? this.random.generateTraceId(); + } + + generateSpanId(): string { + return this.random.generateSpanId(); + } +} + interface SpanEntry { workflow: Span; dependency?: Span; } -/** - * Manufactures the remote parent context that makes a workflow span adopt the - * caller-chosen trace id. The parent is marked sampled only when `isSampled` - * accepts the trace id; a `ParentBasedSampler` then honours that decision for - * the whole trace, so the W3C sampled flag stays consistent end to end. - */ -function remoteParentContext( - traceId: string, - spanId: string, - sampled: boolean, -): Context { - const spanContext: SpanContext = { - isRemote: true, - spanId, - traceFlags: sampled ? TraceFlags.SAMPLED : TraceFlags.NONE, - traceId, - }; - return otelTrace.setSpanContext(ROOT_CONTEXT, spanContext); -} - /** Terminal outcomes that mark a span failed rather than ok. */ function isFailureOutcome(outcome: GatewayProbe["fields"]["outcome"]): boolean { return outcome !== "started" && outcome !== "succeeded"; @@ -116,26 +124,30 @@ function applyTerminalOutcome(span: Span, probe: GatewayProbe): void { /** * Builds a domain probe sink that projects gateway workflow and dependency - * probes onto OpenTelemetry spans. The sink adopts the trace id carried on each - * probe context so a workflow span joins the same trace the browser propagates - * to the BFF and API. Span names are drawn from a bounded action template so - * cardinality stays fixed. + * probes onto OpenTelemetry spans. A workflow span is a true root that adopts + * the trace id carried on each probe context (through the primed + * {@link RootTraceIdGenerator}), so a workflow span joins the same trace the + * browser propagates to the BFF and API while remaining the origin of that + * trace. Span names are drawn from a bounded action template so cardinality + * stays fixed. */ export function createGatewayTraceSink( tracer: Tracer, options: GatewayTraceSinkOptions = {}, ): GatewayTraceSink { - const generateSpanId = - options.generateSpanId ?? (() => new RandomIdGenerator().generateSpanId()); - const isSampled = options.isSampled ?? (() => true); + const beginTrace = options.beginTrace ?? ((): void => undefined); const spansByCorrelation = new Map(); function startWorkflow(probe: GatewayProbe): void { const { correlationId, traceId } = probe.context; - const parent = - traceId === undefined - ? otelContext.active() - : remoteParentContext(traceId, generateSpanId(), isSampled(traceId)); + // A workflow with a chosen trace id is the trace root: prime the generator + // and start it with no parent. Without a chosen id, fall back to the active + // context so any caller-established parent still nests. + let parent = otelContext.active(); + if (traceId !== undefined) { + beginTrace(traceId); + parent = ROOT_CONTEXT; + } const workflow = tracer.startSpan( `gateway.workflow.${probe.fields.action}`, { @@ -236,7 +248,10 @@ export function createGatewayTraceSink( * same-origin OTLP/HTTP, and returns the gateway trace sink bound to it. The * provider is not registered as the global tracer; the sink owns every span * explicitly, keyed by correlation identifier, so no implicit context is - * needed. + * needed. Sampling is a per-trace decision made once at the workflow root by a + * `TraceIdRatioBasedSampler` and inherited by child spans, so the browser and + * the BFF (which uses the same OTel sampler) agree on each trace without + * sharing a decision. * * `BatchSpanProcessor` exports on a timer, which a browser can discard when a * tab is closed or navigated away, losing the tail of a workflow. The provider @@ -247,17 +262,23 @@ export function createGatewayTracing( config: GatewayTracingConfig, ): GatewayTracing { const ratio = config.sampleRatio ?? 1; + const idGenerator = new RootTraceIdGenerator(); const exporter = new OTLPTraceExporter({ url: config.tracesEndpoint }); const provider = new BasicTracerProvider({ + idGenerator, resource: resourceFromAttributes({ [ATTR_SERVICE_NAME]: config.serviceName, }), - sampler: new ParentBasedSampler({ root: new AlwaysOnSampler() }), + sampler: new ParentBasedSampler({ + root: new TraceIdRatioBasedSampler(ratio), + }), spanProcessors: [new BatchSpanProcessor(exporter)], }); const tracer = provider.getTracer(tracerName); const { sink, traceParentFor } = createGatewayTraceSink(tracer, { - isSampled: (traceId) => sampledByRatio(traceId, ratio), + beginTrace: (traceId) => { + idGenerator.primeTraceId(traceId); + }, }); const flushBufferedSpans = (): void => { @@ -288,20 +309,3 @@ export function createGatewayTracing( traceParentFor, }; } - -/** - * Deterministic per-trace sampling decision. The leading 4 bytes of the trace - * id map to a value in [0, 1); a trace records when that value is below the - * configured ratio. Deterministic keying keeps the browser, BFF, and API in - * agreement without sharing a decision. - */ -function sampledByRatio(traceId: string, ratio: number): boolean { - if (ratio >= 1) { - return true; - } - if (ratio <= 0) { - return false; - } - const bucket = Number.parseInt(traceId.slice(0, 8), 16); - return bucket / 0x1_0000_0000 < ratio; -} diff --git a/specs/web-console/tracing.spec.md b/specs/web-console/tracing.spec.md index 62547763..b4a15d86 100644 --- a/specs/web-console/tracing.spec.md +++ b/specs/web-console/tracing.spec.md @@ -16,6 +16,8 @@ The API server is instrumented separately (HYPERSHELL-26). This specification re The browser SHALL produce spans from the typed domain probes published through the `DomainProbePublisher` fan-out, not from ad hoc instrumentation. A trace sink SHALL implement the `DomainProbeSink` contract and register through the existing `additionalSinks` extension point of the gateway observability adapter. The sink SHALL map a workflow `started` probe and its matching terminal probe (`succeeded`, `failed`, `cancelled`, `denied`, or `conflicted`) to one span, and each dependency-attempt probe pair to one child span. Span status SHALL reflect the terminal outcome. The sink SHALL NOT create spans from React renders, pointer movement, or pure computation. +The workflow span SHALL be the root of the browser-originated trace. It MAY adopt the caller-chosen trace identifier so the trace joins the id the browser propagates to the BFF and API, but it SHALL NOT descend from a synthetic or remote parent span. A manufactured parent that no service records would leave the exported trace decapitated (a workflow span whose parent is missing from the trace), so the sink SHALL make the workflow span a true root while still adopting the chosen trace identifier. + Browser application, domain, and API-adapter code SHALL NOT import an OpenTelemetry package. Only the observability adapter and the composition root MAY import OTel, as enforced by `eslint.architecture.mjs`. Experimental browser auto-instrumentation SHALL NOT be a foundational dependency. **Verification:** Run each gateway use case with a recording publisher plus the trace sink; assert one workflow span per invocation, one child span per dependency attempt, and a span status that matches the terminal outcome. Confirm no span is emitted for a rerender. Run the architecture lint and confirm an OTel import outside the approved paths fails. From 079ca644f2515fdc19f8e96c6845715dbe94dfd7 Mon Sep 17 00:00:00 2001 From: user Date: Tue, 18 Aug 2026 14:38:18 -0400 Subject: [PATCH 13/33] fix(web-console): name BFF proxy spans by method and templated route Every proxied request produced a span named "/api/*" -- the Fastify catch-all pattern -- because startProxySpan used request.routeOptions.url for both the span name and http.route. In Jaeger this collapsed all API calls onto one operation, erasing per-endpoint grouping and latency breakdowns. Pass the request path (no query string) to the tracing adapter, which now renders a bounded route template: after the /api//v prefix, resource ids collapse to {id} while collection and action segments stay literal. The span is named "