From 87d2b3417da98142cec03d2c2754f55ecba869ae Mon Sep 17 00:00:00 2001 From: "David Muto (pseudomuto)" Date: Fri, 14 Aug 2026 07:22:16 -0400 Subject: [PATCH] docs(temporal-proxy): update for v0.5.0 v0.5.0 adds a service allowlist and turns the inbound auth extension point into an authorization one. Neither was covered here. The allowlist matters most for anyone upgrading, since the proxy forwards only WorkflowService and OperatorService by default. A deployment that relies on reflection has to opt in, and a service left out answers _Unimplemented_ rather than passing the call through. The gateway now publishes a health entry per allowed service, which is what makes a gRPC probe or the Go SDK's CheckHealth work, so the Kubernetes page has some probe guidance next to the drain budgets a probe has to fit inside. --- .../temporal-proxy/configure.mdx | 129 ++++++++++++++++-- .../temporal-proxy/deploy-kubernetes.mdx | 63 +++++++-- .../temporal-proxy/index.mdx | 27 ++-- 3 files changed, 182 insertions(+), 37 deletions(-) diff --git a/docs/production-deployment/temporal-proxy/configure.mdx b/docs/production-deployment/temporal-proxy/configure.mdx index d187d355a5..6df329ff26 100644 --- a/docs/production-deployment/temporal-proxy/configure.mdx +++ b/docs/production-deployment/temporal-proxy/configure.mdx @@ -18,14 +18,15 @@ import { ReleaseNoteHeader } from '@site/src/components'; Temporal Proxy is under active development and evolving quickly. Behavior and configuration can change between - releases. See the [temporal-proxy repository](https://github.com/temporalio/temporal-proxy) for the current status - and the definitive configuration schema. + releases. See the [temporal-proxy repository](https://github.com/temporalio/temporal-proxy) for the current status and + the definitive configuration schema. The proxy reads a single YAML file. Three sections are the core of it: the gateway listener (`hostPort`), `routing`, and -the `upstreams` it forwards to. The optional `tls`, `auth`, `encryption`, and `extensionServers` sections add inbound -TLS, inbound authentication, and payload encryption. Values support `${VAR}` and `$VAR` environment variable expansion, -and an upstream's `hostPort` can be a template that resolves per request (for example, `{{ .RemoteNamespace }}`). +the `upstreams` it forwards to. The optional `allowedServices`, `tls`, `auth`, `encryption`, and `extensionServers` +sections narrow which gRPC services the proxy forwards and add inbound TLS, inbound authentication and authorization, +and payload encryption. Values support `${VAR}` and `$VAR` environment variable expansion, and an upstream's `hostPort` +can be a template that resolves per request (for example, `{{ .RemoteNamespace }}`). The example below is the proxy's [Temporal Cloud example](https://github.com/temporalio/temporal-proxy/tree/main/examples/cloud), which connects a Worker @@ -66,6 +67,45 @@ upstreams: The [chart `values.yaml`](https://github.com/temporalio/helm-charts/tree/main/charts/temporal-proxy) and the [temporal-proxy repository](https://github.com/temporalio/temporal-proxy) hold the complete, current set of options. +## Restrict forwarded services + +The top-level `allowedServices` list names the gRPC services the proxy forwards, by proto full name. Omit it and the +proxy forwards `WorkflowService` and `OperatorService`, which is what an SDK Client, a Worker, the CLI, and the Web UI +need. + +```yaml +allowedServices: + - temporal.api.workflowservice.v1.WorkflowService + - temporal.api.operatorservice.v1.OperatorService + - grpc.reflection.v1.ServerReflection +``` + +| Service | Backs | In the default set | +| ------------------------------------------------- | ------------------------------------------------------------------------------- | ------------------ | +| `temporal.api.workflowservice.v1.WorkflowService` | Every SDK Client, Worker, and the Web UI | Yes | +| `temporal.api.operatorservice.v1.OperatorService` | `temporal operator` commands, and the Web UI's Search Attribute and Nexus views | Yes | +| `grpc.reflection.v1.ServerReflection` | Service discovery for tools such as `grpcurl` | No | + +Those three are the whole set to select from. A name the proxy cannot forward fails the configuration at startup rather +than at the first request. + +A service you leave out is never forwarded. The gateway answers a call to one with `Unimplemented` and +`unknown service ""`, decided before any upstream work, so the proxy behaves as a server that does not implement +it rather than passing the call through. + +Service discovery is opt-in, so name the reflection service only when you want tools such as `grpcurl` to probe the +gateway. Allowing `grpc.reflection.v1.ServerReflection` also allows the superseded +`grpc.reflection.v1alpha.ServerReflection`, because clients probe `v1` and fall back to it. The relation runs one way: +allowing only the superseded spelling does not allow the current one. + +### Health checks {/* #health-checks */} + +The gateway serves the standard gRPC health service and publishes an entry for every allowed service under its proto +full name, alongside the usual empty service name. Each entry reports the same status, since the proxy's health is +process-wide. A Client health check that names a service, such as the Go SDK's `CheckHealth`, therefore gets a status, +and so does a per-service liveness probe. A service your configuration does not allow returns `NOT_FOUND` instead of +claiming a status. + ## Route requests The `routing` section selects an upstream for each request: @@ -148,11 +188,12 @@ Namespaces. Available variables: Upstreams with a static `hostPort` connect eagerly at startup; templated ones connect lazily on first use. -## Authenticate inbound requests +## Authenticate and authorize inbound requests Inbound authentication runs on the gateway and is off by default: omit the top-level `auth` block to accept all -requests. When present, `auth` must select exactly one authenticator, `staticToken` or `jwks`. The gateway validates the -credential on each request and strips it before forwarding upstream. +requests. When present, `auth` must select exactly one authenticator: `staticToken`, `jwks`, or `external`. The gateway +decides before the request is routed, so nothing reaches an upstream until the caller is admitted, and it strips the +credential before forwarding upstream. Compare an inbound bearer token against a fixed value with `staticToken`: @@ -177,8 +218,70 @@ auth: scheme: Bearer ``` -`token` (for `staticToken`) and `url` (for `jwks`) are required; the remaining fields are optional. Only `staticToken` -or `jwks` may be set, not both. +`token` (for `staticToken`) and `url` (for `jwks`) are required; the remaining fields are optional. + +### Delegate the decision to an extension server + +For an identity system neither built-in authenticator covers, or for a decision that turns on more than who the caller +is, point `auth.external` at an extension server you run. Declare the server under the top-level `extensionServers` +block, the same way you would a +[key management backend](/production-deployment/temporal-proxy/encrypt-payloads#plug-in-your-own-key-management-backend), +and the proxy asks it about every stream it accepts, forwarding only the ones it is told to admit. + +```yaml +extensionServers: + - name: authz + hostPort: 127.0.0.1:9444 + +auth: + external: + name: authz # Required. Names an entry in extensionServers. + credentialHeaders: + - authorization +``` + +`name` must match a configured extension server. `credentialHeaders` names the metadata headers that carry the caller's +credentials: the proxy lifts those into the request it sends the extension server and removes them from the stream it +forwards upstream, so a credential the server consumes never reaches the Temporal Service. + +Declaring no headers does not hide the caller's credentials from the extension server, since the proxy forwards the +caller's other metadata on the call either way. What you lose is the field naming them, so the server has to know which +metadata to read and cannot tell a header the proxy vouches for from any other. Nothing is stripped before forwarding +upstream either, so the caller's credential continues to the upstream alongside any credential configured for it. + +The server implements `api.auth.v1.AuthService`, one RPC defined in +[`api/auth/v1`](https://github.com/temporalio/temporal-proxy/tree/main/api/auth/v1). Each request carries what the call +is addressing and who is making it: + +| Field | Carries | +| ------------------ | ----------------------------------------------------------------------------------------------------------------- | +| `target.full_name` | The gRPC full method being invoked, leading slash included. Always set. | +| `target.namespace` | The Temporal Namespace the request names. | +| `credentials` | One entry per declared header the caller sent, each with the canonical header name and every value sent under it. | + +Both `target` fields come from the stream rather than from anything the caller claims, so a caller cannot forge either +by sending a header. `target.namespace` is empty when the method names no Namespace, when the caller sent no message to +read one from, or when the service is not one the proxy forwards. Empty means unknown rather than a Namespace called +nothing, so match a Namespace-scoped rule against `full_name` as well. `credentials` is empty when you declared no +headers or the caller sent none of them, which is an unauthenticated caller rather than a trusted one. + +Answer with a `decision`: + +- `DECISION_ALLOW` admits the caller and is the only value that does. A response left unfilled is + `DECISION_UNSPECIFIED`, which denies, so there is no way to admit a caller by omission. +- `DECISION_DENY` refuses the caller, who is told `PERMISSION_DENIED`. +- `reason` is optional and written for whoever operates the server. The proxy records it and keeps it out of what a + refused caller is told, so it can name internal systems or subjects. + +Return a gRPC error only for reaching no verdict at all, such as a backend the server cannot itself reach. The proxy +denies either way, but an error keeps its status code, so `UNAVAILABLE` or `DEADLINE_EXCEEDED` tells a Worker to retry +where a denial tells it not to bother. A server that cannot reach its own backend should report that rather than admit +the caller. + +The [authorization example](https://github.com/temporalio/temporal-proxy/tree/main/examples/authz) maps a JWT to claims +and then decides each call against them, the two steps Temporal Server splits across its `ClaimMapper` and `Authorizer`. +It includes a Worker that cannot reach a second Namespace and an auditor that can read Workflow History but not start a +Workflow. ## Present credentials to upstreams @@ -205,7 +308,8 @@ load. TLS is terminated in two independent places, both using the same keys: `ca`, `cert`, `key`, and `serverName`. `ca`, `cert`, and `key` are paths to PEM files on disk, not inline PEM content. On Kubernetes, let the chart mount them from a -Secret and fill in the paths for you, as described in [Supply TLS material](/production-deployment/temporal-proxy/deploy-kubernetes#supply-tls-material). +Secret and fill in the paths for you, as described in +[Supply TLS material](/production-deployment/temporal-proxy/deploy-kubernetes#supply-tls-material). **Inbound, on the gateway.** The top-level `tls` block secures connections from your applications. Set `cert` and `key` for server TLS, and add `ca` to enforce mutual TLS, which requires each client to present a certificate signed by that @@ -220,4 +324,5 @@ CA. Local development commonly omits `tls` and connects in plaintext. Set `serverName` when the host you dial does not match the common name or SAN on the server's certificate. -For payload encryption (envelope encryption, cloud KMS, and custom key backends), see [Encrypt payloads](/production-deployment/temporal-proxy/encrypt-payloads). +For payload encryption (envelope encryption, cloud KMS, and custom key backends), see +[Encrypt payloads](/production-deployment/temporal-proxy/encrypt-payloads). diff --git a/docs/production-deployment/temporal-proxy/deploy-kubernetes.mdx b/docs/production-deployment/temporal-proxy/deploy-kubernetes.mdx index 69cdbf3be7..8a36e6d2b4 100644 --- a/docs/production-deployment/temporal-proxy/deploy-kubernetes.mdx +++ b/docs/production-deployment/temporal-proxy/deploy-kubernetes.mdx @@ -18,8 +18,8 @@ import { ReleaseNoteHeader } from '@site/src/components'; Temporal Proxy is under active development and evolving quickly. Behavior and configuration can change between - releases. See the [temporal-proxy repository](https://github.com/temporalio/temporal-proxy) for the current status - and the definitive configuration schema. + releases. See the [temporal-proxy repository](https://github.com/temporalio/temporal-proxy) for the current status and + the definitive configuration schema. The [Helm chart](https://github.com/temporalio/helm-charts/tree/main/charts/temporal-proxy) provisions everything the @@ -88,7 +88,7 @@ config: That renders `apiKey: ${TP_UPSTREAM_CLOUD_API_KEY}` into the ConfigMap and adds a matching environment variable backed by the `temporal-cloud` Secret's `api-key` entry. -## Supply TLS material {/* #supply-tls-material */} +## Supply TLS material {/*#supply-tls-material*/} The gateway's `config.tls` block and each `upstreams[].tls` block accept a `secretName`. The chart mounts that Secret at `/etc/temporal-proxy/certs/gateway` or `/etc/temporal-proxy/certs/upstream-` and fills in the file paths for you: @@ -100,8 +100,8 @@ The gateway's `config.tls` block and each `upstreams[].tls` block accept a `secr | `ca` | `/`, only when `caKey` is set | `ca` is opt-in because it changes behavior rather than just adding material. On the gateway, a `ca` enforces mutual TLS, -so only set `caKey` on `config.tls` when you intend to require client certificates. See [Configure TLS](/production-deployment/temporal-proxy/configure#configure-tls) -for what each combination means. +so only set `caKey` on `config.tls` when you intend to require client certificates. See +[Configure TLS](/production-deployment/temporal-proxy/configure#configure-tls) for what each combination means. ```yaml config: @@ -121,23 +121,59 @@ Keep upstream `name` values DNS-safe, meaning lowercase alphanumeric characters name from them. To manage certificate files yourself instead, skip `secretName` and mount them with the chart's generic `volumes` and `volumeMounts` values. +## Probe the gateway + +The gateway serves the standard gRPC health service, so a Kubernetes gRPC probe can read it directly. The chart passes +`livenessProbe` and `readinessProbe` through to the container as written, and its defaults are HTTP GETs against the +gateway port, which a gRPC listener does not answer: + +```yaml +livenessProbe: + grpc: + port: 8443 # Must be numeric. Match service.port. +readinessProbe: + grpc: + port: 8443 + service: temporal.api.workflowservice.v1.WorkflowService +``` + +Naming a service is optional; omit it to check the empty service name. Either way the answer is the same, because the +proxy's health is process-wide, and a service your configuration does not allow returns `NOT_FOUND`. See +[Health checks](/production-deployment/temporal-proxy/configure#health-checks) for what the gateway publishes. + +Kubernetes gRPC probes connect in plaintext, so they cannot reach a gateway that terminates TLS. When `config.tls` is +set, probe the proxy some other way, such as an exec probe running a client that presents the right certificate. + +## Shut down cleanly + +On `SIGTERM` the proxy stops accepting connections and drains in-flight RPCs. Each serving tier, meaning the gateway and +each upstream proxy, drains within five seconds, and the whole stop sequence is bounded at 30 seconds. Long-poll methods +block for much longer than that, so a rollout normally cuts some of them short: the proxy logs a warning naming the +deadline that ended the drain and still exits zero, because the callers re-poll. Treat that warning as ordinary on a +restart rather than a failed shutdown. + +An ordinary drain finishes in about five seconds, well inside the Kubernetes default `terminationGracePeriodSeconds` +of 30. + ## Grant access to your KMS key The one thing the chart cannot do for you is grant the proxy access to your cloud KMS key. When you enable -[payload encryption](/production-deployment/temporal-proxy/encrypt-payloads), the proxy needs a cloud identity that holds the encrypt and decrypt permissions -on the key. Bind the chart's Kubernetes ServiceAccount to that identity with workload identity, so the proxy -authenticates to your KMS with no static credentials. If you do not use encryption, you can skip this: API keys and TLS -material come from the configuration and mounted Secrets, so the proxy needs no cloud identity. +[payload encryption](/production-deployment/temporal-proxy/encrypt-payloads), the proxy needs a cloud identity that +holds the encrypt and decrypt permissions on the key. Bind the chart's Kubernetes ServiceAccount to that identity with +workload identity, so the proxy authenticates to your KMS with no static credentials. If you do not use encryption, you +can skip this: API keys and TLS material come from the configuration and mounted Secrets, so the proxy needs no cloud +identity. Grant the cloud identity the encrypt and decrypt permissions first, as described under -[Encrypt payloads](/production-deployment/temporal-proxy/encrypt-payloads), then bind it to the ServiceAccount as shown below. The bindings reference the -ServiceAccount by name, so set `serviceAccount.name` in your values to keep it stable and matching what you bind on the -cloud side. +[Encrypt payloads](/production-deployment/temporal-proxy/encrypt-payloads), then bind it to the ServiceAccount as shown +below. The bindings reference the ServiceAccount by name, so set `serviceAccount.name` in your values to keep it stable +and matching what you bind on the cloud side. ### AWS IRSA and Pod Identity Create an IAM role whose trust policy lets your cluster's OIDC provider assume it from the proxy's ServiceAccount, and -attach the KMS policy from [AWS KMS](/production-deployment/temporal-proxy/encrypt-payloads#aws-kms). Then annotate the ServiceAccount with the role ARN: +attach the KMS policy from [AWS KMS](/production-deployment/temporal-proxy/encrypt-payloads#aws-kms). Then annotate the +ServiceAccount with the role ARN: ```yaml serviceAccount: @@ -195,4 +231,3 @@ serviceAccount: annotations: iam.gke.io/gcp-service-account: proxy@my-project.iam.gserviceaccount.com ``` - diff --git a/docs/production-deployment/temporal-proxy/index.mdx b/docs/production-deployment/temporal-proxy/index.mdx index fa1e3f4322..aed2871b66 100644 --- a/docs/production-deployment/temporal-proxy/index.mdx +++ b/docs/production-deployment/temporal-proxy/index.mdx @@ -21,8 +21,8 @@ import { ReleaseNoteHeader } from '@site/src/components'; Temporal Proxy is under active development and evolving quickly. Behavior and configuration can change between - releases. See the [temporal-proxy repository](https://github.com/temporalio/temporal-proxy) for the current status - and the definitive configuration schema. + releases. See the [temporal-proxy repository](https://github.com/temporalio/temporal-proxy) for the current status and + the definitive configuration schema. The Temporal Proxy is a gRPC proxy that sits between your Temporal SDK Clients, Workers, and the Temporal Web UI on one @@ -74,11 +74,13 @@ flowchart LR For each request, the gateway: -1. peeks the target Namespace without parsing the payload; it is codec-transparent and relays raw frames in both +1. refuses the call if its gRPC service is not one the configuration allows, before any upstream work. By default that + allows `WorkflowService` and `OperatorService` and nothing else. +2. peeks the target Namespace without parsing the payload; it is codec-transparent and relays raw frames in both directions. -2. picks an upstream: the first matching routing rule, otherwise the system upstream for Namespace-less calls, otherwise +3. picks an upstream: the first matching routing rule, otherwise the system upstream for Namespace-less calls, otherwise the default. -3. hands the request to that upstream's proxy over a unix socket. +4. hands the request to that upstream's proxy over a unix socket. The per-upstream proxy then rewrites the local Namespace to the name the upstream expects, attaches that upstream's TLS and credentials, forwards to the Temporal Service, and translates the Namespace back on responses. When payload @@ -92,7 +94,7 @@ stores ciphertext. | gateway | The single inbound gRPC endpoint that every SDK Client, Worker, and the Web UI connects to. It routes each request to an upstream by Namespace and request metadata, and never parses payloads. | | upstream | A configured destination the proxy forwards to: a Temporal Service (local dev, self-hosted, or Temporal Cloud), or another Temporal Proxy. | | system upstream | The upstream that handles Namespace-less requests, such as the SDK's `GetSystemInfo` call on connect. | -| extension server | A gRPC service you run that the proxy calls out to for a capability it has no built-in backend for. Today that means wrapping data encryption keys as a key management backend. | +| extension server | A gRPC service you run that the proxy calls out to for a capability it has no built-in backend for: wrapping data encryption keys, or deciding whether an inbound call may proceed. | | Temporal Service | A Temporal frontend the proxy connects to. | ## Prerequisites @@ -147,8 +149,12 @@ mounted configuration. See [Observability](#observability) for the remaining fla ## Observability {/* #observability */} -The proxy serves Prometheus metrics at `/metrics` on `:9090` and logs JSON to stderr. Both the metrics listener and the -log level are set with flags on `proxy serve`, each with an environment variable equivalent: +The proxy serves Prometheus metrics at `/metrics` on `:9090` and logs JSON to stderr. The gateway also serves the +standard gRPC health service, with an entry for every allowed service, which is what a Client health check or a liveness +probe reads. See [Health checks](/production-deployment/temporal-proxy/configure#health-checks) for what it reports. + +Both the metrics listener and the log level are set with flags on `proxy serve`, each with an environment variable +equivalent: | Flag | Environment variable | Default | Sets | | --------------------- | -------------------- | ------------- | -------------------------------------------- | @@ -177,8 +183,8 @@ Metric names are `__`, so with the default namespace | `encryption` | `dek_cache_misses_total` | none | Reads that required a KMS unwrap | | `encryption` | `dek_cache_size` | none | Current entries in the decrypted-DEK cache | -The `encryption` metrics only move when [payload encryption](/production-deployment/temporal-proxy/encrypt-payloads) is configured. They are layered, so pick -the one that matches the question you are asking: +The `encryption` metrics only move when [payload encryption](/production-deployment/temporal-proxy/encrypt-payloads) is +configured. They are layered, so pick the one that matches the question you are asking: - `vault_ops_*` is the whole envelope operation end to end, including any KEK call and cache lookup, and is the pair to alert on. It carries the local Namespace. @@ -193,7 +199,6 @@ pre-rotation, and `on_demand` for a DEK replaced at request time because no fres rate means rotation is falling behind, so raise `renewBefore`. Compare the cache counters against `cacheSize` to see whether the cache is absorbing read traffic. - ## Related - [Temporal Proxy repository](https://github.com/temporalio/temporal-proxy)