Feat: add SPIRE operand controller for ZTWIM CR and SPIRE CRs lifecycle

[r3v5]

Summary

• New SpireOperandReconciler + SpireBootstrapRunnable that create and reconcile 5 SPIRE operand CRs (operator.openshift.io/v1alpha1) when ZTWIM CRDs are present
• CRD-gated activation (no feature flag) — mirrors TektonConfigCRDExists / CertManagerCRDExists pattern
• Replaces fragile Helm post-install hooks with proper reconciliation loop (RHAIENG-5468)

Changes

File	Action	Purpose
internal/controller/spire_operand_controller.go	NEW	Reconciler, bootstrap, CRD check, spec builders, mergeNestedMap
internal/controller/spire_operand_controller_test.go	NEW	37 Ginkgo unit tests
test/e2e/spire_operand_test.go	NEW	8 e2e tests + mock CRDs
internal/discovery/discovery.go	MOD	Export ztwimGVK → ZTWIMGVK
cmd/main.go	MOD	CRD-gated registration (no flag)
config/rbac/role.yaml	MOD	SPIRE RBAC from kubebuilder markers
docs/architecture.md	MOD	Controller docs, diagram, RBAC table
docs/operator.md	MOD	Controller listing

Managed CRs

All operator.openshift.io/v1alpha1, cluster-scoped, name "cluster":

CR	Key Spec Fields	Role
ZeroTrustWorkloadIdentityManager	trustDomain, clusterName, bundleConfigMap	Parent — created first
SpiffeCSIDriver	agentSocketPath, pluginName	CSI volume plugin for SVID mounting
SpireServer	caSubject, persistence, datastore, jwtIssuer	SPIRE server configuration
SpireAgent	nodeAttestor, workloadAttestors	Node-level SPIRE agent
SpireOIDCDiscoveryProvider	csiDriverName, jwtIssuer	OIDC endpoint for JWT-SVID

Key Design Decisions

• mergeNestedMap instead of whole-spec replace: preserves ZTWIM operator defaults (e.g. disableMigration), avoids reconcile update loops
• trustDomain immutability: reads existing trustDomain from ZTWIM CR instead of overwriting with auto-discovered value (ZTWIM operator enforces immutability)
• Unstructured CRs: uses unstructured.Unstructured to avoid importing OCP ZTWIM operator Go types as a dependency

Verification Report: RHAIENG-5468 — SPIRE Operand Controller

Date: 2026-06-23
Feature: SPIRE/ZTWIM Operand Controller
Jira: RHAIENG-5468
Branch: create-controller-for-spire-and-ztwim

Environment

Component	Version
OCP (ROSA)	4.19.18
Kubernetes	v1.32.9
ZTWIM Operator	v1.0.1 (stable-v1 channel)
cert-manager	v1.17.2
RHBK Operator	v26.4.12
Operator image	image-registry.openshift-image-registry.svc:5000/kagenti-system/kagenti-operator:spire-operand
Cluster	apps.rosa.ian-cluster.g532.p3.openshiftapps.com

Unit Test Results

Command:

cd kagenti-operator && go test -v -count=1 ./internal/controller/ -ginkgo.focus="SPIRE"

Output:

37 passed, 0 failed

Tests cover: CRD guard, ZTWIM creation, 4 child CR creation, spec validation,
drift correction, idempotency, bootstrap runnable, trustDomain immutability,
spec builder assertions.

Acceptance Criteria

AC1: Controller is no-op when CRDs not installed — PASS

Method: Unit tests + operator startup logs.

Command (unit test):

go test -v -count=1 ./internal/controller/ -ginkgo.focus="ZTWIM CR does not exist"

Output:

SPIRE Operand Controller When ZTWIM CR does not exist
  should return without error (bootstrap pending) [PASSED]

Command (operator startup — CRDs present):

oc -n kagenti-system logs deployment/kagenti-controller-manager | grep -E "ZTWIM CRD|SPIRE operand controller enabled|Auto-discovered SPIRE"

Output:

{"level":"info","ts":"2026-06-23T11:31:21Z","logger":"setup","msg":"Auto-discovered SPIRE trust domain","trustDomain":"localtest.me"}
{"level":"info","ts":"2026-06-23T11:31:21Z","logger":"controller.SpireOperand","msg":"ZTWIM CRD detected: will manage SPIRE operand CRs"}
{"level":"info","ts":"2026-06-23T11:31:21Z","logger":"setup","msg":"SPIRE operand controller enabled","trustDomain":"localtest.me"}

Controller detects CRDs and activates. On clusters without CRDs, it logs
"ZTWIM CRD not registered, SPIRE operand controller will not start" and
returns nil (unit tested).

AC2: Operator creates all 5 SPIRE CRs when CRDs present — PASS

From-scratch creation with auto-discovery (production behavior)

ZTWIM parent CR exists on cluster (created by admin or Helm). No env var overrides.
Operator auto-discovers trustDomain from ZTWIM CR, creates 4 children.

Command (delete children, restart operator):

for kind in spiffecsidrivers spireservers spireagents spireoidcdiscoveryproviders; do
  oc delete $kind.operator.openshift.io cluster --ignore-not-found
done
oc -n kagenti-system delete pod -l control-plane=controller-manager
oc -n kagenti-system rollout status deployment/kagenti-controller-manager --timeout=120s

Command (verify auto-discovery + creation):

oc -n kagenti-system logs deployment/kagenti-controller-manager | \
  grep -E "Auto-discovered SPIRE|Created SPIRE|spire-bootstrap|SPIRE operand controller enabled|SPIRE operand reconcil"

Output:

{"level":"info","ts":"2026-06-23T11:31:21Z","logger":"setup","msg":"Auto-discovered SPIRE trust domain","trustDomain":"localtest.me"}
{"level":"info","ts":"2026-06-23T11:31:21Z","logger":"setup","msg":"SPIRE operand controller enabled","trustDomain":"localtest.me"}
{"level":"info","ts":"2026-06-23T11:31:36Z","logger":"spire-bootstrap","msg":"SPIRE bootstrap: ensuring ZTWIM CR exists"}
{"level":"info","ts":"2026-06-23T11:31:36Z","logger":"spire-bootstrap","msg":"SPIRE bootstrap: ZTWIM CR already exists, skipping creation"}
{"level":"info","ts":"2026-06-23T11:31:36Z","msg":"Created SPIRE operand CR","controller":"spire-operand","controllerGroup":"operator.openshift.io","controllerKind":"ZeroTrustWorkloadIdentityManager","ZeroTrustWorkloadIdentityManager":{"name":"cluster"},"namespace":"","name":"cluster","reconcileID":"a42ed70a-de26-4bf0-8a2b-568539ded430","kind":"SpiffeCSIDriver","name":"cluster"}
{"level":"info","ts":"2026-06-23T11:31:36Z","msg":"Created SPIRE operand CR","controller":"spire-operand","controllerGroup":"operator.openshift.io","controllerKind":"ZeroTrustWorkloadIdentityManager","ZeroTrustWorkloadIdentityManager":{"name":"cluster"},"namespace":"","name":"cluster","reconcileID":"a42ed70a-de26-4bf0-8a2b-568539ded430","kind":"SpireServer","name":"cluster"}
{"level":"info","ts":"2026-06-23T11:31:36Z","msg":"Created SPIRE operand CR","controller":"spire-operand","controllerGroup":"operator.openshift.io","controllerKind":"ZeroTrustWorkloadIdentityManager","ZeroTrustWorkloadIdentityManager":{"name":"cluster"},"namespace":"","name":"cluster","reconcileID":"a42ed70a-de26-4bf0-8a2b-568539ded430","kind":"SpireAgent","name":"cluster"}
{"level":"info","ts":"2026-06-23T11:31:36Z","msg":"Created SPIRE operand CR","controller":"spire-operand","controllerGroup":"operator.openshift.io","controllerKind":"ZeroTrustWorkloadIdentityManager","ZeroTrustWorkloadIdentityManager":{"name":"cluster"},"namespace":"","name":"cluster","reconcileID":"a42ed70a-de26-4bf0-8a2b-568539ded430","kind":"SpireOIDCDiscoveryProvider","name":"cluster"}
{"level":"info","ts":"2026-06-23T11:31:36Z","msg":"SPIRE operand reconciliation complete","controller":"spire-operand","controllerGroup":"operator.openshift.io","controllerKind":"ZeroTrustWorkloadIdentityManager","ZeroTrustWorkloadIdentityManager":{"name":"cluster"},"namespace":"","name":"cluster","reconcileID":"a42ed70a-de26-4bf0-8a2b-568539ded430"}
{"level":"info","ts":"2026-06-23T11:31:37Z","msg":"SPIRE operand reconciliation complete","controller":"spire-operand","controllerGroup":"operator.openshift.io","controllerKind":"ZeroTrustWorkloadIdentityManager","ZeroTrustWorkloadIdentityManager":{"name":"cluster"},"namespace":"","name":"cluster","reconcileID":"3b382ff5-425c-4694-95ef-e48a62ea00db"}

No env vars, no manual steps. Trust domain auto-discovered from existing ZTWIM CR.

Verify all 5 CRs and labels

Command:

for kind in zerotrustworkloadidentitymanagers spiffecsidrivers spireservers spireagents spireoidcdiscoveryproviders; do
  echo "--- $kind ---"
  oc get $kind.operator.openshift.io cluster -o jsonpath='{.metadata.name}' && echo ""
  echo "  managed-by=$(oc get $kind.operator.openshift.io cluster -o jsonpath='{.metadata.labels.app\.kubernetes\.io/managed-by}')"
done

Output:

--- zerotrustworkloadidentitymanagers ---
cluster
  managed-by=kagenti-operator
--- spiffecsidrivers ---
cluster
  managed-by=kagenti-operator
--- spireservers ---
cluster
  managed-by=kagenti-operator
--- spireagents ---
cluster
  managed-by=kagenti-operator
--- spireoidcdiscoveryproviders ---
cluster
  managed-by=kagenti-operator

Command (verify ZTWIM spec):

oc get zerotrustworkloadidentitymanagers.operator.openshift.io cluster -o jsonpath='{.spec}' | python3 -m json.tool

Output:

{
    "bundleConfigMap": "spire-bundle",
    "clusterName": "agent-platform",
    "trustDomain": "localtest.me"
}

Command (verify SpiffeCSIDriver spec):

oc get spiffecsidrivers.operator.openshift.io cluster -o jsonpath='{.spec}' | python3 -m json.tool

Output:

{
    "agentSocketPath": "/run/spire/agent-sockets",
    "pluginName": "csi.spiffe.io"
}

Command (verify SpireAgent has no trustDomain — inherited from parent):

oc get spireagents.operator.openshift.io cluster -o jsonpath='{.spec.trustDomain}'

Output:

(empty — trustDomain inherited from ZTWIM parent, not set on children)

AC3: ZTWIM parent created before children — PASS

Command:

oc -n kagenti-system logs deployment/kagenti-controller-manager | \
  grep -E "Created SPIRE|spire-bootstrap|SPIRE operand reconciliation complete" | head -8

Output:

{"level":"info","ts":"2026-06-23T11:31:36Z","logger":"spire-bootstrap","msg":"SPIRE bootstrap: ensuring ZTWIM CR exists"}
{"level":"info","ts":"2026-06-23T11:31:36Z","logger":"spire-bootstrap","msg":"SPIRE bootstrap: ZTWIM CR already exists, skipping creation"}
{"level":"info","ts":"2026-06-23T11:31:36Z","msg":"Created SPIRE operand CR","controller":"spire-operand","controllerGroup":"operator.openshift.io","controllerKind":"ZeroTrustWorkloadIdentityManager","ZeroTrustWorkloadIdentityManager":{"name":"cluster"},"namespace":"","name":"cluster","reconcileID":"a42ed70a-de26-4bf0-8a2b-568539ded430","kind":"SpiffeCSIDriver","name":"cluster"}
{"level":"info","ts":"2026-06-23T11:31:36Z","msg":"Created SPIRE operand CR","controller":"spire-operand","controllerGroup":"operator.openshift.io","controllerKind":"ZeroTrustWorkloadIdentityManager","ZeroTrustWorkloadIdentityManager":{"name":"cluster"},"namespace":"","name":"cluster","reconcileID":"a42ed70a-de26-4bf0-8a2b-568539ded430","kind":"SpireServer","name":"cluster"}
{"level":"info","ts":"2026-06-23T11:31:36Z","msg":"Created SPIRE operand CR","controller":"spire-operand","controllerGroup":"operator.openshift.io","controllerKind":"ZeroTrustWorkloadIdentityManager","ZeroTrustWorkloadIdentityManager":{"name":"cluster"},"namespace":"","name":"cluster","reconcileID":"a42ed70a-de26-4bf0-8a2b-568539ded430","kind":"SpireAgent","name":"cluster"}
{"level":"info","ts":"2026-06-23T11:31:36Z","msg":"Created SPIRE operand CR","controller":"spire-operand","controllerGroup":"operator.openshift.io","controllerKind":"ZeroTrustWorkloadIdentityManager","ZeroTrustWorkloadIdentityManager":{"name":"cluster"},"namespace":"","name":"cluster","reconcileID":"a42ed70a-de26-4bf0-8a2b-568539ded430","kind":"SpireOIDCDiscoveryProvider","name":"cluster"}
{"level":"info","ts":"2026-06-23T11:31:36Z","msg":"SPIRE operand reconciliation complete","controller":"spire-operand","controllerGroup":"operator.openshift.io","controllerKind":"ZeroTrustWorkloadIdentityManager","ZeroTrustWorkloadIdentityManager":{"name":"cluster"},"namespace":"","name":"cluster","reconcileID":"a42ed70a-de26-4bf0-8a2b-568539ded430"}
{"level":"info","ts":"2026-06-23T11:31:37Z","msg":"SPIRE operand reconciliation complete","controller":"spire-operand","controllerGroup":"operator.openshift.io","controllerKind":"ZeroTrustWorkloadIdentityManager","ZeroTrustWorkloadIdentityManager":{"name":"cluster"},"namespace":"","name":"cluster","reconcileID":"3b382ff5-425c-4694-95ef-e48a62ea00db"}

Bootstrap ensures ZTWIM first, then children in order: SpiffeCSIDriver → SpireServer → SpireAgent → SpireOIDCDiscoveryProvider.

AC4: Drift reconciliation — PASS

Command:

oc delete spireagent.operator.openshift.io cluster
sleep 10
oc get spireagent.operator.openshift.io cluster -o jsonpath='{.metadata.name}'
echo "managed-by=$(oc get spireagent.operator.openshift.io cluster -o jsonpath='{.metadata.labels.app\.kubernetes\.io/managed-by}')"
oc -n kagenti-system logs deployment/kagenti-controller-manager --since=15s | grep "Created SPIRE"

Output:

spireagent.operator.openshift.io "cluster" deleted
cluster
managed-by=kagenti-operator
{"level":"info","ts":"2026-06-23T11:50:37Z","msg":"Created SPIRE operand CR","controller":"spire-operand","controllerGroup":"operator.openshift.io","controllerKind":"ZeroTrustWorkloadIdentityManager","ZeroTrustWorkloadIdentityManager":{"name":"cluster"},"namespace":"","name":"cluster","reconcileID":"8f9feffc-cf99-4e44-a913-c0e03caffbdc","kind":"SpireAgent","name":"cluster"}

SpireAgent recreated within 10 seconds with correct managed-by=kagenti-operator label.

AC5: Zero trust — SPIFFE volumes mounted — PASS

Command:

oc -n test-zero-trust get pod -l app.kubernetes.io/name=weather-service \
  -o jsonpath='{.items[0].spec.volumes[*].name}' | tr ' ' '\n' | grep -E "spire|spiffe"

Output:

spire-agent-socket
spiffe-helper-config

Command (verify CSI driver is the source):

oc -n test-zero-trust get pod -l app.kubernetes.io/name=weather-service \
  -o jsonpath='{.items[0].spec.volumes[?(@.name=="spire-agent-socket")].csi.driver}'

Output:

csi.spiffe.io

Command (verify SPIRE infrastructure pods running):

oc -n zero-trust-workload-identity-manager get pods

Output:

NAME                                                              READY   STATUS    RESTARTS   AGE
spire-agent-24k85                                                 1/1     Running   0          15m
spire-agent-dkt27                                                 1/1     Running   0          15m
spire-agent-rw2sf                                                 1/1     Running   0          15m
spire-server-0                                                    2/2     Running   0          18m
spire-spiffe-csi-driver-j8vqc                                     2/2     Running   0          18m
spire-spiffe-csi-driver-p2kq2                                     2/2     Running   0          18m
spire-spiffe-csi-driver-thrzr                                     2/2     Running   0          18m
spire-spiffe-oidc-discovery-provider-cd4f6c6ff-ls6r7              1/1     Running   0          18m
zero-trust-workload-identity-manager-controller-manager-75n6vtq   1/1     Running   0          20h

All SPIRE components reconciled by ZTWIM operator from CRs our controller created.
3 agents + 3 CSI drivers = 3 worker nodes (DaemonSets).

AC6: AgentCard signature verification — PASS

Setup: Built agentcard-signer image (cmd/agentcard-signer/Dockerfile) for
linux/amd64, pushed to OCP internal registry. Deployed signed-agent stack with
SPIRE CSI volume, ServiceAccount, Role/RoleBinding, unsigned card ConfigMap.

Command (verify signer fetched SVID and signed card):

oc -n test-zero-trust logs deployment/signed-agent -c sign-agentcard

Output:

{"level":"info","msg":"starting agentcard signer","signed_path":"/app/.well-known/agent-card.json","socket":"unix:///run/spire/agent-sockets/spire-agent.sock","timeout":"30s","ts":"2026-06-23T10:48:18Z","unsigned_path":"/etc/agentcard/agent.json"}
{"level":"info","msg":"fetched SVID","spiffe_id":"spiffe://localtest.me/ns/test-zero-trust/sa/signed-agent-sa","ts":"2026-06-23T10:48:19Z"}
{"level":"info","msg":"signed card written successfully","output_path":"/app/.well-known/agent-card.json","spiffe_id":"spiffe://localtest.me/ns/test-zero-trust/sa/signed-agent-sa","ts":"2026-06-23T10:48:19Z"}
{"configMap":"signed-agent-card-signed","level":"info","msg":"signed card written to ConfigMap","namespace":"test-zero-trust","ts":"2026-06-23T10:48:19Z"}

Command (verify AgentCard auto-created and synced):

oc -n test-zero-trust get agentcard signed-agent-deployment-card

Output:

NAME                           PROTOCOL   KIND         TARGET         AGENT          VERIFIED   SIGSTORE   BOUND   SYNCED   LASTSYNC   AGE
signed-agent-deployment-card   a2a        Deployment   signed-agent   Signed Agent                         false   True     62m        67m

Command (decode JWS protected header — verify x5c chain and algorithm):

oc -n test-zero-trust get agentcard signed-agent-deployment-card \
  -o jsonpath='{.status.card.signatures[0].protected}' | \
  python3 -c "import sys,json,base64; p=sys.stdin.read(); p+='='*(4-len(p)%4); \
  d=json.loads(base64.urlsafe_b64decode(p)); print(json.dumps(d,indent=2))"

Output:

{
  "alg": "ES256",
  "typ": "JOSE",
  "kid": "2d644473ae350c16",
  "x5c": [
    "MIIDCDCCAfCgAwIBAgIQGHtPqrZcdMerCKwt+W19NzANBgkqhkiG9w0BAQsFADBjMQswCQYDVQQGEwJVUzELMAkGA1UEChMCUkgxFTATBgNVBAMTDGxvY2FsdGVzdC5tZTEwMC4GA1UEBRMnMTQyNDQ2MTUxMDE5NjcxODQ1MzE3Njg5Mzg0MTQ0MDU3MDM1Mjg0MB4XDTI2MDYyMzEwNDc1NloXDTI2MDYyMzExNDgwNlowHTELMAkGA1UEBhMCVVMxDjAMBgNVBAoTBVNQSVJFMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEiqfrEEpkPcfUZawqEQ80SCg00aRTaEFpExS1L8y1+JN08CVIfoqHy2TPRVcaAMxL4SogBzpv3rprJFqtbIwoh6OByDCBxTAOBgNVHQ8BAf8EBAMCA6gwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYEFLDP2T3i607VTY/DAjvBPCKFOSCtMB8GA1UdIwQYMBaAFFas9DHmieY4aBBjITaXheKgV2WjMEYGA1UdEQQ/MD2GO3NwaWZmZTovL2xvY2FsdGVzdC5tZS9ucy90ZXN0LXplcm8tdHJ1c3Qvc2Evc2lnbmVkLWFnZW50LXNhMA0GCSqGSIb3DQEBCwUAA4IBAQA45s3Ry4BpP4sTd3StWiv4pM/Y97Ge41caoJHHAZwZkm0et3oRlFWRwGh4HXaQp73+x1KsPqc3Sg4RJ5PdXuMKC20HM7V6AZbUlvpsaHaj2Imm4H6aIdlziaJqHBrUlwHJvmYG7XEo6i9nw0BX/5lUBv9vnoTD9EewcBGBrTA50NgOyMH843z9tU4d4RsZvY7LUpTf/B9ZG8OfHZ7MRJsIYznBk1NRFhHn3u+R7HU/woJ5VV8XfFDFOAUhWJIarpax+XYCxxaRsYu/d9efxdAFzl5hqmnMi6/olvX+lXeoUUwQrFbT8rURd5Sz25dGc2uEqRsVp0fukMozqmU7rCqV"
  ]
}

Command (verify signature count):

oc -n test-zero-trust get agentcard signed-agent-deployment-card \
  -o jsonpath='{.status.card.signatures}' | \
  python3 -c "import sys,json; print(f'signatures: {len(json.load(sys.stdin))}')"

Output:

signatures: 1

Full chain verified: SPIRE operand CRs (our controller) → ZTWIM operator →
SPIRE running → CSI driver mounts socket → agentcard-signer fetches SVID
(spiffe://localtest.me/ns/test-zero-trust/sa/signed-agent-sa) → signs card
with ES256/x5c → operator fetches signed card → Synced=True.

AC7: Full stack — weather agent tells weather in Dublin — PASS

Setup: weather_service (A2A agent) + weather_tool (MCP server) deployed in
test-zero-trust namespace. OpenAI gpt-4o-mini as LLM backend. LLM credentials
via llm-credentials Secret.

Command:

oc -n test-zero-trust port-forward deploy/weather-service 9998:8000 &
curl -s -m 120 -X POST http://localhost:9998/ \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"message/send","params":{"message":{"role":"user","messageId":"dublin-7","parts":[{"kind":"text","text":"What is the weather in Dublin?"}]}},"id":"8"}'

Output:

{"id":"8","jsonrpc":"2.0","result":{"artifacts":[{"artifactId":"abbd39b2-c374-49c4-9578-30fc3147c355","parts":[{"kind":"text","text":"The current weather in Dublin is as follows:\n- **Temperature:** 77.1°F\n- **Windspeed:** 8.3 mph\n- **Wind Direction:** 99° (east)\n- **Daytime:** Yes\n\nIf you need more specific details or forecasts, feel free to ask!"}]}],"contextId":"96f5f508-2030-4471-a006-e388d95f067f","history":[{"contextId":"96f5f508-2030-4471-a006-e388d95f067f","kind":"message","messageId":"dublin-7","parts":[{"kind":"text","text":"What is the weather in Dublin?"}],"role":"user","taskId":"a1a9e40c-b51b-43cd-9dc9-c368123d3af9"},{"contextId":"96f5f508-2030-4471-a006-e388d95f067f","kind":"message","messageId":"427d1599-1c0d-47be-b700-40977c816b4c","parts":[{"kind":"text","text":"🚶‍♂assistant: {'messages': [HumanMessage(content='What is the weather in Dublin?', additional_kwargs={}, response_metadata={}, id='6190bded-83a3-4f48-96f4-e0447e70978e'), AIMessage(content='', additional_kwargs={'refusal': None}, response_metadata={'token_usage': {'comp..."}],"role":"agent","taskId":"a1a9e40c-b51b-43cd-9dc9-c368123d3af9"},{"contextId":"96f5f508-2030-4471-a006-e388d95f067f","kind":"message","messageId":"86a288ec-8d14-41c0-bcc6-3277bff0d332","parts":[{"kind":"text","text":"🚶‍♂tools: {'messages': [ToolMessage(content=[{'type': 'text', 'text': '{\"time\": \"2026-06-23T11:45\", \"interval\": 900, \"temperature\": 77.1, \"windspeed\": 8.3, \"winddirection\": 99, \"is_day\": 1, \"weathercode\": 3}', 'id': 'lc_28e5fa64-1490-4e44-9c92-70cbce9beb31'}], name=..."}],"role":"agent","taskId":"a1a9e40c-b51b-43cd-9dc9-c368123d3af9"}],"id":"a1a9e40c-b51b-43cd-9dc9-c368123d3af9","kind":"task","status":{"state":"completed","timestamp":"2026-06-23T11:48:24.314130+00:00"}}}

Full A2A trace visible: User → AIMessage (tool call decision) → ToolMessage
(MCP get_weather returned {"temperature": 77.1, "windspeed": 8.3}) →
AIMessage (formatted response). Task state: completed.

mTLS Identity Layer Verification (weather-service A2A agent)

SPIFFE CSI Volume Injection — VERIFIED

AuthBridge mutating webhook injects SPIRE volumes into weather-service pod.

Command (list all pod volumes):

oc -n test-zero-trust get pod -l app.kubernetes.io/name=weather-service \
  -o jsonpath='{.items[0].spec.volumes[*].name}' | tr ' ' '\n'

Output:

cache
kube-api-access-6ltdg
shared-data
spire-agent-socket
spiffe-helper-config
svid-output
envoy-config
authproxy-routes
authbridge-runtime-config
kagenti-keycloak-client-credentials-c2b91e18c24b321b

Command (verify CSI driver source):

oc -n test-zero-trust get pod -l app.kubernetes.io/name=weather-service \
  -o jsonpath='{.items[0].spec.volumes[?(@.name=="spire-agent-socket")]}' | python3 -m json.tool

Output:

{
    "csi": {
        "driver": "csi.spiffe.io",
        "readOnly": true
    },
    "name": "spire-agent-socket"
}

Command (verify CSI driver name):

oc -n test-zero-trust get pod -l app.kubernetes.io/name=weather-service \
  -o jsonpath='{.items[0].spec.volumes[?(@.name=="spire-agent-socket")].csi.driver}'

Output:

csi.spiffe.io

CSI driver csi.spiffe.io matches our SpiffeCSIDriver CR (pluginName: csi.spiffe.io).

AuthBridge Sidecar Containers — VERIFIED

Command:

oc -n test-zero-trust get pod -l app.kubernetes.io/name=weather-service \
  -o jsonpath='{range .items[0].spec.containers[*]}{.name}{"\n"}{end}'

Output:

agent
authbridge-proxy

Two containers: agent (application) and authbridge-proxy (mTLS sidecar injected
by AuthBridge mutating webhook).

AuthBridge Volume Mounts — VERIFIED

Command:

oc -n test-zero-trust get pod -l app.kubernetes.io/name=weather-service \
  -o jsonpath='{.items[0].spec.containers[?(@.name=="authbridge-proxy")].volumeMounts}' | python3 -m json.tool

Output:

[
    {
        "mountPath": "/shared",
        "name": "shared-data"
    },
    {
        "mountPath": "/etc/authbridge",
        "name": "authbridge-runtime-config",
        "readOnly": true
    },
    {
        "mountPath": "/etc/authproxy",
        "name": "authproxy-routes",
        "readOnly": true
    },
    {
        "mountPath": "/opt",
        "name": "svid-output"
    },
    {
        "mountPath": "/etc/spiffe-helper",
        "name": "spiffe-helper-config",
        "readOnly": true
    },
    {
        "mountPath": "/spiffe-workload-api",
        "name": "spire-agent-socket",
        "readOnly": true
    },
    {
        "mountPath": "/shared/client-secret.txt",
        "name": "kagenti-keycloak-client-credentials-c2b91e18c24b321b",
        "readOnly": true,
        "subPath": "client-secret.txt"
    },
    {
        "mountPath": "/shared/client-id.txt",
        "name": "kagenti-keycloak-client-credentials-c2b91e18c24b321b",
        "readOnly": true,
        "subPath": "client-id.txt"
    },
    {
        "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount",
        "name": "kube-api-access-6ltdg",
        "readOnly": true
    }
]

Key mTLS mounts on authbridge-proxy:

• /spiffe-workload-api — SPIRE Workload API socket (CSI csi.spiffe.io)
• /etc/spiffe-helper — spiffe-helper rotation config
• /opt — SVID output (svid.pem, svid_key.pem, svid_bundle.pem)
• /shared/client-secret.txt, /shared/client-id.txt — Keycloak credentials for token exchange

spiffe-helper Configuration — VERIFIED

Command:

oc -n test-zero-trust get configmap spiffe-helper-config -o jsonpath='{.data.helper\.conf}'

Output:

agent_address = "/spiffe-workload-api/spire-agent.sock"
cmd = ""
cert_dir = "/certs"
svid_file_name = "svid.pem"
svid_key_file_name = "svid_key.pem"
svid_bundle_file_name = "svid_bundle.pem"

spiffe-helper connects to SPIRE Workload API socket and rotates X.509 SVIDs
(svid.pem, svid_key.pem, svid_bundle.pem) into the svid-output emptyDir volume.

AuthBridge mTLS Configuration — VERIFIED

Command:

oc -n test-zero-trust get configmap authbridge-config-weather-service -o jsonpath='{.data.config\.yaml}'

Output:

listener:
  forward_proxy_addr: :8081
  reverse_proxy_addr: :8000
  reverse_proxy_backend: http://127.0.0.1:8001
mode: proxy-sidecar
mtls:
  mode: permissive
pipeline:
  inbound:
    plugins:
    - config:
        issuer: https://keycloak-keycloak.apps.rosa.ian-cluster.g532.p3.openshiftapps.com/realms/kagenti
        keycloak_realm: kagenti
        keycloak_url: http://keycloak-service.keycloak.svc:8080
      name: jwt-validation
  outbound:
    plugins:
    - config:
        identity:
          type: client-secret
        keycloak_realm: kagenti
        keycloak_url: http://keycloak-service.keycloak.svc:8080
      name: token-exchange
plugins:
  inbound:
    jwt-validation:
      issuer: https://keycloak-keycloak.apps.rosa.ian-cluster.g532.p3.openshiftapps.com/realms/kagenti
      keycloak_realm: kagenti
      keycloak_url: http://keycloak-service.keycloak.svc:8080
  outbound:
    token-exchange:
      grant_type: client_credentials
      jwt_audience: https://keycloak-keycloak.apps.rosa.ian-cluster.g532.p3.openshiftapps.com/realms/kagenti
      keycloak_realm: kagenti
      keycloak_url: http://keycloak-service.keycloak.svc:8080

AuthBridge runs as proxy-sidecar with mtls.mode: permissive. Inbound: JWT
validation against Keycloak. Outbound: token exchange via client_credentials.

SPIRE Infrastructure Pods — VERIFIED

Command:

oc -n zero-trust-workload-identity-manager get pods

Output:

NAME                                                              READY   STATUS    RESTARTS   AGE
spire-agent-6gljb                                                 1/1     Running   0          9m12s
spire-agent-kfgt5                                                 1/1     Running   0          52m
spire-agent-tjfq2                                                 1/1     Running   0          52m
spire-server-0                                                    2/2     Running   0          19m
spire-spiffe-csi-driver-rkgph                                     2/2     Running   0          53m
spire-spiffe-csi-driver-w6rbb                                     2/2     Running   0          9m12s
spire-spiffe-csi-driver-zwp5q                                     2/2     Running   0          53m
spire-spiffe-oidc-discovery-provider-cd4f6c6ff-f88r5              1/1     Running   0          19m
zero-trust-workload-identity-manager-controller-manager-75n6vtq   1/1     Running   0          24h

All SPIRE components reconciled by ZTWIM operator from CRs our controller created.
3 agents + 3 CSI drivers = 3 worker nodes (DaemonSets). OIDC discovery provider running.

AuthBridge Startup with mTLS — VERIFIED

Command (authbridge-proxy logs after fix):

oc -n test-zero-trust logs deploy/weather-service -c authbridge-proxy --tail=15

Output:

[entrypoint] Starting authbridge-proxy...
time=2026-06-23T15:54:01.144Z level=WARN msg="token-exchange: keycloak_url is deprecated; use provider_url + provider=keycloak instead"
time=2026-06-23T15:54:01.144Z level=WARN msg="token-exchange: keycloak_realm is deprecated; use provider_realm + provider=keycloak instead"
time=2026-06-23T15:54:01.144Z level=WARN msg="token-exchange: no_token_policy defaulted to \"deny\"; prior defaults were mode-specific (envoy-sidecar: client-credentials, waypoint: allow, proxy-sidecar: deny). Set no_token_policy explicitly (allow | deny | client-credentials) to silence this warning and pin the behavior."
time=2026-06-23T15:54:01.146Z level=INFO msg="reloader: watching for config changes" path=/etc/authbridge/config.yaml dir=/etc/authbridge drainWindow=30s
time=2026-06-23T15:54:01.146Z level=INFO msg="session tracking enabled" ttl=30m0s maxEvents=500 maxSessions=100
time=2026-06-23T15:54:01.146Z level=INFO msg="mTLS enabled" mode=permissive
time=2026-06-23T15:54:01.146Z level=INFO msg="HTTP server listening" name=reverse-proxy addr=:8000 mtls=true
time=2026-06-23T15:54:01.146Z level=INFO msg="HTTP server listening" name=forward-proxy addr=:8081
time=2026-06-23T15:54:01.146Z level=INFO msg="transparent proxy listening" addr=:8082
time=2026-06-23T15:54:01.146Z level=INFO msg="authbridge-proxy starting" mode=proxy-sidecar logLevel=INFO
time=2026-06-23T15:54:01.146Z level=INFO msg="stat server listening" addr=:9093
time=2026-06-23T15:54:01.146Z level=WARN msg="session API listening — UNAUTHENTICATED; contains raw user content; never expose via ingress" addr=:9094
time=2026-06-23T15:54:01.146Z level=INFO msg="health server listening" addr=:9091

Command (pod status — 2/2 Running):

oc -n test-zero-trust get pod -l app.kubernetes.io/name=weather-service

Output:

NAME                               READY   STATUS    RESTARTS        AGE
weather-service-7494d468c7-xjjf7   2/2     Running   4 (2m16s ago)   2m57s

SVID Verification — VERIFIED

spiffe-helper fetched X.509 SVID from SPIRE and wrote it to /opt.

Command (list SVID files):

oc -n test-zero-trust exec deploy/weather-service -c authbridge-proxy -- ls -la /opt/

Output:

total 12
drwxrwsrwx    2 root     10011100        65 Jun 23 15:54 .
dr-xr-xr-x    1 root     root            69 Jun 23 15:54 ..
-rw-r--r--    1 1337     10011100      1111 Jun 23 15:54 svid.pem
-rw-r--r--    1 1337     10011100      2692 Jun 23 15:54 svid_bundle.pem
-rw-------    1 1337     10011100       241 Jun 23 15:54 svid_key.pem

Command (decode SVID certificate):

oc -n test-zero-trust exec deploy/weather-service -c authbridge-proxy -- cat /opt/svid.pem | \
  openssl x509 -text -noout

Output:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            e4:99:84:74:c8:8b:8b:6b:02:cf:a9:85:cf:2c:e7:22
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, O=RH, CN=localtest.me, serialNumber=86676098741177021800508327787105255062
        Validity
            Not Before: Jun 24 15:45:01 2026 GMT
            Not After : Jun 24 16:45:11 2026 GMT
        Subject: C=US, O=SPIRE
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:13:2c:2b:1f:b0:56:89:a5:10:21:8c:a8:cf:0e:
                    e1:5d:7b:db:c2:81:38:06:59:99:e2:3a:c3:8d:42:
                    28:b7:0c:7f:b7:50:38:f5:dc:ae:84:38:63:c5:5e:
                    d5:21:37:15:d0:3d:0c:b8:f5:fc:1b:97:31:33:8e:
                    60:fc:dc:69:da
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment, Key Agreement
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier:
                CF:A0:22:28:28:93:8F:40:A9:A2:D0:BA:A6:23:43:BA:19:7E:4B:B3
            X509v3 Authority Key Identifier:
                E1:E7:EE:A8:71:4E:9A:3A:96:C9:0D:3C:A4:08:AA:3F:37:87:93:CB
            X509v3 Subject Alternative Name:
                URI:spiffe://localtest.me/ns/test-zero-trust/sa/weather-service
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:
        28:89:a9:4a:82:34:ea:18:f6:43:cb:05:28:6c:36:28:00:0d:
        48:2b:97:3a:8c:c4:34:9a:e0:bb:8e:68:1f:55:81:19:5a:33:
        99:01:ef:60:4c:ca:85:78:5f:f9:34:bf:4e:44:a6:ff:ed:f6:
        83:e5:6f:0b:f1:4a:ab:0c:08:2d:b2:ea:c2:14:91:7a:42:e1:
        27:18:3f:23:f7:f8:2a:bd:7c:e7:bd:d0:86:09:50:18:67:f8:
        03:6d:f4:96:e4:12:c9:8b:2b:02:b6:aa:d4:b8:04:2d:b7:e0:
        09:37:52:de:a4:bd:12:84:72:e1:75:97:d0:41:f3:77:7c:69:
        5b:48:94:c5:c6:9c:ee:0d:e6:d9:70:fc:d3:1c:bf:1a:80:b3:
        53:dc:ef:da:04:a1:d6:65:68:96:5b:2a:07:80:bc:34:78:23:
        f3:20:f2:9c:16:54:db:e8:3b:6f:e0:a6:ec:c1:dd:36:82:88:
        f8:84:15:14:f2:cd:b1:13:a1:cd:9a:40:33:19:68:b0:5f:35:
        cc:fa:a6:a6:91:d0:45:b4:ad:cd:95:c1:ce:17:7a:8d:13:07:
        b9:95:9e:31:bb:e7:f3:ed:8f:58:f9:85:86:67:12:95:33:42:
        fc:ae:89:6c:6b:03:3b:fb:6b:cc:dc:9f:31:94:65:d8:33:36:
        05:f5:65:8c

SVID issued by SPIRE CA (CN=localtest.me) with SPIFFE ID
spiffe://localtest.me/ns/test-zero-trust/sa/weather-service.
1-hour validity, auto-rotated by spiffe-helper. ECDSA P-256 key.

Full mTLS chain verified: Our SPIRE operand CRs → ZTWIM operator runs SPIRE →
CSI driver mounts Workload API socket → spiffe-helper fetches X.509 SVID →
writes svid.pem/svid_key.pem/svid_bundle.pem → AuthBridge uses them for
mTLS (permissive mode) + Keycloak token exchange.

Summary

AC	Description	Result
AC1	No-op when CRDs absent	PASS
AC2	All 5 CRs created, managed-by=kagenti-operator	PASS
AC3	ZTWIM created before children	PASS
AC4	Drift correction (delete + spec change)	PASS
AC5	SPIFFE volumes mounted in agent pods	PASS
AC6	AgentCard signature (JWS/x5c via SPIRE SVID)	PASS
AC7	Weather agent tells weather in Dublin	PASS

All 7 acceptance criteria verified on ROSA OCP 4.19.18.

---

Test plan

• Unit tests pass (37/37)
• E2E tests compile (go build ./test/e2e/)
• go vet ./... clean
• AC1-AC7 verified on ROSA OCP 4.19.18
• CI pipeline (automated on PR)

🤖 Generated with Claude Code

[rh-dnagornuks]

Solid changes. Seems to work based on my tests using the image built from this branch. One blocker (in an inline comment) and a nit.

Nit (for the overall PR): commit history is a bit messy and rebase squash could have been used in fix up commits such as fix(e2e): correct deployment name and reduce CRD retry delays

[pdettori]

Well-structured controller implementation with good test coverage (37 unit + 8 e2e), proper CRD gating, and thoughtful trustDomain immutability handling. The mergeNestedMap approach for drift correction without reconcile loops is solid design. CI fully green (all 16 checks pass).

A couple of non-blocking suggestions for robustness and configurability below.

Areas reviewed: Go controller, Go unit tests, E2E tests, Helm RBAC, Discovery, Docs
Commits: 13 commits, all signed-off with DCO
CI status: All 16 checks passing

[rh-dnagornuks]

Feedback addressed. LGTM

[rh-dnagornuks]

Could the drift test for ZTWIM delete the controller itself rather than a field within it? The main 3 fields (trustDomain, clusterName, bundleConfigMap) are immutable by default on OpenShift. Function ztwimSpec() only sets these 3 immutable fields, so it might make more sense to test deletion of ZTWIM.

Source: https://docs.redhat.com/en/documentation/openshift_container_platform/4.21/html/security_and_compliance/zero-trust-workload-identity-manager#zero-trust-manager-ztwim-cr_zero-trust-manager-configuration