Skip to content

fix(keycloak): stabilize audience mapper registration for existing scopes - #360

Merged
pdettori merged 3 commits into
rossoctl:mainfrom
pdettori:fix/358-verify-audience-mapper
May 14, 2026
Merged

fix(keycloak): stabilize audience mapper registration for existing scopes#360
pdettori merged 3 commits into
rossoctl:mainfrom
pdettori:fix/358-verify-audience-mapper

Conversation

@pdettori

Copy link
Copy Markdown
Member

Summary

  • Match Python AuthBridge sidecar pattern: for existing scopes, never POST a new mapper — only verify via GET and update via PUT
  • Prevent ghost-409 cascades: repeated mapper POSTs on every reconcile corrupted Keycloak's internal name index, requiring Keycloak restarts to recover
  • Add createAudienceMapperBestEffort: on 409, verifies mapper state via GET before returning success; gracefully handles Keycloak ghost-conflicts

Root Cause

The operator called ensureAudienceMapper (a POST) on every reconcile for existing scopes. The Python AuthBridge sidecar only POSTs a mapper when creating a brand-new scope. Repeated POSTs trigger 409 conflicts, and if a mapper ever enters a corrupted state, the 409 cascades into ghost-conflict entries in Keycloak's DB that persist across pod restarts.

Test plan

  • Unit tests pass (go test ./internal/keycloak/ -run TestEnsureAudienceScope)
  • Built and deployed to Kind cluster (localhost/kagenti-operator:fix-358)
  • Verified stable re-registration: delete/recreate AgentRuntime → no 409 errors
  • Verified concurrent reconciles: rapid back-to-back deployment restarts → all succeed cleanly
  • Verified mapper integrity: correct oidc-audience-mapper with SPIFFE audience after all operations
  • Verified both agents (weather-service, a2a-contact-extractor) register cleanly in parallel

Fixes #358

Assisted-By: Claude Code

…opes

Match the Python AuthBridge sidecar pattern: for existing scopes, never
POST a new mapper—only verify via GET and update via PUT. This prevents
Keycloak ghost-409 cascades that corrupted mapper state on every reconcile.

Key changes:
- getOrCreateAudienceClientScope: skip ensureAudienceMapper for existing
  scopes; only POST mappers when creating a brand-new scope
- verifyAudienceMapper: use createAudienceMapperBestEffort (treats 409 as
  success after verifying actual state) instead of ensureAudienceMapper
- createAudienceMapperBestEffort: on 409, verify mapper exists via GET
  before returning success; handles ghost-conflict gracefully
- updateAudienceMapperIfNeeded: use best-effort POST for delete+recreate
  path to avoid recursive ensureAudienceMapper calls

Fixes rossoctl#358

Signed-off-by: Paolo Dettori <paolo@us.ibm.com>
Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Paolo Dettori <dettori@us.ibm.com>
Signed-off-by: Paolo Dettori <paolo@us.ibm.com>
Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Paolo Dettori <dettori@us.ibm.com>

@cwiklik cwiklik left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clean fix that correctly aligns the operator with the Python AuthBridge sidecar pattern: only POST mappers for brand-new scopes, verify via GET+PUT for existing ones. The createAudienceMapperBestEffort + ghost-409 handling is sound. Good test coverage for the corrupted-mapper and repair scenarios.

All 15 CI checks pass. 2 commits, both DCO-signed.

Comment thread kagenti-operator/internal/keycloak/audience.go Outdated
Comment thread kagenti-operator/internal/keycloak/audience.go Outdated
- Return error for non-409/non-2xx responses so the controller re-queues
  immediately rather than waiting for the next periodic reconcile
- Fix comment: verifyAudienceMapper runs in the same reconcile, not next

Signed-off-by: Paolo Dettori <paolo@us.ibm.com>
Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Paolo Dettori <dettori@us.ibm.com>
@pdettori
pdettori merged commit 9a70be2 into rossoctl:main May 14, 2026
15 checks passed
@pdettori
pdettori deleted the fix/358-verify-audience-mapper branch May 14, 2026 13:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

fix(keycloak): verifyAudienceMapper defense-in-depth never runs due to early return in getOrCreateAudienceClientScope

3 participants