Skip to content

feat: Add control plane telemetry - workerpool desired and ready workers gauges - #826

Merged
Jeff Luo (JeffLuoo) merged 4 commits into
agent-substrate:mainfrom
Angelawork:feature/workerpool-telemetry
Aug 11, 2026
Merged

feat: Add control plane telemetry - workerpool desired and ready workers gauges#826
Jeff Luo (JeffLuoo) merged 4 commits into
agent-substrate:mainfrom
Angelawork:feature/workerpool-telemetry

Conversation

@Angelawork

Copy link
Copy Markdown
Collaborator

Part of #564 (Part 4, Fixes #564)

  • Tests pass
  • Appropriate changes to documentation are included in the PR

Description

This PR implements Part 4 of #564 by adding OpenTelemetry gauge instrumentation for ate.workerpool.desired_workers and ate.workerpool.ready_workers in atecontroller.

Key Changes:

  • Added ReadyReplicas int32 to WorkerPoolStatus with +kubebuilder:printcolumn:name="Ready" annotation in pkg/api/v1alpha1/workerpool_types.go and regenerated the CRD manifest.
  • Updated WorkerPoolReconciler.syncStatus to synchronize dep.Status.ReadyReplicas into wp.Status.ReadyReplicas.
  • Implemented InitMetrics(meter) registering ate.workerpool.desired_workers and ate.workerpool.ready_workers as OpenTelemetry Observable UpDownCounters ({worker}) with an asynchronous observer callback that samples controller-runtime's local informer cache (r.Client.List).
  • Labeled metric datapoints using centralized attributes ateattr.WorkerPoolNamespaceKey and ateattr.WorkerPoolNameKey.

Testing

  • go test -buildvcs=false ./cmd/ateapi/internal/controlapi/...
  • go test -buildvcs=false ./cmd/atenet/internal/router/...
  • make test

E2E Test

  • ./hack/create-kind-cluster.sh
  • ./hack/install-ate-kind.sh --deploy-ate-system --deploy-demo-counter
  • ./hack/run-e2e.sh ./internal/e2e/suites/metrics/...

@Angelawork Angela (Angelawork) changed the title Feature/workerpool telemetry feat: Add control plane telemetry - workerpool desired and ready workers gauges Aug 10, 2026
@Angelawork
Angela (Angelawork) force-pushed the feature/workerpool-telemetry branch from ecffc4b to b64c940 Compare August 10, 2026 17:59
Comment thread cmd/atecontroller/internal/controllers/workerpool_controller_test.go Outdated
Comment thread cmd/atecontroller/internal/controllers/workerpool_controller.go Outdated
Comment thread cmd/atecontroller/internal/controllers/workerpool_controller_test.go Outdated
Comment thread cmd/atecontroller/internal/controllers/workerpool_controller.go
Comment thread cmd/atecontroller/internal/controllers/workerpool_controller.go Outdated
Comment thread docs/observability.md Outdated
@Angelawork
Angela (Angelawork) force-pushed the feature/workerpool-telemetry branch from a00dee9 to 901b616 Compare August 11, 2026 19:48
@JeffLuoo
Jeff Luo (JeffLuoo) merged commit d4f2cf1 into agent-substrate:main Aug 11, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature request: More system metrics for debuggability (Activation SLI, crash accounting, and capacity signals)

2 participants