Skip to content

feat: Add control plane telemetry - scheduler eligible workers histogram - #682

Merged
Jeff Luo (JeffLuoo) merged 6 commits into
agent-substrate:mainfrom
Angelawork:feature/eligible-workers-telemetry
Aug 7, 2026
Merged

feat: Add control plane telemetry - scheduler eligible workers histogram#682
Jeff Luo (JeffLuoo) merged 6 commits into
agent-substrate:mainfrom
Angelawork:feature/eligible-workers-telemetry

Conversation

@Angelawork

@Angelawork Angela (Angelawork) commented Aug 1, 2026

Copy link
Copy Markdown
Collaborator

Part of #564 (Part 3)

  • Tests pass
  • Appropriate changes to documentation are included in the PR

Description

This PR implements Part 3 of #564 by adding telemetry histogram instrumentation for ate.scheduler.eligible_workers in ateapi. It measures unassigned free worker capacity remaining after all scheduling constraint filters are applied, sampled at every scheduling decision.

Key Changes:

  • Updated Scheduler.Schedule() to record eligible candidate workers per pool (recordEligibleWorkers).
    Defined SchedulingConstraintKey (ate.scheduling.constraint) and constraint classification values (none, required_nodes, selector) in internal/ateattr/ateattr.go.
  • Added unit tests in scheduling_test.go covering candidate counts, namespaced attributes, zero-capacity fleet states, empty fleets, sandbox class mismatches, draining workers, and constraint classifications.

Testing

  • go test -buildvcs=false ./cmd/ateapi/internal/controlapi/...
  • go test -buildvcs=false ./cmd/atenet/internal/router/...
  • make test

E2E Test

  • ./hack/create-kind-cluster.sh
  • ./hack/install-ate-kind.sh --deploy-ate-system --deploy-demo-counter
  • ./hack/run-e2e.sh ./internal/e2e/suites/metrics/...

@Angelawork
Angela (Angelawork) force-pushed the feature/eligible-workers-telemetry branch from 5b29a8f to 1ff8de8 Compare August 4, 2026 15:52
@Angelawork
Angela (Angelawork) marked this pull request as draft August 4, 2026 17:51
@JeffLuoo

Copy link
Copy Markdown
Collaborator

Please rebase and fix e2e tests

@Angelawork
Angela (Angelawork) force-pushed the feature/eligible-workers-telemetry branch 2 times, most recently from 2cc4a44 to 59434d7 Compare August 4, 2026 18:23
@Angelawork
Angela (Angelawork) marked this pull request as ready for review August 4, 2026 18:23
@Angelawork
Angela (Angelawork) force-pushed the feature/eligible-workers-telemetry branch from ad9eac4 to 74dce74 Compare August 4, 2026 18:27
@Angelawork
Angela (Angelawork) force-pushed the feature/eligible-workers-telemetry branch from 74dce74 to 8f82992 Compare August 4, 2026 18:29
Comment thread internal/e2e/suites/metrics/metrics_test.go Outdated
Comment thread cmd/ateapi/internal/scheduling/scheduling.go Outdated
Comment thread cmd/ateapi/internal/scheduling/scheduling.go Outdated
Comment thread cmd/ateapi/internal/scheduling/scheduling.go Outdated
Comment thread cmd/ateapi/internal/scheduling/scheduling.go Outdated
Comment thread cmd/ateapi/internal/scheduling/scheduling.go Outdated
@Angelawork
Angela (Angelawork) force-pushed the feature/eligible-workers-telemetry branch 2 times, most recently from abf9345 to d3eb608 Compare August 5, 2026 21:21
Comment thread cmd/ateapi/internal/scheduling/metrics.go Outdated
@Angelawork
Angela (Angelawork) force-pushed the feature/eligible-workers-telemetry branch from d3eb608 to 85d4b22 Compare August 6, 2026 13:46
@Angelawork
Angela (Angelawork) force-pushed the feature/eligible-workers-telemetry branch 3 times, most recently from 22155a2 to 1a7faeb Compare August 6, 2026 17:00
@Angelawork
Angela (Angelawork) marked this pull request as draft August 6, 2026 18:13
@Angelawork
Angela (Angelawork) marked this pull request as ready for review August 6, 2026 18:13
@JeffLuoo

Copy link
Copy Markdown
Collaborator

Please do not use Fixes #564 (Part 3) in the description as it will close the bug. Use Part of #564 (Part 3)

@Angelawork
Angela (Angelawork) force-pushed the feature/eligible-workers-telemetry branch from 1a7faeb to b21e461 Compare August 6, 2026 18:15
@JeffLuoo
Jeff Luo (JeffLuoo) self-requested a review August 6, 2026 18:51
@JeffLuoo

Jeff Luo (JeffLuoo) commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

For some reasons the CI is not running.. I close and reopen the PR to see if it will trigger the presubmit again.

@JeffLuoo
Jeff Luo (JeffLuoo) merged commit 058b104 into agent-substrate:main Aug 7, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants