test: phase 1 coverage improvements (StickyRouting + Metrics.Exporters + E2E host coverage)#86
Merged
Merged
Conversation
New test project tests/ExperimentFramework.StickyRouting.Tests targeting net8.0;net9.0;net10.0 with coverlet.collector instrumentation. Covers: - StickyTrialRouter: determinism, single-key short-circuit, empty list exception, key-order independence, distribution uniformity (2-way and 3-way), unicode / empty / very-long identities - StickyRoutingProvider: ModeIdentifier, SelectTrialKeyAsync happy path and fallback cases (no provider, empty identity, throwing provider), GetDefaultSelectorName, exception swallowing - DI registration: AddExperimentStickyRouting registers ISelectionModeProviderFactory with the correct mode identifier Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…tests) New test project tests/ExperimentFramework.Metrics.Exporters.Tests targeting net8.0;net9.0;net10.0 with coverlet.collector instrumentation. Covers PrometheusExperimentMetrics: - Counter accumulation, TYPE declaration line, label rendering, independent tracking per tag-set - Gauge last-value-wins semantics, TYPE declaration - Histogram sum accumulation, _sum/_count output lines, TYPE declaration - Summary _sum/_count output, TYPE declaration - Label escaping: newline -> \n, backslash -> \, double-quote -> \" - Clear removes all metric types; empty state returns empty/whitespace Covers OpenTelemetryExperimentMetrics: - Construction with default and custom parameters (including null version) - IncrementCounter / RecordHistogram / SetGauge / RecordSummary smoke tests - Dispose idempotency; multiple instances with different meter names coexist Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Changes to .github/workflows/ci.yml e2e-tests job: - Install dotnet-coverage tool before launching the host - Wrap DashboardHost launch with dotnet-coverage collect -f cobertura so that server-side code is instrumented during the Playwright run - Send SIGTERM to the wrapper after the test run to flush the cobertura report before uploading - Upload e2e-dashboard-coverage.xml to Codecov with flag "e2e" - Upload the coverage file as a build artifact for inspection - Removed continue-on-error: true so E2E failures now block PRs - Increased job timeout-minutes from 25 to 30 to account for the dotnet-coverage startup overhead Changes to .codecov.yml: - Added "e2e" flag covering src/ and the DashboardHost sample project - Both flags use carryforward: true Added tests/ExperimentFramework.E2E.Tests/README.md: - Documents how to run the E2E suite locally - Documents how to reproduce the coverage collection locally using dotnet-coverage collect in one terminal and dotnet test in another Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
github-actions
Bot
added
documentation
Contributor
Dependency ReviewThe following issues were found:
Snapshot WarningsEnsure that dependencies are being submitted on PR branches and consider enabling retry-on-snapshot-warnings. See the documentation for more information and troubleshooting advice. License Issuestests/ExperimentFramework.StickyRouting.Tests/ExperimentFramework.StickyRouting.Tests.csproj
OpenSSF Scorecard
Scanned Files
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #86 +/- ##
==========================================
+ Coverage 83.39% 89.59% +6.19%
==========================================
Files 193 193
Lines 7035 7035
Branches 1213 996 -217
==========================================
+ Hits 5867 6303 +436
+ Misses 859 732 -127
+ Partials 309 0 -309
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
Same InteractiveServer reconnect-race pattern as the previous GovernanceAudit fix (0b5412c) — scenarios were interacting with conditionally-rendered elements before the Blazor circuit had finished hydrating. RolloutStepDefinitions.SelectFirstExperimentAsync: Previous implementation counted options and indexed into them without waiting for option[1] to attach. During the Blazor reconnect phase the select has only the placeholder option (value=""); the helper would then fire SelectOptionAsync with the empty value, leaving _selectedExperiment empty and the rollout-manager panel (with its stage-name input) never rendered. Apply the Nth(1).WaitForAsync(Attached, 15s) pattern so we only select after a real experiment option exists. Fixes "Start a rollout" and "Add rollout stages". RolloutPage.AddStageAsync: Belt-and-suspenders: wait for the stage-name input to become visible before FillAsync, since it is only rendered once _selectedExperimentInfo != null on the server side. GovernanceVersionsPage.AssertVersionHistoryVisibleAsync: Previous 30 s default visible-wait was not enough in CI — after SelectOptionAsync the page transitions through _loadingVersions=true before .version-item renders, and InteractiveServer @onchange round-trip is noticeably slower than SSR. Split the wait into Attached (15 s budget) then Visible, so the Blazor circuit has a full budget to complete LoadVersions before visibility is checked. Fixes "Versions page shows version history". Verified locally against a seeded DashboardHost (--seed=docs): Rollout + GovernanceVersions feature files — 8/8 passing, including the three previously-failing scenarios. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previous fix's 15s visibility wait was not enough under CI load — the SignalR circuit handshake can race SelectOptionAsync, so the server never processes the change event and the downstream UI never renders. Apply a three-part hardening to the helpers that select the first experiment in the dashboard dropdowns (Rollout, GovernanceVersions, GovernanceAudit): 1. Wait for window.Blazor._internal.navigationManager to exist before SelectOptionAsync — this only becomes truthy once the circuit is initialized and ready to receive events. 2. After SelectOptionAsync, verify the server actually processed the change by waiting for a visible side-effect (loading indicator, rollout-manager panel, version list, info-message, etc). If the side-effect does not appear within 5s, re-fire SelectOptionAsync up to 3 times total. This tolerates a single lost change event during circuit handshake without failing the scenario. 3. Widen the per-interaction visibility waits in AddStageAsync, FilterByTypeAsync, SearchAsync, and AssertVersionHistoryVisibleAsync from 15s to 30s so that the server-side re-render after the change event has a full budget under CI load. Verified locally against a seeded DashboardHost (--seed=docs): Rollout + GovernanceVersions + GovernanceAudit feature files — 11/11 passing, including the three previously-failing scenarios (Start a rollout, Add rollout stages, Versions page shows version history) and the newly-flaky Search audit entries. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
Code Coverage |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Deliverable 1 — E2E host coverage plumbing: The
e2e-testsCI job now launches DashboardHost underdotnet-coverage collect -f coberturaso that server-side code is instrumented during the Playwright run. After the suite completes the wrapper is sent SIGTERM to flushe2e-dashboard-coverage.xml, which is uploaded to Codecov under a newe2eflag.continue-on-error: truehas been removed so E2E failures now block PRs. The.codecov.ymlgains thee2eflag definition. AREADME.mdintests/ExperimentFramework.E2E.Tests/documents how to reproduce coverage collection locally.Deliverable 2 — StickyRouting test project: New
tests/ExperimentFramework.StickyRouting.Tests(net8.0;net9.0;net10.0, coverlet.collector). 23 tests across two classes:StickyTrialRouterTests(12 tests): determinism, single-key short-circuit, empty-list exception, key-order independence, 2-way and 3-way distribution uniformity, unicode/empty/very-long identities, different users get different buckets.StickyRoutingProviderTests(11 tests): ModeIdentifier constant, SelectTrialKeyAsync happy path + fallback cases (no provider registered, empty identity, throwing provider → exception swallowed), GetDefaultSelectorName, DI registration viaAddExperimentStickyRouting.Deliverable 3 — Metrics.Exporters test project: New
tests/ExperimentFramework.Metrics.Exporters.Tests(net8.0;net9.0;net10.0, coverlet.collector). 30 tests:PrometheusExperimentMetricsTests(20 tests): counter accumulation, TYPE declarations, label rendering, independent tracking per tag-set, gauge last-value-wins, histogram sum/count, summary sum/count, label escaping (newline, backslash, double-quote), Clear, empty-state output.OpenTelemetryExperimentMetricsTests(10 tests): construction (default, custom meter name, null version), IncrementCounter/RecordHistogram/SetGauge/RecordSummary smoke tests, Dispose idempotency, multiple instances coexist.Both new projects are added to
ExperimentFramework.slnxin the/tests/folder. All tests pass locally (53 total new tests).Test plan
e2e-dashboard-coverage.xmlto Codecov withe2eflagunittestsande2eflagsdotnet test tests/ExperimentFramework.StickyRouting.Tests— 23 passeddotnet test tests/ExperimentFramework.Metrics.Exporters.Tests— 30 passed🤖 Generated with Claude Code