Conversation
Contributor
|
No changes needing a change description found. |
Contributor
|
You can try these changes here
|
Contributor
⚡ Benchmark Results
Full details – comparing
|
| Metric | Baseline | Current | Change |
|---|---|---|---|
| total | 🔴 1.03s | 🔴 1.06s | +3.8% |
| loader | 🟡 310.9ms | 🟡 326.2ms | +4.9% |
| resolver | 🟢 40.5ms | 🟢 41.2ms | +1.9% |
| checker | 🟡 369.3ms | 🟡 383.6ms | +3.9% |
| validation | 🟢 85.6ms | 🟢 87.6ms | +2.3% |
| ↳ validation/@azure-tools/typespec-azure-core | 🟡 12.8ms | 🟡 13.4ms | +4.8% |
| ↳ validation/@typespec/http | 🟡 15.7ms | 🟡 16.3ms | +3.7% |
| ↳ validation/@typespec/rest | 🟢 1.6ms | 🟢 1.7ms | +3.5% |
| ↳ validation/@typespec/versioning | 🔴 51.5ms | 🔴 52.1ms | +1.3% |
| ↳ validation/compiler | 🟢 3.9ms | 🟢 4.0ms | +2.7% |
| linter | 🟡 218.5ms | 🟡 225.4ms | +3.2% |
| ↳ linter/@azure-tools/typespec-azure-core/auth-required | 🟢 0.1ms | 🟢 0.1ms | +3.7% |
| ↳ linter/@azure-tools/typespec-azure-core/bad-record-type | 🟢 0.6ms | 🟢 0.6ms | -1.2% |
| ↳ linter/@azure-tools/typespec-azure-core/byos | 🟢 8.5ms | 🟢 8.7ms | +3.1% |
| ↳ linter/@azure-tools/typespec-azure-core/casing-style | 🟢 1.5ms | 🟢 1.5ms | +1.6% |
| ↳ linter/@azure-tools/typespec-azure-core/composition-over-inheritance | 🟢 0.2ms | 🟢 0.2ms | +0.1% |
| ↳ linter/@azure-tools/typespec-azure-core/documentation-required | 🟢 2.0ms | 🟢 2.0ms | +1.6% |
| ↳ linter/@azure-tools/typespec-azure-core/friendly-name | 🟢 1.4ms | 🟢 1.4ms | +2.4% |
| ↳ linter/@azure-tools/typespec-azure-core/key-visibility-required | 🟢 0.4ms | 🟢 0.4ms | +2.1% |
| ↳ linter/@azure-tools/typespec-azure-core/known-encoding | 🟢 0.5ms | 🟢 0.5ms | +4.6% |
| ↳ linter/@azure-tools/typespec-azure-core/long-running-polling-operation-required | 🟢 0.8ms | 🟢 0.8ms | +0.4% |
| ↳ linter/@azure-tools/typespec-azure-core/no-case-mismatch | 🟢 0.7ms | 🟢 0.7ms | +1.1% |
| ↳ linter/@azure-tools/typespec-azure-core/no-closed-literal-union | 🟢 1.0ms | 🟢 1.0ms | -0.7% |
| ↳ linter/@azure-tools/typespec-azure-core/no-enum | 🟢 0.2ms | 🟢 0.2ms | +0.9% |
| ↳ linter/@azure-tools/typespec-azure-core/no-error-status-codes | 🟢 0.3ms | 🟢 0.3ms | +1.3% |
| ↳ linter/@azure-tools/typespec-azure-core/no-explicit-routes-resource-ops | 🟢 0.2ms | 🟢 0.2ms | +2.5% |
| ↳ linter/@azure-tools/typespec-azure-core/no-format | 🟢 0.8ms | 🟢 0.8ms | +0.5% |
| ↳ linter/@azure-tools/typespec-azure-core/no-generic-numeric | 🟢 0.8ms | 🟢 0.9ms | +2.9% |
| ↳ linter/@azure-tools/typespec-azure-core/no-header-explode | 🔴 27.2ms | 🔴 28.5ms | +4.9% |
| ↳ linter/@azure-tools/typespec-azure-core/no-legacy-usage | 🟢 2.1ms | 🟢 2.1ms | +1.3% |
| ↳ linter/@azure-tools/typespec-azure-core/no-multiple-discriminator | 🟢 0.2ms | 🟢 0.3ms | +6.0% |
| ↳ linter/@azure-tools/typespec-azure-core/no-nullable | 🟢 0.4ms | 🟢 0.4ms | +4.0% |
| ↳ linter/@azure-tools/typespec-azure-core/no-offsetdatetime | 🟢 2.0ms | 🟢 2.0ms | +3.0% |
| ↳ linter/@azure-tools/typespec-azure-core/no-openapi | 🟢 2.3ms | 🟢 2.3ms | -0.2% |
| ↳ linter/@azure-tools/typespec-azure-core/no-private-usage | 🟢 3.2ms | 🟢 3.3ms | +1.7% |
| ↳ linter/@azure-tools/typespec-azure-core/no-query-explode | 🔴 28.3ms | 🔴 29.5ms | +4.3% |
| ↳ linter/@azure-tools/typespec-azure-core/no-response-body | 🔴 33.9ms | 🔴 35.1ms | +3.7% |
| ↳ linter/@azure-tools/typespec-azure-core/no-rest-library-interfaces | 🟢 0.1ms | 🟢 0.1ms | +9.7% |
| ↳ linter/@azure-tools/typespec-azure-core/no-route-parameter-name-mismatch | 🟢 7.8ms | 🟢 8.2ms | +4.7% |
| ↳ linter/@azure-tools/typespec-azure-core/no-rpc-path-params | 🟢 0.4ms | 🟢 0.4ms | +2.1% |
| ↳ linter/@azure-tools/typespec-azure-core/no-string-discriminator | 🟢 0.1ms | 🟢 0.2ms | +3.4% |
| ↳ linter/@azure-tools/typespec-azure-core/no-unknown | 🟢 0.4ms | 🟢 0.4ms | +2.3% |
| ↳ linter/@azure-tools/typespec-azure-core/no-unnamed-union | 🟢 0.8ms | 🟢 0.8ms | +0.4% |
| ↳ linter/@azure-tools/typespec-azure-core/operation-missing-api-version | 🟢 0.4ms | 🟢 0.4ms | -1.3% |
| ↳ linter/@azure-tools/typespec-azure-core/request-body-problem | 🟢 0.5ms | 🟢 0.5ms | +0.3% |
| ↳ linter/@azure-tools/typespec-azure-core/require-versioned | 🟢 0.1ms | 🟢 0.1ms | +3.8% |
| ↳ linter/@azure-tools/typespec-azure-core/response-schema-problem | 🔴 33.2ms | 🔴 34.6ms | +4.1% |
| ↳ linter/@azure-tools/typespec-azure-core/rpc-operation-request-body | 🟢 0.7ms | 🟢 0.7ms | +4.0% |
| ↳ linter/@azure-tools/typespec-azure-core/spread-discriminated-model | 🟢 0.5ms | 🟢 0.5ms | -0.2% |
| ↳ linter/@azure-tools/typespec-azure-core/use-standard-names | 🟢 7.7ms | 🟢 8.0ms | +3.9% |
| ↳ linter/@azure-tools/typespec-azure-core/use-standard-operations | 🟢 0.3ms | 🟢 0.3ms | -2.3% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-agent-base-type-child-resources | 🔴 21.7ms | 🔴 21.7ms | -0.2% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-agent-base-type-lifecycle-operations | 🟢 0.3ms | 🟢 0.3ms | -2.0% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-common-types-version | 🟡 12.3ms | 🟡 12.5ms | +1.3% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-custom-resource-no-key | 🟢 0.2ms | 🟢 0.2ms | +0.8% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-custom-resource-usage-discourage | 🟢 0.1ms | 🟢 0.1ms | -2.3% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-delete-operation-response-codes | 🟢 5.0ms | 🟢 4.5ms | -10.7% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-no-path-casing-conflicts | 🟡 17.2ms | 🟡 17.5ms | +1.4% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-no-record | 🟢 0.6ms | 🟢 0.7ms | +4.8% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-post-operation-response-codes | 🟢 1.4ms | 🟢 1.4ms | +4.2% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-put-operation-response-codes | 🟢 0.2ms | 🟢 0.2ms | -4.8% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-action-no-segment | 🟢 0.5ms | 🟢 0.5ms | +2.3% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-duplicate-property | 🟢 0.4ms | 🟢 0.4ms | +2.3% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-interface-requires-decorator | 🟢 0.1ms | 🟢 0.1ms | -0.9% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-invalid-action-verb | 🟢 0.2ms | 🟢 0.2ms | +5.7% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-invalid-envelope-property | 🟢 0.3ms | 🟢 0.3ms | +0.9% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-invalid-version-format | 🟢 0.2ms | 🟢 0.2ms | +0.6% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-key-invalid-chars | 🟢 0.5ms | 🟢 0.5ms | +3.6% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-name-pattern | 🟢 0.1ms | 🟢 0.1ms | -1.0% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-operation | 🟢 0.6ms | 🟢 0.6ms | +3.6% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-operation-response | 🟢 9.1ms | 🟢 9.5ms | +4.1% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-patch | 🟢 0.8ms | 🟢 0.9ms | +4.1% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-path-segment-invalid-chars | 🟢 0.4ms | 🟢 0.4ms | +3.4% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-provisioning-state | 🟢 0.4ms | 🟢 0.4ms | +2.4% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/beyond-nesting-levels | 🟢 0.2ms | 🟢 0.2ms | +2.3% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/empty-updateable-properties | 🟢 0.4ms | 🟢 0.4ms | -3.5% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/improper-subscription-list-operation | 🟢 0.1ms | 🟢 0.1ms | +7.0% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/lro-location-header | 🔴 20.0ms | 🔴 20.9ms | +4.5% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/missing-operations-endpoint | 🟢 0.1ms | 🟢 0.2ms | +6.1% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/missing-x-ms-identifiers | 🟢 1.0ms | 🟢 1.0ms | +1.1% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/no-empty-model | 🟢 0.3ms | 🟢 0.3ms | +3.5% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/no-override-props | 🟢 0.3ms | 🟢 0.3ms | +3.3% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/no-reserved-resource-property | 🟢 0.2ms | 🟢 0.2ms | +2.1% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/no-resource-delete-operation | 🟢 0.5ms | 🟢 0.5ms | +1.4% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/no-response-body | 🔴 30.4ms | 🔴 32.2ms | +6.2% 🔴 |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/patch-envelope | 🟢 0.4ms | 🟢 0.4ms | +4.1% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/resource-name | 🟢 0.4ms | 🟢 0.4ms | +0.1% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/secret-prop | 🟢 5.4ms | 🟢 5.5ms | +1.7% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/unsupported-type | 🟢 0.7ms | 🟢 0.7ms | -1.4% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/version-progression | 🟢 0.3ms | 🟢 0.3ms | +2.3% |
| ↳ linter/@azure-tools/typespec-client-generator-core/property-name-conflict | 🟢 2.3ms | 🟢 2.4ms | +1.8% |
| ↳ linter/@azure-tools/typespec-client-generator-core/require-client-suffix | 🟢 1.7ms | 🟢 1.7ms | +1.9% |
| emit | 🔴 6.13s | 🔴 6.28s | +2.4% |
| ↳ emit/@Azure-Tools | 🟢 0.0ms | 🟢 0.0ms | +0.0% |
| ↳ emit/@azure-tools/typespec-autorest | 🟢 127.0ms | 🟢 129.7ms | +2.2% |
| ↳ emit/@azure-tools/typespec-python | 🔴 2.28s | 🔴 2.34s | +2.6% |
| ↳ emit/@typespec | 🟢 0.0ms | 🟢 0.0ms | +0.0% |
| ↳ emit/@typespec/http-client-js | 🔴 578.5ms | 🔴 590.3ms | +2.0% |
| ↳ emit/@typespec/openapi3 | 🟢 109.5ms | 🟢 112.2ms | +2.5% |
| ↳ emit/@typespec/openapi3/compute | 🟢 95.1ms | 🟢 97.4ms | +2.4% |
| ↳ emit/@typespec/openapi3/write | 🟢 14.0ms | 🟢 14.5ms | +3.3% |
Averaged across 3 specs (azure-arm-resource-manager, azure-core-dataplane, azure-full).
Threshold: changes > ±5% are highlighted.
🟢 Fast · 🟡 Moderate (stages >200ms, rules >10ms) · 🔴 Slow (stages >400ms, rules >20ms)
Add support for benchmarking specs that live in another repo (e.g. azure-rest-api-specs). A spec source is either a local directory with main.tsp or an external-spec directory with spec.json; the runner discovers both via --specs-dir and produces one results file. External specs are sparse-checked-out into .external (git-ignored) and compiled against this workspace's packages, so results reflect local library changes; each external spec's tspconfig.yaml (copied into the checkout) controls the emitters/linter measured. store-results and upload-pr-comment gain --results-dir (and --title/--comment-file) so groups can be stored/commented separately; the data-branch push rebases and retries to tolerate concurrent writers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Each external spec is a directory with spec.json (source) and tspconfig.yaml (emit TCGC + ARM ruleset), pulled from azure-rest-api-specs at latest. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add a Main baseline / External specs switch to the benchmarks page. The External view reads external-results/history.json and shows a single chart switchable between by-emitter (one line per spec) and by-spec (one line per emitter) comparison modes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Factor the run and comment logic into reusable workflows (benchmark-run.yml, benchmark-comment-run.yml). benchmark.yml (main) and benchmark-external.yml instantiate the run template with their own parameters; benchmark-comment.yml and benchmark-comment-external.yml instantiate the comment template. Each spec group is an independent workflow, avoiding duplicated steps. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
tadelesh
changed the title
tadelesh
requested review from
bterlson,
markcowl and
timotheeguerin
as code owners
Member
Author
|
@timotheeguerin this is ready for review — implements the external-spec benchmarking we discussed in #4801. Summary of the approach:
Included external specs: |
timotheeguerin
approved these changes
Jul 13, 2026
timotheeguerin
left a comment
timotheeguerin
left a comment
Member
There was a problem hiding this comment.
Approving, didn't check too much the code, will check what the benchmark looks like after it runs and we'll tweak as needed
github-merge-queue
Bot
removed this pull request from the merge queue due to no response for status checks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements external-spec benchmarking, designed in #4801.
What this adds
Benchmark real specs from another repository (e.g. azure-rest-api-specs) as a separate, opt-in group alongside the built-in specs, with their own dashboard view — kept out of the main baseline.
Runner (generic)
A benchmark spec is anything the runner can compile: a local spec (a directory with
main.tsp) or an external spec (a directory withspec.json).--specs-dirselects which set to run; both are treated uniformly and produce one results file. There is no built-in/external split in the tool itself — the distinction is made by CI (which directory to run) and the dashboard (which dataset to show).External specs are sparse-checked-out into
packages/benchmark/.external/(git-ignored) and compiled against this workspace's packages, so results reflect local library changes (TCGC, client emitters, linters).reftracks latest (no pinning). Each external spec directory carries atspconfig.yaml(copied into the checkout) that selects the emitters and linter measured — e.g. TCGC + the ARM ruleset — exactly like a local spec.Storage
store-results --results-dir <dir>stores each group under its own directory onbenchmark-data(results/for the baseline,external-results/for external). The push rebases-and-retries so the two workflows writing the branch concurrently don't fail.Dashboard
The benchmarks page gains a Main baseline / External specs switch. The External view reads
external-results/history.jsonand shows a single chart switchable between two modes:PR comments
The external group posts its own sticky PR comment ("⚡ External Spec Benchmark Results"), separate from the main one, comparing against the
external-resultsrolling baseline.Workflows (reusable)
benchmark-run.ymlandbenchmark-comment-run.ymlare reusable templates holding the run/comment logic.benchmark.yml(main) andbenchmark-external.ymlinstantiate the run template with their own parameters;benchmark-comment.ymlandbenchmark-comment-external.ymlinstantiate the comment template. Adding a group is a small instance file, not duplicated logic.Included external specs
compute,web(AppService), andnetwork— large ARM specs from azure-rest-api-specs, measuring TCGC + the ARM ruleset.Notes
@azure-tools/typespec-benchmarkand the website package are private, so no changeset is required.external-resultshistory only exists after the external workflow has run onmain; until then the external dashboard/PR comment shows "no baseline".