[Draft] Add generated C# full-generation benchmarks

[live1206]

Summary

Benchmark/profiling PR for generated C# full generation and post-processing performance. This PR carries the benchmark harness plus the production-shaped hybrid provider reference-map logic from #10976 so we can measure it against the Roslyn reference-map path.

What Changed

• Added full-generation BenchmarkDotNet coverage for Sample-TypeSpec.
• Added optional per-step profiling around provider/code writing, PostProcessAsync(), GetGeneratedFilesAsync(), and file writes.
• Added benchmark modes comparing Roslyn reference-map construction with provider reference-map analysis.
• Updated the shadow provider reference-map analyzer to avoid stale previous-generated-file roots and instead model the missing reachability from current generation data:
  • current union/variant roots participate in internalization, matching Roslyn _typesToKeep behavior;
  • discriminator derived-model edges are added for remove reachability after current root discovery.

Latest Benchmark Data

Latest data from benchmark PR #10885 after porting the current #10976 hybrid analyzer changes (including generated body invocation edges and base-preserved reachability).

Benchmark output root: /tmp/typespec-final-hybrid-bench-20260624-0943.

Full-generation BenchmarkDotNet, averaged across 3 local runs:

Mode	Avg Mean	Avg Allocated
Roslyn reference maps	895.8 ms	67.78 MB
Provider reference map	859.9 ms	57.75 MB

Approximate full-generation improvement:

Time:       ~4.0% faster
Allocation: ~14.8% less

Focused profile data, using 72 Roslyn-mode and 72 provider-mode invocations from the same runs (median):

Path	Median Time	Median Allocated
Roslyn reference-map construction	357.2 ms	23.70 MB
Provider map analysis + candidate consumption	263.2 ms	17.83 MB
Provider candidate consumption only	0.79 ms	0.06 MB

Approximate focused reference-map improvement:

Time:       ~26.3% faster
Allocation: ~24.8% less

Profile notes:

• Data comes from POSTPROCESSING_BENCHMARK_PROFILE_STEPS=true on the full-generation benchmark so both paths run against real TypeProvider output.
• Roslyn rows are PostProcessor.Internalize.BuildPublicReferenceMapAsync and PostProcessor.Remove.BuildAllReferenceMapAsync.
• Provider rows are Generation.ProviderReferenceMapShadowAnalysis, PostProcessor.Internalize.UseShadowCandidates, PostProcessor.Internalize.UseShadowPublicizeCandidates, PostProcessor.Remove.UseShadowCandidates, and PostProcessor.Remove.BuildShadowReferencedSet.
• Runtime: .NET 10.0.9, Ubuntu 26.04, AMD EPYC 7763.

How To Run

cd packages/http-client-csharp/generator/Microsoft.TypeSpec.Generator/perf
POSTPROCESSING_BENCHMARK_PROFILE_STEPS=true \
POSTPROCESSING_BENCHMARK_PROFILE_DIR=/tmp/typespec-post-processing-profiles \
  dotnet run -c Release --framework net10.0 --filter '*FullGenerationBenchmark*'

Latest Data

Latest local run: 3 BenchmarkDotNet runs after removing stale generated-file scanning and replacing it with current-generation roots/edges, commit 32840a556. Artifacts: /tmp/typespec-current-generated-roots-bench/run-{1,2,3}.

Full-generation BenchmarkDotNet

Mode	Run means	Avg mean	Avg allocated
Roslyn reference maps	833.5, 977.7, 923.0 ms	911.4 ms	67.75 MB
Provider reference map	836.4, 792.4, 782.7 ms	803.8 ms	52.30 MB

Approximate full-generation improvement:

Time:       ~11.8% faster
Allocation: ~22.8% less

Focused reference-map profile

Profile data below uses all 72 Roslyn-mode and 72 provider-mode invocations across the same three runs, with trimmed averages.

Path	Time	Allocated
Roslyn combined maps	437.7 ms	23.74 MB
Provider analysis, computes both	243.3 ms	12.33 MB
Provider candidate consumption	0.88 ms	negligible

Focused reference-map replacement improvement:

Time:       ~44.4% faster
Allocation: ~48.1% less

Validation

• npm run build:generator
• Provider-mode generation with zero checked-in diffs for:
  • http/type
  • http/documentation
  • http/parameters/collection-format
  • http/payload/multipart
• npm run format
• npm run lint

Conclusion

The previous final-parity benchmark regressed because it scanned stale src/Generated files and over-rooted old public generated declarations. The latest version removes that stale scan and fixes the actual current-generation gaps, restoring both correctness on targeted scenarios and an end-to-end full-generation performance win.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

Latest Safe Provider-Map Optimization (2026-06-24)

Pushed safe optimization commits:

• Production PR: 88bea5758
• Benchmark PR: 7b3c0f1e5

Changes:

• Removed the global Roslyn SymbolFinder.FindReferencesAsync helper lookup from generated-body dependency discovery.
• Kept correctness by scanning current generated body syntax for both type references and invocation containing types, even when provider body dependencies are available.
• Reused the flattened generated-provider list during graph construction/base-preserved dependency traversal.
• Skipped XML doc parsing unless the doc contains a type cref marker.

Safe two-pass focused profile aggregate (/tmp/typespec-provider-map-safe-final-20260624-1320, /tmp/typespec-provider-map-safe-final2-20260624-1320) compared with the previous hotspot baseline (/tmp/typespec-provider-map-hotspots-20260624-1320):

Path	Before Median	After Median	Delta
Provider map analysis + candidate consumption	231.5 ms, 18.03 MB	188.8 ms, 7.89 MB	~18.4% faster, ~56.2% less allocation
Provider analysis only	231.3 ms, 17.98 MB	188.0 ms, 7.83 MB	~18.7% faster, ~56.4% less allocation

Validation after the safe correction:

• npm run build:generator
• pwsh ./eng/scripts/Generate.ps1 -filter Sample-TypeSpec with no checked-in generated diffs

Latest 3-Run Benchmark Data (2026-06-24 14:03 UTC)

Benchmark branch: mtg-manual-name-reduction-experiment at 7b3c0f1e5. Artifacts: /tmp/typespec-latest-safe-bench-20260624-1403/run-{1,2,3}.

Full-generation BenchmarkDotNet

Mode	Run means	Avg mean	Avg allocated
Roslyn reference maps	1295.0, 1029.8, 1633.4 ms	1319.4 ms	67.52 MB
Provider reference map	1008.0, 785.1, 795.0 ms	862.7 ms	47.95 MB

Approximate full-generation improvement:

Time:       ~34.6% faster
Allocation: ~29.0% less

Focused reference-map profile

Profile data uses 71 Roslyn-mode and 71 provider-mode invocations from the same three runs (median):

Path	Median Time	Median Allocated
Roslyn reference-map construction	429.7 ms	23.65 MB
Provider map analysis + candidate consumption	198.7 ms	7.87 MB
Provider analysis only	197.0 ms	7.81 MB
Provider candidate consumption only	0.98 ms	0.06 MB

Approximate focused reference-map replacement improvement:

Time:       ~53.8% faster
Allocation: ~66.7% less

Latest Helper-Fix Benchmark Data (2026-06-24 14:36 UTC)

Benchmark branch: mtg-manual-name-reduction-experiment at b14b1f282. Artifacts: /tmp/typespec-helper-fix-bench-20260624-1436/run-{1,2,3}. This includes the CI helper-root fix that explicitly preserves generated ClientUriBuilder, ClientPipelineExtensions, and CancellationTokenExtensions without restoring the broad Roslyn SymbolFinder helper lookup.

Full-generation BenchmarkDotNet

Mode	Run means	Avg mean	Avg allocated
Roslyn reference maps	1025.7, 881.6, 1007.9 ms	971.7 ms	67.64 MB
Provider reference map	809.8, 693.8, 842.7 ms	782.1 ms	48.12 MB

Approximate full-generation improvement:

Time:       ~19.5% faster
Allocation: ~28.9% less

Focused reference-map profile

Profile data uses 72 Roslyn-mode and 72 provider-mode invocations from the same three runs (median):

Path	Median Time	Median Allocated
Roslyn reference-map construction	349.0 ms	23.57 MB
Provider map analysis + candidate consumption	172.6 ms	7.88 MB
Provider analysis only	171.3 ms	7.82 MB
Provider candidate consumption only	0.80 ms	0.06 MB

Approximate focused reference-map replacement improvement:

Time:       ~50.6% faster
Allocation: ~66.6% less

[pkg-pr-new]

Open in StackBlitz

npm i https://pkg.pr.new/@typespec/http-client-csharp@10885

commit: e4019d3

[github-actions]

No changes needing a change description found.

[live1206]

Network post-processing performance experiment results

Documenting the current experiment state before switching back to the Network MPG migration work.

Network generator-only timing

All runs used the saved Network generator inputs from sdk/network/Azure.ResourceManager.Network/{Configuration.json,tspCodeModel.json} and produced 3,542 .cs files.

Variant	Wall time	Roslyn/post-processing time	Result
main (ccc3f6004)	712.482s (~11m52s)	11m48.286s	Fastest baseline
PR #10846 / scoped simplifier (53124ea7f)	851.905s (~14m12s)	14m07.350s	~19.6% slower than main
This PR / manual no-Roslyn reducer (e4019d359)	1152.665s (~19m13s)	19m08.268s	~61.8% slower than main

Artifacts: /tmp/network-three-branch-20260604052728/.

Bounded parallelism follow-up

I also tried an env-var-gated bounded document-processing experiment (TYPESPEC_GENERATOR_POSTPROCESSING_PARALLELISM=16). The synthetic BenchmarkDotNet corpus looked promising, but real Network generation did not beat main:

Variant	Wall time	Roslyn/post-processing time	Files
main baseline	712s	11m48s	3542
manual branch + parallelism=16	722s	11m57s	3542
main + parallelism=16	733s	12m08s	3542

Artifacts: /tmp/network-parallelism16-20260604062933/ and /tmp/network-main-parallelism16-20260604064307/.

Conclusion

This PR is useful as an experiment, but the manual reducer approach is not currently a Network performance improvement. The bounded-parallelism experiment was removed from the working tree after it failed to beat main on Network.

Recommended next direction is deeper phase/document-size instrumentation around Roslyn workspace post-processing, then target reducing the number or size of documents that go through semantic simplification rather than replacing Roslyn simplification wholesale.

[live1206]

Latest shadow-map replacement results from local BenchmarkDotNet runs.

Correctness Shadow Comparison

Replacement mode still compares the provider/custom-code hybrid candidates against the Roslyn candidates before using them.

Internalize:
Roslyn candidates: 32
Provider candidates: 32
Missing: 0
Extra: 0

Remove:
Roslyn candidates: 37
Provider candidates: 37
Missing: 0
Extra: 0

This is for Sample-TypeSpec.

Full-Generation Benchmark

Benchmark command shape:

DOTNET_ROOT="$HOME/.dotnet" PATH="$HOME/.dotnet:$PATH" \
dotnet run --project packages/http-client-csharp/generator/Microsoft.TypeSpec.Generator/perf/Microsoft.TypeSpec.Generator.Tests.Perf.csproj \
  -c Release --framework net10.0 --filter "*FullGenerationBenchmark*"

Replacement mode additionally used:

TYPESPEC_PROVIDER_REFERENCE_MAP_SHADOW=true
TYPESPEC_PROVIDER_REFERENCE_MAP_USE_SHADOW=true
TYPESPEC_PROVIDER_REFERENCE_MAP_SHADOW_DIR="/tmp/typespec-provider-reference-map-shadow"

Mode	Mean	Error	StdDev	Allocated
Baseline Roslyn reference-map path	1.042 s	0.363 s	0.418 s	63.98 MB
Hybrid provider/custom map replacement	641.3 ms	98.37 ms	113.3 ms	44.41 MB

Approximate improvement:

Time:       ~38.5% faster
Allocation: ~30.6% less

Interpretation

The hybrid provider/custom-code map is now exact for Sample-TypeSpec in shadow comparison and shows a meaningful local benchmark improvement when used to replace Roslyn reference-map construction.

Next step: create a clean PR from latest main with this replacement path and run a proper regeneration across SDK services to validate output correctness broadly.

[azure-sdk-automation]

You can try these changes here

🛝 Playground	🌐 Website	🛝 VSCode Extension