Conversation
Specifically:
* Median is reported, in addition to the existing mean+stdev, which is
significantly more resistant to skew by outliers.
* --metric {wall,cpu} (default wall): Enables profiling using CPU time
rather than wall-clock time. CPU profiling has roughly half the coefficient
of variation as wall-clock profiling equal run count.
* --workers1: Forces MYPY_NUM_WORKERS=1 (rather than the default 4) to
cut CPU scheduling variance. Strongly recommended when using --metric cpu.
* --warmup-runs N (default 1): Configurable number of leading cold runs to discard.
Previously was always 1. Higher run counts decrease outliers that skew
the reported mean.
* A new "Paired deltas vs <first commit>" section is added to the report,
showing per-round paired differencing against the first commit
to cancel round-level common-mode noise, reducing variance.
Reported as median +/-95% CI.
Also:
* --cache-binaries (default false): Caches each commit's compiled clone
to avoid ~5min recompile whenever comparing the same commit multiple times.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…_parse_as_type_expression() Specifically: - If you set MYPY_TYPEFORM_PROFILE_FULL_PARSE environment variable, mypy will output a .tsv to that filepath which characterizes the kinds of Expressions that try_parse_as_type_expression() in semanal.py was forced to do a full parse of, which was not rejected early. - A misc/analyze_typeform_full_parse_profile.py script is added which takes those .tsvs and prints an expression-time summary (by total time) plus top-N descriptors per FAIL class. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Contributor
|
According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅ |
This was referenced Aug 11, 2026
Member
|
@JukkaL I will leave this one up to you, as IIRC you wrote most of the |
ilevkivskyi
pushed a commit
that referenced
this pull request
Sep 18, 2026
References #21262. Replaces #21585 and obsoletes #21596. ## Summary Enabling `TypeForm` by default (referenced #21262) made `SemanticAnalyzer.try_parse_as_type_expression` run eagerly on every expression in certain syntactic positions. The cost is concentrated in the expensive full-parse block (`expr_to_analyzed_type` + `isolated_error_analysis`), which **fails ~87% of the time** - pure wasted work. This branch adds early-reject filters that eliminate **74% of full parses (2570 → 666)** on mypy's self-check, recovering **~46% of the regression**: **+1.57% → +0.84%** CPU time. **No new regexes** - per review feedback on replaced #21585. Every filter here is plain string/`isinstance` work, and the two shape tests that were regexes are now helper functions. <details> <summary>Why not do a type-context check?</summary> Review of replaced #21585 suggested skipping the call to `SemanticAnalyzer.try_parse_as_type_expression` entirely when the type context cannot be a `TypeForm`. That optimization already exists, but in the other *type checker* pass at `ExpressionChecker.try_parse_as_type_expression`. The same skip cannot be used in the *semantic analyzer* pass's function because the type context is not yet known. So cheaply filtering the inputs to `SA.try_parse_as_type_expression` is the only remaining (obvious) lever to reduce its runtime contribution. </details> ## Optimization Results CPU time, single worker, paired per-round deltas, n=300: ``` python misc/perf_compare.py --warmup-runs 3 --num-runs 300 -j 3 \ --metric cpu --workers1 \ <TypeForm-disabled-commit> 5bb72b7 <tip-of-this-pr-branch> ``` | Commit | Mean | Median | Δ vs baseline (paired median ±95% CI) | |---|---:|---:|---:| | baseline, `<TypeForm-disabled-commit>` | 2.679 s | 2.676 s | - | | master, `5bb72b788` | 2.720 s | 2.720 s | **+41.9 ms ±2.9 (+1.57%)** | | all filters, `<tip-of-this-pr-branch>` | 2.703 s | 2.699 s | **+22.4 ms ±2.9 (+0.84%)** | The feature branch recovers **19.5 ms of the 41.9 ms regression (~46% by paired median)** - leaving **+22.4 ms (~54%)**. Derivation: - +41.9 ms - +22.4 ms == 19.5 ms recovered - 19.5 ms / +41.9 ms == 46.5% (~46%) recovered - +22.4 ms / +41.9 ms == 53.5% (~54%) left A separate 2-way run of master vs `<tip-of-this-pr-branch>` measured **−21.1 ms ±3.2**, consistent with the 19.5 ms recovered that was derived above. <details> <summary>Notes on the measurement</summary> The baseline (`<TypeForm-disabled-commit>`) is current master (`5bb72b788`) with referenced #21262 (SHA: `dd851f559`) reverted, so all three arms share today's code and differ only in `TypeForm`. Measuring against the original pre-#21262 master commit instead of today's master would have conflated optimizations made during the following ~80 commits, including notably `c0cced35c`, which optimised `SA.try_parse_as_type_expression` specifically. Thus runtime regression measured here (+41.9 ms) is smaller than the +50.2 ms reported in replaced #21585: part of the original regression has already been absorbed upstream. </details> Full parses per self-check, identical corpus: | | master | branch | | |---|---:|---:|---| | full parses | 2570 | 666 | **−74.1%** | | - succeeded (produced a type) | 345 | 345 | **±0** ✅ | | - failed (wasted work) | 2225 | 321 | **−85.6%** | The successful-parse count is unchanged at every commit on the branch, as expected: No expression that previously parsed as a type stopped doing so. ## Overview of changes - Most changes are made to the `SemanticAnalyzer.try_parse_as_type_expression` function. All other changes occur within the same file. - 5 commits, each individually profiled: - 4 commits add a filter - 1 commit reorders existing filters - Any added filter can be dropped (if needed) without disturbing the other filters ### The filter commits Bare-identifier strings (`"Foo"`): 1. Reject a `Var` whose declared type is a concrete `Instance` - a value, not a type. 2. Reject `FuncDef` / `OverloadedFuncDef` / `MypyFile` - functions and modules are never types. 3. Reorder the mutually-exclusive checks by measured rejection frequency. Other strings: 4. Reject strings containing a character or boundary pattern that never appears in a type expression - leading/trailing `.`, or one of ``!:/<>@%$^?;&~`\``, or a `-` that is not a `Literal[...]` unary minus. Catches `"utf-8"`, `".pyi"`, `"error:"`, `"pkg/mod.py"`. 5. Dotted-name strings (`"builtins.tuple"`, `"typing.Mapping"`): look up the leftmost component and reject when it does not resolve, or resolves to a placeholder or a value `Var`. Filters 4 and 5 replace `_NONTYPE_PATTERN_RE` and `_DOTTED_IDENTIFIER_RE` from replaced #21585 with the helpers `has_nontype_char()` and `dotted_identifier_leftmost()`. Each was verified to agree with the regex it replaces on all 1171 distinct strings the full-parse profiler observes during a self-check. <details> <summary>Two specific hazards, and how they are handled</summary> `var_is_typing_special_form` was extended to recognize `typing.Self` / `typing_extensions.Self`, so filter 1 does not reject a stringified `'Self'` annotation (otherwise `testSelfRecognizedInOtherSyntacticLocations` regresses). In filter 4, `-` is treated as a unary minus wherever the preceding non-space character is `[` or `,`, so `"Literal[-1, -2]"` and `"Literal[1, -2]"` are still recognized. (`_NONTYPE_PATTERN_RE` in replaced #21585 used `(?<!\[)-`, which rejected those.) On the strings observed during a self-check the two rules reject identical sets, so the (improved) soundness costs nothing. </details> ## Notes - I don't think it's worth trying to recover the remaining +22.4 ms: - The 321 surviving failed parses are spread across four categories with no common cheap/obvious shape left - I experimented with adding some fancy `OpExpr` filters that actually gave a net *slowdown* of 1.9ms. - The profiling instrumentation and the `misc/perf_compare.py` improvements used to produce these numbers are in a separate PR: #21832. Happy to fold them in here instead if that is easier to review. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Contains 2 commits:
See the individual commit messages for more details.
I'm planning to use these instrumentation enhancements 1-3 upcoming PRs aimed at reducing the slowdown that TypeForm recognition introduces when it is enabled. Related: