Skip to content

Introduce CompileData parity markers and align with RPython - #52

Merged
youknowone merged 7 commits into
youknowone:mainfrom
lifthrasiir:compile-data-parity
May 15, 2026
Merged

Introduce CompileData parity markers and align with RPython#52
youknowone merged 7 commits into
youknowone:mainfrom
lifthrasiir:compile-data-parity

Conversation

@lifthrasiir

@lifthrasiir lifthrasiir commented May 15, 2026

Copy link
Copy Markdown
Contributor

Summary

This pull request introduces a set of new data structures to standardize and clarify the data passed into the JIT compilation pipeline, closely modeling RPython's CompileData and its subclasses. It refactors how trace and runtime information (such as snapshots, input arguments, and optimization options) are bundled and passed throughout the compilation and optimization process. The changes improve maintainability and parity with the original RPython design, and ensure that all relevant context is consistently available to downstream consumers.

Key changes include:

Introduction of CompileData Bundles:

  • Added new structs (CompileData, PreambleCompileData, SimpleCompileData, BridgeCompileData, and UnrolledLoopData) in compile.rs to encapsulate trace, runtime, and optimization state, mirroring RPython's object model for JIT compilation inputs. These bundles are now used to pass data to optimizers and other compilation stages, replacing ad-hoc argument lists.

Refactoring Compilation Call Sites:

  • Updated all major compilation entry points in pyjitpl/mod.rs to construct and use the new CompileData bundles, ensuring that trace operations, snapshots, input arguments, and optimization options are accessed via these unified interfaces. This includes retrace, simple compilation, and bridge compilation paths.
  • All optimizer invocations now receive call_pure_results and other relevant state from the appropriate CompileData bundle, ensuring consistency.

Runtime Boxes and Bridge Compilation:

  • Ensured that runtime_boxes (the original live argument boxes from the closing JUMP) are passed separately and consistently into optimizer and bridge compilation routines, matching RPython's model and fixing previous inconsistencies.
  • Clarified documentation and comments to explain the parity with RPython and the purpose of these changes.

Miscellaneous Improvements:

  • Ensured that all relevant state (such as call_pure_results, enable_opts, and snapshots) is cloned or moved from the context at the correct time to avoid accidental mutation or loss of data.

These changes bring the codebase closer to RPython's design, reduce code duplication, and make the flow of compilation data more explicit and maintainable.

Self-review

Prompt & Model

Model: gpt-5.5

Prompt:

Assess by static analysis whether our changes in git diff main are equivalent
to the corresponding RPython/PyPy source code. The RPython and PyPy sources
are available locally.

If anything was ported incorrectly, report every instance in detail. After
collecting all differences, organize the report into separate sections:

1. Cases where our patch regressed PyPy parity compared to main
2. Other mismatches introduced by our patch
3. Mismatches that already existed before this patch
4. Structural adaptations

Exceptions: some differences cannot be ported 1:1 because of Python 3.11 vs
3.14 differences, opcode mismatches caused by using a CPython-compatible
compiler, GIL/free-threading differences, and fundamental implementation-
language differences between RPython and Rust. Mark those separately under
“Structural adaptations.”

Answer

  1. Cases Where Our Patch Regressed PyPy Parity Compared To Main
  None found.

  The main behavioral change in majit/majit-metainterp/src/optimizeopt/optimizer.rs:3035 improves parity: runtime_boxes is now
  threaded separately into jump_to_existing_trace, matching RPython unroll.py:183-207, where runtime_boxes is distinct from the fresh
  trace iterator boxes.

  2. Other Mismatches Introduced By Our Patch
  None found.

  The new CompileData structs in majit/majit-metainterp/src/compile.rs:223 are constructor-payload mirrors only. They do not fully
  port CompileData.optimize_trace, but the file documents this as a Pyre adaptation, and the existing Rust optimizer dispatch remains
  flattened in pyjitpl/mod.rs.

  3. Mismatches That Already Existed Before This Patch
  Entry bridges still do not receive call_pure_results.

  RPython compile.py:1048-1060 passes metainterp.call_pure_results into both BridgeCompileData and SimpleCompileData; then
  unroll.py:193-194 passes it to propagate_all_forward. Current Rust captures call_pure_results in majit/majit-metainterp/src/
  pyjitpl/mod.rs:4733, and this patch now passes it into guard bridges via majit/majit-metainterp/src/pyjitpl/mod.rs:8333. But the
  interpreter entry bridge path still calls majit/majit-metainterp/src/pyjitpl/mod.rs:4842 without call_pure_results, and majit/
  majit-metainterp/src/pyjitpl/mod.rs:7963 never sets optimizer.call_pure_results.

  This mismatch was already present on upstream/main; the patch fixes the guard-bridge half but not entry bridges.

  4. Structural Adaptations
  CompileData is not ported as a polymorphic optimizer driver. RPython CompileData.optimize_trace builds the optimization chain,
  logs, dispatches to subclass optimize, and clears forwarded boxes. Pyre keeps that flow flattened in pyjitpl/mod.rs, with the new
  structs used as typed payload bundles.

  Rust uses HashMap<Vec<Value>, Value> for call_pure_results; this is acceptable parity because RPython stores it in an args-dict-
  like mapping, not on box forwarding state.

  Bridge trace preparation still uses a Rust-specific fresh OpRef namespace before optimization. That is the existing adaptation for
  RPython trace.get_iter() creating fresh InputArg / ResOperation object identities. The patch correctly takes runtime_boxes from the
  original pre-prepared JUMP args before that transformation.

Summary by CodeRabbit

  • Chores
    • Internal compilation infrastructure refactored to improve data handling and pipeline efficiency across preamble, simple, and bridge compilation stages.

Review Change Stack

Add Rust structs mirroring RPython compile.py's CompileData hierarchy: CompileData, PreambleCompileData, SimpleCompileData, BridgeCompileData, and UnrolledLoopData. The structs are intentionally small input bundles so the sensitive compile path gains named parity boundaries without a broad control-flow rewrite.

Thread the new bundles through the existing pyjitpl compile paths for preamble/unrolled loops, simple loops, bridge compilation, retrace, and finish compilation. Existing backend APIs and optimizer calls remain structurally unchanged; inputs such as trace ops, snapshots, inputargs, call_pure_results, and inline_short_preamble now flow through the corresponding parity marker where practical.

Verification: cargo fmt; cargo check --features dynasm; cargo test --features dynasm; python3 pyre/check.py --backend dynasm (14/14 benchmarks passed).
Keep the new CompileData structs honest about what they model: they are Rust input bundles for the flattened pyjitpl optimizer flow, not a separate reimplementation of RPython's optimize_trace dispatch. Document that adaptation explicitly.

Thread the RPython constructor payloads through the Rust bundles instead of using placeholder state. The compile data now carries enable_opts, UnrolledLoopData requires the real JitCellToken and ExportedState, compile_retrace uses the tracing session's call_pure_results and existing loop token, and compile_bridge receives the bridge trace's call_pure_results, runtime boxes, and resume storage.

This removes the bogus None/empty HashMap/empty runtime-box cases found in the static parity review while preserving the existing flattened optimizer entry points.
BridgeCompileData.runtime_boxes should mirror RPython compile.py:1056-1060: the runtime boxes passed into optimize_bridge are the live boxes from the bridge-closing JUMP, before trace.get_iter() allocates fresh input boxes for optimization.

The previous Rust plumbing built runtime_boxes from prepared.inputargs. Those are the fresh TraceIterator input boxes, not the original live_arg_boxes that RPython threads through BridgeCompileData and unroll.py:183-184. That was currently dormant because optimize_bridge still derives its active runtime boxes from the pre-opt JUMP args, but it made the new CompileData bundle structurally misleading and unsafe for future readers.

Read the closing JUMP args before shadowing bridge_ops with the prepared trace, and pass that vector into BridgeCompileData instead. If the path has no terminal JUMP, keep the existing empty fallback.

Verification: cargo fmt --check; cargo check --features dynasm; cargo test --features dynasm; python3 pyre/check.py --backend dynasm (14/14 passed).
Keep CompileData itself limited to the shared trace payload, matching RPython's base class, and store call_pure_results on each concrete CompileData variant instead.

Construct BridgeCompileData from the original bridge trace and runtime boxes before the Rust TraceIterator preparation step, so the wrapper mirrors compile.py while the explicit preparation remains the unroll.py trace.get_iter adaptation.

Look up the retrace loop token before taking call_pure_results or closing/consuming the active trace, matching compile.py's get_procedure_token/assert ordering and avoiding trace mutation on a missing token.
RPython compile_retrace resolves the existing loop_jitcell_token before it records the closing JUMP for the retrace. Pyre had moved that lookup inside the block that had already taken self.tracing, so a rare missing-token path returned false after draining the active trace context.

Move the token lookup to the start of compile_retrace, before partial_trace, exported_state, retracing_from, or tracing are consumed. If the token is absent, the retrace state remains intact instead of turning a pre-JUMP failure into a destructive cancellation.

Keep the CompileData payload structs in place. They still document the RPython compile.py data-shape adaptation, while this commit fixes the concrete parity regression without deleting that structural marker.

Verification: cargo check -p majit-metainterp --features dynasm; cargo test -p majit-metainterp --features dynasm.
RPython's BridgeCompileData keeps runtime_boxes separate from the trace iterator and passes them to UnrollOptimizer.optimize_bridge. The Rust bridge path was constructing BridgeCompileData but then discarded that field, leaving Optimizer::optimize_bridge to reuse the prepared bridge JUMP args for both fallback jump emission and virtual-state guard generation.

Pass runtime_boxes explicitly into Optimizer::optimize_bridge. Keep the prepared pre-optimization JUMP args for send_extra_operation fallback jumps, but use the original runtime boxes when calling jump_to_existing_trace so guard generation follows unroll.py's runtime_boxes contract.

Also thread the same runtime box payload through the entry-bridge path, where Pyre has a flattened compile_entry_bridge adaptation instead of the exact RPython CompileData object path.
@coderabbitai

coderabbitai Bot commented May 15, 2026

Copy link
Copy Markdown

Warning

Rate limit exceeded

@lifthrasiir has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 47 minutes and 14 seconds before requesting another review.

You’ve run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 14e6bed6-54c6-434d-be89-f7403fe264b6

📥 Commits

Reviewing files that changed from the base of the PR and between 7392004 and 479f511.

📒 Files selected for processing (1)
  • majit/majit-metainterp/src/pyjitpl/mod.rs

Walkthrough

This PR introduces compile-data bundle types to encapsulate trace metadata and propagates them through the compilation pipeline. New struct types (CompileData, PreambleCompileData, SimpleCompileData, BridgeCompileData, UnrolledLoopData) bundle trace references with call-pure results and compilation options, replacing direct trace cloning. The optimizer's bridge path is updated to accept and thread runtime boxes separately. Integration across preamble, retrace, simple, and bridge compilation phases derives trace operations from bundles instead of cloning directly.

Changes

Compile Data Bundling and Pipeline Integration

Layer / File(s) Summary
Compile data type contracts
majit/majit-metainterp/src/compile.rs
Five new public struct types (CompileData, PreambleCompileData, SimpleCompileData, BridgeCompileData, UnrolledLoopData) bundle trace references with compile-time context: runtime boxes, call-pure results, enable options, resume storage, and unroll state. Each provides constructors and accessors for trace operations, input arguments, and snapshots.
Optimizer bridge signature and runtime_boxes threading
majit/majit-metainterp/src/optimizeopt/optimizer.rs
Optimizer::optimize_bridge is extended to accept runtime_boxes: &[OpRef] and threads this parameter through both jump-to-existing-trace attempts (force_boxes=false and force_boxes=true paths) for runtime-box-aware guard and jump optimization.
Preamble compilation with compile data
majit/majit-metainterp/src/pyjitpl/mod.rs (lines 3968–4022)
During trace compilation, PreambleCompileData is constructed before garbage collection, and trace operations are extracted from it rather than cloned directly. Optimizer call-pure results and trace input-arg types are set from the preamble compile data.
Retrace compilation with unrolled loop data
majit/majit-metainterp/src/pyjitpl/mod.rs (lines 4912–5052)
Loop JIT cell token is resolved early before state consumption. UnrolledLoopData is constructed with the token and call-pure results, and trace operations are derived from unrolled data instead of cloned from the original trace. Call-pure results are threaded through the retrace pipeline.
Simple and Task#70 compilation with simple compile data
majit/majit-metainterp/src/pyjitpl/mod.rs (lines 4733–5962)
SimpleCompileData is constructed for both standard simple compilation and Task#70 special handling, deriving trace snapshots and operations from the compile data. Call-pure results are taken from context and passed through compile data to configure the optimizer. Task#70 rebuilds the trace with snapshots before compile data construction.
Bridge compilation with bridge compile data and runtime boxes
majit/majit-metainterp/src/pyjitpl/mod.rs (lines 7998–8576)
bridge_runtime_boxes is extracted from the last Jump operation's arguments. BridgeCompileData is constructed with original bridge trace data. Bridge-specific call-pure results and runtime boxes are returned and used to configure the optimizer with bridge inputarg types and passed to the optimizer bridge call site.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~75 minutes

Possibly related PRs

  • youknowone/pyre#34: Both PRs modify the bridge optimization path in majit/majit-metainterp/src/optimizeopt/optimizer.rs—the main PR extends Optimizer::optimize_bridge/try_jump_to_existing_trace to thread runtime_boxes, while the retrieved PR changes optimize_bridge to stop passing the pre-forced virtual-state (pre_vs) into those jump attempts.

Suggested reviewers

  • youknowone

Poem

🐰 With data bundled tight and boxed just right,
The traces flow through optimization's light,
From preamble to bridge, each checkpoint clear,
The call-pure results propagate without fear!
A refactor most neat, the pipeline's complete!

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: introducing CompileData structs to achieve parity with RPython, which is the central objective of this PR.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@majit/majit-metainterp/src/pyjitpl/mod.rs`:
- Around line 5518-5531: The code duplicates construction of a compile bundle
(SimpleCompileData::new(...) plus deriving trace_snapshots and trace_ops) in
multiple places; extract that sequence into a single helper (e.g.,
build_simple_compile_data or make_simple_compile_views) that takes the shared
inputs (trace, &call_pure_results, enable_opts from
self.warm_state.get_enable_opts()) and returns the SimpleCompileData and the
derived trace_snapshots and trace_ops (or at least the snapshots and operations
vectors). Replace the direct calls to SimpleCompileData::new,
simple_data.base.snapshots().to_vec(), and
simple_data.base.operations().to_vec() with calls to this helper in both places
(including where ctx.constants.refresh_from_gc() / into_inner_with_types() is
used) so both paths use the identical assembly.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: a559a674-26b8-4860-857e-4c7420742d69

📥 Commits

Reviewing files that changed from the base of the PR and between de6346f and 7392004.

📒 Files selected for processing (3)
  • majit/majit-metainterp/src/compile.rs
  • majit/majit-metainterp/src/optimizeopt/optimizer.rs
  • majit/majit-metainterp/src/pyjitpl/mod.rs

Comment thread majit/majit-metainterp/src/pyjitpl/mod.rs Outdated
Extract the repeated SimpleCompileData setup in pyjitpl into make_simple_compile_views. Both finish compilation and compile_simple_loop now use the same helper to assemble the SimpleCompileData plus the derived trace snapshots and operation vector.

This keeps the compile.py parity marker wiring consistent across the two simple compile paths while leaving constant-pool refresh and optimizer behavior unchanged.

Verification: cargo fmt; cargo check --features dynasm; cargo test --features dynasm.
@youknowone
youknowone merged commit b200775 into youknowone:main May 15, 2026
22 of 25 checks passed
youknowone added a commit that referenced this pull request Aug 11, 2026
Adds `.claude/skills/cel-unboxed-values/SKILL.md`, the design of record for
task #52: one CEL bytecode VM as `Program::execute`, traced by
majit-translate, over a `#[repr(C)]` header-first `Arc`-free class family.

Force-added: `.claude` is in `.gitignore`, matching how `codex-review`,
`merge`, `parity` and `rebase` are tracked.

Records, among others:

- four shapes that do not lower and must not be re-derived — a
  function-pointer vtable struct, `virtualizables: []` on a stack VM, a raw
  `*mut CelRef` slot array, and `Result<CelRef, CelError>` inside the traced
  VM (the `?`-to-exception-link rewrite is keyed on `PyError`);
- CEL errors carried out-of-band and host builtins compiled to integer
  opcodes, both consequences of those two;
- a majit change budget M1..M8, with M4 (`guard_class`) off the critical
  path and M8 (a user-registered-function residual path) newly on it;
- write-barrier sites, the root set, and the `Value: !Send` break that
  follows from cel having no GIL;
- the P0.d run of 2026-08-07: assertions 1 and 3 hold, assertion 2 holds
  once the class-static address is a constant — which pins M1 as its sole
  gate — and assertion 4 is not checkable below the codewriter.

Assisted-by: Claude
youknowone added a commit that referenced this pull request Aug 11, 2026
`compiles >= 1` could not be tightened while the count was not the fixture's
own: before #52 the probe read `compiled 2` in 10 of 10 runs, because the
counters are process-wide and the two result-only tests entered the JIT
without taking PROBE_LOCK.

With #52 landed the count is deterministic -- 8 consecutive full-suite runs
read `compiled 1` -- so the inequality can become an equality. That also makes
the assertion gate the contamination #52 closed: a future test entering the JIT
inside the probe's window reads 2 and fires here, where previously any count
>= 1 passed and the ratio below was computed from whichever trace arrived last.

The ratio and ops_before figures are unchanged (4805 -> 5, ratio 961): both
fixtures run away to trace_limit and fold identically, which is why the
contamination was invisible to every assertion in the file.

Assisted-by: Claude
youknowone added a commit that referenced this pull request Aug 24, 2026
…nd residual fnaddr shims

Move `StringBuilderBox`/`StringPieceBox` and the `rbuilder_runtime` module
(`ll_new`, `ll_grow_by`, `ll_grow_and_append`, `ll_append`, `ll_append_char`,
`ll_getlength`, `ll_shrink_final`, `ll_fold_pieces`, `ll_build`, plus the #52
semantics tests) from `pyre-jit::eval` to `pyre_object::rbuilder`. Move the raw
low-level-string helpers `bh_lowlevel_chars_offset`, `bh_read_lowlevel_string`,
`bh_write_lowlevel_char` to `pyre_object::lowlevel_string` beside their existing
`bh_alloc`/`bh_free`/`bh_len` siblings. The relocated crate is reachable from
`pyre-interpreter`, where the fnaddr registry lives; `pyre-jit::eval` was not.

Add `pub extern "C"` shims `jit_ll_append_res0(builder, ll_str)` and
`jit_ll_append_res_slice(builder, ll_str, start, end)` delegating to `ll_append`
with `item_size = 1`, and register them in `jit_trace_fnaddrs` under the
un-prefixed leaf keys `ll_append_res0` / `ll_append_res_slice`. With the fnaddr
bound, `guess_call_kind`'s `dont_look_inside` residual arm (the append helper
recognizer) resolves these callsites.

`build_gc`'s builder/piece tid registration now reads the
`pyre_object::rbuilder` offset/size consts instead of `size_of`/`offset_of` on
the moved structs, so `pyre-jit::eval` no longer references the struct types.

Assisted-by: Claude
youknowone added a commit that referenced this pull request Aug 24, 2026
…nd residual fnaddr shims

Move `StringBuilderBox`/`StringPieceBox` and the `rbuilder_runtime` module
(`ll_new`, `ll_grow_by`, `ll_grow_and_append`, `ll_append`, `ll_append_char`,
`ll_getlength`, `ll_shrink_final`, `ll_fold_pieces`, `ll_build`, plus the #52
semantics tests) from `pyre-jit::eval` to `pyre_object::rbuilder`. Move the raw
low-level-string helpers `bh_lowlevel_chars_offset`, `bh_read_lowlevel_string`,
`bh_write_lowlevel_char` to `pyre_object::lowlevel_string` beside their existing
`bh_alloc`/`bh_free`/`bh_len` siblings. The relocated crate is reachable from
`pyre-interpreter`, where the fnaddr registry lives; `pyre-jit::eval` was not.

Add `pub extern "C"` shims `jit_ll_append_res0(builder, ll_str)` and
`jit_ll_append_res_slice(builder, ll_str, start, end)` delegating to `ll_append`
with `item_size = 1`, and register them in `jit_trace_fnaddrs` under the
un-prefixed leaf keys `ll_append_res0` / `ll_append_res_slice`. With the fnaddr
bound, `guess_call_kind`'s `dont_look_inside` residual arm (the append helper
recognizer) resolves these callsites.

`build_gc`'s builder/piece tid registration now reads the
`pyre_object::rbuilder` offset/size consts instead of `size_of`/`offset_of` on
the moved structs, so `pyre-jit::eval` no longer references the struct types.

Assisted-by: Claude
youknowone added a commit that referenced this pull request Aug 24, 2026
…nd native fnaddr (#1447)

* majit rbuilder: residualize dont_look_inside append helpers when a native fnaddr is bound

guess_call_kind classifies a call to ll_append_res0 / ll_append_res_slice as
Residual once a function_fnaddrs entry is registered for the path, matching the
@jit.dont_look_inside decoration on those helpers in rbuilder.py. The append /
append_slice jit arms route through these general residuals; residualizing keeps
the grow/malloc body out of the trace.

Gated on the function_fnaddrs entry: without a bound native address the
symbolic-fallback address is not callable, so the call stays Regular and executes
the generated jitcode. This is the codewriter half; a later change registers the
runtime helper fnaddr to activate it.

Assisted-by: Claude

* majit rbuilder: relocate builder runtime to pyre-object; bind ll_append residual fnaddr shims

Move `StringBuilderBox`/`StringPieceBox` and the `rbuilder_runtime` module
(`ll_new`, `ll_grow_by`, `ll_grow_and_append`, `ll_append`, `ll_append_char`,
`ll_getlength`, `ll_shrink_final`, `ll_fold_pieces`, `ll_build`, plus the #52
semantics tests) from `pyre-jit::eval` to `pyre_object::rbuilder`. Move the raw
low-level-string helpers `bh_lowlevel_chars_offset`, `bh_read_lowlevel_string`,
`bh_write_lowlevel_char` to `pyre_object::lowlevel_string` beside their existing
`bh_alloc`/`bh_free`/`bh_len` siblings. The relocated crate is reachable from
`pyre-interpreter`, where the fnaddr registry lives; `pyre-jit::eval` was not.

Add `pub extern "C"` shims `jit_ll_append_res0(builder, ll_str)` and
`jit_ll_append_res_slice(builder, ll_str, start, end)` delegating to `ll_append`
with `item_size = 1`, and register them in `jit_trace_fnaddrs` under the
un-prefixed leaf keys `ll_append_res0` / `ll_append_res_slice`. With the fnaddr
bound, `guess_call_kind`'s `dont_look_inside` residual arm (the append helper
recognizer) resolves these callsites.

`build_gc`'s builder/piece tid registration now reads the
`pyre_object::rbuilder` offset/size consts instead of `size_of`/`offset_of` on
the moved structs, so `pyre-jit::eval` no longer references the struct types.

Assisted-by: Claude

* majit rbuilder: UnicodeBuilder append_slice arm; shrink_array selects width by GC tid

UnicodeBuilderRepr::rtype_method gains the append_slice arm StringBuilderRepr
already had, dispatching the shared width-generic rtype_builder_append_slice with
the UNICODE piece/pointer lltypes and mallocunicode/copyunicodecontent. Adds a
unicodebuilder_rtype_method_append_slice_emits_direct_call test mirroring the STR
one (asserts the UNICODE callee tree mints mallocunicode).

jit_ll_shrink_array splits into a width-parametric shrink_lowlevel_array core and
a shrink_array_width helper that reads the buffer's GC type id (lowlevel_unicode
tid -> item_size 4, else STR width 1), so a single residual target serves both
builder widths without a jtransform change. A raw-fallback buffer has no GC owner
and keeps STR width. Adds shrink tests for both widths.

Assisted-by: Claude

* rbuilder: add pending end-to-end StringBuilder fixture (join/repeat/concat/replace)

pyre/bench/synth/_pending/rbuilder_string_build_hot.py drives the builder-mode
string paths (unicodeobject.rs Wtf8Buf accumulators lowered via
graph_has_builder_accumulator to ll_new/ll_append*/ll_build) from four hot loops
with deterministic integer output (join/repeat/concat/replace). Kept in _pending/
— run_synthetic_suite globs *.py non-recursively, so the gate skips it — until
its jitstats snapshots and max-pypy-ratio are recorded and tuned on a
charon-capable host and it is promoted to pyre/bench/synth/. Verified
deterministic on CPython (1800000/420000/256000/960000).

Assisted-by: Claude

* majit rbuilder: port ll_grow_by growth ovfchecks; abort on overflow

Native `ll_grow_by` computed the growth size with plain `i64` arithmetic,
which wraps on overflow in release and would allocate an undersized buffer.
Factor the three `ovfcheck`s of `rbuilder.py ll_grow_by` into `grow_by_sizes`
using `checked_add`; on overflow the caller aborts with a message instead of
wrapping. `raise MemoryError` propagation and the JIT graph-builder's
`int_add_ovf` exception edge remain unported.

Add a `grow_by_sizes` unit test covering the 64-round-up and each of the
three overflow points.

Assisted-by: Claude

* majit rbuilder: model grow_by/grow_and_append ovfcheck exception edges

ll_grow_by: split the three growth ovfchecks into one block each so each
int_add_ovf is its block's trailing op; the OverflowError edge lands in a
shared handler that raises MemoryError via the graph exceptblock.

ll_grow_and_append: split the fast path into block_fast_check (the trailing
ovfcheck) and block_fast_body; the OverflowError edge routes to the slow
path (`except OverflowError: pass`).

Add close_ovfcheck_block to attach the c_last_exception exit shape
flatten insert_exits requires for a block containing an _ovf op (2-3 exits
switched on last-exception). Update the two graph-structure tests for the
multi-block layout.

Assisted-by: Claude

* majit: preserve exception-link extravars in opname-spine lowering; harden grow_by test

jtransform_opname::lower_graph rebuilt each exit link with Link::new_mixed,
which leaves last_exception/last_exc_value None. A canraise block's
OverflowError exit (as close_ovfcheck_block emits for ll_grow_by/
ll_grow_and_append) then reaches flatten's make_exception_link with None
extravars, which asserts they are Some and unwraps them for an empty-target
handler. Copy the flowspace link's extravars onto the model link. Add
lower_graph_preserves_ovfcheck_exception_extravars covering a trailing
int_add_ovf block with an OverflowError edge.

build_ll_grow_by_ovf_arith_mallocs_piece_and_relinks_buffer asserted only
op-name counts and the MemoryError edge; add block_d data-flow assertions:
the mallocfn size is the same value written to current_end, total_size is a
distinct value, and current_pos is 0.

Assisted-by: Claude

* majit rbuilder: thread StringBuilder with_capacity size through the newstringbuilder marker

The builder-mode ctor marker (front `mir.rs`) and its flowspace_adapter
lowering both emitted `newstringbuilder` with no operands, so
`rtype_builder_new` could only take the `nb_args() == 0` branch and called
`ll_new(INIT_SIZE)`. Forward the ctor operands verbatim: `new()` carries
none and `with_capacity(n)` carries the size `n`, so the operand count
mirrors `len(hop.args_v)` in `AbstractStringBuilderRepr.rtyper_new` and a
requested capacity threads `ll_new(n)`.

Add the size-carrying cases: a flowspace_adapter test that the marker
forwards its operand to `newstringbuilder`, and an rtyper test that
`rtyper_new` with a size operand emits `direct_call ll_new` carrying the
requested size rather than the default.

Assisted-by: Claude
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants