builtins: fold sum() over the live iterator, and carry a wide int through the compensated fast paths - #1036
Conversation
…ough the compensated fast paths `builtin_sum` materialised the iterable with `collect_iterable` before folding. `app_functional.py:54 _regular_sum` folds over the live iterator (`for x in sequence: last = last + x`), so `next` and `__add__` interleave; materialising first ran every `next` before every `__add__`, and allocated O(n) for an iterable the fold consumes in O(1). Pump the iterator instead, holding the running total, the pending item and the iterator in shadow-stack slots — `next` and `__add__` both run allocating Python code, and the fold previously kept the accumulator and the item `Vec` as raw locals across those calls. The four phases (exact int, compensated float, compensated complex, generic fold) and the handling of an item that leaves a phase are unchanged. The float and complex fast paths accepted only a machine-word `int`, so a wider one left the compensated loop and flushed the compensation term before the large value arrived: `sum([1.0, 10**100, 1.0, -(10**100)])` read 0.0 where 3.14 reads 2.0. Convert an `int` of any width to a double inside the loop, raising OverflowError only when the magnitude exceeds f64 range. `RootScope::set` is the cached-cell twin of `shadow_stack_set`, which re-resolves the thread local on each call; this fold reads and writes a slot per item. Assisted-by: Claude
|
Warning Review limit reached
Next review available in: 6 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
Walkthrough
ChangesStreaming sum
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant builtin_sum
participant RootScope
participant iterator
participant accumulator
builtin_sum->>RootScope: retain iterator, item, and accumulator
builtin_sum->>iterator: request next item
iterator-->>builtin_sum: return item
builtin_sum->>accumulator: add item using numeric or generic path
builtin_sum->>RootScope: update accumulator root
Possibly related PRs
Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
💡 Codex Review
https://github.com/youknowone/pyre/blob/a0c018c9bde230ad636fc5dcb1abcf66643ada88/pyre-interpreter/src/builtins.rs#L15297
When the running integer has already exceeded the machine-word fast path, this new is_int_or_long arm incorrectly lets a later float transition re-enter compensated summation. For example, CPython 3.14 returns 0.0 for sum([2**63, 0.1, 1, -(2**63)]) because 2**63 permanently switches the operation to the generic fold, whereas this implementation consumes the final wide integer here and preserves the compensation, producing 1.0. Preserve the generic-mode state once the initial integer path encounters a wide integer rather than deciding solely from the accumulator's current runtime type.
AGENTS.md reference: AGENTS.md:L231-L233
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@pyre/extra_tests/parity_tests/builtin_sum_python314.py`:
- Around line 63-76: Add coverage in the existing complex-phase assertions for a
float subclass whose overridden __radd__ must be honored, and for input ordered
as float followed by complex to exercise compensated complex accumulation. Keep
the checks focused on the expected sums and use lightweight local test values
without changing the existing exact-operand cases.
In `@pyre/pyre-interpreter/src/builtins.rs`:
- Around line 15345-15346: Update the complex fast-path branch around
compensated_sum_add to require both pyre_object::is_float(item) and
pyre_object::is_exact_builtin_instance(item), matching the existing float phase.
Ensure float subclasses bypass this accumulator and remain pending for the
generic fold.
- Around line 15312-15321: Update the float-accumulation loop and transition
into the exact-complex path so a pending exact complex item is consumed into the
compensated real_sum and imag_sum before writing the float result to last_slot
and breaking. Ensure the accumulator becomes complex afterward, allowing
is_exact_complex_operand and the complex compensated path to run for mixed
float-then-complex inputs while preserving the existing all-float behavior.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 7c5f076b-18c2-4527-860f-4ef72c9bd4f0
📒 Files selected for processing (3)
pyre/extra_tests/parity_tests/builtin_sum_python314.pypyre/pyre-interpreter/src/builtins.rspyre/pyre-object/src/gc_roots.rs
| roots.set( | ||
| last_slot, | ||
| pyre_object::w_float_new(compensated_sum_to_double(real_sum)), | ||
| ); | ||
| } | ||
| // CPython 3.14's complex fast path carries independent compensated real | ||
| // and imaginary accumulators. Exact complex items, all ints, and all | ||
| // floats stay on the path; an arbitrary numeric object falls back to the | ||
| // generic left-fold below. | ||
| if unsafe { is_exact_complex_operand(last) } { | ||
| if unsafe { is_exact_complex_operand(roots.get(last_slot)) } { |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
CPython bltinmodule.c builtin_sum float loop fallback PyNumber_Add then PyComplex_CheckExact result complex compensated block order
💡 Result:
In CPython, the sum() built-in function (implemented in Python/bltinmodule.c) utilizes specialized loops to enhance performance and accuracy for specific types, such as floats and complexes [1][2][3]. When processing inputs, sum() attempts to use these specialized loops. If it encounters types that fall outside the specialization (such as when a non-float or non-complex type is introduced in a sequence), it performs a fallback to the generic PyNumber_Add() operation to ensure correct behavior according to Python's general arithmetic rules [2][4][5]. Specifically, regarding compensated summation and type handling: 1. Compensated Summation: Recent improvements (e.g., in Python 3.12+ and subsequent updates) introduced compensated summation (a Kahan-like algorithm) for both float and complex accumulations to reduce floating-point rounding errors [4][6][7]. 2. Fallback Mechanism: If the specialization loop (e.g., for PyFloat_CheckExact or PyComplex_CheckExact) can no longer proceed—either because the input type changes or because the accumulation result is no longer compatible with the specialized storage—the implementation falls back to PyNumber_Add() [2][4][5]. 3. Interaction with Complex Numbers: For complex summation, the code specifically checks PyComplex_CheckExact [5]. If the accumulated result is a complex number, the logic ensures that subsequent additions are handled correctly, often using PyNumber_Add() to combine the current result with the next item if the item itself is not a simple float or complex that can be handled by the specialized in-place accumulation [5][3]. 4. Block Order/Compensated Block: The compensated summation logic maintains high-order and low-order components (often referred to as a CompensatedSum structure or similar in internal implementations) to track accumulated values and compensation for lost bits [3]. If the type of the sum result or the item requires moving out of this compensated loop, the implementation stops the compensated summation and reverts to the standard, non-compensated PyNumber_Add() flow [2][4][5]. These changes were part of ongoing efforts to improve the precision of sum() for floating-point and complex data while maintaining the flexibility to handle arbitrary object types through PyNumber_Add() [2][6][7].
Citations:
- 1: https://github.com/python/cpython/blob/main/Python/bltinmodule.c
- 2: Accuracy issues of sum() specialization for floats/complexes python/cpython#122234
- 3: gh-121149: improve accuracy of builtin sum() for complex inputs python/cpython#121176
- 4: Improve accuracy of builtin sum() for float inputs python/cpython#100425
- 5: Specialization for accurate complex summation in sum()? python/cpython#121149
- 6: python/cpython@169e713
- 7: gh-122234: fix accuracy issues for sum() python/cpython#122236
🏁 Script executed:
#!/bin/sh
set -eu
# Locate the relevant builtin implementation and inspect the sum loop around the cited region.
printf 'Files named builtins.rs:\n'
fd -a 'builtins\.rs$' . | sed 's#^\./##'
printf '\nLine count:\n'
wc -l pyre/pyre-interpreter/src/builtins.rs
printf '\nRelevant region (15260-15365):\n'
sed -n '15260,15365p' pyre/pyre-interpreter/src/builtins.rs | nl -ba -v15260
printf '\nSearch for exact_complex_operand and sum implementation context:\n'
rg -n "is_exact_complex_operand|PyComplex_CheckExact|compensated_sum_to_double|generic left-fold|sum\\(" pyre/pyre-interpreter/src/builtins.rsRepository: youknowone/pyre
Length of output: 365
🏁 Script executed:
#!/bin/sh
set -eu
printf 'Relevant region (15260-15365):\n'
awk '{printf "%8d\t%s\n", NR, $0}' pyre/pyre-interpreter/src/builtins.rs | sed -n '15260,15365p'
printf '\nSearch for exact_complex_operand and sum implementation context:\n'
rg -n "is_exact_complex_operand|PyComplex_CheckExact|compensated_sum_to_double|generic left-fold|sum\\(" pyre/pyre-interpreter/src/builtins.rsRepository: youknowone/pyre
Length of output: 7115
🌐 Web query:
CPython bltinmodule.c builtin_sum 3.14 float item PyComplex_CheckExact result add fallback order
💡 Result:
In CPython's builtin_sum (implemented in Python/bltinmodule.c), the summation logic has been progressively optimized with specialized fast paths to handle numeric types like integers, floats, and complex numbers efficiently [1][2][3]. When builtin_sum processes items, it maintains a running result. If the internal specialized summation detects an item type that does not match the current specialized path (e.g., encountering a complex object while summing float items), the implementation breaks out of the optimized loop and falls back to a more generic addition mechanism [1][2]. Regarding the specific interaction with PyComplex_CheckExact: 1. Logic Flow: Recent optimizations have introduced specialized paths for complex numbers, using PyComplex_CheckExact to identify complex inputs directly [1][3]. When the sum contains complex numbers, the implementation tracks the real and imaginary components separately using compensated summation techniques (e.g., cs_add helpers) to maintain precision [4][3]. 2. Fallback Mechanism: If the current item is not a simple numeric type that can be handled by the specialized loop (or if a type mismatch occurs, such as encountering a non-numeric type or a complex number where only floats were expected), the specialized loop terminates [1][2]. It then falls back to using standard Python object operations, typically PyNumber_Add(result, item), to compute the next step of the sum [2]. 3. Item Handling: The implementation attempts to avoid the overhead of generic PyNumber_Add by using specialized checks like PyFloat_CheckExact or PyLong_Check for performance [1][2]. If an item fails these exact type checks but is still a valid numeric type, it may be handled by the generic fallback or, in newer versions, by broader type checks (like PyFloat_Check) if the complexity is deemed acceptable for that specific execution path [1][3]. In summary, the fallback order generally prioritizes: - Specialized exact-type summation loops (e.g., using PyFloat_CheckExact or PyComplex_CheckExact) [1][2][3]. - Falling back to the generic PyNumber_Add() method when the item type deviates from the current specialized accumulator's type [1][2]. This design ensures that sum() remains high-performance for homogeneous numeric iterables while maintaining correctness for mixed-type inputs by deferring to the standard Python numeric protocol [2][3].
Citations:
- 1: Specialization for accurate complex summation in sum()? python/cpython#121149
- 2: Improve accuracy of builtin sum() for float inputs python/cpython#100425
- 3: brettcannon/cpython@169e713
- 4: gh-121149: improve accuracy of builtin sum() for complex inputs python/cpython#121176
🌐 Web query:
site:github.com/python/cpython/bltinmodule.c builtin_sum PyObject_Add PyComplex_CheckExact cs_add
💡 Result:
Based on the search results, I found information about the code you're looking for in CPython's bltinmodule.c.
In the builtin_sum function, when PyComplex_CheckExact(result) is true, the code uses compensated summation with cs_add to accumulate complex number values by separately summing the real and imaginary parts.
Specifically, the implementation checks if items are exact complex types using PyComplex_CheckExact(item), and if so, uses cs_add to add the real and imaginary components to running sums (re_sum and im_sum).
The code utilizes a specialized CompensatedSum structure (cs_add) to improve numerical accuracy when summing complex numbers, which is more sophisticated than the basic PyObject_Add approach used for general object addition. You can view the complete implementation in the CPython repository's bltinmodule.c file.
Citations:
🌐 Web query:
github.com python cpython Python/bltinmodule.c "PyComplex_CheckExact(result)" "cs_add" result PyObject_Add(item)
💡 Result:
In the CPython source code, PyComplex_CheckExact is a macro defined in Include/complexobject.h that checks if a given PyObject is exactly an instance of the complex type, explicitly excluding its subclasses [1][2]. It is defined as: #define PyComplex_CheckExact(op) (Py_TYPE(op) == &PyComplex_Type) This macro is widely used within CPython's internal implementation (such as in Objects/complexobject.c) to optimize operations by avoiding the overhead or safety concerns associated with handling arbitrary subclasses [3][4][5]. When CPython code uses PyComplex_CheckExact, it ensures that it is dealing with the standard, native complex number type, allowing it to safely access the underlying structure (e.g., the cval member) without needing to worry about overridden methods or non-standard behavior that might be introduced by a user-defined subclass [3][4][5]. While your query mentions Python/bltinmodule.c, the primary usage of PyComplex_CheckExact is concentrated in Objects/complexobject.c for handling complex arithmetic and internal object management [3][4][5]. The function mentioned in your query—PyObject_Add—is a general C-API function used to perform addition on Python objects, though it is not typically part of a direct pattern named cs_add in the context of PyComplex_CheckExact. Most internal arithmetic operations involving complexes utilize specific complex-type functions or macros to ensure correct dispatching [4][5]. top_results: [1][3][2][4][5]
Citations:
- 1: https://github.com/python/cpython/blob/2.7/Include/complexobject.h
- 2: https://docs.python.org/3/c-api/complex.html
- 3: https://github.com/python/cpython/blob/9b6c60cbce4ac45e8ccd7934babff465e9769509/Objects/complexobject.c
- 4: https://github.com/python/cpython/blob/main/Objects/complexobject.c
- 5: https://github.com/python/cpython/blob/3.12/Objects/complexobject.c
🏁 Script executed:
#!/bin/sh
set -eu
python3 - <<'PY'
try:
import math
from _pyre_interpreter import builtin_sum
except Exception as e:
print("IMPORT_ERROR", repr(e))
import sys
sys.exit(0)
cases = [
("float_float_complex", [0.1] + [[complex(1.0)])],
("float_complex", [0.1 + 0.0, 1+2j],
("complex_float_float", [1+0j] + [0.1, 0.2, 0.3]),
]
missing = []
for name, args in cases:
try:
print(name, "REPR", repr(builtin_sum(list(args))))
except Exception as e:
print(name, "EXCEPTION", type(e).__name__, repr(e))
print("PYTHON_FLOAT_FLOAT_COMPLEX", repr(sum([0.1, complex(1.0, 2.0)])))
print("PYTHON_FLOAT_COMPLEX", repr(sum([0.1+0j, 1+2j])))
print("PYTHON_COMPLEX_FLOAT_FLOAT", repr(sum([1+0j, 0.1, 0.2, 0.3])))
PYRepository: youknowone/pyre
Length of output: 363
🏁 Script executed:
#!/bin/sh
set -eu
python3 - <<'PY'
try:
import math
from _pyre_interpreter import builtin_sum
except Exception as e:
print("IMPORT_ERROR", repr(e))
import sys
sys.exit(0)
cases = [
("float_float_complex", [0.1, complex(1.0, 2.0)]),
("float_complex", [0.1+0j, 1+2j]),
("complex_float_float", [1+0j, 0.1, 0.2, 0.3]),
]
for name, args in cases:
try:
print(name, "REPR", repr(builtin_sum(list(args))))
except Exception as e:
print(name, "EXCEPTION", type(e).__name__, repr(e))
for args in [
[0.1, complex(1.0, 2.0)],
[0.1+0j, 1+2j],
[1+0j, 0.1, 0.2, 0.3],
]:
print("python", repr(args), "=>", repr(sum(args)))
PYRepository: youknowone/pyre
Length of output: 225
🏁 Script executed:
#!/bin/sh
set -eu
# Read the next phase to understand how the pending complex falls into the generic fold and any post-fold checks.
printf 'Generic fold and end checks (15361-15415):\n'
awk '{printf "%8d\t%s\n", NR, $0}' pyre/pyre-interpreter/src/builtins.rs | sed -n '15361,15415p'
printf '\nOperand predicates and helpers:\n'
rg -n "is_exact_float_operand|is_exact_int_operand|is_exact_complex_operand|w_complex_get_real|w_complex_get_imag" pyre/pyre-interpreter/src -A8 -B4
printf '\nBuiltins tests mentioning sum or complex:\n'
rg -n "sum\\(|complex|isnan|float\\+complex|float.*complex|complex.*float" pyre -g '*.py' -g '*.rs' -A4 -B4Repository: youknowone/pyre
Length of output: 50372
🏁 Script executed:
#!/bin/sh
set -eu
printf 'Generic fold and post-general checks (15361-15445):\n'
awk '{printf "%8d\t%s\n", NR, $0}' pyre/pyre-interpreter/src/builtins.rs | sed -n '15361,15445p'
printf '\nOperand predicates (15404-15430):\n'
awk '{printf "%8d\t%s\n", NR, $0}' pyre/pyre-interpreter/src/builtins.rs | sed -n '15404,15430p'Repository: youknowone/pyre
Length of output: 5501
🌐 Web query:
github.com python cpython Python/bltinmodule.c lines "compensated" "PyComplex_CheckExact(result)" "PyObject_Add"
💡 Result:
The functions PyComplex_CheckExact and PyObject_Add are both part of the CPython C API, but they serve different purposes and are used in different contexts within the CPython source code [1]. PyComplex_CheckExact(op) is a macro defined in the CPython C API that returns true if its argument is a PyComplexObject but not a subtype of PyComplexObject [1]. In the context of Python/bltinmodule.c, it is used to perform type checks on objects, often to allow for specialized, faster handling of complex numbers without invoking general, slower methods [2]. For example, development discussions have included proposing the use of PyComplex_CheckExact within the builtin sum function to specialize summation for complex numbers [2]. PyObject_Add(o1, o2) is a C API function that adds two objects together, essentially performing the equivalent of the Python expression o1 + o2. It is a general-purpose function that handles various object types by invoking their type-specific number methods (e.g., nb_add) [1]. While both may appear in Python/bltinmodule.c—which implements many of Python's built-in functions—they are not typically used together as a pair [3][4][2]. PyComplex_CheckExact is used to verify an object's exact type, while PyObject_Add is used to perform arithmetic operations [2][1]. Their proximity in search results or code discussions generally reflects distinct operations being performed on objects within built-in functions, rather than them being linked as a single mechanism.
Citations:
- 1: https://docs.python.org/3/c-api/complex.html
- 2: Specialization for accurate complex summation in sum()? python/cpython#121149
- 3: https://github.com/python/cpython/blob/main/Python/bltinmodule.c
- 4: https://github.com/python/cpython/blob/master/Python/bltinmodule.c
🌐 Web query:
raw "cs_add" "PyComplex_CheckExact(result)" "PyNumber_Add(result, item)" bltinmodule.c
💡 Result:
The terms and code snippet provided originate from two distinct and unrelated domains: the Python C API and the CSparse library. There is no native function named cs_add in Python's bltinmodule.c or any other core CPython source file that uses PyComplex_CheckExact or PyNumber_Add. 1. Python C API (PyComplex_CheckExact, PyNumber_Add) These are standard functions within the CPython source code [1][2]. - PyComplex_CheckExact(result) is a C API function used to determine if an object is an exact instance of the built-in complex type, excluding subclasses [1][3]. - PyNumber_Add(result, item) is a generic C API function that performs addition between two Python objects, equivalent to the expression result + item in Python [2][4]. 2. CSparse Library (cs_add) The symbol cs_add belongs to the CSparse library, a suite of sparse matrix algorithms [5][6]. The function signature is typically cs *cs_add (const cs A, const cs B, double alpha, double beta) [5][6]. It performs sparse matrix addition (C = alphaA + betaB) [5][6] and is implemented in C, completely unrelated to Python's complex number or object addition APIs [5][7]. The snippet likely represents a user-defined function or a misunderstanding of merging two separate codebases. No such integration exists in the standard CPython distribution [8][9].
Citations:
- 1: https://docs.python.org/3/c-api/complex.html
- 2: https://github.com/python/cpython/blob/main/Doc/c-api/number.rst
- 3: https://runebook.dev/en/docs/python/c-api/complex/c.PyComplex_CheckExact
- 4: https://runebook.dev/en/docs/python/c-api/number/c.PyNumber_Add
- 5: https://github.com/DrTimothyAldenDavis/SuiteSparse/blob/stable/CSparse/Source/cs_add.c
- 6: https://people.sc.fsu.edu/%7Ejburkardt/c_src/csparse/csparse.c
- 7: https://fossies.org/dox/SuiteSparse-7.12.2/CXSparse_2Source_2cs__add_8c_source.html
- 8: https://github.com/python/cpython/blob/master/Python/bltinmodule.c
- 9: https://github.com/python/cpython/blob/v3.12.11/Python/bltinmodule.c
🌐 Web query:
site:raw.githubusercontent.com python cpython Python/bltinmodule.c "PyComplex_CheckExact(item)"
💡 Result:
PyComplex_CheckExact(item) is a C macro in the CPython API used to determine if a given PyObject* is specifically an instance of the complex type (PyComplex_Type), excluding subclasses [1][2][3]. It is defined in the Python header files, typically within Include/complexobject.h, as a check against the object's type pointer [1][4]. The macro essentially performs the following logic: #define PyComplex_CheckExact(op) Py_IS_TYPE(op, &PyComplex_Type) It is widely used across the CPython codebase, including in modules like Python/bltinmodule.c (often in specialized logic or optimizations), Python/ast.c, and various internal parsers to verify that an object is exactly a complex number without triggering potential subclass methods or overhead [4][2][5]. Code using this macro typically follows it with functions like PyComplex_AsCComplex to access the underlying data [6][1][4].
Citations:
- 1: https://github.com/python/cpython/blob/3b3b83c965447a8329b34cb4befe6e9908880ee5/Objects/codeobject.c
- 2: https://github.com/python/cpython/blob/97cf0828727ac2a269c89c5aa09570a69a22c83c/Python/ast.c
- 3: https://github.com/python/cpython/blob/8370e07e1e5b626e78ddc7aadbfaf248976c4454/Python/ast_opt.c
- 4: Specialization for accurate complex summation in sum()? python/cpython#121149
- 5: https://github.com/python/cpython/blob/0274d830/Parser/action_helpers.c
- 6: https://github.com/@raw/opencv/opencv/3.1.0/modules/python/src2/cv2.cpp
🌐 Web query:
"builtin_sum" "PyComplex_CheckExact" "PyNumber_Add" "builtin_sum" "bltinmodule.c"
💡 Result:
The functions PyComplex_CheckExact and PyNumber_Add are part of the Python/C API and are utilized within the implementation of the built-in sum function located in Python/bltinmodule.c [1][2]. The sum function employs fast-path optimizations for specific types (such as integers, floats, and complex numbers) to improve performance [3][2]. 1. PyComplex_CheckExact: This function is used to verify if an object is an exact instance of a complex number, excluding subclasses [4]. In the context of builtin_sum_impl within bltinmodule.c, it is used to guard or enter specialized loops that perform high-performance, in-place summation for complex numbers [3]. 2. PyNumber_Add: When the inputs do not meet the criteria for specialized fast-path handling (e.g., they are not exact instances of the expected types or a fallback is required), the implementation falls back to the generic PyNumber_Add function [3][2]. This ensures that the summation remains compatible with arbitrary Python objects that support the addition protocol [3]. Recent developments in the CPython source code, such as PR #150869, have highlighted the importance of using exact type checks (like PyFloat_CheckExact or PyComplex_CheckExact) alongside specialized deallocation routines (like _Py_DECREF_SPECIALIZED with _PyFloat_ExactDealloc) to avoid incorrect deallocation of object subclasses [2]. Failure to use exact checks can cause specialized deallocation to bypass an object's actual deallocator, potentially leading to memory leaks or inconsistent internal states [2].
Citations:
- 1: https://github.com/python/cpython/blob/main/Python/bltinmodule.c
- 2: gh-150868: Fix incorrect specialized deallocation of float subclasses in sum() python/cpython#150869
- 3: Specialization for accurate complex summation in sum()? python/cpython#121149
- 4: https://docs.python.org/3/c-api/complex.html
🌐 Web query:
"builtin_sum_impl" "PyComplex_CheckExact" "PyNumber_Add(result, item)"
💡 Result:
The identifiers builtin_sum_impl, PyComplex_CheckExact, and PyNumber_Add are associated with the implementation of the built-in sum() function in the CPython source code (Python/bltinmodule.c) [1][2]. Recent development work has introduced optimizations and accuracy improvements for the sum() function, particularly for complex number inputs [3][4][5]. In the CPython implementation, builtin_sum_impl performs the summation by iterating through the input [5]. To improve performance and floating-point accuracy, the implementation checks if the accumulator (result) is a complex number using PyComplex_CheckExact [3][5]. When PyComplex_CheckExact identifies the accumulator as a complex type, the implementation can branch into a specialized, high-accuracy summation path that avoids repeated, generic calls to PyNumber_Add [3][5]. When this optimization is not applicable or when falling back to a more general case, the code uses PyNumber_Add(result, item) to add each subsequent item to the running total [3][5]. Historically, the use of PyNumber_Add (rather than PyNumber_InPlaceAdd) in sum() has been a deliberate design choice to prevent unintended side effects, such as mutating objects passed in as the start argument when performing operations like list concatenation [6]. While PyNumber_InPlaceAdd could improve performance for certain types like list or collections.Counter, it would represent a breaking change in semantics for those types [6]. Consequently, recent improvements have focused on type-specific specializations (like the complex number case) rather than a global switch to in-place addition [3][4].
Citations:
- 1: https://github.com/python/cpython/blob/v3.12.11/Python/bltinmodule.c
- 2: https://github.com/python/cpython/blob/master/Python/bltinmodule.c
- 3: Specialization for accurate complex summation in sum()? python/cpython#121149
- 4: gh-121149: improve accuracy of builtin sum() for complex inputs python/cpython#121176
- 5: fedora-python/cpython@d4faa7b
- 6: https://bugs.python.org/issue39440
🌐 Web query:
"cs_add" "PyComplex_CheckExact(result)" "fs"
💡 Result:
The terms you are referencing appear to be related to the internal implementation of Python's C API, specifically concerning numeric operations and object checking. PyComplex_CheckExact(p) is a macro in the Python/C API that returns a non-zero value if the provided object p is exactly a standard Python complex number object, and zero if it is a subtype or a different type [1][2]. It is commonly used in CPython's internal code to optimize operations by ensuring the object has the expected standard memory layout, avoiding the overhead or complexity associated with subclasses [2][3]. The terms "cs_add" and "fs" (often seen as part of fsum or similar accuracy-improving logic) are related to internal helper functions used within CPython to improve the numerical precision of summation operations [4][5]. Specifically: - cs_add: This is an internal helper function (Compensated Summation) used in recent CPython implementations (notably within the builtin sum function) to maintain higher precision when adding numbers [4][5]. It uses a compensated summation algorithm to track high-order and low-order bits, mitigating precision loss during floating-point or complex number accumulation [4]. - Context: These are separate components. PyComplex_CheckExact is for type verification, while cs_add is an internal algorithm for accurate numerical summation [4][3]. They appear together in CPython's codebase when the sum function handles complex or float inputs, ensuring that the inputs are correctly identified and processed using these high-precision helpers [5].
Citations:
- 1: https://docs.python.org/3/c-api/complex.html
- 2: https://runebook.dev/en/docs/python/c-api/complex/c.PyComplex_CheckExact
- 3: https://github.com/python/cpython/blob/main/Objects/complexobject.c
- 4: gh-121149: improve accuracy of builtin sum() for complex inputs python/cpython#121176
- 5: brettcannon/cpython@169e713
Fix the mixed float-then-complex sum() path.
sum([0.1, 1+2j]) writes the compensated float total back to last_slot, makes the accumulating object a float, and then skips the exact complex fast path. CPython adds the pending complex item first while still in the float loop, and only later enters the complex compensated path with a complex accumulator. Promote the pending exact complex to real_sum/imag_sum before breaking and writing the float accumulator back.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@pyre/pyre-interpreter/src/builtins.rs` around lines 15312 - 15321, Update the
float-accumulation loop and transition into the exact-complex path so a pending
exact complex item is consumed into the compensated real_sum and imag_sum before
writing the float result to last_slot and breaking. Ensure the accumulator
becomes complex afterward, allowing is_exact_complex_operand and the complex
compensated path to run for mixed float-then-complex inputs while preserving the
existing all-float behavior.
| } else if pyre_object::is_float(item) { | ||
| real_sum = compensated_sum_add(real_sum, pyre_object::w_float_get_value(item)); |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Require an exact float in the complex fast path.
Line 15345 tests pyre_object::is_float(item) without an exactness test. A float subclass therefore enters the compensated complex accumulator, so its __add__/__radd__ override never runs. The float phase at line 15295 pairs is_float with is_exact_builtin_instance for that reason, and CPython's complex loop uses PyFloat_CheckExact after issue gh-122234 reported the merged specialization's non-exact PyFloat_Check(item) as a bug.
A float subclass item must leave the fast path and stay pending for the generic fold.
🐛 Proposed fix
- } else if pyre_object::is_float(item) {
+ } else if pyre_object::is_float(item)
+ && pyre_object::is_exact_builtin_instance(item)
+ {
real_sum = compensated_sum_add(real_sum, pyre_object::w_float_get_value(item));📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| } else if pyre_object::is_float(item) { | |
| real_sum = compensated_sum_add(real_sum, pyre_object::w_float_get_value(item)); | |
| } else if pyre_object::is_float(item) | |
| && pyre_object::is_exact_builtin_instance(item) | |
| { | |
| real_sum = compensated_sum_add(real_sum, pyre_object::w_float_get_value(item)); |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@pyre/pyre-interpreter/src/builtins.rs` around lines 15345 - 15346, Update the
complex fast-path branch around compensated_sum_add to require both
pyre_object::is_float(item) and pyre_object::is_exact_builtin_instance(item),
matching the existing float phase. Ensure float subclasses bypass this
accumulator and remain pending for the generic fold.
…fitting a machine word The C fast path accumulates into a `long`, and `PyLong_AsLongAndOverflow` signalling overflow leaves it for good: the value that leaves is still an `int`, so neither compensated phase can claim it and the rest of the fold is generic. The int loop here ran on `baseobjspace::add`, so a wide total stayed in it and a later float item handed the fold to the compensated float phase — `sum([2**63, 0.1, 1, -(2**63)])` read 1.0 where 3.14 reads 0.0. Leave the loop once the total is no longer a machine-word `int`. A wide item reached when the total is already a float is unaffected and stays on the compensated path: `sum([0.1, 2**63, 1, -(2**63)])` is 1.1. Pins the complex fast path's `PyFloat_Check` / `PyLong_Check` asymmetry too: a `float` or `int` subclass stays on it and its reflected addition is skipped, while a `complex` subclass leaves. Measured against 3.14, which returns `1+1j` for `sum([1j, SubFloat(1.0)])`. Assisted-by: Claude
Review responsesCodex P1 — valid, fixed in Confirmed against 3.14 before changing anything: The mechanism is the one Codex names. The C fast path accumulates into a This was introduced by the widening in the first commit, so it is mine, not CodeRabbit "Require an exact Measured on 3.14: class SubFloat(float):
def __radd__(self, other): return "radd-float"
sum([1j, SubFloat(1.0)]) # 3.14 -> (1+1j), pyre -> (1+1j)3.14's complex loop takes CodeRabbit test-coverage suggestion — taken, including the float-subclass Re-verified at the current base: — commented by Claude |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 709aef73ce
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| /// loop. pyre splits a wide `int` into its own `LONG_TYPE`, so overflow is | ||
| /// the type test rather than a range check. | ||
| unsafe fn is_machine_word_int_operand(obj: PyObjectRef) -> bool { | ||
| unsafe { is_exact_int_operand(obj) && !pyre_object::is_long(obj) } |
There was a problem hiding this comment.
When start is an exact integer stored as W_LongObject but still fits in i64—for example (2**63) - (2**63 - 1), because long_sub returns w_long_new even for the value 1—this type-only check incorrectly exits the integer fast path. CPython's PyLong_AsLongAndOverflow accepts that value, so sum([0.1] * 10, start) uses compensated summation and returns 2.0; pyre instead takes the generic float fold and returns 2.000000000000001. Test w_long_fits_int rather than treating every LONG_TYPE as overflow.
AGENTS.md reference: AGENTS.md:L231-L233
Useful? React with 👍 / 👎.
…ct operation guard on its cached cell `RootScope` resolves the thread-local root-stack cell once and caches it so a bracket need not resolve it again per slot access, but only `get` / `pin_root` / `set` used it. `DictOperationGuard` opened a scope and then went back through the free `publish_roots` / `normalize_roots` / `shadow_stack_get`, each of which resolves the thread local again — on Darwin an out-of-line `_tlv_get_addr` — and `normalize_roots` resolves once per root. A three-operand store resolved it ten times. Add the two missing cached-cell twins and route the guard through them, so one dict operation resolves the cell once. `normalize`'s per-index re-read of the slot is kept: a collection triggered by an earlier query may already have rewritten every published root in place. `sample`(1) over a dict-comprehension loop shows `normalize_roots` leaving the profile entirely and the shadow-stack frames halving. The wall-clock effect is below this machine's noise floor — median 0.997 against a 0.991-1.019 self-A/B band — so this is an instruction-count change, not a measured speedup. Assisted-by: Claude
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 341ea33f08
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| roots.pin_root(start); | ||
| let iterable_slot = last_slot + 1; | ||
| roots.pin_root(iterable); |
There was a problem hiding this comment.
If another thread starts a moving collection while the first pin_root calls try_gc_current_object_address, iterable is still only a raw local; the collection can move it, after which the second pin_root publishes the stale address. The DictOperationGuard added in this same change documents and avoids this exact race by publishing the complete live set before normalizing it. Publish start and iterable together, then normalize both before either is read.
Useful? React with 👍 / 👎.
🤖 Codex parity reviewStatic analysis of this diff vs the local RPython/PyPy sources (commit 341ea33). Files in the reviewed diff1. Regressions to PyPy parity introduced by this patchNone. 2. Other mismatches introduced by this patchNone. 3. Pre-existing mismatches (already present before this patch)
4. Structural adaptations
|
builtin_summaterialised the iterable withcollect_iterableand thenindexed the
Vec.app_functional.py:54 _regular_sumis a fold over thelive iterator, so
nextand__add__interleave.The ordering is observable
Both oracles agree with each other and disagree with pyre:
Three consequences: user-visible side effects run in the wrong order, an
iterable the fold consumes in O(1) costs O(n) (
sum(itertools.count())neverreturns a
Stopfrom a user__radd__), and — the one that matters most —the running total and the whole item
Vecwere raw locals acrossbaseobjspace::add, which allocates.collect_iterator's own doc commentwarns about exactly that hazard for the
Vecit builds; the fold reintroducedit by keeping that
Vecalive across everyadd.Pump the iterator instead, holding the total, the pending item and the
iterator in shadow-stack slots. The four phases (exact int, compensated
float, compensated complex, generic fold) and the "an item that leaves a
phase stays pending for the next one" handoff are preserved exactly.
A second, pre-existing defect
Porting CPython's own
test_sumandtest_sum_accuracy(54 checks) into astandalone fixture found two failures that are not caused by the rewrite —
established from the committed source, where the float phase has only
is_float/is_intarms andbreaks otherwise:The fast paths took only a machine-word
int, so a wider one left thecompensated loop and flushed the compensation term before the large value
arrived. Convert an
intof any width to a double in the loop, raisingOverflowErroronly past f64 range.Neither oracle gates this:
test_sum_accuracyis@support.cpython_only,and the
pypy3oracle is 3.11, which has no compensated sum at all.Over-range is signalled through a non-finite
f64rather than aResult,because
descroperation.rsdocuments that the JIT codewriter cannot flattena payload mixing
Float(Ok) andRef(Err) into one register kind.RootScope::setRootScopecaches the thread-local root-stack cell precisely so a loop neednot re-resolve it per access, but had only
get/pin_root; the freeshadow_stack_setresolves the thread local three times per call. Added thesymmetric
setand put the fold on the cached-cell accessors.Cost
Measured control-free, since a sibling-worktree control arm proved invalid:
+21ns per
sum()call (rootingstartand the item slot) and +3.8nsper item (one forwarding query for the now-rooted accumulator).
sumon alarge list remains ~40x
pypy3either way — untouched here.Verification
test_sum+test_sum_accuracychecks.messages included.
check.py374/374 on dynasm and cranelift, with no jit-stats movement;loops_comprehensionreads 10.1x against its 144x ratio gate.cpython_tests46/46, no regressions.cargo test799 pass.cargo fmtclean; the
clippyerrors inpyre-objectare pre-existing and in filesthis branch does not touch.
parity_tests/builtin_sum_python314.py, greenon cpython, dynasm and cranelift.
Note: the check.py run above was made against the pre-rebase base. #1034 has
since re-recorded 43 baselines and changed the gate, so CI's run is the
authoritative one at this base.
— authored by Claude
Summary by CodeRabbit
New Features
sumto process values incrementally, supporting large and unbounded iterables more efficiently.Bug Fixes
sumnow raisesOverflowErrorwhen values exceed floating-point limits.