Consider FOR_ITER family for specialization?

[Fidget-Spinner]

I haven't properly profiled anything. Throwing this idea here to see what people think.

Consider this fairly common pattern in Python:

for i in range(len(x)):
    ...

Bytecode snippet:

           6 GET_ITER
     >>    8 FOR_ITER                 2 (to 14)

Anecdotally, some users are surprised at how much overhead this has. For most simple for loops, users intend for the range objects to be used as the equivalent for(int i=0; i < x; i++) loop in C. The range objects are created then thrownaway immediately.

In those cases, calling next(range_iterator) in FOR_ITER is unnecessary overhead. We can unbox this into a simple PyLong object, then PyLong_Add on it. This can be a FOR_ITER_RANGE opcode.

This will have to be extremely conservative. We can only implement this on range objects with a reference count of 1 (ie used only for the for loop) and they must be the builtin range and not some monkeypatched version. Without the compiler’s help, the following would be very dangerous to optimize:

x = iter(range(10))
for _ in x:
    next(x)

We can also do the same with lists and tuples (FOR_ITER_LIST and FOR_ITER_TUPLE), but use the the native PyTuple/List_GetItem instead of iterator protocol. But I'm not sure how common something like for x in (1,2,3) is. So maybe those aren't worth it (they're also much harder to rollback if the optimization breaks halfway).

FOR_ITER isn't a common instruction. But I think when it's used it matters, because it tends to be rather loopy code.

What do y'all think?

[gvanrossum]

I definitely don't think that for x in (1, 2, 3) or for x in [1, 2, 3] are common enough to bother. I have been eyeing for i in range() myself, but before the specialization framework was implemented it seemed tricky.

There may have been some previous hacks, see comments by user @heeres in #42. IIRC the idea was to reuse the integer object rather than allocating a new one. However, making sure that there are no other users of that same object is tricky.

Of course, if we ever get tagged integers, that would be the right approach.

Also, in case you hadn't looked yet, range() already has two entirely distinct implementations depending on whether the end points fit in a machine word. But of course this just saves doing the increment using integer objects, the result still has to be allocated. (A free list might help too?)

[Fidget-Spinner]

Oops I didn't see that issue. It's cool that the solutions and problems somewhat converge with this issue. Also those are some really nice hacks by Heeres!

You mentioned trying out FOR_ITER/{STORE_NAME|STORE_FAST} combined instruction. This makes me wonder if the super instruction would be faster than individually specializing FOR_ITER and STORE. Or maybe the fastest would be specializing that combined instruction ;)?

[gvanrossum]

We’ll, there’s not much to specialize about STORE_FAST, and STORE_NAME isn’t worth it (a for loop at the module level?!) So the super-instruction seems a good idea; specializing it for range() seems good too.

You might try to measure how often ‘for i in range(…)’ actually occurs. I think in my own code I use enumerate() more than range(), but that is harder to specialize (or is it?). It would be a nice win to avoid needing the tuple there (this would need help from the compiler).

[markshannon]

One possible way to speed up tight for loops is to add a FOR_END opcode, which would be like FOR_ITER but reverse the direction of branch. It would not branch when the iterator is exhausted.
So for x: body would compile to:

    FOR_ITER end
loop:
    body 
    FOR_END loop
end:

instead of

loop:
    FOR_ITER end
    body 
    JUMP_ABSOLUTE loop
end:

I don't know if this would be profitable, but might be worth an experiment.

[Fidget-Spinner]

After some hacking around, I think my original approach is pretty naive -- next(iter) seems to be very little overhead (just calling tp_iternext which increments and creates a PyLong).

User heeres is right that most of the overhead is from the memory allocation. I'm surprised though -- I expected with how pymalloc works, that _PyObject_Malloc would be very cheap.

I will attempt to use the specializing interpreter features to apply their optimizations in a safer manner (with full credits given to them of course). And hopefully, it will have better results.

[Fidget-Spinner]

I'm abandoning this idea. I integrated the PyLong reuse hacks above into a specialized instruction, and it produces a significant pessimization on ranges with small longs [1].

Since there's a small int cache in CPython, any long from -5 to 256 is essentially free. So the overhead of an adaptive instruction is significant. I would expect such small ranges to cover most use cases.

Additionally, to get the full test suite to pass, I had to be extremely conservative. This means more than half the time it still had to allocate a new long. I don't see this going too far without the compiler's help.

There is some speedup on very large ranges, but it was only 10% on a microbench [2]. I'm not sure what settings user heeres was testing with (eg. debug, release, PGO?) for 50%, and I might have implemented things wrongly, so I can't provide an explanation. One guess is that pymalloc is quite efficient after PGO. Also if you're dealing with such large ranges, you're probably better off using numpy, cython, numba etc. Overall, even if we can fix the other shortcomings, for the complexity it introduces it's not profitable.

TLDR: Combined instructions or Mark's rearranged instructions look like a better fit. I would think FOR_ITER + LOAD_FAST (woops, STORE_FAST) would show benefits. But they cross basic blocks, so optimization is more difficult. Maybe interpreter specializations can be re-looked at again for late-stage optimizations.

Note: All results were obtained on Windows with PGO. Also I know I should use pyperf instead, but I was short on time 😉 . Sorry.

[1]

python.exe -m timeit "for _ in range(0, 256): pass"
Before:
100000 loops, best of 5: 2.17 usec per loop
Specialized:
100000 loops, best of 5: 2.45 usec per loop

[2]

python.exe -m timeit "for _ in range(256, 10000): pass"
Before:
2000 loops, best of 5: 138 usec per loop
Specialized:
2000 loops, best of 5: 126 usec per loop

Commit: Fidget-Spinner/cpython@e5393d6

[gvanrossum]

Thanks for doing all the research!

If you want to combine FOR_ITER and LOAD_FAST in the compiler, that could be a super-instruction (#16) added in the assembly phase. I have done a few of these, but never merged any upstream (https://github.com/faster-cpython/cpython/tree/super-instr). Also check out Mark's recent attempt (https://github.com/faster-cpython/cpython/tree/super-instructions).

[Fidget-Spinner]

I'm on a roll producing things with 0 speedups 😉 . FOR_ITER + STORE_FAST showed no speedups too. Seems like the existing PREDICT macro is doing a really good job. On Windows with PGO:

python.exe -m timeit -s
>> "def f(n):
>>  for _ in range(n):
>>   pass
>> " "f(10_000)"

Before:
2000 loops, best of 5: 138 usec per loop
2000 loops, best of 5: 141 usec per loop
2000 loops, best of 5: 138 usec per loop

Superinstruction:
2000 loops, best of 5: 140 usec per loop
2000 loops, best of 5: 137 usec per loop
2000 loops, best of 5: 137 usec per loop

Commit: Fidget-Spinner/cpython@c8a1338

[Fidget-Spinner]

To close the loop:

It seems that Larry tried something similar (though with a different approach) back in 2015: https://bugs.python.org/issue24138. He decided it wasn't worth it in the end.

That spawned some attempts at a long freelist in https://bugs.python.org/issue24165. But the pyperformance numbers were a little murky, so that was abandoned too.

[gvanrossum]

Thanks for the links! I wonder if maybe we should look at that integer freelist again.

[Fidget-Spinner]

Once other operations are faster, the overhead of the integer allocation will be a bigger percentage. So maybe it will be noticeable in 3.11 :).

Oh, and one more really cool idea by Serhiy, which I think will probably get merged https://bugs.python.org/issue45026. In short, the PyLong to return is currently calculated by (start + index * step) (index is the index of the iterator, not the return value). The change stores the previous Pylong in start, so the new PyLong uses just (start + step). Iterating ranges are 2x faster in his PR!

[markshannon]

I'm reopening this. Specialization of FOR_ITER is worthwhile, if only for generators and coroutines. A specialization of FOR_ITER for generators can avoid consuming C stack in the same way as we currently do for calls to Python functions.
If we are going to specialize for generators anyway, additional specializations may be worthwhile.

A specialization for list iterators might be worthwhile.
Handling range is probably out of scope for PEP 659 specialization, as we want to perform scalar replacement of the iterator which is going to be a job for a higher tier optimizer.