Remove deep-freezing of code objects and modules.

[markshannon]

Deep-freezing, as implemented has a number of problems:

• It is slow to build
• It does not fit into the normal build system, make will not recreate deep-frozen modules
• Deep-freezing only makes sense if the objects are immutable, but code objects are not (at the VM level)
• It is a bit slower
• It gets in the way of other optimizations, notable faster loading from pyc files.

So let's remove it.

But let's keep the ability to deep-freeze objects.
The ability to deep-freeze objects into the executable is valuable, we already do it with strings, and deep-freezing whole immutable object graphs like the sys and builtin modules could improve start up considerably.

Linked PRs

• GH-108716: Turn off deep-freezing of modules. #108722
• gh-108716: make regen-global-objects no longer builds deepfreeze.c #110078
• gh-108716: Cleanup remaining deepfreeze infrastructure #116919
• gh-108716:: Remove _PyStaticCode_Init/Fini #117141

[malemburg]

Out of interest and unrelated to the other arguments you brought up: Why do you believe that removing deep-freeze for code objects slows down startup time ?

If I look at https://github.com/faster-cpython/benchmarking-public/blob/main/results/bm-20230827-3.13.0a0-1ad9bed/bm-20230827-linux-x86_64-faster%252dcpython-no_deep_freeze_3-3.13.0a0-1ad9bed-vs-3.11.0.md which measures removal of deep-freezing code objects against 3.11 which introduced this, the startup time has gone up by 18%.

The other differences and overall speedup shown in that benchmark are most likely not related to the deep-freeze change.

Background: I'm looking into using deep freeze for eGenix PyRun, which freezes (not deep freezes) most of the stdlib. I haven't done experiments yet (still trying to get things ported to 3.11, which is proving harder than usual due to the many CPython internal changes), so it'd be good to know whether it's worth starting into that adventure.

Thanks.

[markshannon]

The main advantage of freezing is having to avoid I/O finding and reading .pyc files.
Because code objects are now mutable at the VM level, having them deep frozen adds a fair bit of complexity to initialization and finalization.

We are looking to deep freeze more objects, though. Strings, and tuples of strings are the obvious candidates.

[erlend-aasland]

Is there more work to be done, or can we close this?

[markshannon]

There is more to be done with deep freezing in general, but this specific task is done.

[malemburg]

The main advantage of freezing is having to avoid I/O finding and reading .pyc files.

Actually, no 🙂 That's the main point of freezing modules using the freeze tool: you avoid I/O and reading PYC files by placing the byte code into a C array.

The purpose and intent of deep-freeze was to additionally avoid having to create the code objects from the byte code and put the ready to use objects into memory (again as a C array). So you skip the "read the byte code step" to create the objects and go straight to the objects. Well, at least that's what I had in mind when I discussed this with Guido some time ago.

Because code objects are now mutable at the VM level, having them deep frozen adds a fair bit of complexity to initialization and finalization.

Ok, mutable objects are probably not well stored in static C arrays 😉

We are looking to deep freeze more objects, though. Strings, and tuples of strings are the obvious candidates.

I'd add bytes to that list, since the byte code in code objects is stored as a bytes object, AFAIK.

In any case, if this logic will change again for 3.13, it's probably better to wait some more before experimenting with deep-freezing the stdlib as I had planned for eGenix PyRun in one of the next releases.

Thanks.