Sub-interpreter shared modules appear to be broken in Python 3.12

[ndjensen]

Shared modules appear to be broken in Python 3.12. It is presumed they are broken in later releases of Python too.

Here is some sample code that illustrates the problem about 50% of the time. It is inconsistent, sometimes it works, sometimes it doesn't.

import jep.Jep;
import jep.JepConfig;
import jep.NDArray;
import jep.SubInterpreter;

public class TestHgt2Pres {

    public static void main(String[] args) {
        float[] floats = new float[300_000];
        try (Jep j = new SubInterpreter(
                new JepConfig().addSharedModules("numpy"))) {
            j.set("testarg", new NDArray<>(floats, 300_000));
            j.exec("import numpy");
            j.exec("numpy.piecewise(testarg, [testarg < 5, testarg >= 5], [lambda z: z + 1, lambda z: z - 1])");
            j.exec("print('finished')");
        }
    }
}

The error that is produced by this example is:

Exception in thread "main" jep.JepException: <class 'TypeError'>: _abc_impl is set to a wrong type
	at <frozen abc>.__subclasscheck__(<frozen abc>:123)
	at <frozen abc>.__instancecheck__(<frozen abc>:119)
	at /home/njensen/tmp/python3.12-sub-interpreters/jep_venv/lib64/python3.12/site-packages/numpy/core/numeric.isscalar(numeric.py:1933)
	at /home/njensen/tmp/python3.12-sub-interpreters/jep_venv/lib64/python3.12/site-packages/numpy/lib/function_base.piecewise(function_base.py:734)
	at <string>.<module>(<string>:1)
	at jep.Jep.exec(Native Method)
	at jep.Jep.exec(Jep.java:384)
	at TestHgt2Pres.main(TestHgt2Pres.java:23)

The same code runs fine if you switch it to a shared interpreter. The code also works fine on Python 3.11. We have seen other unexplainable errors but this one is the most reproducible one we have narrowed down so far.

Based on the error message also showing up on this ticket, python/cpython#134591, and the ensuing discussion there, I think that CPython changed sub-interpreter internals that shared modules was relying on.

• OS Platform, Distribution, and Version: RHEL 8
• Python Distribution and Version: 3.12.13
• Java Distribution and Version: OpenJDK 17 and 21
• Jep Version: 4.3.1
• Python packages used (e.g. numpy, pandas, tensorflow): numpy

I tested with numpy 1.26.4 and numpy 2.4.6 and it made no difference.

[bsteffensmeier]

I've been able to replicate the problem using your test case. To ensure this isn't numpy specific I have also managed to create a test case that does not involve numpy:

    public static void main(String[] args) throws JepException {
        try (Jep j = new SharedInterpreter()) {
            j.exec("import abc, sys, types");
            j.exec("shared = types.ModuleType('shared')");
            j.exec("class NewClass(metaclass=abc.ABCMeta):\n  pass");
            j.exec("shared.NewClass = NewClass");
            j.exec("sys.modules['shared'] = shared");

            // Either of the next two lines fixes it in many cases.
            //j.exec("isinstance([], shared.NewClass)");
            j.exec("issubclass(list, shared.NewClass)");
        }
        try (Jep j = new SubInterpreter(
                new JepConfig().addSharedModules("shared"))) {
            j.exec("import shared");

            // isinstance is fixed by the subclass cache in abc.
            j.exec("isinstance([], shared.NewClass)");
            // but issubclass fails most of the time, not all, even though it uses the same cache.
            j.exec("issubclass(list, shared.NewClass)");

        }
        System.out.println("Done");
    }

The bug happens when an abstract class is created in a shared interpreter and then isinstance() or issubclass() is used in a sub interpreter. For abstract classes there's a cache of classes that are instances and populating that cache in the shared interpreter by calling isinstance() seems to partially "fix" the problem but not entirely.

The attribute lookup on the shared NewClass seems be doing weird things. For example in the sub interpreter NewClass.__subclasscheck__.__func__ is pointing at a different function than type(NewClass).__dict__['__subclasscheck__']. In the shared interpreter those are the same object and for instancecheck they are also the same object which partly explains why instancecheck seems to work in more cases than subclasscheck.

I traced through the cpython code to find where the attribute lookup might be going wrong and ended up in _PyType_Lookup() which maintains a per-interpreter cache of attributes on type objects. Disabling this cache fixes the problem. The cache uses a version tag in the key and (there was a change in Python 3.12 to make the version tag specific to sub-interpreters([https://github.com/python/cpython/pull/102343] so I am pretty sure that is what broke this. I don't understand exactly what is happening but it makes sense to me that using a cache with interpreter specific keys would lead to inconsistency when bringing objects from different interpreters

I don't see any way to keep shared modules working with the changes cpython is making to add extra layers of isolation to sub-interpreters. I think we should deprecate shared modules in Jep 4.4 and drop them entirely whenever we drop support for Python 3.11, which may be as soon as Jep 4.5. Hopefully now that (sub-interpreters are available in python)[https://peps.python.org/pep-0734/] it will provide incentive for more module maintainers to support sub-interpreters and workarounds like shared modules won't be necessary anymore but that is more of a long term outlook. Anyone using shared modules in jep will have to stick to Python 3.11 or move to SharedInterpreters.