Implement PEP 652 -- Maintaining the Stable ABI

[encukou]

BPO	43795
Nosy	@orsenthil, @vstinner, @encukou, @ambv, @willingc, @corona10, @pablogsal, @miss-islington, @erlend-aasland
PRs	bpo-43795: Sort PC/python3dll.c #25312 bpo-43795: Generate python3dll.c and doc data from manifest (PEP 652) #25315 bpo-43795: PEP-652: Clean up the stable ABI/limited API #25482 bpo-43795: PEP-652: Simplify headers for easier static analysis #25483 bpo-43795: PEP 652 user documentation #25668 bpo-43795 - Update Stable ABI dat. #25716 bpo-43795: Mark PyCodec_Unregister as a function, not data, in stable ABI #25920 [3.10] bpo-43795: Mark PyCodec_Unregister as a function, not data, in stable ABI (GH-25920) #25925 [3.10] bpo-43795: PEP 652 user documentation (GH-25668) #26034 bpo-43795: Reword paragraph on specific value for Py_LIMITED_API #26101 [3.10] bpo-43795: Reword paragraph on specific value for Py_LIMITED_API (GH-26101) #26123 bpo-43795: Remove Py_FrozenMain from the Limited API & Stable ABI  #26241 [3.10] bpo-43795: Remove Py_FrozenMain from the Limited API & Stable ABI (GH-26241) #26353 bpo-43795: Add a test for Stable ABI symbol availability using ctypes #26354 bpo-43795: Note Stable ABI PEP in What's New #26479 [3.10] bpo-43795: Note Stable ABI PEP in What's New (GH-26479) #26603 bpo-43795: Don't list private names in the limited API #26740 [3.10] bpo-43795: Don't list private names in the limited API (GH-26740) #26753 [3.10] bpo-43795: Add a test for Stable ABI symbol availability using ctypes (GH-26354) #29148 bpo-43795: Document stable_abi.txt format and contents #29956

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-04-09.15:43:50.940>
labels = ['3.10']
title = 'Implement PEP 652 -- Maintaining the Stable ABI'
updated_at = <Date 2021-12-10.23:34:50.985>
user = 'https://github.com/encukou'

bugs.python.org fields:

activity = <Date 2021-12-10.23:34:50.985>
actor = 'lukasz.langa'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2021-04-09.15:43:50.940>
creator = 'petr.viktorin'
dependencies = []
files = []
hgrepos = []
issue_num = 43795
keywords = ['patch']
message_count = 26.0
messages = ['390638', '391081', '391217', '391223', '391235', '391258', '391432', '391435', '391436', '391684', '391686', '392212', '392222', '392296', '393031', '393484', '393496', '393509', '393624', '394335', '395335', '395337', '395913', '395915', '404722', '408267']
nosy_count = 9.0
nosy_names = ['orsenthil', 'vstinner', 'petr.viktorin', 'lukasz.langa', 'willingc', 'corona10', 'pablogsal', 'miss-islington', 'erlendaasland']
pr_nums = ['25312', '25315', '25482', '25483', '25668', '25716', '25920', '25925', '26034', '26101', '26123', '26241', '26353', '26354', '26479', '26603', '26740', '26753', '29148', '29956']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue43795'
versions = ['Python 3.10']

[encukou]

This issue tracks implementation of PEP-652.

CPython's Limited C-API and Stable ABI, introduced in PEP-384, will be formalized in a single definitive file, tested, and documented.

[encukou]

New changeset 92eebf6 by Petr Viktorin in branch 'master':
bpo-43795: Sort PC/python3dll.c (GH-25312)
92eebf6

[encukou]

Opened as a separate issue: bpo-43868, Remove PyOS_ReadlineFunctionPointer from the stable ABI list

[encukou]

I unsure about the following symbols.
Victor, do you think they should be part of the stable ABI?

• PyThreadState_DeleteCurrent
• Py_GetArgcArgv
• PyInterpreterState_GetID

[vstinner]

PyThreadState_DeleteCurrent

No.

bpo-37878 explicitly excluded it from the limited C API. IMO it's too close to the CPython implementation and so must not be added to the stable ABI.

Py_GetArgcArgv

No.

Python 3.10 now provides sys.orig_argv which can be read in C.
https://docs.python.org/dev/library/sys.html#sys.orig_argv

Py_GetArgcArgv() is part of the "legacy" C API to initialize Python. The "new" PyConfig API provides PyConfig.orig_argv.

FYI in the PyPI top 4000 projects, the function seems to be called by the following projects:

2021-02-18/uWSGI-2.0.19.1.tar.gz
2021-02-18/stem-1.8.0.tar.gz
2021-02-18/sphinx-gallery-0.8.2.tar.gz
2021-02-18/setproctitle-1.2.2.tar.gz
2021-02-18/pyuwsgi-2.0.19.1.post0.tar.gz
2021-02-18/mercurial-5.7.tar.gz
2021-02-18/httpretty-1.0.5.tar.gz
2021-02-18/CherryPy-18.6.0.tar.gz
2021-02-18/aws-encryption-sdk-cli-2.1.0.tar.gz

PyInterpreterState_GetID

Yes.

It's part of the limited C API. IMO technically, it's ok to add to the stable ABI. I added it to PC/python3dll.c on purpose. Sadly, it's not available on PyPy, but I don't think that the stable ABI must care about PyPy which is a different use case.

[encukou]

Thanks! I'll send PRs later.

[encukou]

Thanks for linking to the issues/documentation!

I see that Py_GetArgcArgv it was exported in python3.def (i.e. stable ABI) in bpo-40910, which got backported to 3.9.

And the removal of PyThreadState_DeleteCurrent was reverted: bpo-38266

While we would like to deprecate/remove them in the future, they are now part of the stable ABI.
They are now in internal headers, so I will exclude them from the limited API.

[erlend-aasland]

Should we mention PEP-652 in Include/README.rst, now that the PEP is accepted?

[encukou]

Should we mention PEP-652 in Include/README.rst, now that the PEP is accepted?

No, we should link to the documentation (when it's written). The PEP is a design document; it'll become outdated.

[encukou]

New changeset e7cc64e by Petr Viktorin in branch 'master':
bpo-43795: PEP-652: Simplify headers for easier static analysis (GH-25483)
e7cc64e

[encukou]

New changeset 9d6a2d0 by Petr Viktorin in branch 'master':
bpo-43795: PEP-652: Clean up the stable ABI/limited API (GH-25482)
9d6a2d0

[vstinner]

Right now, running "make regen-limited-abi" adds again functions which were removed from Doc/data/stable_abi.dat:
#25687 (comment)

diff --git a/Doc/data/stable_abi.dat b/Doc/data/stable_abi.dat
index cdc7160250..6fe61743d6 100644
--- a/Doc/data/stable_abi.dat
+++ b/Doc/data/stable_abi.dat
@@ -351,11 +351,17 @@ PyMapping_Length
 PyMapping_SetItemString
 PyMapping_Size
 PyMapping_Values
+PyMarshal_ReadObjectFromString
+PyMarshal_WriteLongToFile
+PyMarshal_WriteObjectToFile
+PyMarshal_WriteObjectToString
 PyMem_Calloc
 PyMem_Free
 PyMem_Malloc
 PyMem_Realloc
 PyMemberDescr_Type
+PyMember_GetOne
+PyMember_SetOne
 PyMemoryView_FromMemory
 PyMemoryView_FromObject
 PyMemoryView_GetContiguous

I guess that you should not be added until this issue is solved.

[encukou]

I hope the PR fixes that. I plan to merge tomorrow if there ar no objections to it.

On April 28, 2021 5:15:19 PM GMT+02:00, STINNER Victor <report@bugs.python.org> wrote:

STINNER Victor <vstinner@python.org> added the comment:

Right now, running "make regen-limited-abi" adds again functions which
were removed from Doc/data/stable_abi.dat:
#25687 (comment)

diff --git a/Doc/data/stable_abi.dat b/Doc/data/stable_abi.dat
index cdc7160250..6fe61743d6 100644
--- a/Doc/data/stable_abi.dat
+++ b/Doc/data/stable_abi.dat
@@ -351,11 +351,17 @@ PyMapping_Length
PyMapping_SetItemString
PyMapping_Size
PyMapping_Values
+PyMarshal_ReadObjectFromString
+PyMarshal_WriteLongToFile
+PyMarshal_WriteObjectToFile
+PyMarshal_WriteObjectToString
PyMem_Calloc
PyMem_Free
PyMem_Malloc
PyMem_Realloc
PyMemberDescr_Type
+PyMember_GetOne
+PyMember_SetOne
PyMemoryView_FromMemory
PyMemoryView_FromObject
PyMemoryView_GetContiguous

I guess that you should not be added until this issue is solved.

----------

---

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue43795\>

---