PEP 793, 803, 820: Update & mark as Final

Author: encukou

peps/pep-0793.rst

@@ -2,14 +2,16 @@ PEP: 793
 Title: PyModExport: A new entry point for C extension modules
 Author: Petr Viktorin <encukou@gmail.com>
 Discussions-To: https://discuss.python.org/t/93444
-Status: Accepted
+Status: Final
 Type: Standards Track
 Created: 23-May-2025
 Python-Version: 3.15
 Post-History: `14-Mar-2025 <https://discuss.python.org/t/84498/>`__,
               `27-May-2025 <https://discuss.python.org/t/93444/>`__,
 Resolution: `23-Oct-2025 <https://discuss.python.org/t/93444/46>`__
 
+.. canonical-doc:: :ref:`py3.15:extension-modules`
+
 
 Abstract
 ========
@@ -21,7 +23,7 @@ This allows extension authors to avoid using a statically allocated
 ``PyObject``, lifting the most common obstacle to making one compiled library
 file usable with both regular and free-threaded builds of CPython.
 
-To make this viable, we also specify new module slot types to replace
+To make this viable, we also specify new module slot IDs to replace
 ``PyModuleDef``'s fields, and to allow adding a *token* similar to the
 ``Py_tp_token`` used for type objects.
 
@@ -131,8 +133,8 @@ This proposal does away with fixed fields and proposes using a slots array
 directly, without a wrapper struct.
 
 The ``PyModuleDef_Slot`` struct does have some downsides compared to fixed fields.
-We believe these are fixable, but leave that out of scope of this PEP
-(see “Improving slots in general” in the Possible Future Directions section).
+We believe these are fixable, but leave that out of scope of this PEP.
+(Note: this was done in :pep:`820`, still in Python 3.15.)
 
 
 Tokens
@@ -187,6 +189,8 @@ like this:
 
     PyModuleDef_Slot *PyModExport_<NAME>(void);
 
+.. note:: :pep:`820` changed the return type to ``PySlot *``.
+
 where ``<NAME>`` is the name of the module.
 For non-ASCII names, it will instead look for ``PyModExportU_<NAME>``,
 with ``<NAME>`` encoded as for existing ``PyInitU_*`` hooks
@@ -223,14 +227,13 @@ A new function will be added to create a module from an array of slots:
 
     PyObject *PyModule_FromSlotsAndSpec(const PyModuleDef_Slot *slots, PyObject *spec)
 
+.. note:: :pep:`820` changed the first argument type to ``PySlot *``.
+
 The *slots* argument must point to an array of ``PyModuleDef_Slot`` structures,
 terminated by a slot with ``slot=0`` (typically written as ``{0}`` in C).
-There are no required slots, though *slots* must not be ``NULL``.
-It follows that minimal input contains only the terminator slot.
-
-.. note::
-
-   If :pep:`803` is accepted, the ``Py_mod_abi`` slot will be mandatory.
+The ``Py_mod_abi`` slot is required (see :pep:`803`); all other slots
+are optional.
+It follows that *slots* must not be ``NULL``.
 
 The *spec* argument is a duck-typed ModuleSpec-like object, meaning that any
 attributes defined for ``importlib.machinery.ModuleSpec`` have matching
@@ -373,7 +376,7 @@ Bits & Pieces
 -------------
 
 A ``PyMODEXPORT_FUNC`` macro will be added, similar to the ``PyMODINIT_FUNC``
-macro but with ``PyModuleDef_Slot *`` as the return type.
+macro but with ``PySlot *`` as the return type.
 
 A ``PyModule_GetStateSize`` function will be added to retrieve the size set
 by ``Py_mod_state_size`` or ``PyModuleDef.m_size``.
@@ -397,6 +400,9 @@ The ``PyInit_*`` export hook will be
 
 New API summary
 ---------------
+
+.. note:: This summary was updated with a change from :pep:`820`.
+
 Python will load a new module export hook, with two variants:
 
 .. code-block:: c
@@ -408,7 +414,7 @@ The following functions will be added:
 
 .. code-block:: c
 
-    PyObject *PyModule_FromSlotsAndSpec(const PyModuleDef_Slot *, PyObject *spec)
+    PyObject *PyModule_FromSlotsAndSpec(const PySlot *, PyObject *spec)
     int PyModule_Exec(PyObject *)
     int PyModule_GetToken(PyObject *, void**)
     PyObject *PyType_GetModuleByToken(PyTypeObject *type, const void *token)
@@ -477,6 +483,15 @@ Here is a guide to convert an existing module to the new API, including
 some tricky edge cases.
 It should be moved to a HOWTO in the documentation.
 
+.. note::
+
+   The guide is available at :ref:`py3.15:abi3t-howto-modexport`.
+   (It is part of the ``abi3t`` migration HOWTO since switching to
+   ``PyModExport`` doesn't bring benefits in 3.15 if you don't also
+   adopt ``abi3t``.)
+
+   This section contains the original, outdated guide.
+
 This guide is meant for hand-written modules. For code generators and language
 wrappers, the :ref:`pep793-shim` below may be more useful.
 
@@ -585,24 +600,29 @@ The following implementation can be copied and pasted to a project; only the
 names ``PyInit_examplemodule`` (twice) and ``PyModExport_examplemodule`` should
 need adjusting.
 
-When added to the :ref:`pep793-example` below and compiled with a
-non-free-threaded build of this PEP's reference implementation, the resulting
-extension is compatible with non-free-threading 3.9+ builds, in addition to a
+.. note::
+
+   This section was updated for :pep:`820`.
+
+When compiled together with the :ref:`pep793-example` below on a
+non-free-threaded build of Python 3.15, the resulting
+extension is compatible with non-free-threading 3.11+ builds, in addition to a
 free-threading build of the reference implementation.
 (The module must be named without a version tag, e.g. ``examplemodule.so``,
 and be placed on ``sys.path``.)
 
 Full support for creating such modules will require backports of some new
 API, and support in build/install tools. This is out of scope of this PEP.
 (In particular, the demo “cheats” by using a subset of Limited API 3.15 that
-*happens to work* on 3.9; a proper implementation would use Limited API 3.9
-with backport shims for new API like ``Py_mod_name``.)
+*happens to work* on 3.11, and includes a few hacks.
+A proper implementation would use Limited API 3.11 with cleaner backport shims
+for new API like ``Py_mod_name``.)
 
 This implementation places a few additional requirements on the slots array:
 
-- Slots that correspond to ``PyModuleDef`` members must come first.
+- ``Py_mod_slots`` and ``Py_slot_subslots`` are not supported.
 - A ``Py_mod_name`` slot is required.
-- Any ``Py_mod_token`` must be set to ``&module_def_and_token``, defined here.
+- Any ``Py_mod_token`` must be set to the ``MOD_TOKEN`` defined here.
 
 .. literalinclude:: pep-0793/shim.c
    :language: c
@@ -626,6 +646,10 @@ be added as a new HOWTO.
 Example
 =======
 
+.. note::
+
+   The example was updated for :pep:`820`.
+
 .. literalinclude:: pep-0793/examplemodule.c
    :language: c
 
@@ -693,6 +717,10 @@ These ideas are out of scope for *this* proposal.
 Improving slots in general
 --------------------------
 
+.. note::
+
+   This idea was implemented in :pep:`820`.
+
 Slots -- and specifically the existing ``PyModuleDef_Slot`` -- do have a few
 shortcomings. The most important are:
 

peps/pep-0793/examplemodule.c

@@ -35,7 +35,13 @@ typedef struct {
     int value;
 } examplemodule_state;
 
-static PyModuleDef_Slot examplemodule_slots[];
+static PySlot examplemodule_slots[];
+
+#ifndef MOD_TOKEN
+// Module token: normally set to the slots array,
+// but a backwards-compatibility shim will redefine it.
+#define MOD_TOKEN (&examplemodule_slots)
+#endif
 
 // increment_value function
 
@@ -59,20 +65,21 @@ exampletype_repr(PyObject *self)
 {
     /* To get module state, we cannot use PyModule_GetState(Py_TYPE(self)),
      * since Py_TYPE(self) might be a subclass defined in an unrelated module.
-     * So, use PyType_GetModuleByToken.
+     * So, we should use use PyType_GetModuleByToken.
+     * For pre-3.15 compatibility, we use PyType_GetModuleByDef instead:
+     * this needs a cast and returns a borrowed reference.
      */
-    PyObject *module = PyType_GetModuleByToken(
-        Py_TYPE(self), examplemodule_slots);
+    PyObject *module = PyType_GetModuleByDef(
+        Py_TYPE(self), (PyModuleDef*)MOD_TOKEN);
     if (!module) {
         return NULL;
     }
     examplemodule_state *state = PyModule_GetState(module);
-    Py_DECREF(module);
     if (!state) {
         return NULL;
     }
-    return PyUnicode_FromFormat("<%T object; module value = %d>",
-                                self, state->value);
+    return PyUnicode_FromFormat("<ExampleType object; module value = %d>",
+                                state->value);
 }
 
 static PyType_Spec exampletype_spec = {
@@ -105,13 +112,17 @@ examplemodule_exec(PyObject *module) {
 
 PyDoc_STRVAR(examplemodule_doc, "Example extension.");
 
-static PyModuleDef_Slot examplemodule_slots[] = {
-    {Py_mod_name, "examplemodule"},
-    {Py_mod_doc, (char*)examplemodule_doc},
-    {Py_mod_methods, examplemodule_methods},
-    {Py_mod_state_size, (void*)sizeof(examplemodule_state)},
-    {Py_mod_exec, (void*)examplemodule_exec},
-    {0}
+PyABIInfo_VAR(abi_info);
+
+static PySlot examplemodule_slots[] = {
+    PySlot_STATIC_DATA(Py_mod_abi, &abi_info),
+    PySlot_STATIC_DATA(Py_mod_name, "examplemodule"),
+    PySlot_STATIC_DATA(Py_mod_doc, (char*)examplemodule_doc),
+    PySlot_STATIC_DATA(Py_mod_methods, examplemodule_methods),
+    PySlot_SIZE(Py_mod_state_size, sizeof(examplemodule_state)),
+    PySlot_FUNC(Py_mod_exec, examplemodule_exec),
+    PySlot_STATIC_DATA(Py_mod_token, MOD_TOKEN),
+    PySlot_END
 };
 
 // Avoid "implicit declaration of function" warning:

peps/pep-0793/shim.c

@@ -1,57 +1,91 @@
-#include <string.h>     // memset
+#include <Python.h>
 
-PyMODINIT_FUNC PyInit_examplemodule(void);
+// Hack: Restore old definition of Py_TYPE
+#undef Py_TYPE
+#define Py_TYPE(OBJ) (((PyObject*)OBJ)->ob_type)
 
+// PyModuleDef, also reused as module token
 static PyModuleDef module_def_and_token;
+#define MOD_TOKEN (&module_def_and_token)
+
+#include "examplemodule.c"
+
+extern PySlot *PyModExport_examplemodule(void);
+
+PyMODINIT_FUNC PyInit_examplemodule(void);
 
 PyMODINIT_FUNC
 PyInit_examplemodule(void)
 {
-    PyModuleDef_Slot *slot = PyModExport_examplemodule();
-
     if (module_def_and_token.m_name) {
         // Take care to only set up the static PyModuleDef once.
         // (PyModExport might theoretically return different data each time.)
         return PyModuleDef_Init(&module_def_and_token);
     }
-    int copying_slots = 1;
-    for (/* slot set above */; slot->slot; slot++) {
-        switch (slot->slot) {
+
+    static PyModuleDef_Slot module_slots[5] = {{0}};
+    module_def_and_token.m_slots = module_slots;
+    int current_m_slot = 0;
+
+    PySlot *slot = PyModExport_examplemodule();
+
+    for (/* slot set above */; slot->sl_id; slot++) {
+        switch (slot->sl_id) {
         // Set PyModuleDef members from slots. These slots must come first.
-#       define COPYSLOT_CASE(SLOT, MEMBER, TYPE)                            \
-            case SLOT:                                                      \
-                if (!copying_slots) {                                       \
-                    PyErr_SetString(PyExc_SystemError,                      \
-                                    #SLOT " must be specified earlier");    \
-                    goto error;                                             \
-                }                                                           \
-                module_def_and_token.MEMBER = (TYPE)(slot->value);          \
-                break;                                                      \
-            /////////////////////////////////////////////////////////////////
-        COPYSLOT_CASE(Py_mod_name, m_name, char*)
-        COPYSLOT_CASE(Py_mod_doc, m_doc, char*)
-        COPYSLOT_CASE(Py_mod_state_size, m_size, Py_ssize_t)
-        COPYSLOT_CASE(Py_mod_methods, m_methods, PyMethodDef*)
-        COPYSLOT_CASE(Py_mod_state_traverse, m_traverse, traverseproc)
-        COPYSLOT_CASE(Py_mod_state_clear, m_clear, inquiry)
-        COPYSLOT_CASE(Py_mod_state_free, m_free, freefunc)
+#       define COPYSLOT_CASE(SLOT, DEF_MEMBER, SL_MEMBER, TYPE)               \
+            case SLOT:                                                        \
+                if (slot->sl_flags & PySlot_INTPTR) {                         \
+                    module_def_and_token.DEF_MEMBER = (TYPE)(slot->sl_ptr);   \
+                } else {                                                      \
+                    module_def_and_token.DEF_MEMBER = (TYPE)(slot->SL_MEMBER);\
+                }                                                             \
+                break;                                                        \
+            ///////////////////////////////////////////////////////////////////
+        COPYSLOT_CASE(Py_mod_name, m_name, sl_ptr, char*)
+        COPYSLOT_CASE(Py_mod_doc, m_doc, sl_ptr, char*)
+        COPYSLOT_CASE(Py_mod_state_size, m_size, sl_size, Py_ssize_t)
+        COPYSLOT_CASE(Py_mod_methods, m_methods, sl_ptr, PyMethodDef*)
+        COPYSLOT_CASE(Py_mod_state_traverse, m_traverse, sl_func, traverseproc)
+        COPYSLOT_CASE(Py_mod_state_clear, m_clear, sl_func, inquiry)
+        COPYSLOT_CASE(Py_mod_state_free, m_free, sl_func, freefunc)
+        COPYSLOT_CASE(Py_mod_slots, m_slots, sl_ptr, PyModuleDef_Slot*)
+#       undef COPYSLOT_CASE
+        case Py_mod_create:
+        case Py_mod_exec:
+        case Py_mod_multiple_interpreters:
+        case Py_mod_gil:
+            int old_slot_id = (int)slot->sl_id;
+            if (old_slot_id > 83) {
+                // Hack: slots were renumbered; use old IDs here
+                old_slot_id -= 83;
+            }
+            module_slots[current_m_slot].slot = old_slot_id;
+            module_slots[current_m_slot].value = slot->sl_ptr;
+            current_m_slot++;
+            if (current_m_slot >= 4) {
+                PyErr_SetString(PyExc_SystemError,
+                                "Too many slots for array");
+                goto error;
+            }
+            break;
         case Py_mod_token:
             // With PyInit_, the PyModuleDef is used as the token.
-            if (slot->value != &module_def_and_token) {
+            if (slot->sl_ptr != &module_def_and_token) {
                 PyErr_SetString(PyExc_SystemError,
                                 "Py_mod_token must be set to "
                                 "&module_def_and_token");
                 goto error;
             }
             break;
+        case Py_mod_abi:
+            // ABI checking skipped here
+            break;
         default:
-            // The remaining slots become m_slots in the def.
-            // (`slot` now points to the "rest" of the original
-            //  zero-terminated array.)
-            if (copying_slots) {
-                module_def_and_token.m_slots = slot;
+            if (!(slot->sl_flags & PySlot_OPTIONAL)) {
+                PyErr_Format(PyExc_SystemError,
+                             "Unknown slot ID %d.", (int)slot->sl_id);
+                goto error;
             }
-            copying_slots = 0;
             break;
         }
     }
@@ -63,6 +97,6 @@ PyInit_examplemodule(void)
     return PyModuleDef_Init(&module_def_and_token);
 
 error:
-    memset(&module_def_and_token, 0, sizeof(module_def_and_token));
+    module_def_and_token.m_name = NULL;
     return NULL;
 }