Adding `PyMethod_New` to stable ABI / limited API

[NicolasT]

When implementing, in some extension, a custom callable type similar to PyFunction_Type, one may want to set Py_TPFLAGS_METHOD_DESCRIPTOR in the tp_flags. In many such cases, the implementation of tp_descr_get is trivial and essentially that of PyFunction_Type:

     if (obj == Py_None || obj == NULL) {
          return Py_NewRef(func);
      }
      return PyMethod_New(func, obj);

PyMethod_New is currently not part of the stable ABI (I believe), and hence not part of the limited API. Since the functionality provided by PyMethod_Type is quite extensive, providing a local re-implementation of a similar type seems a bit wasteful (iff possible under limited API at all).

Assuming the prototype of PyMethod_New seems unlikely to change and only depends on PyObject * values, would the workgroup consider adding PyMethod_New to the stable ABI / limited API?

For reference, I believe nanobind has a local implementation of a similar type, though a cursory check seems to indicate it's less "advanced" than PyMethod_Type (e.g., it doesn't come with a __reduce__ method).

[encukou]

You should be able to do this by calling PyMethod_Type as a Python callable (e.g. with PyObject_CallFunction). (Perhaps we should document the arguments for types.MethodType, though.)

Is that not enough?

[NicolasT]

That'd work if PyMethod_Type were part of the limited API, which I don't believe it is? Though agree, if it'd be, that should be a good fit.

There's types.MethodType but triggering imports etc. from C code and then somehow getting a reference to that type object into the tp_descr_get callback isn't trivial.

A similar question applies to PyInstanceMethod_Type, by the way, which isn't exported to the Python layer at all (I believe).

[vstinner]

When implementing, in some extension, a custom callable type similar to PyFunction_Type, one may want to set Py_TPFLAGS_METHOD_DESCRIPTOR in the tp_flags. In many such cases, the implementation of tp_descr_get is trivial and essentially that of PyFunction_Type

I'm sorry, I don't understand well the use case. Is there a full example in the wild where I can look at the code to understand? Or is it possible to implement it in Python for the example?

For reference, I believe nanobind has a local implementation of a similar type

Do you have a URL to this nanobind code?

[da-woods]

Cython essentially does the same thing for our Cyfunction type. We set the Py_TPFLAGS_METHOD_DESCRIPTOR flag and then set tp_descr_get to something very close to the code given in the issue. See

https://github.com/cython/cython/blob/b8e835a322439ff79dd03108e542576753720047/Cython/Utility/CythonFunction.c#L1466

And

https://github.com/cython/cython/blob/b8e835a322439ff79dd03108e542576753720047/Cython/Utility/ObjectHandling.c#L2784

However, in the Limited API we have to go though a (cached) lookup in the types module and then call the method type instead.

[NicolasT]

@vstinner Sorry, I guess when I wrote "in some extension" this was misleading.

We're building an SDK to implement Python modules in OCaml. This is not based on code-generation (a-la Cython), but instead we populate module objects when imported at runtime (from OCaml code). We expose functions implemented in OCaml to Python by wrapping them ("as a value") in instances of a custom-built Python type (implemented in C), called OCamlFunction. Instances of this type (which carry a reference to the OCaml value of the implementation function) are callable (through the vectorcall interface, and tp_call using PyVectorcall_Call).

We also want to set the Py_TPFLAGS_METHOD_DESCRIPTOR flag in OCamlFunctions tp_flags, and hence need to provide an implementation for tp_descr_get. If PyMethod_Type is available, this is quite easy to achieve, but if it's not, implementing a type similar to PyMethod_Type for use from such tp_descr_get essentially implies duplicating PyMethod_Types implementation (since that one is very generic), which seems suboptimal (also because the interpreter seems to special-case PyMethod_Type instances in some cases).

For reference, the nanobind code (note our project is not related to nanobind at all) is at

• https://github.com/wjakob/nanobind/blob/73223b97e28909eb8fc7e83213c352c020a854f6/src/nb_internals.cpp#L138-L147
• https://github.com/wjakob/nanobind/blob/73223b97e28909eb8fc7e83213c352c020a854f6/src/nb_internals.cpp#L100C33-L100C52
• https://github.com/wjakob/nanobind/blob/73223b97e28909eb8fc7e83213c352c020a854f6/src/nb_func.cpp#L1341

[NicolasT]

@da-woods Thanks for the pointers! Indeed, that's roughly what we're doing as well (but breaking out of limited API to achieve this). The "cached import" would work but also feels a bit not-very-nice vs. having PyMethod_Type being part of the (limited) API.

[NicolasT]

I started https://discuss.python.org/t/adding-pymethod-new-and-or-pymethod-type-to-stable-abi-limited-api/108482 to gather more feedback.

[vstinner]

Thanks for Cython and nanobind examples, it helps me to understand the use case. So you're not the first one who has the issue, the limited C API implementation of Cython already has code to get types.MethodType at startup:
https://github.com/cython/cython/blob/b8e835a322439ff79dd03108e542576753720047/Cython/Utility/ObjectHandling.c#L2758-L2769

I'm not sure about exposing the PyMethod_Type type. It seems like exposing PyMethod_New() in the stable ABI would be enough.

So I'm +1 on adding PyMethod_New() to the stable ABI.

[NicolasT]

Thanks Victor! While loading types.MethodType is possible, that approach seems slightly unfortunate. I agree having PyMethod_New is "cleaner" (since it's an actual C API) than exposing PyMethod_Type and calling it.

I'll file an issue on the cpython project and send a PR to make this change, awaiting votes / a decision here.

[encukou]

-0 from me -- it feels like in limited API this should be a Python call, and we should focus on exposing the right callable. In my opinion a better way to solve this would be exposing a Py_GetBuiltinType_Borrow function, as discussed in #105. (Or even adding types to the set of constants for Py_GetConstantBorrowed?)

But I won't block adding PyMethod_New if I'm outvoted.

[NicolasT]

@encukou I'm a bit confused as to how stable ABI, the API and the intent of abi3(t) compatibility relate, then.

When the ABI/limited API exposes PyObject *PyMethod_New(PyObject *, PyObject *) this is fixed. As long as this symbol is provided, for that prototype, a built extension can be used in future versions of Python supporting abi3 (assuming no significant behavior changes).

When the ABI/limited API exposes PyTypeObject PyMethod_Type, there are much less guarantees: future versions may change the arguments its constructor takes, and hence an extension built today and PyObject_Calls PyMethod_Type may fail in future versions of Python. Unless the type being part of the stable ABI/limited API itself would imply its signature(s) can't change (which seems unlikely?) Though as you mentioned before, its constructor arguments are currently not even documented at all.

Of course, one could argue calling PyMethod_Type is alike importing some module (from C code), then getting some attribute from it and calling that ("writing Python in C"), and future changes to stdlib APIs could hence break modules built today, but that seems like... "more expected" vs. using plain values exposed by the C API?