Expose Py_SETREF and Py_XSETREF in the limited API

[ngoldbaum]

See python/cpython#99574 for the last time this came up.

Victor told me on Discord that he originally planned to re-propose this in 2023 but simply forgot.

I noticed these macros are missing while reviewing a NumPy PR that is part of the effort to make it possible to build NumPy under the limited API. The contributor replaced Py_SETREF calls with the "deadly" incorrect version that SETREF exists to avoid.

https://github.com/numpy/numpy/pull/32256/changes#diff-22d279e0a87948dce92784ff43557067e20213f6f4c06669a0a6315f80478f85L157-R161

[vstinner]

I'm +1 on adding Py_SETREF() and Py_XSETREF(). They help to write correct code managing reference counting.

There are documented at:

• https://docs.python.org/dev/c-api/refcounting.html#c.Py_SETREF
• https://docs.python.org/dev/c-api/refcounting.html#c.Py_XSETREF

Py_CLEAR() is already part of the limited C API.

Py_CLEAR(), Py_SETREF() and Py_XSETREF() are only macros, they are not implemented as function, so they would not be added to the stable ABI in practice.

[encukou]

As for other limited API macros, to guide non-C/C++ reimplementations the docs should include an implementation (so, a C23-only version with typeof is fine -- users of older C can just use the macro itself).

With that, +1 :)

Also, let's formally add all of them to stable_abi.toml, so they get the "Part of the Limited API" line in docs. That is, add the following to limited API 3.2+:

• Py_INCREF
• Py_XINCREF
• Py_DECREF
• Py_XDECREF
• Py_CLEAR

Hopefully that's not controversial.

• @encukou
• @erlend-aasland
• @serhiy-storchaka
• @vstinner
• @zooba

[vstinner]

As for other limited API macros, to guide non-C/C++ reimplementations the docs should include an implementation (so, a C23-only version with typeof is fine -- users of older C can just use the macro itself).

These macros are special and I'm not sure that it's a good idea to provide similar functions doing the same. There were multiple attempts to implement such functions and they all failed so far.

[ngoldbaum]

For PyO3/Rust, I think we'd probably implement Py_SETREF as:

#[inline]
pub unsafe fn Py_SETREF(op: *mut *mut PyObject, new: *mut PyObject) {
    let old = *op;
    *op = new;
    Py_DECREF(old);   // Py_XDECREF for Py_XSETREF
}

This is a function, so no need to worry about type safety or use __typeof__ or worry about C aliasing rules: the operands are PyObject pointers. Rust doesn't have the type punning concerns that make the C API want to abstract all the type punning into the macro.

Another neat aspect of this is that using *mut *mut instead of &mut *mut for the first argument avoids exactly the Rust-side re-entrancy concern that the SETREF macro exists in C to avoid.

So anyway, I personally think it's fine to just document the C23-ism macros and don't provide function variants at all. The functions just force callers to do the type punning that the macro exists partially to avoid.

Ping @davidhewitt in case you're interested.

[encukou]

To clarify, for the docs I want something along the lines of:

Py_SETREF(dst, src) expands roughly to the following, except it only evaluates each argument once and it doesn't need the C23 keyword typeof:

do {
    typeof(dst) old = dst;
    dst = src;
    Py_DECREF(old);
} while (0)

That's enough code to show how to use it and how to implement it in a non-C/C++ language. (I'm not too happy about the do/while(0), but it's easier to keep it than explain in prose.)