Add PyMemoryView::from_owned_buffer for zero-copy memoryview creation by alex · Pull Request #5937 · PyO3/pyo3

alex · 2026-04-02T12:59:48Z

Adds a new method to create a Python memoryview that exposes a read-only view of byte data owned by a frozen PyClass instance without copying. This is useful for libraries like pyca/cryptography that need to expose internal buffers efficiently.

The method uses PyBuffer_FillInfo + PyMemoryView_FromBuffer to create a memoryview backed by the owner's data, with the owner kept alive via the buffer's obj reference.

Safety is enforced at compile time:

T: PyClass<Frozen = True> prevents mutation that could invalidate pointers
for<'a> FnOnce(&'a T) -> &'a [u8] ensures the slice borrows from T or is 'static

Closes #5871

https://claude.ai/code/session_01EEP1DaqJwHGCoNufi2JT9H

codspeed-hq · 2026-04-02T13:21:48Z

Merging this PR will not alter performance

✅ 105 untouched benchmarks
⏩ 1 skipped benchmark¹

_{Comparing alex:claude/implement-pyo3-api-5871-kzVaG (ddec6dd) with main (dd63a6a)}

1 benchmark was skipped, so the baseline result was used instead. If it was deleted from the codebase, click here and archive it to remove it from the performance reports. ↩

alex · 2026-04-02T23:14:35Z

netlify stuff looks unrelated?

davidhewitt · 2026-04-03T09:37:34Z

Indeed, hopefully #5939 will resolve that.

davidhewitt · 2026-04-03T12:19:49Z

#5941 might make the call_method_one_arg benchmark more stable...

alex · 2026-04-04T01:08:07Z

No luck with netlify...

davidhewitt · 2026-04-04T07:06:40Z

#5945 has finally got things green, one more rebase / merge should fix.

davidhewitt

Thanks for moving this forward and sorry to be a touch slow to review. Please forgive me for being slightly critical in comments, as per the OP hinting this was generated by Claude I do feel that the code has a somewhat AI-bloaty feel.

src/types/memoryview.rs

davidhewitt · 2026-04-13T10:46:10Z

tests/test_memoryview.rs

+        // Force GC to ensure the owner is kept alive by the memoryview
+        py.run(c"import gc; gc.collect()", None, None).unwrap();
+        let bytes: Vec<u8> = view.call_method0("tobytes").unwrap().extract().unwrap();
+        assert_eq!(bytes, b"kept alive");


We perhaps need to test the opposite, that after the MemoryView is dropped the owner is also freed.

src/types/memoryview.rs

davidhewitt · 2026-04-13T10:53:34Z

tests/test_memoryview.rs

I am generally trending towards having tests inside the library, gated on cfg(feature = "macros") where necessary. Perhaps can do the same here?

davidhewitt · 2026-04-13T10:57:29Z

src/types/memoryview.rs

+        // SAFETY: PyMemoryView_FromBuffer creates a memoryview that takes
+        // ownership of the buffer (it will call PyBuffer_Release when the
+        // memoryview is deallocated, which will decref view.obj).


Is this definitely the case? I don't see documentation or tests which demonstrate that view is stolen by the call.

I'm pretty sure it is with respect to the Py_buffer.... however in going back to triple check I spotted https://github.com/python/cpython/blob/main/Objects/memoryobject.c#L769-L788 which I'm now confused/concerned about the correctness.

Yup, turns out CPython throws away the owner 🙃. Going to file this upstream.

python/cpython#148493

Mmm, good catch! I guess possibly also relevant is https://github.com/python/cpython/blob/d24ce178a2a413d3b5bb032264434e8cb416d8fe/Objects/memoryobject.c#L832 - the implementation of __buffer__ wrapper for bf_getbuffer implementations.

That might hint that an alternative implementation we could use is to support __buffer__ in #[pymethods] (we would like that anyway, see #3148). We could perhaps allow two signatures:

low-level signature equivalent to our current __getbuffer__ method (or maybe with small tweaks)

fn __buffer__(&self) -> &[u8], i.e. the equivalent of what's here, but packaged up to call PyBuffer_FillInfo somewhere in PyO3 machinery

src/types/memoryview.rs

davidhewitt · 2026-04-13T10:59:34Z

pytests/src/buf_and_str.rs

I'm not so convinced there's value in adding tests here if we already added tests at the Rust level.

Adds a new method to create a Python memoryview that exposes a read-only view of byte data owned by a frozen PyClass instance without copying. This is useful for libraries like pyca/cryptography that need to expose internal buffers efficiently. The method uses PyBuffer_FillInfo + PyMemoryView_FromBuffer to create a memoryview backed by the owner's data, with the owner kept alive via the buffer's obj reference. Safety is enforced at compile time: - T: PyClass<Frozen = True> prevents mutation that could invalidate pointers - for<'a> FnOnce(&'a T) -> &'a [u8] ensures the slice borrows from T or is 'static Closes PyO3#5871 https://claude.ai/code/session_01EEP1DaqJwHGCoNufi2JT9H

alex · 2026-04-13T11:34:39Z

Thanks for the review -- sorry I hadn't caught the sloppy comments prior to putting this up.

I think this may be on pause while I work through the CPython bug, or figure out an alternative path. Which is frustrating, because conceptually the buffer machinery supports everything we need for this...

alex force-pushed the claude/implement-pyo3-api-5871-kzVaG branch 2 times, most recently from ac672ac to 800c7dc Compare April 2, 2026 13:05

alex force-pushed the claude/implement-pyo3-api-5871-kzVaG branch 2 times, most recently from 710f6b2 to 2353d5d Compare April 2, 2026 23:12

alex force-pushed the claude/implement-pyo3-api-5871-kzVaG branch from 2353d5d to fba259b Compare April 3, 2026 12:27

alex force-pushed the claude/implement-pyo3-api-5871-kzVaG branch 2 times, most recently from 4de5647 to 7f86fc8 Compare April 4, 2026 18:31

davidhewitt reviewed Apr 13, 2026

View reviewed changes

alex force-pushed the claude/implement-pyo3-api-5871-kzVaG branch from 7f86fc8 to ddec6dd Compare April 13, 2026 11:22

alex mentioned this pull request Apr 13, 2026

PyMemoryView_FromBuffer discards owner reference from Py_buffer python/cpython#148493

Open

Conversation

alex commented Apr 2, 2026

Uh oh!

codspeed-hq bot commented Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Footnotes

Uh oh!

alex commented Apr 2, 2026

Uh oh!

davidhewitt commented Apr 3, 2026

Uh oh!

davidhewitt commented Apr 3, 2026

Uh oh!

alex commented Apr 4, 2026

Uh oh!

davidhewitt commented Apr 4, 2026

Uh oh!

davidhewitt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alex commented Apr 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codspeed-hq bot commented Apr 2, 2026 •

edited

Loading