add support for abi3t

[Icxolu]

This adds abi3t support by

• switching to opaque structs
• exposing data access helpers from slots 369-378
• redirect all field accesses through the access helpers

Todo:

• changelog entry
• CI testing for 3.15/3.15t

See numpy/numpy#31091
Closes #555

[ngoldbaum]

Ping @kumaraditya303 - we were just talking about this today.

[neutrinoceros]

I confirmed that this resolved the runtime error I reported in #555
example logs (remaining failures are unrelated, they're just due to numpy not having cp315 nightlies for windows yet)

[kumaraditya303]

I think we should also add a check that when abi3t is used, the minimum numpy version should be 2.5.

[Icxolu]

Looks like 32bit windows still has some issues... not sure why though
There was also https://github.com/PyO3/rust-numpy/actions/runs/27866655836/job/82471861978 earlier

[neutrinoceros]

My guess is that'd be because there are no wheels for cp315*-*win32
xref numpy/numpy-release#53

[Icxolu]

To me it looks like there are cp315-win32 wheels here and they installed fine. The version specific run had no issues as well. It's just the abi3t run that failed, so I think this is a genuine failure, but I'm not sure which fault it could be: ours, pyo3, numpy or python

[kumaraditya303]

I think it could alignment issue on 32 bit MSVC, I am not very familiar with rust but it seems flags is defined as npy_uint64 which rust gives 8 bytes alignment whereas in C I think it is 4 bytes which could be an issue.

[kumaraditya303]

@Icxolu Can you try forcing a 4 byte alignment layout using #[cfg_attr(all(Py_LIMITED_API, Py_GIL_DISABLED), repr(C, packed(4)))]?

[ngoldbaum]

If you all don't sort this out in the meantime I'll try to look on Monday. While win32 isn't so important per se, I do find it useful to detect and sort out CPU architecture-sensitive issues. It's possible this issue is in any one of CPython, NumPy, or rust-numpy.

Glad to see everything mostly works otherwise!

[Icxolu]

I did a bit of digging with compiler explorer: https://godbolt.org/z/qYe3Ghz7a
Both rustc and msvc agree on both architectures on the structure size and that flags is at offset 16 (and thus 8 byte aligned as it should), so I think there is no issue with our definitions here. I did however also find this: rust-lang/rust#112480, which does not really paint a great picture about win32... perhaps we ran into that...

[ngoldbaum]

I think I know what the issue is.

PyArray_Descr_fields has a field that causes alignment issues on 32bit builds:

https://github.com/numpy/numpy/blob/1d656e454dea882b2f2a20ccdd93b02f3c7880c1/numpy/_core/include/numpy/ndarraytypes.h#L650

It's explicitly npy_uint64 - 8 bytes. In this case, it happens to lead to a field access with incorrect alignment and you get a segfault.

I think the fix needs to live in NumPy's headers. There probably needs to be a mirrored change here to ensure correct alignment for the flags field in PyArray_Descr_fields and _PyArray_LegacyDescr_fields. I'll have a PR open on NumPy shortly.

[ngoldbaum]

I opened numpy/numpy#31759

[ngoldbaum]

It turns out my first attempt at a fix had a flaw and I think the only real fix requires NumPy to add new a C API function to fix this issue: numpy/numpy#31762

[kumaraditya303]

I am looking at this on numpy side as well but the following diff fixes the win32 issues on rust side:

diff --git a/src/npyffi/objects.rs b/src/npyffi/objects.rs
index 6cd72ca..d3189f7 100644
--- a/src/npyffi/objects.rs
+++ b/src/npyffi/objects.rs
@@ -78,7 +78,14 @@ pub struct PyArray_DescrProto {
     pub hash: npy_hash_t,
 }
 
-#[repr(C)]
+#[cfg_attr(
+    all(target_arch = "x86", target_os = "windows", Py_LIMITED_API, Py_GIL_DISABLED),
+    repr(C, packed(4))
+)]
+#[cfg_attr(
+    not(all(target_arch = "x86", target_os = "windows", Py_LIMITED_API, Py_GIL_DISABLED)),
+    repr(C)
+)]
 pub struct _PyArray_DescrNumPy2 {
     #[cfg(not(all(Py_LIMITED_API, Py_GIL_DISABLED)))]
     pub ob_base: PyObject,
@@ -96,7 +103,14 @@ pub struct _PyArray_DescrNumPy2 {
     pub reserved_null: [*mut std::ffi::c_void; 2],
 }
 
-#[repr(C)]
+#[cfg_attr(
+    all(target_arch = "x86", target_os = "windows", Py_LIMITED_API, Py_GIL_DISABLED),
+    repr(C, packed(4))
+)]
+#[cfg_attr(
+    not(all(target_arch = "x86", target_os = "windows", Py_LIMITED_API, Py_GIL_DISABLED)),
+    repr(C)
+)]
 pub(crate) struct _PyArray_LegacyDescr_fields {
     #[cfg(not(all(Py_LIMITED_API, Py_GIL_DISABLED)))]
     pub ob_base: PyObject,

[kumaraditya303]

See https://github.com/kumaraditya303/rust-numpy/actions/runs/28233899128/job/83644222553, with the above change CI passes for windows 32 bit builds.

[Icxolu]

Thank you both for continuing to look into this, much appreciated ❤️. That patch seems to make it run indeed. What I don't really understand is, why this should work. If I got it correctly, then from numpy's definition there are padding bytes inserted to align the flags field on 32bit. If we now tell Rust to use a packed layout, wouldn't we read the padding bytes instead of the field? I'd also assume that this than effects all 32bit platforms, not just windows and not just x86, we just don't have any CI configured for other cases.

[ngoldbaum]

I think we need a similar fix on the NumPy side too.

This does affect all 32 bit platforms, I think.