Add C-level buffered array writer with direct file descriptor writes

[Copilot]

NPYConvert.to_py currently writes array payloads via Python np.nditer chunks and chunk.tobytes('C'), creating many short-lived Python objects. This change adds a C-extension path that streams buffered array bytes directly to file objects using OS-level writes when possible, while preserving C/Fortran write-order semantics used by NPY serialization.

• New C API for buffered file writes

  • Added write_array_to_file(array, file, *, fortran_order=False, buffersize=8192) to arraykit.
  • Implementation uses NumPy NpyIter (EXTERNAL_LOOP | BUFFERED | ZEROSIZE_OK).
  • For real file objects: Uses PyObject_AsFileDescriptor() to get the file descriptor and writes directly via C write() system call (_write() on Windows), completely eliminating Python method calls and object creation for the I/O path.
  • For file-like objects (BytesIO, StringIO, custom classes): Falls back to calling file.write() with PyMemoryView (zero-copy for contiguous data).
  • For non-unit inner strides, bytes are packed into a reusable contiguous buffer in C before writing.

• Module + typing surface

  • Registered and exported the new function through _arraykit and arraykit.__init__.
  • Added stub signature in __init__.pyi with matching defaults.

• Contract and edge handling

  • Preserves prior order behavior: Fortran iteration is used only when requested and the array is not C-contiguous.
  • Validates buffersize >= 1 with explicit error messaging.
  • Handles partial writes with explicit error reporting.
  • Properly checks PyErr_Occurred() after iteration (standard NumPy pattern).

Performance characteristics:

• Best case (contiguous data + real file): Pure C path with zero Python objects or method calls
• Good case (non-contiguous data + real file): Single reusable buffer + direct file descriptor writes
• Fallback (file-like objects): PyMemoryView (zero-copy) + Python method call

# before (Python-level per-chunk allocation pattern)
for chunk in np.nditer(array, flags=('external_loop', 'buffered', 'zerosize_ok'),
                       buffersize=buffersize, order='C'):
    file.write(chunk.tobytes('C'))

# after (C-level buffered streaming with direct fd writes in arraykit)
arraykit.write_array_to_file(array, file, fortran_order=False, buffersize=buffersize)

[Copilot]

Pull request overview

This PR adds a new C-extension API (write_array_to_file) to stream NumPy array bytes to a file-like object using NumPy’s iterator API, reducing Python-level per-chunk tobytes() allocations while preserving C/Fortran iteration semantics used by NPY-style serialization.

Changes:

• Implemented write_array_to_file(array, file, *, fortran_order=False, buffersize=8192) in the C extension using NpyIter with EXTERNAL_LOOP | BUFFERED | ZEROSIZE_OK.
• Exported the new function via _arraykit, arraykit.__init__, and added a matching stub in __init__.pyi.
• Added unit tests validating C-order and Fortran-order output against a Python np.nditer(...).tobytes('C') reference.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
test/test_util.py	Adds tests for write_array_to_file output and basic argument validation.
src/methods.h	Declares the new C-extension entry point.
src/methods.c	Implements buffered iteration + streaming writes via file.write(...).
src/_arraykit.c	Registers write_array_to_file in the module method table.
src/init.pyi	Adds typed stub signature for the new API.
src/init.py	Re-exports the new API from the top-level package.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[flexatone]

@copilot : In this implementation we are still calling Python methods on the Python file object, passing it Python byte objects. Is there any more direct way that we can write the bytes to the file, without creating Python Bytes objects or calling Python file methods?

[Copilot]

@flexatone Unfortunately I hit an unexpected error while processing your comment. I've automatically reported this to GitHub.

You can ask me to try again later by mentioning me in a new comment.

If you want to contact GitHub about this error, please mention the following identifier so they can better serve you: bc2dc4ae-9b35-4c70-9ed8-5e3743e8528c

Sorry for the inconvenience!

[Copilot]

@flexatone Unfortunately I hit an unexpected error while processing your comment. I've automatically reported this to GitHub.

You can ask me to try again later by mentioning me in a new comment.

If you want to contact GitHub about this error, please mention the following identifier so they can better serve you: bc2dc4ae-9b35-4c70-9ed8-5e3743e8528c

Sorry for the inconvenience!

[Copilot]

@flexatone Unfortunately I hit an unexpected error while processing your comment. I've automatically reported this to GitHub.

You can ask me to try again later by mentioning me in a new comment.

If you want to contact GitHub about this error, please mention the following identifier so they can better serve you: bc2dc4ae-9b35-4c70-9ed8-5e3743e8528c

Sorry for the inconvenience!

[Copilot]

@flexatone Unfortunately I hit an unexpected error while processing your comment. I've automatically reported this to GitHub.

You can ask me to try again later by mentioning me in a new comment.

If you want to contact GitHub about this error, please mention the following identifier so they can better serve you: bc2dc4ae-9b35-4c70-9ed8-5e3743e8528c

Sorry for the inconvenience!

[Copilot]

@flexatone Unfortunately I hit an unexpected error while processing your comment. I've automatically reported this to GitHub.

You can ask me to try again later by mentioning me in a new comment.

If you want to contact GitHub about this error, please mention the following identifier so they can better serve you: bc2dc4ae-9b35-4c70-9ed8-5e3743e8528c

Sorry for the inconvenience!

[Copilot]

@flexatone Unfortunately I hit an unexpected error while processing your comment. I've automatically reported this to GitHub.

You can ask me to try again later by mentioning me in a new comment.

If you want to contact GitHub about this error, please mention the following identifier so they can better serve you: bc2dc4ae-9b35-4c70-9ed8-5e3743e8528c

Sorry for the inconvenience!

[flexatone]

@copilot : In this implementation we are still calling Python methods on the Python file object, passing it Python byte objects. Is there any more direct way that we can write the bytes to the file, without creating Python Bytes objects or calling Python file methods?

[flexatone]

@copilot In this implementation we are still calling Python methods on the Python file object, passing it Python byte objects. Is there any more direct way that we can write the bytes to the file, without creating Python Bytes objects or calling Python file methods?

[Copilot]

@copilot In this implementation we are still calling Python methods on the Python file object, passing it Python byte objects. Is there any more direct way that we can write the bytes to the file, ...

Yes! I've optimized the implementation to use direct file descriptor writes when possible.

What changed:

• For real file objects (opened with open()), the code now uses PyObject_AsFileDescriptor() to get the underlying file descriptor and writes directly via the C write() system call (or _write() on Windows).
• This completely eliminates Python method calls and Python object creation for the file write path when using real files.
• For file-like objects without a file descriptor (e.g., BytesIO, StringIO, custom classes), the code gracefully falls back to calling file.write() with a PyMemoryView (which is zero-copy for contiguous data).

Performance improvements:

1. Best case (contiguous data + real file): Pure C path with zero Python objects or method calls
2. Good case (non-contiguous data + real file): Single reusable C buffer + direct fd writes
3. Fallback (file-like objects): PyMemoryView (zero-copy) + Python method call

The implementation also now properly handles partial writes and checks for iteration errors per the code review feedback.

Committed in [latest commit].