Hardening: GC-safe lifecycle, 64-bit integers, exception-safe VFS, power-fail recovery - #41
Open
UKTailwind wants to merge 16 commits into
Open
Hardening: GC-safe lifecycle, 64-bit integers, exception-safe VFS, power-fail recovery#41UKTailwind wants to merge 16 commits into
UKTailwind wants to merge 16 commits into
Conversation
mp_obj_is_type(v, &mp_type_float) now trips a compile-time static assert (mp_type_assert_not_float) in MicroPython 1.29, since float may be a value-encoded type. Use mp_obj_is_float() instead. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The default allocator hands SQLite one gc_alloc() block per allocation,
so SQLite's structures live on MicroPython's GC heap. The conservative
collector cannot follow SQLite's interior/tagged pointers, so a
gc.collect() -- explicit, or automatic under memory pressure -- while a
connection is open frees live SQLite memory: SQLITE_CORRUPT ("malformed
database schema") or a hard hang. Bulk inserts die once auto-GC fires
under the accumulated garbage.
Switch to SQLite's MEMSYS5 pool (SQLITE_CONFIG_HEAP): one block the GC
never sub-collects, so gc.collect() is safe with a live connection. The
pool is reserved lazily on the first connect() -- a program that never
opens a database reserves nothing -- rooted via MP_REGISTER_ROOT_POINTER
so the GC keeps it. MEMSYS5_HEAP_SIZE defaults to a small 128 KB so it
fits constrained targets (a plain RP2040/ESP32 has only a few hundred KB
of RAM); a board with more RAM raises it, e.g. -DMEMSYS5_HEAP_SIZE=0x400000.
The page cache defaults to half the pool so it always fits inside it.
Also fix two bugs that stopped the MEMSYS5 branch working at all: the
undefined HEAP_SIZE (-> MEMSYS5_HEAP_SIZE), and the SQLITE_CONFIG_HEAP
minimum-allocation argument, which was 0 -- MEMSYS5 turns that into a
1-byte atom and cripples the buddy allocator; pass 64.
Verified on rp2 (RP2350, 8 MB PSRAM, 4 MB pool) and the unix port: an
8000-row insert with gc.collect() forced mid-transaction runs at flat
memory with no corruption.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
execute() registered every cursor in a strong-referenced list on the connection and removed it only in __del__; but cursors have no GC finaliser and are pinned by that list, so a loop of `con.execute(insert, row)` accumulated one prepared statement (~2 KB VDBE) per row until the connection closed, and the GC could never reclaim any of it. (With SQLite on the GC heap this also meant auto-GC fired ever sooner and corrupted the database.) Track a cursor in the connection list only while it holds a live statement -- idempotent register/deregister guarded by a `registered` flag -- and autoclose statements that return no rows: after the first step, if sqlite3_column_count()==0 (INSERT/UPDATE/DELETE/DDL) finalize the statement inline and drop the cursor. rowcount is captured first and lastrowid reads from the connection, so both survive on the returned cursor; SELECT cursors keep their statement for fetching. A finalized or result-less cursor now iterates empty and reports description=None instead of raising. Also fix connection.close(): it cleared cursors.items[0] on every iteration instead of items[i] and never reset len, leaving the list full of dangling pointers. With this (and the MEMSYS5 heap), a bulk insert in one transaction runs at flat memory with no chunk/close/reopen workaround -- ~3x faster on rp2 in testing (610 vs ~212 rows/s). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A Row is meant to be a tuple of the column values with one extra hidden slot holding the cursor, which usqlite_row_attr reads at items[len] to build .keys. Three bugs left it unusable (row_type defaults to tuple, so they went unnoticed): - the row factory never stamped usqlite_row_type on the object, so it was a plain tuple and .keys raised AttributeError; - it left len = columns+1, so the trailing cursor slot leaked into indexing / len / iteration, and .keys read items[columns+1] out of bounds; - keys() used sqlite3_data_count() (0 once the fetch has stepped past the row) instead of sqlite3_column_count(). Stamp the type, set len=columns (the block stays columns+1 wide so the GC still keeps the cursor referenced), and count columns with sqlite3_column_count(). Now `row.keys` returns the column-name tuple and `dict(zip(row.keys, row))` works, with the cursor slot hidden. Note .keys is a property, not a method. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
bindParameter() passed a NULL destructor to sqlite3_bind_text/blob, which is SQLITE_STATIC: SQLite keeps the caller-supplied pointer and does not copy. That pointer is into a MicroPython str/bytes object on the GC heap, so if the argument is transient -- built inline for the call and dropped afterwards -- the collector can free it while the prepared statement is still bound to it, and a later step() reads freed memory. Pass SQLITE_TRANSIENT so SQLite copies the bytes at bind time, matching what CPython's sqlite3 does and what callers expect. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
An open database file is a MicroPython stream object (io.open()), but the only reference to it lives in MPFILE.stream inside SQLite's sqlite3_file, which SQLite allocates from its own dedicated heap (MEMSYS5). The MicroPython GC does not walk SQLite's heap as object memory, so nothing strongly reachable keeps the stream alive: after a gc.collect() while a connection is open the stream could be reclaimed, and the next read/write would touch a freed object. It survived only by luck -- the collector conservatively scanning SQLite's heap and happening to spot the pointer -- which is memory-layout dependent and unreliable. Keep an explicit strong reference instead: pin every stream in a GC-visible list held from a MP_REGISTER_ROOT_POINTER on open, and swap-remove it on close. The unpin is pure C (no allocation, cannot raise) so it is safe on the close/finalize path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Every SELECT kept its prepared statement open, pinned by the connection's cursor list, until the cursor or the connection was closed. Because the list holds a strong reference the cursor is never garbage, so its __del__ never runs and gc.collect() cannot reclaim it -- a long-lived connection running many SELECTs without closing each cursor accumulated open statements in the fixed SQLite heap until it hit "out of memory". The idiomatic `for row in con.execute(sql): ...` leaked one statement per call (observed: OOM after ~2200 iterations on an RP2350B with a 4 MB heap). Free the statement the moment stepping reaches SQLITE_DONE (cursor_finish in stepExecute), and drop the cursor from the connection's list. rowcount is left untouched and the result-column names are cached first, so the cursor stays usable for rowcount/lastrowid/.keys afterwards. This also subsumes the previous "finalize result-less statements after execute" special case, which is removed. Fully-consumed cursors (for/list/fetchall/fetchmany, and empty or result-less statements) now run at flat memory; a partially fetched cursor that is dropped without close() still holds its statement until the connection closes, as before. Verified on an RP2350B (flash and SD): 6000 idiomatic-loop iterations at constant sqlite3_memory_used(), .keys intact on rows from an exhausted cursor, rowcount preserved, and PRAGMA integrity_check ok throughout. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Turn on SQLITE_DEFAULT_MEMSTATUS so sqlite3_memory_used() and sqlite3_memory_highwater() -- exposed as usqlite.mem_current() and usqlite.mem_peak() -- return real figures instead of 0. With the engine confined to a fixed dedicated heap this is the natural way to watch how full that heap is and to catch leaks. The cost is negligible here: SQLITE_THREADSAFE is 0, so it is an unlocked counter updated per malloc/free, not a contended atomic. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The page cache defaulted to half the dedicated pool (2 MB on a 4 MB board). SQLite ties the sorter's spill threshold to the cache, so a big cache let an unindexed ORDER BY / GROUP BY / index build accumulate ~3 MB of temp b-tree in RAM before spilling -- climbing to the edge of the pool and thrashing (a 20k-row report could hang the board until reset). The sorter already spills to temp files through the VFS; it just spilled far too late. Cap the cache low (~256 KB, and never more than 1/8 of a small pool) so the temp b-tree spills to disk early and stays bounded. Measured on an RP2350B (4 MB pool): the 20k-row join+group+order report that previously thrashed for 150s+ now completes in ~19s at ~530 KB peak -- a >6x drop in peak memory with no speed cost (spilling to flash is cheap; even on SD the same query went from thrash-to-reset to a clean 20s). Because large sorts are now bounded sub-MB rather than pool-sized, the pool no longer has to be large to run them. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
SQLite asks the VFS for anonymous temporary files -- the sorter's
merge-spill and temp b-trees for large ORDER BY / GROUP BY / DISTINCT /
index builds -- by calling xOpen with a NULL name and expecting the VFS to
invent one. usqlite_file_open dereferenced that NULL (strlen/strcpy),
hard-faulting the board: a big enough sort didn't fail gracefully, it
crashed the machine (leaving orphaned journals behind).
Give anonymous temp opens a unique name in a temp directory instead, so
the sorter actually spills to disk and the sort completes at bounded
memory. The name defaults to USQLITE_TEMP_DIR (the flash root) and honours
sqlite3_temp_directory, so PRAGMA temp_store_directory can redirect temp
files (e.g. to '/sd' to spare flash). SQLite sets SQLITE_OPEN_DELETEONCLOSE
on these, so the existing close path removes them -- no temp files leak.
Supporting changes:
- xAccess now answers the SQLITE_ACCESS_READWRITE probe (via os.stat)
so PRAGMA temp_store_directory validates instead of being rejected as
"not a writable directory". Existence probes still return 0, so
file-creation and journal behaviour are unchanged.
- xFullPathname guards against a NULL name and bounds the copy.
- temp files (fresh unique names, and root-level paths that confuse the
existence probe) skip the exists() check and are created write-new.
Verified in the emulator (same 4 MB dedicated heap as the board): a
cross-join producing a 7.7 MB sort that previously segfaulted now
completes at ~1.2 MB peak, repeatably, with no leaked temp files and
PRAGMA integrity_check ok; an unwritable temp dir now raises a catchable
OSError instead of crashing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Large sorts can spill several megabytes to a temp file, and the churn is
better aimed at the removable SD card than at the soldered flash. When
neither the caller (PRAGMA temp_store_directory) nor sqlite3_temp_directory
has chosen a location, prefer USQLITE_TEMP_SD ('/sd') if it is mounted,
falling back to the flash root (USQLITE_TEMP_DIR) otherwise -- so a board
with a card spares its flash automatically, and one without still works.
Verified in the emulator (same dedicated heap as the board): with /sd
mounted a 7.7 MB sort spills to /sd (nothing on flash) and completes,
cleaned up after; with no /sd it falls back to the flash root without
crashing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
SQLite C statics survive a soft reset while the MEMSYS5 pool dies with the heap, so the next session allocated from memory the new Python heap owned, corrupting it and hard-locking the board at the first close. initialize() now runs a full shutdown/configure/initialize cycle per session, keyed off a session marker that the module __init__ hook clears on the first import of each session. The hook matters: root pointers are NOT auto-zeroed on soft reset (mp_init never memsets VM state), so a root-pointer marker survives Ctrl-D exactly like a C static. The stale sqlite3_temp_directory pointer and the usqlite_heap/usqlite_files roots are reset the same way. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- connection.close() no longer m_free()s cursor objects Python may still reference (use-after-free); it finalizes their statements and lets the GC own the memory. cursor.close() always untracks; deregistration is a non-raising swap-remove; sqlite3_close_v2. - Connections and cursors are allocated with mp_obj_malloc_with_finaliser so their __del__ actually runs: a dropped connection now returns its pool memory on collect instead of leaking it for the session. - executemany copies and frees the SQLite error message before raising (the raise-then-free order leaked it in the fixed pool). - 64-bit INTEGER both ways (bind_int64/column_int64): binds over 2^31 raised OverflowError and reads silently wrapped -- fatal for epoch-ms timestamps and large ids. - .description no longer hard-faults on computed columns (NULL decltype) and works before the first fetch (column_count, not data_count). - A raising trace callback is swallowed (CPython semantics) instead of longjmping through sqlite3_step; expanded-sql buffer freed either way. - Repr-A-only pointer/mp_obj_t punning fixed; per-access qstr interning removed from the cursor attr handler. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Robustness: every VFS path that calls into Python (open, delete, the existence probe, stream close) is nlr-guarded and returns a SQLite error code -- an OSError (missing dir, full flash, SD card pulled) used to longjmp through the pager mid-operation. The existence probe is a single os.stat instead of an ilistdir walk (faster, never raises, and correct for root-level names, which used to resolve against the cwd and could truncate a database). File deletion refuses to run while the GC holds the heap locked, so a finaliser-driven close survives the sweep. Recovery: the VFS used to answer no to every xAccess existence probe, which silently disabled hot-journal detection -- a power cut mid-commit left a torn database presented as valid. Enabled as a set, each part required: honest xAccess; zero-filled short reads (VFS contract; recovery reads into lost tails); a real truncate-to-zero via reopen (locking-mode EXCLUSIVE finalizes the journal on every commit by truncation, and a successful-but-fake truncate plus honest detection would replay stale journals over committed data); SQLITE_OMIT_WAL (journal_mode=WAL would call the NULL xShm methods; saves 17 KB). xRandomness now actually fills its buffer instead of returning stack garbage. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Five host-side suites (run against the unix build, heap size matched to the target): lifecycle smoke, soft-reset session cycle, 64-bit ints + description + VFS errors, snapshot-based power-cut recovery, and the historical GC churn pattern. See tests/README.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
One branch carrying the full set of fixes we needed to make usqlite production-solid on a MicroPython appliance (RP2350 + 8 MB PSRAM, SQLite 3.47 amalgamation, MicroPython 1.29). Every fix is validated twice: on real hardware and on a unix build with the bundled regression suite (
tests/, 5 suites). Supersedes #38 and #40 and includes #39 — those PRs are subsets of this branch; happy to rebase/split however you prefer.Memory / GC safety
gc_allocper SQLite allocation: the conservative collector cannot follow SQLite's interior pointers, so anygc.collect()under an open connection could free live engine memory (issue RPI PICO is freezing #24 freezing, Possible memory leak #23 leak). Pool is lazily reserved on first connect, size overridable (MEMSYS5_HEAP_SIZE, default 128 KB).__init__hook (the runtime calls it on the first import of each session) clears the session state;initialize()then runs a fullsqlite3_shutdown()/configure/initialize cycle. Note: MicroPython root pointers are NOT auto-zeroed on soft reset, so a root-pointer guard alone is not enough — this one cost us a debugging odyssey.mp_obj_malloc_with_finaliser, so dropped-without-close objects are reclaimed.connection.close()no longer frees cursor objects Python may still reference (use-after-free); statements are finalized and the GC owns the objects. Statements are also finalized as soon as a cursor is exhausted, so long-lived connections do not accumulate VDBEs.SQLITE_TRANSIENT(wasSQLITE_STATICpointers into GC-owned buffers); open db streams pinned in a GC root;executemany's error message freed before raising.API correctness
sqlite3_bind_int64/sqlite3_column_int64) — was 32-bit: binds past 2^31 raised OverflowError, reads silently wrapped..descriptionno longer crashes on computed columns (strlen(NULL)decltype) and works before the first fetch;row_type="row"fixed (issue row_type="row" is unusable: .keys raises, cursor slot leaks into the tuple #37).xOpenno longer dereferences the NULL pathname; large sorts spill to disk (PR Spill large sorts to disk; cap the page cache for fixed-heap builds #40), and the default page cache is capped so the sorter spills early instead of thrashing a small pool.VFS: exception safety + power-fail recovery
io.open,os.remove, the existence probe, stream close) is nlr-guarded and returns a SQLite error code — an OSError (missing directory, full or removed media) used to longjmp through the pager mid-operation. Same for a raising trace callback.xAccessanswered "no" to every existence probe, so a hot journal left by a power cut was never detected and a torn database was served as valid. Enabled as a set (each part is required): honestxAccess(a singleos.stat), zero-filled short reads (VFS contract), a real truncate-to-zero via reopen (withSQLITE_DEFAULT_LOCKING_MODE=1the journal is finalized by truncation on every commit — honest detection with the old no-op truncate would replay stale journals over committed data), andSQLITE_OMIT_WAL(journal_mode=WAL would call the NULL xShm methods). Validated with snapshot-based crash tests plus physical power pulls mid-commit on hardware.xRandomnessactually fills its buffer now.🤖 Generated with Claude Code