Skip to content

chore: sqlite framework#2265

Open
SantiagoPittella wants to merge 17 commits into
nextfrom
santiagopittella-sqlite-framework-phase1
Open

chore: sqlite framework#2265
SantiagoPittella wants to merge 17 commits into
nextfrom
santiagopittella-sqlite-framework-phase1

Conversation

@SantiagoPittella

@SantiagoPittella SantiagoPittella commented Jun 19, 2026

Copy link
Copy Markdown
Collaborator

closes #2248
closes #2249
closes #1465

Summary

Introduces a light SQLite framework in crates/db (miden-node-db) and migrates the validator onto it, removing Diesel. It handles a connection pool, type-enforced read/write transactions, always-cached queries, a type codec, and cacheable IN-list helpers.

  • The framework keeps a small pool of connections alive and hands them out. Because SQLite is blocking, the actual query work runs on a background thread.
  • We dont handle raw connections. The struct provides either a read or a write transaction and get a small
    handle:
    • read(...) gives you a ReadTx. It begins a read-only transaction that is never committed, so
      nothing it does can persist.
    • write(...) gives you a WriteTx. It begins a write transaction.
  • Added verbs like query_rows, query_opt, query_one, exists, count and execute (write only). Every one of them prepares its statement through SQLite's cache.
  • Added a codec to convert between native data types and sql's.

The usage is like this:

// a read
db.read("chain_tip", |tx| {
    tx.query_opt(
        "SELECT block_header FROM block_headers ORDER BY block_num DESC LIMIT 1",
        &[],
        |row| row.get::<BlockHeader>(0),
    )
}).await?;

// a write (several statements in one atomic transaction)
db.write("apply", move |tx| {
    tx.execute("INSERT INTO ... VALUES (?1, ?2)", &[&a, &b])?;
    tx.execute("UPDATE ... SET ... WHERE id = ?1", &[&id])?;
    Ok::<(), DatabaseError>(())
}).await?;

Changelog

[[entry]]
scope       = "internal"
impact      = "added"
description = "Bespoke SQLite framework aimed at replacing Diesel"

[[entry]]
scope       = "internal"
impact      = "changed"
description = "Validator now uses our own SQLite framework instead of Diesel"

@SantiagoPittella SantiagoPittella force-pushed the santiagopittella-sqlite-framework-phase1 branch from 31dc8cc to a954f57 Compare June 19, 2026 15:36
Comment thread crates/db/src/sqlite/in_list.rs Outdated
Comment thread crates/db/src/sqlite/pool.rs Outdated
Comment thread crates/db/src/sqlite/pool.rs Outdated
Comment thread crates/db/src/sqlite/tx.rs Outdated
Comment thread crates/db/src/sqlite/tx.rs Outdated
Comment thread crates/db/src/lib.rs

@kkovaacs kkovaacs left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apart from the single nit I've added I think this looks good!

Thanks!

Comment thread crates/db/src/sqlite/in_list.rs Outdated
//! (so an index on the column can be used):
//!
//! ```sql
//! ... WHERE col IN (SELECT value FROM rarray(?1))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Neat!

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah super cool!

//! (so an index on the column can be used):
//!
//! ```sql
//! ... WHERE col IN (SELECT value FROM rarray(?1))

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah super cool!

Comment on lines +111 to +115
"INSERT INTO validated_transactions \
(id, block_num, account_id, account_patch, input_notes, output_notes, \
initial_account_hash, final_account_hash) \
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8) \
ON CONFLICT DO NOTHING",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should consider moving the sql into dedicated .sql files.

This allows for:

  • SQL syntax highlighting and LSP
  • linting
  • formatting
  • not fighting the rust indentations

This is slightly less relevant here, since we have so few; but I'm thinking a structure like:

bin/validator/src/db/queries
  /insert_transaction
    mod.rs
    tests.rs
    insert_transaction.sql
  /transaction_exists
    mod.rs
    tests.rs
    transaction_exists.sql
  /...

Doesn't need to be part of this PR though.

Comment on lines +129 to +139
fn in_memory() -> Connection {
let conn = Connection::open_in_memory().expect("open in-memory db");
// `rarray()` is provided by rusqlite's `array` extension, which must be loaded per
// connection (the pool does this in `configure_connection`).
rusqlite::vtab::array::load_module(&conn).expect("load array module");
conn.execute_batch(
"CREATE TABLE items (id INTEGER PRIMARY KEY, payload BLOB, label TEXT);",
)
.expect("create table");
conn
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would caution against this; there are some edge cases here that are non-obvious. See my impl from previous job.

Instead, I would suggest using tmpdir/tmpfile to create a concrete but temporary db.

path: PathBuf,
/// When set, connections are configured `PRAGMA query_only = ON` and skip the writer-only
/// `journal_mode` setup — used for the reader pool.
query_only: bool,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: read_only seems like a better name to me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Migrate validator off diesel SQLite: Design framework db/diesel: Explicitness of Transaction vs Connection for database queries

3 participants