Skip to content

Improve Chunked Insert Performance - #167

Open
GALHP wants to merge 11 commits into
clonio-dev:mainfrom
GALHP:feat/improve-chunked-insert-performance
Open

Improve Chunked Insert Performance#167
GALHP wants to merge 11 commits into
clonio-dev:mainfrom
GALHP:feat/improve-chunked-insert-performance

Conversation

@GALHP

@GALHP GALHP commented Aug 13, 2026

Copy link
Copy Markdown

Summary

Replaces LIMIT/OFFSET chunk pagination with keyset (seek) pagination for
chunked reads, removing the deep-OFFSET scan cost that made large-table clones
slow down as they progressed.

Stacked PR — base: feat/progress-and-performance-tracking · 3rd of 6 ·
Part of #164. Depends on #165 — uses its per-phase timing model.

Motivation

A ~25M-row table cloned for hours and was aborted. #165's timings showed why:
SELECT … LIMIT ? OFFSET ? slows as the offset grows — the DB scans and
discards all skipped rows, so per-chunk select time climbs with the page number
regardless of chunk size. (Refs:
hackernoon,
mariadb,
readyset.)

Result: the same table now clones in ~7 min at constant pace, independent
of page number.

What's changed

CloningRunOrchestrator::buildChunkQuery() now builds a seek query — order
by a stable key, fetch the next chunk via a bound row-value comparison
(WHERE (k1, k2, …) > (?, …) ORDER BY … LIMIT n) instead of a growing OFFSET,
advancing a cursor from each chunk's last row. Applies to both the live-transfer
and dump paths.

  • Key selection (keysetColumns()): full seeks by PK; first/last seek
    by sort_by (default id) with PK columns appended as a tiebreaker for a
    total order. Direction follows the strategy (ASC/DESC); the row limit is
    still respected across chunks.
  • Fallback: when a total order isn't guaranteed — no PK or SQL Server
    — it transparently uses the original LIMIT/OFFSET (offsetChunkQuery()).
    Keyset covers MySQL, MariaDB, PostgreSQL, SQLite.
  • Adds a per-driver identifier-quoting helper for ordering columns.

Values are passed as bindings, not interpolated. Behavior is unchanged for
no-PK tables and SQL Server (old path); the speedup targets the common PK case.

Testing

CloningRunOrchestratorTest covers the keyset query construction and the OFFSET
fallback. Verified end-to-end that row output matches the previous behavior.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant