feat: pluggable RangeJob sources for repackage (cdx / sql{athena,duckdb} / csv)#73

Draft

malteos wants to merge 10 commits into

feat/warc-by-cdxfrom

feat/warc-range-sources

Commits on Jun 12, 2026

feat: pluggable RangeJob sources for repackage (cdx/sql{athena,duckdb}/csv)
malteos
committed
feat: add --domains (url_host_registered_domain) guided filter
malteos
committed

Commits on Jun 17, 2026

feat: carry extra SQL --query columns into range-jobs CSV
malteos
committed
feat: sort range jobs by (warc_filename, warc_record_offset) for read locality
malteos
committed

Commits on Jun 19, 2026

add notes for warc fetcher performance
malteos
committed

Commits on Jun 20, 2026

feat(repackage): multi-process fetching with single merged WARC output
malteos
committed

Commits on Jun 22, 2026

Commits on Jun 24, 2026

perf(repackage): concurrent multipart merge for S3 shards
cdxt-bench
committed
fix(repackage): honor --size in multi-process mode (multiple output WARCs)
cdxt-bench
committed