Add a validate subcommand for post-export sanity checks

[gaurav]

Summary

Adds pubmed2db validate <dir>, which inspects a finished JSON export (a
directory of NDJSON shards) and writes an archivable, gated
validation_report.json. It answers "does this export make sense?" after an HPC
run. Targets initial-implementation so it can be reviewed after the base branch.

Five checks, split into an offline phase (fast, deterministic) and an
online phase (Entrez eutils cross-checks; skip with --offline):

1. structure — every line parses as JSON and matches the exporter's exact
   10-field record shape; flags malformed lines, missing/extra fields, nulls,
   bad ids, invalid months, and cross-shard duplicate PMIDs. Per-shard
   reservoir sampling keeps memory bounded.
2. coverage — exported count vs. two denominators: the live Entrez total
   (einfo) and the local latest_article count (a shortfall is an error —
   rows were dropped); plus drift vs. --previous-report.
3. field_validation — a seeded sample re-fetched via batched efetch and
   compared field-by-field (fuzzy abstract via difflib). Journal name/abbrev
   are warning-only (different source); a sampled PMID PubMed no longer serves is
   an error.
4. deletions — samples DB deleted_pmids not reinstated by a later version,
   confirms they're absent from the export and gone from PubMed.
5. drops_since_previous — diffs this export's PMID set against a previous
   export's manifest (see below).

PMID manifest sidecar

Coverage counts can't detect two same-sized exports whose PMID sets differ, so
--manifest writes a sorted gzipped pmids.txt.gz and --previous-manifest
diffs against an earlier one:

uv run pubmed2db validate data/json --manifest data/json/pmids.txt.gz
# next month
uv run pubmed2db validate data/json-new \
    --previous-manifest data/json/pmids.txt.gz --manifest data/json-new/pmids.txt.gz

A drop the deleted_pmid table explains is expected; an unexplained drop is
an error — records were lost rather than retired. Without a database the drops
can't be attributed, so they degrade to a warning. The manifest is written from
the PMID set check_structure already holds, so it costs a sort and a write
rather than another pass over the shards.

Report & gating

The report leads with errors/warnings arrays that are empty on a clean
run; the stdout summary is quiet on success and loud on findings. Exit is
non-zero on errors (--fail-on-warn extends to warnings) so it can gate a
pipeline. The DuckDB database, a previous report, and a previous manifest are all
optional inputs used when present and left blank (noted in skipped_checks)
when not.

Coverage band is calibrated, not guessed

The default --entrez-low/--entrez-high band is ±5%, derived from a real
full-corpus run: the 2026-07-30 export held 40,901,984 documents against a live
Entrez total of 40,944,369 — a ratio of 0.9990. The band absorbs Entrez
growth between export and validation (PubMed adds roughly 4% a year) while still
catching a materially short export.

Note for reviewers: a partial export (from a --limit test download) is
legitimately far below the band and will warn. Pass --entrez-low 0.001 or
--offline when validating one. This is documented in the README.

Notes for review

• All network funnels through one validate._eutils seam (rate-limited,
  retrying), which tests monkeypatch — the suite stays fully offline.
• "Expected" is always defined by the exporter, never restated: month_to_abbrev
  is imported from export, and EXPECTED_FIELDS is derived by calling
  export._document on a placeholder row, so the record shape can't drift.
  test_expected_fields_matches_spec additionally locks the ten field names,
  since they're an external contract with Node Annotator / ElasticSearch.
• validate uses the group-level _connect, so --threads/--temp-dir apply
  to it too — it reads latest_article over a 40M-row database, which is exactly
  where the spill directory matters.
• Deletion status still treats "efetch returns nothing" as deleted; labelling
  merges via esummary remains deferred in FUTURE.md.
• No new dependencies (requests, lxml already present).
• initial-implementation has been merged in twice as it advanced, so the diff
  here is only the validate work. Both conflicts were docs lines the two branches
  had each edited.

Testing

• uv run pytest — 58 passing (43 existing + 15 new), no network.
• Verified end-to-end beyond the unit tests: a clean export reports PASS/exit 0;
  an export seeded with a malformed line, a missing field, a duplicate PMID and a
  bad month reports FAIL/exit 1 with those four itemized; the manifest
  round-trips sorted and gzipped; a 3-PMID drop against a previous manifest is
  reported (warning without a DB, error for unexplained drops with one); and
  --threads 2 reaches validate's connection.

TODO (undecided: fix here or file)

• Check the refactor against the existing export first. data/json was
  produced before the pub_year backfill, so its blank years are still on disk
  and the verdict must come back unchanged — WARN, 0 errors, 1 warning, the
  same 20 mismatches, now rendered as 18x pub_year / 1x article_title / 1x issue,
  all exported_blank, with 0 exported a different value printed and the two
  previously-invisible skipped checks shown. This is the behaviour-preservation
  check: a different verdict means the renderer rewrite changed behaviour and is
  a bug, not an improvement.
• Then re-export and re-run. With PR Initial implementation: download, store, and export PubMed abstracts #1 merged in, this branch's exporter
  recovers the year from MedlineDate, so a fresh export should drop core-fields
  from 20 mismatches to ~2. The residual issue = "Suppl" (PMID 10137601) and
  article_title = "[Not Available]." (PMID 28972331) are expected to survive —
  they are unrelated to MedlineDate and tracked on PR Initial implementation: download, store, and export PubMed abstracts #1.
• Merge PR Initial implementation: download, store, and export PubMed abstracts #1 into this branch once it lands. Done — this branch now
  carries PR Initial implementation: download, store, and export PubMed abstracts #1, and Export DOIs and PMCIDs as an identifiers field #7 has restacked on top of it (diff back to +266/−23 from an
  inflated +664/−228). The merge exposed two real breakages, both fixed here: a
  hardcoded placeholder arity in EXPECTED_FIELDS that crashed validate at
  import once the export query gained a column, and validate comparing efetch's
  raw <Year> against an export that now recovers one from MedlineDate — a
  false-mismatch source whenever efetch returns the archival form.

🤖 Generated with Claude Code