Skip to content

perf: cut Windows CI test fixture overhead#174

Merged
ScriptedAlchemy merged 1 commit into
masterfrom
perf/windows-slow-tests
Jul 1, 2026
Merged

perf: cut Windows CI test fixture overhead#174
ScriptedAlchemy merged 1 commit into
masterfrom
perf/windows-slow-tests

Conversation

@ScriptedAlchemy

Copy link
Copy Markdown
Owner

Summary

Windows CI junit data from the latest green master run (28503571229) shows 10 shards at 11-13.5 min each with 3182 tests summing to ~235 CPU-minutes, and a hard ~15s fixture floor: any test that initializes a TraceDecay project or DB costs 15-65s on Windows (p50 for mcp_handler_test is 22.4s; schema-only tests run in 0.02s). This PR attacks the floor from four directions:

  • Optimized SQLite in test builds (Cargo.toml): the bundled SQLite C sources in libsql-ffi were compiled at -O0 in dev/test builds. Scoped opt-level = 2 overrides for libsql-ffi/libsql-sys/libsql-rusqlite cut measured per-test time 2.8x on Linux (1.69s to 0.60s); the MSVC delta should be at least as large.
  • TRACEDECAY_SQLITE_UNSAFE_FAST=1 escape hatch (src/db/connection.rs): tests/CI-only env var that switches every connection to journal_mode=MEMORY + synchronous=OFF, eliminating the per-transaction rollback-journal create/write/fsync/delete cost of the Windows DELETE+FULL pairing. In-memory journals never enter WAL mode, so the Windows WAL-teardown crash stays avoided. Tests that assert durable journal/synchronous modes pin the var off via env guards.
  • Runner tuning (ci.yml, windows-test-shard only): disable Defender real-time monitoring plus exclusion-path fallback for workspace/temp/cargo dirs, redirect TMP/TEMP off the slow C: user temp onto the RUNNER_TEMP drive, and export the unsafe-fast env for the nextest step.
  • Fixture trims in the slowest junit offenders: drop dead session-store seeding from scheduler gate tests (gate is evaluated before any store is opened); pin TRACEDECAY_GLOBAL_DB at each skill-writer runner test's own store (evidence building was opening the shared per-user global DB, contended by every parallel test process — the 9-of-10-slow signature of that binary); pre-create dashboard GlobalDb-schema stores from a cached empty template so server startup and seeding skip full schema creation (the curation restart test starts the server twice); seed branch-drift tracking metadata directly instead of building and syncing a branch DB only to delete it.

Verification

  • cargo fmt --all -- --check, git diff --check, cargo clippy --workspace --all-targets --locked -- -D warnings
  • cargo nextest run --workspace --profile ci --locked with TRACEDECAY_SQLITE_UNSAFE_FAST=1: 3272 passed, 0 failed
  • Mode-asserting suites green both with the var set and unset (migration_test, profile_storage_migration_test, corruption_test, src/db/connection.rs unit tests)
  • scripts/check-conventional-commits.sh origin/master..HEAD
  • One earlier full-run failure (dashboard_session_and_skill_runs_emit_activity_when_evidence_is_unavailable) reproduces identically with these changes stashed under heavy machine load and passes reliably otherwise — pre-existing load-sensitivity flake, not a regression.

Test plan

  • Windows shards green on this PR's CI run
  • Compare shard wall-clock times and junit per-test times against master run 28503571229 (baseline: 11-13.5 min/shard, mcp_handler_test p50 22.4s)

Windows shards spend most of their 11-13.5 min on a ~15s per-test fixture
floor: every test that touches a DB pays unoptimized bundled SQLite,
DELETE+FULL journal fsyncs, Defender scanning, and repeated full schema
creations. Attack all four:

- Cargo.toml: compile libsql-ffi/libsql-sys/libsql-rusqlite at opt-level 2
  in dev/test builds (2.8x per-test speedup measured on Linux).
- TRACEDECAY_SQLITE_UNSAFE_FAST=1 env hatch (tests/CI only) switching all
  connections to journal_mode=MEMORY + synchronous=OFF; mode-asserting
  tests pin the var off via env guards.
- ci.yml windows-test-shard: disable Defender real-time scanning, exclude
  hot paths, redirect TMP/TEMP to the fast RUNNER_TEMP drive, and export
  the unsafe-fast env for the test step.
- Trim per-test fixture excess in the slowest junit offenders: drop dead
  session-store seeding from scheduler gate tests, isolate the shared
  per-user global DB out of skill-writer runner tests, pre-create
  dashboard GlobalDb-schema stores from a cached empty template, and seed
  branch-drift tracking metadata directly instead of building a branch DB
  just to delete it.
@changeset-bot

changeset-bot Bot commented Jul 1, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: e8a0406

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@ScriptedAlchemy ScriptedAlchemy merged commit 31eed9d into master Jul 1, 2026
35 of 36 checks passed
@ScriptedAlchemy ScriptedAlchemy deleted the perf/windows-slow-tests branch July 1, 2026 22:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant