DEVPROD-XXXX Add --batch-test-discovery and cache suiteconfig results#116
Draft
lavigneer wants to merge 5 commits into
Draft
DEVPROD-XXXX Add --batch-test-discovery and cache suiteconfig results#116lavigneer wants to merge 5 commits into
lavigneer wants to merge 5 commits into
Conversation
version_gen invokes resmoke test-discovery once per generated task, but only ~55% of those suites are distinct (the rest are multiversion/variant repeats of the same suite). Each invocation costs ~2s of python startup, ~10 CPU-min per run. Memoize results per suite name; per-entry locks make concurrent duplicate lookups wait for the in-flight discovery instead of spawning a duplicate resmoke process.
There was a problem hiding this comment.
Pull request overview
This PR reduces resmoke.py subprocess startup overhead during config generation by (1) optionally batching resmoke test discovery across many suites and pre-seeding the in-process discovery cache, and (2) memoizing suiteconfig results per suite to avoid repeated resmoke startups across generated tasks.
Changes:
- Add
--batch-test-discoveryCLI flag and plumbing to optionally prewarm the discovery cache before generation. - Implement batched
test-discovery(--suiterepeated) inResmokeProxy::prewarmand seed the per-suite discovery cache from the multi-document YAML stream. - Add a per-suite memoization cache for
get_suite_configto avoid redundantresmoke.py suiteconfiginvocations.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| src/resmoke/resmoke_proxy.rs | Adds prewarm batching, suiteconfig memoization, and shared command construction for discovery. |
| src/main.rs | Introduces --batch-test-discovery flag and passes it into the execution configuration. |
| src/lib.rs | Wires the flag through dependencies and triggers prewarming prior to task generation. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Recover from poisoned mutexes instead of panicking the run, make the test temp dir unique per invocation, and quote the counter file path in the fake resmoke script.
With this flag, all resmoke-generated suites are discovered up front via batched 'resmoke.py test-discovery --suite A --suite B ...' invocations (chunks of 100) that seed the per-suite discovery cache, instead of paying resmoke's ~2s python startup once per suite. Requires a resmoke version whose test-discovery accepts repeated --suite arguments; older branches simply don't pass the flag and keep per-suite discovery. Fuzzer tasks, multiversion-generate tasks, and burn_in tasks are excluded from the prewarm list: their 'suite' is not a plain resmoke suite and a single SuiteNotFound fails the whole batch. Multiversion per-old-version suites are still discovered lazily. A failed batch falls back to lazy per-suite discovery rather than failing generation. On mongodb-mongo-master this replaces ~323 sequential ~2s resmoke calls with 4 batched invocations totalling ~30s.
write_standard_suite creates a fresh ResmokeConfigCache per generated task, so 'resmoke.py suiteconfig' ran once per task (~800 python startups, ~160s of a ~200s generation run). Cache the parsed config in ResmokeProxy keyed by suite name, same per-entry locking scheme as the discovery cache, and log suiteconfig invocation durations.
Replace run_command unwraps with error propagation in run_test_discovery and get_suite_config, log the requested suite name rather than a misleading suite_config field, discard a batch whose document count does not match the request instead of zipping by order blindly, and use poison-safe lock recovery for the caches.
lavigneer
force-pushed
the
lavigneer/batch-test-discovery
branch
from
July 15, 2026 18:37
ae8c8fa to
5639808
Compare
lavigneer
force-pushed
the
lavigneer/memoize-test-discovery
branch
from
July 16, 2026 18:47
c3a9e21 to
f522577
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Stacked on #115. Two changes:
--batch-test-discoveryflag (src/main.rs,src/lib.rs,src/resmoke/resmoke_proxy.rs): before generation, collect the suite of every resmoke-generated task (bazel targets included; fuzzer, multiversion-generate, and burn_in tasks excluded since their "suite" is not a plain resmoke suite), dedupe, and discover them via batchedresmoke.py test-discovery --suite A --suite B ...calls in chunks of 100, seeding the discovery cache from the multi-document YAML output. A failed batch logs and falls back to lazy per-suite discovery — generation cannot be broken by a bad suite. Multiversion per-old-version suites remain lazily discovered.src/resmoke/resmoke_proxy.rs):get_suite_configresults are now memoized per suite with the same per-entry-lock scheme.write_standard_suitebuilds a freshResmokeConfigCacheper generated task, soresmoke.py suiteconfigpreviously ran once per task (~800 python startups, 666s CPU measured).Why
Resmoke subprocess startup (~2s) dominated version_gen generation time. Measured on mongodb-mongo-master patches: 323 suites batch-discovered in ~30s (vs ~11 CPU-minutes), and generation dropped 201s → 142s with the suiteconfig cache. Combined with #115, the
generate version shstep went 7m26s → 4m05s on patch builds (~357s → ~142s of generator time at ~2,160 runs/week ≈ 129 machine-hours/week).Compatibility
The flag requires resmoke
test-discoveryto accept repeated--suiteargs (10gen/mongo#58681). Branches that don't pass the flag keep exactly the old behavior; release branches pin older generator versions anyway.Testing
Unit tests for prewarm seeding and suiteconfig caching; full
cargo testpasses; verified end-to-end on mongodb-mongo-master patch builds (all 4 batches seeded, generated config identical task set).