diff --git a/changelog.d/7799-retain-survival-adaptive-pacing.md b/changelog.d/7799-retain-survival-adaptive-pacing.md new file mode 100644 index 0000000000..94014a698d --- /dev/null +++ b/changelog.d/7799-retain-survival-adaptive-pacing.md @@ -0,0 +1,173 @@ +### GC: a heap whose young generation is not dying no longer schedules full mark-sweeps that free nothing + +`retain.ts` **0.542 s → 0.345 s**, `retain_wide.ts` **1.099 s → 0.454 s**, `retain1` 0.299 → 0.134, +`retain_wide1` 0.276 → 0.157, and — not targeted — `deeplist.ts` 0.245 → 0.123. +Peak RSS *fell* on every one of them. Quiet M1 mini, best-of-5, vs `origin/main` @ `0a2bf15bd`. + +These programs build a multi-million-element array of records and keep every one +alive. Nothing is ever garbage. Before this change **79% of `retain` and 88% of +`retain_wide` was GC pause**, and most of that pause was full mark-sweeps that +found the heap fully live: + +| bench | fulls | what each reclaimed | +|---|--:|---| +| `retain` | 1 (161 ms) | 11.9% | +| `retain_wide` | 2 (98 + 512 ms) | 6.8%, then 9.6% | +| `deeplist` | 1 (127 ms) | **0.0%** | +| `tree` / `tree_wide` | 40 each | 87.8% / 92.3% | + +#### 1. Survival-adaptive major pacing + +`arena_growth_full_escalation_due` escalated a minor to a full once the arena +grew past `growth_num` (default 2) times the last full's live set. "Escalate when +the heap doubles" is the right rule for a heap that accumulates garbage and the +wrong rule for one that does not: when **everything allocated stays alive**, +doubling is the program working, and each escalation marked a bigger all-live +heap than the last. + +#7726/#7733's retrospective backoff cannot fix this. It prices a full *after* +paying for it, and on a monotonically growing live heap deferring a full only +makes the next one more expensive — there is no schedule of futile fulls that is +cheap. The useless full has to be predicted. + +The prediction is a measurement the collector already takes. `young_survival_permille` +separates the two populations by two orders of magnitude, with nothing in between: + +| workload | young survival (permille) | +|---|--:| +| `churn`, `churn_alloc`, `push_cls` | 0 – 4 | +| `cycles` | 0 | +| `shapes` | 713 – 920 | +| `retain`, `retain_wide`, `deeplist` | 999 – 1000 | + +So a copying minor that measures ≥ 900 permille now marks the heap RETAINING, +which (a) multiplies the escalation growth band by 4 and (b) re-baselines +arena-growth pacing on the occupancy that survived the collection. (b) is what +makes (a) reachable: before the first full the baseline is 0, so the boundary +degenerates to the absolute `PERRY_GC_MAJOR_PACING_FLOOR_MB`, and **any** program +retaining more than 32 MB paid a whole-heap mark-sweep for doing so, once, +unconditionally. + +The same signal and the same multiplier are applied to `old_reclaim_pressure_due`'s +growth band. `credit_promoted_bytes_to_old_baseline` (#7592) already exempts +old-gen growth a minor proved live, but a large object is allocated *straight into* +old-gen and never passes through promotion, so its bytes are uncredited growth even +when they are the program's live data — on `retain.ts` that is the element array +itself. With the arena-growth escalation correctly declining, this band became the +binding constraint and fired a 452 ms full that reclaimed 7.6%: the identical +futile-full shape, one trigger over, reached by the same route. + +Two properties keep the blast radius small, and both are asserted: + +* **It can only make fulls rarer, never more frequent.** The baseline only + ratchets up and the multiplier is ≥ 1, so the escalation boundary is never + lower than before. The exposure is deferred reclamation, not extra pauses — + and measured peak RSS went *down* on every affected benchmark, because the + fulls being skipped were reclaiming 0–12%. +* **A single non-retaining minor disarms it**, with no decay window, so a heap + that stops retaining paces tightly again on its very next collection. `tree` / + `tree_wide` run no minors at all, never arm it, and are unchanged (40 fulls + each, 87.8% / 92.3% yield, same wall time). + +`retaining` is emitted in the `major_pacing` GC trace object: a run that never +armed the band must not be indistinguishable from one that did and had nothing +to skip. + +#### 2. An all-pointer array's dirty-card scan was O(live array), not O(dirty pages) + +`scan_dirty_object_slots` has two arms. Its `Range` arm intersects a slot range +with the remembered set's dirty-page set directly; its `Slot` arm answers "is +this slot on a dirty page?" with a **hash-set probe per slot**. A JS array whose +elements are all pointers selects `LayoutSlotMask::AllPointers`, which reported +itself as `Masked` and therefore emitted one `Slot` descriptor per element — so a +3M-element array cost 3M probes on every minor, to find the few hundred pages the +remembered set already knew were dirty. `dirty_slot_ranges_scanned == 0` in every +`retain.ts` GC trace was recording exactly this: the cheap arm was never reached. + +`AllPointers` selects every index in `0..slot_count`, which *is* a contiguous +range, so it now emits one `Range`. Worth 9% on `retain` on its own, before any +pacing change. + +`dirty_slot_ranges_for` gained a second traversal arm at the same time: it walked +the whole dirty-page set per range, which is right for one huge array and +quadratic for a heap holding many small pointer ranges. It now walks whichever +side is smaller. + +#### 3. Two hot-path readings that were paid for and not used + +* `classify_heap_space_in_range` is split into an `#[inline(always)]` cache-hit + arm and an `#[inline(never)]` miss arm, exactly as #7469 did for its sibling + `classify_heap_generation` and for the same reason: with the map lookup inlined + alongside, the whole function stayed out of line and every call paid its own + `_tlv_get_addr` for the cache base. This one is the copying minor's inner loop. +* `CopyingPointerSet::classify_arena` read both survivor-space thread-locals + before the match that selects a kind. On Darwin there is no local-exec TLS, so + each is a real `_tlv_get_addr` — two per classified pointer, on workloads that + never touch a survivor. They can only answer `Survivor0`/`Survivor1`/`Unknown`, + so the non-survivor arms are hoisted above them; no verdict changes. + +#### Refuted along the way + +* Batching the per-slot `old_page_account_dirty_slot` map probe into one update + per 4 KB page (and hoisting the per-slot weak-target check to a per-object one) + measured as **exactly zero** — `retain` 0.344 vs 0.345 s. Not shipped. +* The gc-ratchet's `11_collect_at_depth` reports six gating REGRESSION rows here + (6,150 promoted objects becoming 6,139 copied ones, and `heap_used_bytes` + +4.12%), plus `04_dead_after_deep_stack`'s `copied_objects`. **None of them is + this change.** A gc_ratchet run of the `origin/main` @ `0a2bf15bd` reference + build on the same host produces the identical six rows, byte for byte, and a + cell-by-cell comparison of the two artifacts finds **every gating metric across + all 13 probes identical** — only `wall_ms` / `rss_bytes` / `peak_rss_bytes` + differ, which is exactly the set the `shared_ci` profile deliberately does not + gate. `12_large_live_set` holds (`heap_used_bytes` −2.39%, an improvement) with + `copied_objects` 61,851, so the probe's subject was live. The rows are red on + main on this host and want their own investigation. +* A first attempt to "fix" those rows scoped the retaining multiplier away from + `copied_minor_promotion_handoff_pressure_due`, on the theory that survivor + *placement* should not read a band derived from full-GC-yield evidence. Reverted: + the premise was the phantom above, and the principle was wrong too — that + predicate's own doc says it decides "whether an imminent promotion justifies a + full old reclaim FIRST", and `gc::mod.rs` answers it with + `note_survivor_promotion_handoff_full`. All three callers of + `old_reclaim_pressure_due` are full-collection decisions, so one signal and one + multiplier is the coherent shape, not a carve-out. + +#### Validation + +All 19 corpus programs byte-identical to `node --experimental-strip-types` with +exit 0. `gc-handoff/apps/iso_miss.ts` prints `checksum 437840 misses 0`, including +under `PERRY_GC_PROTECT_FROMSPACE=1 PERRY_GC_PROTECT_FROMSPACE_DEPTH=800` and +`PERRY_GC_VERIFY_EVACUATION=1`. Protected set unchanged within noise: `churn` +0.421, `churn_alloc` 0.374, `push_cls` 0.358, `push_num` 0.137, `churn_read` +0.022, `cycles` 0.193, `tree` 1.631, `tree_wide` 2.113, `fib40` 0.394, `interp` +1.889, `asyncpipe` 0.710, `shapes` 0.219, `pipeline` 0.543. + +#### Gap suite + +Every gap failure outside `test-parity/gap_snapshot.json` reproduces **identically +on the `origin/main` @ `0a2bf15bd` reference build**, on the same host: + +* `test_gap_gc_rest_argument_rooting`, `test_gap_gc_same_module_call_argument_rooting` + — byte-identical output from both builds, zeal verdict line included + (`forced_collections=379 copying_minors=379 moved_objects=115536` and + `749/749/231012`). The mismatch is the `[gc-zeal]` exit verdict against a node + oracle that does not print it. +* `test_gap_gc_alloc_point_no_move` — timed out against the harness's 10 s limit + on a dev host at load 65+, where both builds take 9–11 s. On the quiet mini: + main 2.35 / 1.88 / 1.89 s, this branch 2.23 / 1.88 / 1.88 s. +* `test_gap_fetch_request_from_node_incoming_message`, + `test_gap_http_client_no_redirect_follow`, `test_gap_http_overloads_3226plus`, + `test_gap_http_req_async_iterator`, + `test_gap_http_res_socket_writable_onfinished`, `test_gap_net_connect_bound_value` + — SIGABRT (exit 134) on both builds. +* `test_gap_specabi_reassign`, `test_gap_zlib_3285_params` — diverge from the node + oracle byte-identically on both builds. + +The harness also reports ten `node_fail -> parity_fail` status changes +(`test_gap_4510_enum_forward_ref`, `..._backoff_options`, `..._cron_cronjob`, +`..._dayjs_factory_arg`, `..._derived_param_props`, `..._enum_in_function_body`, +`..._moment_methods`, `..._prop_plan_cache_invalidation`, `..._ratelimiter_memory`, +`..._slugify_options`) and one improvement (`test_gap_iterator_helpers_2874`). +Those are the oracle's environment, not Perry's output: the snapshot records +`node_fail` for files whose node run cannot resolve an npm import or whose +TypeScript syntax strip-only mode refuses, and this host resolves some of them. diff --git a/crates/perry-runtime/src/arena/page_meta.rs b/crates/perry-runtime/src/arena/page_meta.rs index 0905fca734..3aff15405a 100644 --- a/crates/perry-runtime/src/arena/page_meta.rs +++ b/crates/perry-runtime/src/arena/page_meta.rs @@ -690,7 +690,13 @@ pub(crate) fn classify_heap_space(addr: usize) -> HeapSpace { /// a range base); the base is precisely the guard that keeps a *garbage* /// candidate address at the very start of a range from turning into a read of /// the unmapped page below it (#7742). -#[inline] +/// Split hit/miss exactly like [`classify_heap_generation`] above, and for the +/// same reason (#7469): with the map-lookup arm inlined alongside it, the whole +/// function stayed out of line and every call paid its own `_tlv_get_addr` for +/// the cache base. This one is the copying minor's inner loop — +/// `CopyingPointerSet::classify_arena` calls it once per visited slot — so on a +/// promotion-heavy cycle it runs millions of times per collection. +#[inline(always)] pub(crate) fn classify_heap_space_in_range(addr: usize) -> Option<(HeapSpace, usize)> { if addr == 0 { return None; @@ -700,18 +706,20 @@ pub(crate) fn classify_heap_space_in_range(addr: usize) -> Option<(HeapSpace, us if let Some(range) = unsafe { (*hot_page_generation_cache()).lookup(key, addr) } { return Some((range.space, range.base)); } + classify_heap_space_in_range_uncached(addr, key) +} +/// Cache-miss arm of [`classify_heap_space_in_range`]. +#[inline(never)] +fn classify_heap_space_in_range_uncached(addr: usize, key: usize) -> Option<(HeapSpace, usize)> { let found = { let pages = hot_page_generations().borrow(); pages.get(&key).and_then(|slot| slot.find(addr)) }; - if let Some(range) = found { - // SAFETY: as above. - unsafe { (*hot_page_generation_cache()).insert(key, range) }; - Some((range.space, range.base)) - } else { - None - } + let range = found?; + // SAFETY: thread-local, single-threaded, and the borrow ends here. + unsafe { (*hot_page_generation_cache()).insert(key, range) }; + Some((range.space, range.base)) } pub(crate) fn old_object_page_overlaps( diff --git a/crates/perry-runtime/src/gc/barrier.rs b/crates/perry-runtime/src/gc/barrier.rs index cb34ce3c86..1f08466ec7 100644 --- a/crates/perry-runtime/src/gc/barrier.rs +++ b/crates/perry-runtime/src/gc/barrier.rs @@ -241,21 +241,43 @@ fn dirty_slot_ranges_for( return Vec::new(); }; + // Walk whichever side is smaller. Iterating the dirty-page set is O(dirty + // pages) regardless of the range's size, which is the right shape for the + // one huge array this exists for — but it is quadratic when a heap holds + // MANY small pointer ranges (each would rescan the whole set). Enumerating + // the range's own pages instead is O(range pages) with one set probe each. + // Both arms produce the same ranges; only the traversal order differs, and + // the merge below sorts. + let range_pages = (slots_end - 1).saturating_sub(slots) / PAGE_SIZE + 1; let mut ranges = Vec::new(); - for &page in dirty_pages { + let push_page = |page: usize, ranges: &mut Vec<(usize, usize)>, stats: &mut _| { let page_start = page << PAGE_SHIFT; let page_end = page_start + PAGE_SIZE; let start = slots.max(page_start); let end = slots_end.min(page_end); if start >= end { - continue; + return; } + let stats: &mut RememberedSetTraceStats = stats; stats.dirty_slot_pages_considered += 1; let first = (start - slots) / std::mem::size_of::(); let last = (end - slots).div_ceil(std::mem::size_of::()); if first < last { ranges.push((first.min(slot_count), last.min(slot_count))); } + }; + if range_pages <= dirty_pages.len() { + let first_page = slots >> PAGE_SHIFT; + let last_page = (slots_end - 1) >> PAGE_SHIFT; + for page in first_page..=last_page { + if dirty_pages.contains(&page) { + push_page(page, &mut ranges, stats); + } + } + } else { + for &page in dirty_pages { + push_page(page, &mut ranges, stats); + } } if ranges.is_empty() { diff --git a/crates/perry-runtime/src/gc/copying.rs b/crates/perry-runtime/src/gc/copying.rs index 52761c04aa..fc411e7930 100644 --- a/crates/perry-runtime/src/gc/copying.rs +++ b/crates/perry-runtime/src/gc/copying.rs @@ -121,15 +121,21 @@ impl CopyingPointerSet { if unsafe { !plausible_gc_header(header, true) } { return None; } - let active_survivor = crate::arena::active_survivor_space(); - let inactive_survivor = crate::arena::inactive_survivor_space(); + // The two survivor-space readings are TLS loads, and Darwin has no + // local-exec TLS — each is a real `_tlv_get_addr` call. Reading them + // eagerly cost two per classified pointer on workloads that never touch + // a survivor at all (`retain.ts` classifies Eden / PromotedYoung / Old + // and nothing else). They can only ever answer `Survivor0`, `Survivor1` + // or `Unknown`, and `space` is already narrowed to the six accepted + // spaces, so hoisting the non-survivor arms above them changes no + // verdict — it just stops paying for an answer the arm does not use. let kind = match space { crate::arena::HeapSpace::NurseryEden => CopyingPointerKind::Eden, crate::arena::HeapSpace::PromotedYoung => CopyingPointerKind::PromotedYoung, - s if s == active_survivor => CopyingPointerKind::FromSurvivor, - s if s == inactive_survivor => CopyingPointerKind::ToSurvivor, crate::arena::HeapSpace::Longlived => CopyingPointerKind::Longlived, crate::arena::HeapSpace::Old => CopyingPointerKind::Old, + s if s == crate::arena::active_survivor_space() => CopyingPointerKind::FromSurvivor, + s if s == crate::arena::inactive_survivor_space() => CopyingPointerKind::ToSurvivor, _ => return None, }; Some(CopyingPointer { header, kind }) @@ -1578,6 +1584,11 @@ pub(super) fn gc_collect_minor_copying_fast_path_with_eligibility( // the stale baseline and schedules a full that is guaranteed to free // nothing (see `credit_promoted_bytes_to_old_baseline`). credit_promoted_bytes_to_old_baseline(collector.stats.promoted_bytes); + // The same argument one trigger over: a young generation that did not die + // is a heap growing by LIVE data, so arena-growth pacing must not read that + // growth as garbage accumulating. Fed here, after the reset, so the + // re-baseline sees post-collection occupancy. + note_copying_minor_young_survival(collector.stats.young_survival_permille); maybe_schedule_old_reclaim_after_copied_minor(); retune_after_scavenge( collector.stats.eden_live_bytes, diff --git a/crates/perry-runtime/src/gc/layout.rs b/crates/perry-runtime/src/gc/layout.rs index 0422eb48fa..bb55cee590 100644 --- a/crates/perry-runtime/src/gc/layout.rs +++ b/crates/perry-runtime/src/gc/layout.rs @@ -1570,6 +1570,22 @@ pub(super) enum HeapPayloadSlotScan { raw_numeric_array: bool, raw_numeric_object_slots: usize, }, + /// [`LayoutSlotMask::AllPointers`]: the mask selects EVERY live payload + /// slot, so the slot set is a contiguous range and the descriptor visitor + /// emits one `Range` rather than `slot_count` individual `Slot`s. + /// + /// This is not a micro-optimisation. `scan_dirty_object_slots`'s `Slot` arm + /// answers "is this slot on a dirty page?" with a hash-set probe **per + /// slot**, so a 3M-element array of pointers cost 3M probes on every minor + /// — O(live array) rather than O(dirty pages) — even though the remembered + /// set knew only a few hundred of its pages were dirty. Its `Range` arm + /// intersects the range with the dirty-page set directly + /// (`dirty_slot_ranges_for`), which is what `dirty_slot_ranges_scanned == 0` + /// in every `retain.ts` GC trace was recording: the cheap arm was never + /// reached, because an all-pointer array is `Masked`, not `All` (#7787). + AllPointers { + raw_numeric_object_slots: usize, + }, Masked, All(HeapSlotRange), } @@ -1650,6 +1666,22 @@ impl HeapChildSlotIterator { raw_numeric_array, raw_numeric_object_slots, }, + HeapPayloadSlotSelection::Masked { + mask: LayoutSlotMask::AllPointers, + raw_numeric_object_slots, + raw_numeric_recorded, + .. + } => HeapPayloadSlotScan::AllPointers { + // Mirror the iterator's one-shot accounting: `next` records the + // raw-numeric skip on its first call and never again, so a + // descriptor visit that replaces the whole iteration records it + // exactly once too. + raw_numeric_object_slots: if raw_numeric_recorded { + 0 + } else { + raw_numeric_object_slots + }, + }, HeapPayloadSlotSelection::Masked { .. } => HeapPayloadSlotScan::Masked, HeapPayloadSlotSelection::All { .. } => HeapPayloadSlotScan::All(self.payload), } diff --git a/crates/perry-runtime/src/gc/layout_slot_visit.rs b/crates/perry-runtime/src/gc/layout_slot_visit.rs index 4cc6a18026..adc134a61c 100644 --- a/crates/perry-runtime/src/gc/layout_slot_visit.rs +++ b/crates/perry-runtime/src/gc/layout_slot_visit.rs @@ -38,6 +38,21 @@ pub(super) unsafe fn visit_gc_layout_slot_descriptors( } visit(GcMutableSlotDescriptor::PointerFreeRange); } + HeapPayloadSlotScan::AllPointers { + raw_numeric_object_slots, + } => { + if raw_numeric_object_slots != 0 { + record_layout_raw_numeric_object_field_range_skipped(raw_numeric_object_slots); + } + // Same slot set the `Masked` arm below would emit one-at-a-time + // (`AllPointers` yields every index in `0..slot_count`), handed over + // as a contiguous range so `scan_dirty_object_slots` can intersect + // it with the dirty-page set instead of probing that set per slot. + visit(GcMutableSlotDescriptor::Range { + range: child_slots.payload, + layout_kind: Some(HeapChildSlotReadKind::Masked), + }); + } HeapPayloadSlotScan::Masked => { for child_slot in child_slots { if let HeapChildSlot::Child(slot, layout_kind) = child_slot { diff --git a/crates/perry-runtime/src/gc/policy.rs b/crates/perry-runtime/src/gc/policy.rs index 3a142093a8..85546e81c2 100644 --- a/crates/perry-runtime/src/gc/policy.rs +++ b/crates/perry-runtime/src/gc/policy.rs @@ -906,6 +906,15 @@ thread_local! { /// `OldReclaim` escalation — old-gen garbage still forces a full through /// `old_reclaim_pressure_due` regardless of this backoff. pub(super) static GC_MAJOR_PACING_BACKOFF_SHIFT: Cell = const { Cell::new(0) }; + /// Survival-adaptive arm of major-GC pacing: `true` once a copying minor + /// has measured a young-survival ratio at or above + /// `MAJOR_PACING_RETAINING_SURVIVAL_PERMILLE`, cleared by any minor that + /// measures less. See `MAJOR_PACING_RETAINING_GROWTH_MULTIPLIER`. + /// + /// Deliberately the LAST minor's verdict rather than a running maximum: a + /// heap that stops retaining must pace tightly again on its very next + /// collection, not after a decay window. + pub(super) static GC_MAJOR_PACING_RETAINING: Cell = const { Cell::new(false) }; /// Re-entrancy guard for the #5476 direct old-gen reclaim driven from /// `gc_check_trigger`: the full collection must not recursively trigger /// another reclaim if a hook it runs allocates. @@ -1342,7 +1351,23 @@ const OLD_RECLAIM_GROWTH_DIVISOR: usize = 2; /// the "is it due" predicate and the debt arithmetic cannot diverge (#7024's /// two-predicates-collapse family). pub(super) fn gc_old_reclaim_growth_band_bytes(baseline: usize) -> usize { - gc_old_gen_reclaim_growth_dyn_bytes().max(baseline / OLD_RECLAIM_GROWTH_DIVISOR) + let band = gc_old_gen_reclaim_growth_dyn_bytes().max(baseline / OLD_RECLAIM_GROWTH_DIVISOR); + // Survival-adaptive, the same signal and the same multiplier the + // arena-growth escalation uses (`MAJOR_PACING_RETAINING_GROWTH_MULTIPLIER`). + // + // `credit_promoted_bytes_to_old_baseline` already exempts old-gen growth + // that a minor PROVED live, but a large object is allocated straight into + // old-gen and never passes through promotion, so its bytes are uncredited + // growth even when they are the program's live data. On `retain.ts` that is + // the element array itself: with the arena-growth escalation correctly + // declining, this band became the binding constraint and fired a 452 ms + // full that reclaimed 7.6% — the same futile-full shape one trigger over, + // reached by the same route. While the young generation is not dying, old + // growth is priced as live here too. + if GC_MAJOR_PACING_RETAINING.with(|c| c.get()) { + return band.saturating_mul(MAJOR_PACING_RETAINING_GROWTH_MULTIPLIER); + } + band } #[inline] @@ -1503,6 +1528,46 @@ pub(super) fn credit_promoted_bytes_to_old_baseline(promoted_bytes: usize) { .with(|bytes| bytes.set(bytes.get().saturating_add(promoted_bytes))); } +/// Feed a copying minor's measured young-survival ratio to arena-growth pacing. +/// +/// Two effects, both gated on the same measurement: +/// +/// * It arms/disarms `GC_MAJOR_PACING_RETAINING`, which widens the escalation +/// growth band (see `MAJOR_PACING_RETAINING_GROWTH_MULTIPLIER`). +/// * While retaining, it re-baselines arena-growth pacing on the occupancy that +/// *survived this collection*. Without that the band has nothing to scale: +/// before the first full the baseline is 0, so the boundary degenerates to +/// the absolute `PERRY_GC_MAJOR_PACING_FLOOR_MB` and **any** program that +/// retains more than 32 MB pays a whole-heap mark-sweep for doing so. +/// +/// The re-baseline is a ratchet (`max`), never a decrease, so a minor cannot +/// pull the boundary in below what the last full established — and it is +/// skipped entirely when the heap is not retaining, which is what keeps +/// `churn`/`cycles`/`push_cls` (0–4 permille survival) bit-identical to the +/// previous policy: their baseline stays 0 and their boundary stays the floor. +/// +/// Note the direction of the whole change: because the baseline only ever +/// ratchets UP and the multiplier is ≥ 1, the escalation boundary is never +/// *lower* than it was before. This can only make fulls rarer, never more +/// frequent — the exposure is deferred reclamation (RSS), not extra pauses. +pub(super) fn note_copying_minor_young_survival(survival_permille: u64) { + let retaining = survival_permille >= MAJOR_PACING_RETAINING_SURVIVAL_PERMILLE; + GC_MAJOR_PACING_RETAINING.with(|c| c.set(retaining)); + if !retaining { + return; + } + let survived = pacing_arena_in_use_bytes(); + GC_LAST_FULL_ARENA_IN_USE_BYTES.with(|bytes| bytes.set(bytes.get().max(survived))); +} + +/// Whether the last copying minor measured a retaining heap. Trace/test +/// observability — a gate that cannot see this cannot prove which arm paced a +/// given run. +#[cfg(any(feature = "diagnostics", test))] +pub(super) fn major_pacing_retaining() -> bool { + GC_MAJOR_PACING_RETAINING.with(|c| c.get()) +} + pub(super) fn maybe_schedule_old_reclaim_after_copied_minor() { // #6010: external Map/Set side buffers count toward old-gen pressure — // a tenured-then-dead Map holds its multi-MB buffer until a full @@ -1549,6 +1614,40 @@ const MAJOR_PACING_PRODUCTIVE_YIELD_PCT: usize = 20; /// long run of low-yield fulls cannot disable arena-growth pacing outright. const MAJOR_PACING_BACKOFF_SHIFT_MAX: u32 = 2; +/// Young-survival ratio (permille) at or above which the heap is treated as +/// RETAINING, i.e. growing by data that is alive rather than by garbage. +/// +/// Measured on the GC-benchmark corpus, this separates by two orders of +/// magnitude rather than marginally — the two populations do not overlap: +/// +/// | workload | young survival (permille) | +/// |---|---| +/// | `churn`, `churn_alloc`, `push_cls` | 0 – 4 | +/// | `cycles` | 0 | +/// | `shapes` | 713 – 920 | +/// | `retain`, `retain_wide`, `deeplist` | 999 – 1000 | +const MAJOR_PACING_RETAINING_SURVIVAL_PERMILLE: u64 = 900; + +/// Extra growth allowed before escalating while the heap is RETAINING, on top +/// of `PERRY_GC_MAJOR_PACING_GROWTH` (so 8× with the default 2). +/// +/// This is the survival-adaptive growing factor every generational collector +/// needs and this one lacked. `growth_num = 2` means "escalate when the arena +/// doubles". On a heap where **everything allocated stays alive**, doubling is +/// not evidence of garbage — it is the program working — so the fixed 2× +/// scheduled a full mark-sweep per doubling, each of which marked a bigger +/// all-live heap and freed almost nothing (`retain.ts` 11.9%, `retain_wide.ts` +/// 6.8% then 9.6%, `deeplist.ts` **0.0%**; against `tree.ts` 87.8% and +/// `tree_wide.ts` 92.3%, which run no minors at all and are therefore +/// untouched by this). +/// +/// It is the prospective twin of `MAJOR_PACING_PRODUCTIVE_YIELD_PCT`'s +/// retrospective backoff, and it exists because retrospection cannot help a +/// monotonically growing live heap: every full costs O(live) and delaying one +/// only makes the next bigger, so the useless full has to be *predicted*, not +/// priced after the fact. +const MAJOR_PACING_RETAINING_GROWTH_MULTIPLIER: usize = 4; + /// Record what the just-finished full reclaimed and adjust the pacing backoff. /// /// `pre` is the arena in-use reading captured when the full cycle started @@ -1658,6 +1757,7 @@ pub(super) fn major_pacing_snapshot() -> (usize, u32, Option) { pub(super) fn test_reset_major_pacing_backoff() { GC_MAJOR_PACING_BACKOFF_SHIFT.with(|shift| shift.set(0)); GC_FULL_CYCLE_PRE_IN_USE_BYTES.with(|bytes| bytes.set(0)); + GC_MAJOR_PACING_RETAINING.with(|c| c.set(false)); } /// The pre-full arena reading `arena_growth_full_escalation_due` recorded, or 0 @@ -2813,6 +2913,15 @@ fn major_pacing_escalation_threshold_bytes() -> Option { // escalation out (`GC_MAJOR_PACING_BACKOFF_SHIFT`). Shift the multiplier, // not the baseline, so one productive full restores the original pacing. let shift = GC_MAJOR_PACING_BACKOFF_SHIFT.with(|shift| shift.get()); + // Survival-adaptive: fold the retaining multiplier into `growth_num` rather + // than adding a parameter, so `major_pacing_escalation_threshold_for` stays + // the single pure `(config, state) → boundary` the snapshot and the + // predicate both read (#7733's divergence). + let growth_num = if GC_MAJOR_PACING_RETAINING.with(|c| c.get()) { + growth_num.saturating_mul(MAJOR_PACING_RETAINING_GROWTH_MULTIPLIER) + } else { + growth_num + }; major_pacing_escalation_threshold_for(floor_bytes, growth_num, baseline, shift) } diff --git a/crates/perry-runtime/src/gc/telemetry.rs b/crates/perry-runtime/src/gc/telemetry.rs index e3cc182a02..94cc5755bf 100644 --- a/crates/perry-runtime/src/gc/telemetry.rs +++ b/crates/perry-runtime/src/gc/telemetry.rs @@ -1154,6 +1154,10 @@ impl GcCycleTrace { "baseline_bytes": pacing_baseline, "backoff_shift": pacing_shift, "escalate_at_or_above_bytes": pacing_threshold, + // Which arm paced this cycle. Without it a run that never armed the + // survival-adaptive band is indistinguishable from one that did and + // simply had nothing to skip. + "retaining": super::policy::major_pacing_retaining(), }); serde_json::json!({ "event": "gc_cycle", diff --git a/crates/perry-runtime/src/gc/tests/triggers.rs b/crates/perry-runtime/src/gc/tests/triggers.rs index 86615734d5..d9cf8c26e3 100644 --- a/crates/perry-runtime/src/gc/tests/triggers.rs +++ b/crates/perry-runtime/src/gc/tests/triggers.rs @@ -1223,3 +1223,119 @@ fn the_schedule_holds_the_poll_word_armed_like_zeal() { "dropping the ScheduleGuard must release the arm it took" ); } + +/// The survival-adaptive arm must CHANGE a verdict, in both directions, from +/// the same arena reading. +/// +/// A test that only armed the retaining flag and checked the boundary grew +/// would pass on a build where the flag never reaches the predicate — the +/// #7024/#6942 shape this repo keeps paying for. So the assertion is on +/// `arena_growth_full_escalation_due()` itself, at a reading chosen to sit +/// strictly between the un-retained boundary and the retained one: today's +/// policy escalates there, and the retaining arm is the only thing that can +/// make it decline. +#[test] +fn retaining_survival_widens_the_escalation_band_and_low_survival_restores_it() { + use super::super::policy::{ + arena_growth_full_escalation_due, major_pacing_config, major_pacing_retaining, + note_copying_minor_young_survival, test_major_pacing_pre_in_use_bytes, + test_reset_major_pacing_backoff, test_set_major_pacing_baseline, + test_set_pacing_arena_in_use, + }; + + let (floor_bytes, growth_num) = major_pacing_config(); + if floor_bytes == 0 { + return; // pacing disabled outright: no boundary to widen + } + + test_reset_major_pacing_backoff(); + // Baseline high enough that the growth clause, not the floor, is the + // boundary — the floor would mask the multiplier entirely. + let baseline = floor_bytes; + let previous_baseline = test_set_major_pacing_baseline(baseline); + // Above `growth_num × baseline` (today's boundary), below `4 ×` that. + let reading = baseline * growth_num + 1; + let previous_reading = test_set_pacing_arena_in_use(Some(reading)); + + // 1. Not retaining (the OFF state): this reading escalates, as before. + note_copying_minor_young_survival(0); + let off_retaining = major_pacing_retaining(); + let off_due = arena_growth_full_escalation_due(); + test_reset_major_pacing_backoff(); + test_set_major_pacing_baseline(baseline); + + // 2. Retaining: the same reading no longer escalates. Re-arm the reading + // first — the retaining path re-baselines from it, and `max` keeps the + // baseline where it is here (reading > baseline would raise it, which is + // itself part of the effect being asserted). + note_copying_minor_young_survival(1000); + let on_retaining = major_pacing_retaining(); + let on_due = arena_growth_full_escalation_due(); + let on_recorded = test_major_pacing_pre_in_use_bytes(); + + // 3. A single low-survival minor disarms it again, same reading. + note_copying_minor_young_survival(0); + let back_retaining = major_pacing_retaining(); + + test_set_pacing_arena_in_use(previous_reading); + test_set_major_pacing_baseline(previous_baseline); + test_reset_major_pacing_backoff(); + + assert!(!off_retaining, "survival 0 must not arm the retaining arm"); + assert!( + off_due, + "the reading must escalate WITHOUT the retaining arm, or this test \ + proves nothing about the arm" + ); + assert!(on_retaining, "survival 1000 must arm the retaining arm"); + assert!( + !on_due, + "a retaining heap must not escalate at a reading only the un-widened \ + growth band rejects" + ); + assert_eq!( + on_recorded, 0, + "a declined escalation must leave no pre-full reading behind" + ); + assert!( + !back_retaining, + "one non-retaining minor must disarm the band immediately — a decayed \ + window would keep pacing a churning heap as if it were retaining" + ); +} + +/// The retaining re-baseline is a ratchet, never a decrease: a minor must not +/// be able to pull the boundary in below what the last full established. +#[test] +fn retaining_rebaseline_never_lowers_the_pacing_baseline() { + use super::super::policy::{ + major_pacing_snapshot, note_copying_minor_young_survival, test_reset_major_pacing_backoff, + test_set_major_pacing_baseline, test_set_pacing_arena_in_use, + }; + + test_reset_major_pacing_backoff(); + let high = 512 * 1024 * 1024; + let previous_baseline = test_set_major_pacing_baseline(high); + let previous_reading = test_set_pacing_arena_in_use(Some(1024)); + + note_copying_minor_young_survival(1000); + let (after_low, _, _) = major_pacing_snapshot(); + + test_set_pacing_arena_in_use(Some(high * 2)); + note_copying_minor_young_survival(1000); + let (after_high, _, _) = major_pacing_snapshot(); + + test_set_pacing_arena_in_use(previous_reading); + test_set_major_pacing_baseline(previous_baseline); + test_reset_major_pacing_backoff(); + + assert_eq!( + after_low, high, + "a small post-minor occupancy must not lower the baseline" + ); + assert_eq!( + after_high, + high * 2, + "a larger post-minor occupancy must raise it" + ); +}