perf(gc): predict the futile full mark-sweep instead of pricing it afterwards (retain 3.99x -> 2.54x node, retain_wide 7.01x -> 2.89x)

changelog.d/7799-retain-survival-adaptive-pacing.md

@@ -0,0 +1,173 @@
+### GC: a heap whose young generation is not dying no longer schedules full mark-sweeps that free nothing
+
+`retain.ts` **0.542 s → 0.345 s**, `retain_wide.ts` **1.099 s → 0.454 s**, `retain1` 0.299 → 0.134,
+`retain_wide1` 0.276 → 0.157, and — not targeted — `deeplist.ts` 0.245 → 0.123.
+Peak RSS *fell* on every one of them. Quiet M1 mini, best-of-5, vs `origin/main` @ `0a2bf15bd`.
+
+These programs build a multi-million-element array of records and keep every one
+alive. Nothing is ever garbage. Before this change **79% of `retain` and 88% of
+`retain_wide` was GC pause**, and most of that pause was full mark-sweeps that
+found the heap fully live:
+
+| bench | fulls | what each reclaimed |
+|---|--:|---|
+| `retain` | 1 (161 ms) | 11.9% |
+| `retain_wide` | 2 (98 + 512 ms) | 6.8%, then 9.6% |
+| `deeplist` | 1 (127 ms) | **0.0%** |
+| `tree` / `tree_wide` | 40 each | 87.8% / 92.3% |
+
+#### 1. Survival-adaptive major pacing
+
+`arena_growth_full_escalation_due` escalated a minor to a full once the arena
+grew past `growth_num` (default 2) times the last full's live set. "Escalate when
+the heap doubles" is the right rule for a heap that accumulates garbage and the
+wrong rule for one that does not: when **everything allocated stays alive**,
+doubling is the program working, and each escalation marked a bigger all-live
+heap than the last.
+
+#7726/#7733's retrospective backoff cannot fix this. It prices a full *after*
+paying for it, and on a monotonically growing live heap deferring a full only
+makes the next one more expensive — there is no schedule of futile fulls that is
+cheap. The useless full has to be predicted.
+
+The prediction is a measurement the collector already takes. `young_survival_permille`
+separates the two populations by two orders of magnitude, with nothing in between:
+
+| workload | young survival (permille) |
+|---|--:|
+| `churn`, `churn_alloc`, `push_cls` | 0 – 4 |
+| `cycles` | 0 |
+| `shapes` | 713 – 920 |
+| `retain`, `retain_wide`, `deeplist` | 999 – 1000 |
+
+So a copying minor that measures ≥ 900 permille now marks the heap RETAINING,
+which (a) multiplies the escalation growth band by 4 and (b) re-baselines
+arena-growth pacing on the occupancy that survived the collection. (b) is what
+makes (a) reachable: before the first full the baseline is 0, so the boundary
+degenerates to the absolute `PERRY_GC_MAJOR_PACING_FLOOR_MB`, and **any** program
+retaining more than 32 MB paid a whole-heap mark-sweep for doing so, once,
+unconditionally.
+
+The same signal and the same multiplier are applied to `old_reclaim_pressure_due`'s
+growth band. `credit_promoted_bytes_to_old_baseline` (#7592) already exempts
+old-gen growth a minor proved live, but a large object is allocated *straight into*
+old-gen and never passes through promotion, so its bytes are uncredited growth even
+when they are the program's live data — on `retain.ts` that is the element array
+itself. With the arena-growth escalation correctly declining, this band became the
+binding constraint and fired a 452 ms full that reclaimed 7.6%: the identical
+futile-full shape, one trigger over, reached by the same route.
+
+Two properties keep the blast radius small, and both are asserted:
+
+* **It can only make fulls rarer, never more frequent.** The baseline only
+  ratchets up and the multiplier is ≥ 1, so the escalation boundary is never
+  lower than before. The exposure is deferred reclamation, not extra pauses —
+  and measured peak RSS went *down* on every affected benchmark, because the
+  fulls being skipped were reclaiming 0–12%.
+* **A single non-retaining minor disarms it**, with no decay window, so a heap
+  that stops retaining paces tightly again on its very next collection. `tree` /
+  `tree_wide` run no minors at all, never arm it, and are unchanged (40 fulls
+  each, 87.8% / 92.3% yield, same wall time).
+
+`retaining` is emitted in the `major_pacing` GC trace object: a run that never
+armed the band must not be indistinguishable from one that did and had nothing
+to skip.
+
+#### 2. An all-pointer array's dirty-card scan was O(live array), not O(dirty pages)
+
+`scan_dirty_object_slots` has two arms. Its `Range` arm intersects a slot range
+with the remembered set's dirty-page set directly; its `Slot` arm answers "is
+this slot on a dirty page?" with a **hash-set probe per slot**. A JS array whose
+elements are all pointers selects `LayoutSlotMask::AllPointers`, which reported
+itself as `Masked` and therefore emitted one `Slot` descriptor per element — so a
+3M-element array cost 3M probes on every minor, to find the few hundred pages the
+remembered set already knew were dirty. `dirty_slot_ranges_scanned == 0` in every
+`retain.ts` GC trace was recording exactly this: the cheap arm was never reached.
+
+`AllPointers` selects every index in `0..slot_count`, which *is* a contiguous
+range, so it now emits one `Range`. Worth 9% on `retain` on its own, before any
+pacing change.
+
+`dirty_slot_ranges_for` gained a second traversal arm at the same time: it walked
+the whole dirty-page set per range, which is right for one huge array and
+quadratic for a heap holding many small pointer ranges. It now walks whichever
+side is smaller.
+
+#### 3. Two hot-path readings that were paid for and not used
+
+* `classify_heap_space_in_range` is split into an `#[inline(always)]` cache-hit
+  arm and an `#[inline(never)]` miss arm, exactly as #7469 did for its sibling
+  `classify_heap_generation` and for the same reason: with the map lookup inlined
+  alongside, the whole function stayed out of line and every call paid its own
+  `_tlv_get_addr` for the cache base. This one is the copying minor's inner loop.
+* `CopyingPointerSet::classify_arena` read both survivor-space thread-locals
+  before the match that selects a kind. On Darwin there is no local-exec TLS, so
+  each is a real `_tlv_get_addr` — two per classified pointer, on workloads that
+  never touch a survivor. They can only answer `Survivor0`/`Survivor1`/`Unknown`,
+  so the non-survivor arms are hoisted above them; no verdict changes.
+
+#### Refuted along the way
+
+* Batching the per-slot `old_page_account_dirty_slot` map probe into one update
+  per 4 KB page (and hoisting the per-slot weak-target check to a per-object one)
+  measured as **exactly zero** — `retain` 0.344 vs 0.345 s. Not shipped.
+* The gc-ratchet's `11_collect_at_depth` reports six gating REGRESSION rows here
+  (6,150 promoted objects becoming 6,139 copied ones, and `heap_used_bytes`
+  +4.12%), plus `04_dead_after_deep_stack`'s `copied_objects`. **None of them is
+  this change.** A gc_ratchet run of the `origin/main` @ `0a2bf15bd` reference
+  build on the same host produces the identical six rows, byte for byte, and a
+  cell-by-cell comparison of the two artifacts finds **every gating metric across
+  all 13 probes identical** — only `wall_ms` / `rss_bytes` / `peak_rss_bytes`
+  differ, which is exactly the set the `shared_ci` profile deliberately does not
+  gate. `12_large_live_set` holds (`heap_used_bytes` −2.39%, an improvement) with
+  `copied_objects` 61,851, so the probe's subject was live. The rows are red on
+  main on this host and want their own investigation.
+* A first attempt to "fix" those rows scoped the retaining multiplier away from
+  `copied_minor_promotion_handoff_pressure_due`, on the theory that survivor
+  *placement* should not read a band derived from full-GC-yield evidence. Reverted:
+  the premise was the phantom above, and the principle was wrong too — that
+  predicate's own doc says it decides "whether an imminent promotion justifies a
+  full old reclaim FIRST", and `gc::mod.rs` answers it with
+  `note_survivor_promotion_handoff_full`. All three callers of
+  `old_reclaim_pressure_due` are full-collection decisions, so one signal and one
+  multiplier is the coherent shape, not a carve-out.
+
+#### Validation
+
+All 19 corpus programs byte-identical to `node --experimental-strip-types` with
+exit 0. `gc-handoff/apps/iso_miss.ts` prints `checksum 437840 misses 0`, including
+under `PERRY_GC_PROTECT_FROMSPACE=1 PERRY_GC_PROTECT_FROMSPACE_DEPTH=800` and
+`PERRY_GC_VERIFY_EVACUATION=1`. Protected set unchanged within noise: `churn`
+0.421, `churn_alloc` 0.374, `push_cls` 0.358, `push_num` 0.137, `churn_read`
+0.022, `cycles` 0.193, `tree` 1.631, `tree_wide` 2.113, `fib40` 0.394, `interp`
+1.889, `asyncpipe` 0.710, `shapes` 0.219, `pipeline` 0.543.
+
+#### Gap suite
+
+Every gap failure outside `test-parity/gap_snapshot.json` reproduces **identically
+on the `origin/main` @ `0a2bf15bd` reference build**, on the same host:
+
+* `test_gap_gc_rest_argument_rooting`, `test_gap_gc_same_module_call_argument_rooting`
+  — byte-identical output from both builds, zeal verdict line included
+  (`forced_collections=379 copying_minors=379 moved_objects=115536` and
+  `749/749/231012`). The mismatch is the `[gc-zeal]` exit verdict against a node
+  oracle that does not print it.
+* `test_gap_gc_alloc_point_no_move` — timed out against the harness's 10 s limit
+  on a dev host at load 65+, where both builds take 9–11 s. On the quiet mini:
+  main 2.35 / 1.88 / 1.89 s, this branch 2.23 / 1.88 / 1.88 s.
+* `test_gap_fetch_request_from_node_incoming_message`,
+  `test_gap_http_client_no_redirect_follow`, `test_gap_http_overloads_3226plus`,
+  `test_gap_http_req_async_iterator`,
+  `test_gap_http_res_socket_writable_onfinished`, `test_gap_net_connect_bound_value`
+  — SIGABRT (exit 134) on both builds.
+* `test_gap_specabi_reassign`, `test_gap_zlib_3285_params` — diverge from the node
+  oracle byte-identically on both builds.
+
+The harness also reports ten `node_fail -> parity_fail` status changes
+(`test_gap_4510_enum_forward_ref`, `..._backoff_options`, `..._cron_cronjob`,
+`..._dayjs_factory_arg`, `..._derived_param_props`, `..._enum_in_function_body`,
+`..._moment_methods`, `..._prop_plan_cache_invalidation`, `..._ratelimiter_memory`,
+`..._slugify_options`) and one improvement (`test_gap_iterator_helpers_2874`).
+Those are the oracle's environment, not Perry's output: the snapshot records
+`node_fail` for files whose node run cannot resolve an npm import or whose
+TypeScript syntax strip-only mode refuses, and this host resolves some of them.

crates/perry-runtime/src/arena/page_meta.rs

@@ -690,7 +690,13 @@ pub(crate) fn classify_heap_space(addr: usize) -> HeapSpace {
 /// a range base); the base is precisely the guard that keeps a *garbage*
 /// candidate address at the very start of a range from turning into a read of
 /// the unmapped page below it (#7742).
-#[inline]
+/// Split hit/miss exactly like [`classify_heap_generation`] above, and for the
+/// same reason (#7469): with the map-lookup arm inlined alongside it, the whole
+/// function stayed out of line and every call paid its own `_tlv_get_addr` for
+/// the cache base. This one is the copying minor's inner loop —
+/// `CopyingPointerSet::classify_arena` calls it once per visited slot — so on a
+/// promotion-heavy cycle it runs millions of times per collection.
+#[inline(always)]
 pub(crate) fn classify_heap_space_in_range(addr: usize) -> Option<(HeapSpace, usize)> {
     if addr == 0 {
         return None;
@@ -700,18 +706,20 @@ pub(crate) fn classify_heap_space_in_range(addr: usize) -> Option<(HeapSpace, us
     if let Some(range) = unsafe { (*hot_page_generation_cache()).lookup(key, addr) } {
         return Some((range.space, range.base));
     }
+    classify_heap_space_in_range_uncached(addr, key)
+}
 
+/// Cache-miss arm of [`classify_heap_space_in_range`].
+#[inline(never)]
+fn classify_heap_space_in_range_uncached(addr: usize, key: usize) -> Option<(HeapSpace, usize)> {
     let found = {
         let pages = hot_page_generations().borrow();
         pages.get(&key).and_then(|slot| slot.find(addr))
     };
-    if let Some(range) = found {
-        // SAFETY: as above.
-        unsafe { (*hot_page_generation_cache()).insert(key, range) };
-        Some((range.space, range.base))
-    } else {
-        None
-    }
+    let range = found?;
+    // SAFETY: thread-local, single-threaded, and the borrow ends here.
+    unsafe { (*hot_page_generation_cache()).insert(key, range) };
+    Some((range.space, range.base))
 }
 
 pub(crate) fn old_object_page_overlaps(

crates/perry-runtime/src/gc/barrier.rs

@@ -241,21 +241,43 @@ fn dirty_slot_ranges_for(
         return Vec::new();
     };
 
+    // Walk whichever side is smaller. Iterating the dirty-page set is O(dirty
+    // pages) regardless of the range's size, which is the right shape for the
+    // one huge array this exists for — but it is quadratic when a heap holds
+    // MANY small pointer ranges (each would rescan the whole set). Enumerating
+    // the range's own pages instead is O(range pages) with one set probe each.
+    // Both arms produce the same ranges; only the traversal order differs, and
+    // the merge below sorts.
+    let range_pages = (slots_end - 1).saturating_sub(slots) / PAGE_SIZE + 1;
     let mut ranges = Vec::new();
-    for &page in dirty_pages {
+    let push_page = |page: usize, ranges: &mut Vec<(usize, usize)>, stats: &mut _| {
         let page_start = page << PAGE_SHIFT;
         let page_end = page_start + PAGE_SIZE;
         let start = slots.max(page_start);
         let end = slots_end.min(page_end);
         if start >= end {
-            continue;
+            return;
         }
+        let stats: &mut RememberedSetTraceStats = stats;
         stats.dirty_slot_pages_considered += 1;
         let first = (start - slots) / std::mem::size_of::<u64>();
         let last = (end - slots).div_ceil(std::mem::size_of::<u64>());
         if first < last {
             ranges.push((first.min(slot_count), last.min(slot_count)));
         }
+    };
+    if range_pages <= dirty_pages.len() {
+        let first_page = slots >> PAGE_SHIFT;
+        let last_page = (slots_end - 1) >> PAGE_SHIFT;
+        for page in first_page..=last_page {
+            if dirty_pages.contains(&page) {
+                push_page(page, &mut ranges, stats);
+            }
+        }
+    } else {
+        for &page in dirty_pages {
+            push_page(page, &mut ranges, stats);
+        }
     }
 
     if ranges.is_empty() {

crates/perry-runtime/src/gc/copying.rs

@@ -121,15 +121,21 @@ impl CopyingPointerSet {
         if unsafe { !plausible_gc_header(header, true) } {
             return None;
         }
-        let active_survivor = crate::arena::active_survivor_space();
-        let inactive_survivor = crate::arena::inactive_survivor_space();
+        // The two survivor-space readings are TLS loads, and Darwin has no
+        // local-exec TLS — each is a real `_tlv_get_addr` call. Reading them
+        // eagerly cost two per classified pointer on workloads that never touch
+        // a survivor at all (`retain.ts` classifies Eden / PromotedYoung / Old
+        // and nothing else). They can only ever answer `Survivor0`, `Survivor1`
+        // or `Unknown`, and `space` is already narrowed to the six accepted
+        // spaces, so hoisting the non-survivor arms above them changes no
+        // verdict — it just stops paying for an answer the arm does not use.
         let kind = match space {
             crate::arena::HeapSpace::NurseryEden => CopyingPointerKind::Eden,
             crate::arena::HeapSpace::PromotedYoung => CopyingPointerKind::PromotedYoung,
-            s if s == active_survivor => CopyingPointerKind::FromSurvivor,
-            s if s == inactive_survivor => CopyingPointerKind::ToSurvivor,
             crate::arena::HeapSpace::Longlived => CopyingPointerKind::Longlived,
             crate::arena::HeapSpace::Old => CopyingPointerKind::Old,
+            s if s == crate::arena::active_survivor_space() => CopyingPointerKind::FromSurvivor,
+            s if s == crate::arena::inactive_survivor_space() => CopyingPointerKind::ToSurvivor,
             _ => return None,
         };
         Some(CopyingPointer { header, kind })
@@ -1578,6 +1584,11 @@ pub(super) fn gc_collect_minor_copying_fast_path_with_eligibility(
     // the stale baseline and schedules a full that is guaranteed to free
     // nothing (see `credit_promoted_bytes_to_old_baseline`).
     credit_promoted_bytes_to_old_baseline(collector.stats.promoted_bytes);
+    // The same argument one trigger over: a young generation that did not die
+    // is a heap growing by LIVE data, so arena-growth pacing must not read that
+    // growth as garbage accumulating. Fed here, after the reset, so the
+    // re-baseline sees post-collection occupancy.
+    note_copying_minor_young_survival(collector.stats.young_survival_permille);
     maybe_schedule_old_reclaim_after_copied_minor();
     retune_after_scavenge(
         collector.stats.eden_live_bytes,

crates/perry-runtime/src/gc/layout.rs

@@ -1570,6 +1570,22 @@ pub(super) enum HeapPayloadSlotScan {
         raw_numeric_array: bool,
         raw_numeric_object_slots: usize,
     },
+    /// [`LayoutSlotMask::AllPointers`]: the mask selects EVERY live payload
+    /// slot, so the slot set is a contiguous range and the descriptor visitor
+    /// emits one `Range` rather than `slot_count` individual `Slot`s.
+    ///
+    /// This is not a micro-optimisation. `scan_dirty_object_slots`'s `Slot` arm
+    /// answers "is this slot on a dirty page?" with a hash-set probe **per
+    /// slot**, so a 3M-element array of pointers cost 3M probes on every minor
+    /// — O(live array) rather than O(dirty pages) — even though the remembered
+    /// set knew only a few hundred of its pages were dirty. Its `Range` arm
+    /// intersects the range with the dirty-page set directly
+    /// (`dirty_slot_ranges_for`), which is what `dirty_slot_ranges_scanned == 0`
+    /// in every `retain.ts` GC trace was recording: the cheap arm was never
+    /// reached, because an all-pointer array is `Masked`, not `All` (#7787).
+    AllPointers {
+        raw_numeric_object_slots: usize,
+    },
     Masked,
     All(HeapSlotRange),
 }
@@ -1650,6 +1666,22 @@ impl HeapChildSlotIterator {
                 raw_numeric_array,
                 raw_numeric_object_slots,
             },
+            HeapPayloadSlotSelection::Masked {
+                mask: LayoutSlotMask::AllPointers,
+                raw_numeric_object_slots,
+                raw_numeric_recorded,
+                ..
+            } => HeapPayloadSlotScan::AllPointers {
+                // Mirror the iterator's one-shot accounting: `next` records the
+                // raw-numeric skip on its first call and never again, so a
+                // descriptor visit that replaces the whole iteration records it
+                // exactly once too.
+                raw_numeric_object_slots: if raw_numeric_recorded {
+                    0
+                } else {
+                    raw_numeric_object_slots
+                },
+            },
             HeapPayloadSlotSelection::Masked { .. } => HeapPayloadSlotScan::Masked,
             HeapPayloadSlotSelection::All { .. } => HeapPayloadSlotScan::All(self.payload),
         }

crates/perry-runtime/src/gc/layout_slot_visit.rs

@@ -38,6 +38,21 @@ pub(super) unsafe fn visit_gc_layout_slot_descriptors(
             }
             visit(GcMutableSlotDescriptor::PointerFreeRange);
         }
+        HeapPayloadSlotScan::AllPointers {
+            raw_numeric_object_slots,
+        } => {
+            if raw_numeric_object_slots != 0 {
+                record_layout_raw_numeric_object_field_range_skipped(raw_numeric_object_slots);
+            }
+            // Same slot set the `Masked` arm below would emit one-at-a-time
+            // (`AllPointers` yields every index in `0..slot_count`), handed over
+            // as a contiguous range so `scan_dirty_object_slots` can intersect
+            // it with the dirty-page set instead of probing that set per slot.
+            visit(GcMutableSlotDescriptor::Range {
+                range: child_slots.payload,
+                layout_kind: Some(HeapChildSlotReadKind::Masked),
+            });
+        }
         HeapPayloadSlotScan::Masked => {
             for child_slot in child_slots {
                 if let HeapChildSlot::Child(slot, layout_kind) = child_slot {