fix(gc): refuse the knob combination that emits a rootless binary (#7326)

[proggeramlug]

Fixes #7326.

PERRY_SHADOW_STACK=0 + PERRY_STATEPOINTS=1 produced a binary with no precise frame roots at all — no __perry_gcmap section, same size as a plain shadow-off build, and it ran and printed the correct answer. Nothing distinguished it from a correct build until a collection moved something live.

The cause is structural, not a missing check

The statepoint backends are an alternative lowering of the shadow stack's root-set analysis, not an independent mechanism. reserve_shadow_slot() is the single entry point that, under native_stack_roots_enabled(), allocates a stack-map slot instead — and the caller returns empty maps outright when the shadow stack is off. Switching one off switches the other off with it.

The consequence for the adoption plan

Because they share this analysis, "delete the shadow stack and keep statepoints" is not currently expressible. Any plan treating them as two interchangeable mechanisms — including the one in docs/engine-plan.md — needs that premise corrected before the swap can happen. That is a decoupling job, not a flag flip.

Verified

• The combination now refuses, naming the reason and PERRY_SHADOW_STACK=0 + PERRY_STATEPOINTS=1 silently emits a binary with no precise roots #7326.
• statepoints alone: compiles. shadow-off alone: compiles, so the bisection knob keeps its meaning. default: compiles.
• file-size, GC store-site, addr-class gates green; cargo fmt clean.

Summary by CodeRabbit

• Bug Fixes

  • Added validation to prevent incompatible runtime configuration combinations.
  • The application now reports a clear error when shadow stacks are disabled alongside statepoint or RS4GC modes, avoiding unsupported builds.

• Documentation

  • Added changelog documentation describing the configuration restriction and its impact.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 6fd9ae83-6bf0-4eae-8817-6bf83e9f48d3

📥 Commits

Reviewing files that changed from the base of the PR and between 564cbec and b7c7198.

📒 Files selected for processing (2)

• changelog.d/7331-rootless-combination-refused.md
• crates/perry-codegen/src/codegen/helpers.rs

---

📝 Walkthrough

Walkthrough

The codegen now rejects PERRY_SHADOW_STACK=0 when statepoints or RS4GC are enabled. The changelog documents that these backends depend on shadow-stack root-set analysis.

Changes

Rootless backend validation

Layer / File(s)	Summary
Shadow-stack backend compatibility guard crates/perry-codegen/src/codegen/helpers.rs, changelog.d/7331-rootless-combination-refused.md	shadow_stack_enabled now raises a descriptive panic for incompatible statepoint or RS4GC settings. The changelog documents the shared root-set analysis and hard-error policy.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

• PerryTS/perry#7292: Both changes address GC correctness involving statepoints and shadow-stack root tracking.
• PerryTS/perry#7314: This change extends related statepoint and RS4GC configuration checks in perry-codegen.
• PerryTS/perry#7322: Both changes cover native-root backend compatibility with shadow-stack settings.

Suggested reviewers: andrewtdiz, thehypnoo

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/7326-rootless-combination

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}