feat: report interrupted teardown drains; never subordinate a user abort

[rasdani]

Follow-up to #1920 (merged into this base). Two changes, both about what happens when a user interrupts a run whose shielded teardowns are still draining.

1. Report the drain, in aggregate

The shield keeps teardowns running through Ctrl-C — but silently, so the wait reads as a hang; a user whose interrupt "does nothing" escalates to kill -9, which skips the atexit backstop and guarantees the leak. Now stop() tracks in-flight teardowns in a registry (_STOPPING) and the first teardown to absorb the interrupt logs one aggregate line; one closing line marks the drain finished. One line for 512 in-flight teardowns, not 512 — dedup via a flag that resets when the registry empties, so a later drain in a long-lived process reports again. Per-runtime detail (name + provider descriptor) logs at DEBUG for --verbose.

How the terminal reads

A Ctrl-C'd validate run (non-rich mode; WARNING lines are this PR, the rest is existing behavior):

01:44:03    INFO idx=0 valid=True reason=valid (3.2s)
01:44:04    INFO idx=1 valid=True reason=valid (3.4s)
^C
01:44:10 WARNING finishing 3 in-flight teardown(s) (oldest 2s) — Ctrl-C again to abort; leftover resources are freed best-effort at exit
          (…silence while the deletes finish…)
01:44:11 WARNING all in-flight teardowns finished
Traceback (most recent call last):
  ...
KeyboardInterrupt

The closing KeyboardInterrupt traceback is the pre-existing exit path (asyncio.run re-raises it), not something this PR adds. Pressing Ctrl-C again mid-drain aborts immediately: the loop dies, the atexit backstop frees what's left silently, then the same traceback. With --verbose, each teardown additionally identifies itself:

01:44:10   DEBUG finishing teardown of validate-noop-2-9c7279e1 (qj0gk9ra8h341h6n5e2zwcq3)

("freed best-effort at exit" = the sync atexit backstop, cleanup_at_exit.)

Known gap: under the --rich dashboard the report goes to the log file only (console=False — the dashboard owns the screen). Surfacing drain state in the dashboard is its own piece of work.

2. run_shielded: narrow except BaseException to except Exception

Raised by Cursor Bugbot on #1920 (partially valid). The headline mechanism there was off — a signal-handler KeyboardInterrupt normally lands in the event loop's frame (a suspended coroutine isn't executing) and kills the loop as designed — but the broad catch was still wrong in three narrow windows: a signal delivered while the interpreter executes run_shielded's own frame was swallowed (the abort silently ignored); a KeyboardInterrupt/SystemExit raised out of the teardown itself got chained under the pending CancelledError instead of winning; and a force-closed coroutine would swallow GeneratorExit once and re-await ("coroutine ignored GeneratorExit"). The catch's only job — defer the inner task's failure so it unifies with the cancellation at the bottom — needs Exception only. A user abort now always propagates immediately.

Verification

• Full test matrix -m "not prime and not modal" passes.
• test_cancelled_stop_still_frees_runtime — the e2e trip-wire for the guard, parametrized over runtimes so it actually gates PRs: subprocess (always in CI), docker (CI matrix), prime (local-only, real API DELETE observed server-side). A gate holds teardown open so the cancel deterministically lands mid-stop(); asserts teardown ran to completion, the resource is really gone (workdir / container / sandbox TERMINATED), the drain was reported, and the cancellation propagated. Verified to fail on all CI variants when the run_shielded guard in Runtime.stop is removed.
• Live smoke test (docker, 3-second teardown window, single SIGINT mid-drain): the WARNING lines above are the verbatim output, exit 130, zero leaked containers. A second SIGINT mid-drain exits immediately with the atexit backstop leaving no containers behind.

🤖 Generated with Claude Code

---

Note

Medium Risk
Changes shutdown and cancellation behavior on the path that frees containers and paid sandboxes; fixes are guarded by new e2e leak checks but alter how second Ctrl-C and BaseException interact with shielded work.

Overview
When Ctrl-C hits while shielded runtime teardown is still running, Runtime.stop now tracks in-flight stops and logs one aggregated warning (count, oldest age, hint to Ctrl-C again) on the first absorbed cancellation, then a single closing line when the drain finishes or is aborted. Per-runtime detail stays at DEBUG.

run_shielded gains an optional interrupted callback (used by stop() for that report) and no longer swallows KeyboardInterrupt / SystemExit: user aborts propagate immediately instead of being chained under a pending CancelledError.

With console=False (rich dashboard), WARNING logs still print via Rich’s console so drain messages don’t look like a silent hang.

A new parametrized e2e test (subprocess / docker / prime) cancels mid-stop(), asserts teardown completes, resources are freed, drain text is logged, and cancellation still propagates.

^{Reviewed by Cursor Bugbot for commit b2a917e. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Report interrupted teardown drains and immediately propagate user aborts in Runtime.stop

• Runtime.stop now tracks in-flight teardowns in a module-level _STOPPING registry and, on first absorbed cancellation, logs a single aggregated warning with the in-flight count and oldest teardown age.
• run_shielded in aio.py accepts an optional interrupted callback invoked on the first absorbed CancelledError; BaseException (e.g. KeyboardInterrupt, SystemExit) now propagates immediately instead of being swallowed.
• When console=False, WARNING-level logs are now emitted via the active rich Console rather than suppressed, so warnings surface above a Live region.
• A new e2e test test_cancelled_stop_still_frees_runtime validates that teardown completes and resources are freed even when cancellation arrives during stop(), and that the aggregated warning is logged.
• Behavioral Change: KeyboardInterrupt/SystemExit during a shielded teardown now re-raises immediately; previously these were caught and suppressed by run_shielded.

^{Macroscope summarized b2a917e.}

[macroscopeapp]

Approvability

Verdict: Needs human review

This PR introduces new feature behavior around interrupt handling during teardown, including a significant change from catching BaseException to Exception (allowing KeyboardInterrupt/SystemExit to propagate immediately). Runtime behavior changes of this nature around critical teardown/cleanup paths warrant human review.

^{You can customize Macroscope's approvability policy. Learn more.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit bc98d9e. Configure here.}