Skip to content

feat(ui): adopt freenet-migrate 0.5 for the delegate-secret walk - #617

Draft
sanity wants to merge 7 commits into
mainfrom
feat/migrate-crate-adoption
Draft

feat(ui): adopt freenet-migrate 0.5 for the delegate-secret walk#617
sanity wants to merge 7 commits into
mainfrom
feat/migrate-crate-adoption

Conversation

@sanity

@sanity sanity commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Problem

River hand-rolls its delegate-secret recovery (the legacy sweep in chat_delegate.rs /
response_handler.rs, 27 registered generations). Four Freenet apps independently wrote the
same registry-and-probe loop; freenet-migrate exists to remove that duplication, and Delta
and ghostkeys have already adopted its 0.5 delegate walk. River was the largest remaining
hand-rolled delegate implementation (#398 phase 3, freenet-core#2776 A3).

Separately, River has a concrete recurring cost the hand-rolled sweep cannot fix: in the
gateway iframe localStorage is unavailable, so the sweep's is_legacy_migration_done flag
never persists and every page load re-probes all 27 generations. Durable markers fix that.

Approach

Release 1 of 2: the crate's walk runs ALONGSIDE the existing sweep (the shape Delta
shipped). Nothing is removed from the sweep; the walk adds the shared library's
classification, withholding, and durable per-predecessor markers. Retiring the sweep is
release 2, after field validation.

ui/src/components/app/freenet_api/delegate_migration.rs implements the two crate seams:

  • RiverPredecessorIo reads predecessors through River's own delegate protocol
    (ListRequest/GetRequest over the pending-request oneshot side-table), with a
    concurrent pre-warm so 27 mostly-absent generations cost one 10s timeout window
    per load instead of ~4.5 minutes of sequential probes.
  • RiverSuccessorIo writes the successor through River's own import path and keeps the
    marker bookkeeping in the CURRENT delegate's KV store.

Wired into fire_legacy_migration_request behind a tested once-per-page-load latch, so
the walk inherits the #253 gate (it only starts when the current delegate was
observed empty, or an interrupted migration is being recovered).

Deliberate divergences (each is load-bearing)

  1. freenet-migrate#19 override — a timeout is NOT an absence. The crate's recommended
    semantics treat silence as "the predecessor does not have it"; that is a data-loss default
    and this adapter deliberately does not use it. Every round-trip is three-way: reply /
    silence / transport fault. Silence and faults classify the predecessor Unresponsive
    (retry next load) — they never read as "no data", never write, never seal a marker. A
    node-side DelegateError is uncorrelated with its request, so "delegate errored" and
    "no reply" are indistinguishable — both are UNKNOWN.

  2. Durable markers — the OPPOSITE of ghostkeys' choice, on purpose. ghostkeys bans
    durable markers because its predecessor stores keep receiving writes after the re-key, so
    sealing there can strand late-arriving data. River's legacy delegates are FROZEN after a
    re-key (only the current delegate is ever written), so sealing is safe — and durability is
    what fixes the iframe re-probe waste above. Markers live in the current delegate's KV
    store under __migrate_pred_done__:<hex> / __migrate_pred_wip__:<hex>. The predecessor
    key is hex-encoded, never raw: the delegate's create_origin_key runs storage keys
    through String::from_utf8_lossy, which maps every invalid byte to U+FFFD, so two raw
    32-byte keys could alias onto one marker slot (sealing a predecessor that was never
    migrated).

  3. fetch_secrets UNIONS the legacy ListRequest with fixed rooms_data +
    outbound_dms probes.
    The frozen legacy WASM swallows index decode errors into an
    empty list (handlers.rs — cannot be fixed), so List alone is not trustworthy. The union
    is a floor, not a full fix: dynamic room:<vk> keys are unguessable, so a corrupt index
    still strands those.

  4. One merge, ranked. Every recovered room routes through Rooms::merge_from_source
    with RecoveredSecret::generation as the source rank — the SAME rank scale the sweep
    uses (source_rank_for_delegate_key = registry index; pinned by a test against the real
    registry). Ranked tombstones (A legacy generation's tombstone can permanently delete a room the current delegate holds Present #590) and identity conflicts (Lost IDs and rooms to River Update across two nodes. #527) are therefore handled
    identically to the sweep, with no second merge implementation. Persistence goes through
    the coalesced save_rooms_to_delegate per-room CAS, and flush_predecessor reports that
    save's REAL outcome — a failed flush withholds the completion marker.

  5. Policy = UnionAllGenerations (the crate's loud opt-in): NewestSnapshotWins halts
    at the first silent predecessor, and silence is ordinary here (most of the 27 generations
    were never installed on a given node) — halting would strand every older generation
    (the legacy_delegates.toml V1-V3 entries point at stdlib-incompatible delegates #204 failure class). Union's resurrection hazard is bounded by River's ranked
    tombstones; the run-scoped-withholding hazard (freenet-migrate#15) is acknowledged, not
    fixed here.

Delegate WASM is byte-identical

This is UI-side only. ui/public/contracts/chat_delegate.wasm is untouched:
b3sum = a44c64014d60fd245fe8fb5172f8fd7397039b248b3b6a8e95e89a0bb539fb5e (the required
pin), now enforced by the chat_delegate_wasm_is_byte_identical test.
legacy_delegates.toml is untouched (27 generations).

freenet-migrate 0.3 → 0.5 in ui/ and cli/: the contract-side probe API
(backward_probe.rs, cli/src/api.rs) is source-compatible — both compile unchanged.

Testing

17 new native tests in delegate_migration.rs + the WASM pin in chat_delegate.rs. The
scenario tests drive the REAL freenet_migrate driver through walk_with with a scripted
transport and an in-memory seam that holds a real Rooms + MergeRanks, so classification,
the union fetch, marker bookkeeping and the ranked merge are the production code paths.

Mutation evidence — every mutation applied, watched RED, then reverted (baseline 21/21
re-confirmed green after the final revert). Canary first: a panic! patched into the
crate's migrate_delegate_secrets via [patch.crates-io] failed exactly the 11 scenario
tests (10 unit/pin tests unaffected), proving the harness drives the real crate driver.
No mutation survived.

# Mutation Killed by (count of failing tests)
M1 predecessor GET timeout → "absent" a_silent_get_is_unresponsive_never_absent_never_sealed (1)
M2 probe silence → "executable" a_silent_predecessor_does_not_halt_the_union_walk (1)
M3 ranked merge → raw overwrite the #527 + #590 fixtures (2)
M4 oldest lineage generation dropped (.skip(1)) 9 tests incl. both lineage pins + full walk
M5 GET-reply key-mismatch check removed a_mismatched_get_reply_is_a_fault_not_data (1)
M6 latch always fires arm_walk_latch_fires_exactly_once (1)
M7 call-site latch gate removed crate_walk_is_wired_behind_the_latch_and_the_253_gate (1)
M8 marker read timeout → Ok(None) an_unreadable_marker_stops_the_walk_before_any_import (1) — the fetch-count assert caught 2 predecessor GETs running without marker bookkeeping
M9 marker read fault → fabricated Done same test (1)
M10 flush_predecessor no-op'd a_failed_flush_withholds_the_completion_marker + full walk's flush-count assert (2)
M11 marker keys carry raw predecessor bytes marker_keys_hex_encode_the_predecessor_and_survive_utf8_lossy (1)
M12 fixed probes removed (trust List alone) unit rule + corrupt-index end-to-end (2)
M13 policy weakened to NewestSnapshotWins a_silent_predecessor… + full walk (2)

Suite status on this branch: cargo test -p river-ui --bins 921 passed / 0 failed;
cargo make test (web-container, room-contract, scaffold, common, chat-delegate +
integration) 354 passed / 0 failed; cargo make build-ui succeeds; cargo check -p riverctl and the wasm32 check pass; fmt applied; clippy clean for the new module.

Review round 2 — two blocking bugs, both fixed

All CI was green on this PR — including check-delegate-migration, build, and
Playwright — while the feature could not work at all in production.
It also had 19
tests and the 13-mutation campaign above, with predictions written first and all 13
killed. Both bugs survived every bit of that, because they live in layers that harness
structurally could not reach. Green CI on the earlier head said nothing about whether
the walk functioned.

The generalisable lesson: mutation testing inherits the blind spot of its harness.
Killing every mutation proves discrimination only over paths the harness reaches. Ask
what layer the test double stands in for, and whether the bug you care about is above or
below it.

B1 — the walk's ListRequests could never be answered (feature was dead code)

enqueue_delegate_request_to registered a legacy ListRequest's waiter under
legacy_scoped_correlation(delegate, base), but response_handler.rs completed
ListResponse under a freshly built bare b"__list_request__". It was the only
storage variant that skipped the scoped() closure; Get/Store/Delete/GetVersioned/
CasStore all used it. complete_pending_request is an exact-key removal, so the waiter
was never completed: every prewarm probe and every key_index retry timed out after
10 s, leaving all 27 generations permanently Unresponsive. It fails SAFE (silence never
seals) but recovers nothing, at a cost of 10 s and 27 warnings per page load.

The blocking half: each prewarm ListRequest reaching an installed legacy delegate still
yielded a ListResponse, which was routed unconditionally into migrate_legacy_per_room
— a duplicate concurrent instance beside the one the sweep already spawned. Identical
scoped correlation keys, so single-waiter displacement produced Cancelled reads and a
possible false LoadFailed for a user whose migration actually succeeded.
Deterministic, not a race.

Fixed by root-causing the shape rather than the instance. The correlation base now comes
from one shared list_request_correlation_key(); the response side derives every storage
key through a single response_correlation_base() mirroring get_request_key; and the
legacy scoping is applied in exactly one place instead of six hand-written
per-variant arms, so no variant can be left unscoped again. The duplicate spawn is gated
on !completed — verified safe because both existing ListRequest senders (the
current-delegate load and the sweep) are fire-and-forget with no waiter, so they still
drive their migrations exactly as before.

B2 — flush could seal Done over DM data that was never saved

merge_outbound_dms called hydrate helpers that defer their signal writes through
util::defer (setTimeout(0), a macrotask) and returned Ok immediately, while
flushsave_outbound_dms_to_delegate reads OUTBOUND_DMS synchronously. The
merge→flush→snapshot chain runs on microtasks, which drain first. For a DM-only
predecessor (no rooms, so no intervening network round-trip) the save deterministically
serialised the PRE-merge cache, the driver sealed Done { had_data: true }, and the
marker guarantees the predecessor is never re-fetched. Silent, permanent loss.

Worst in the gateway iframe — no localStorage, so the sweep's flag never persists and
the marker is the only authority. That iframe case is the stated motivation for durable
markers, so the bug was worst exactly where the feature matters most. outbound_dms is
also a whole-blob replace-not-merge StoreRequest, the one store in River with Delta's
dangerous whole-list shape.

It now runs both writes inside one defer and awaits a oneshot signalled from within it,
exactly as merge_rooms already did correctly.

Also in this round

  • S3 — the pre-warm's join_all could overlap the single WEB_API write borrow held
    across api.send().await: a RefCell double-borrow, which in single-threaded WASM is a
    panic. It was survived only by WebApi::send happening not to yield for small payloads,
    a property of the transport rather than of this code. The channel trait gains
    request_all, whose default is today's fan-out (correct for doubles, keeps their
    concurrency assertions unchanged) and whose NodeDelegateChannel override enqueues
    sequentially then awaits together — the shape the startup per-room fan-out already uses.
  • N1 — marker reads logged ~54 Unexpected key in GetResponse warnings per run; now
    swallowed like the per-room keys directly above them.
  • N2 — the walk was armed before its sends, so a load whose every send failed burned
    the once-per-page-load latch on a dead transport. Arming now happens after
    any_dispatched is final and is gated on it, matching the guard-reset precedent
    immediately above it. Still at most one arm per load, so a reconnect cannot stack a
    second concurrent walk.

Reasoning that was load-bearing but undocumented

  • Check 7: sweep and walk are NOT independent, and that is what makes the data path
    safe.
    They deliberately share the merge point, the save chokepoint and the correlation
    map: frozen predecessors, one ranked merge on a shared rank scale, CAS/coalesced saves,
    and a seal attesting only to the walk's own import. No interleaving can be constructed
    that loses a room or seals wrongly. The single exception was outbound_dms — B2.
  • Nothing writes a whole-list rooms_data any more. Storage has been per-room CAS
    since Multi-tab room loss: chat delegate rooms_data is a blind full-blob overwrite (last-write-wins across tabs) #345; the legacy blob is read-only, kept purely as a rollback fallback.
  • The 60 s quiescence wait is load-shedding, not a safety mechanism. PENDING_LOADS
    counts only response-processing workers, so it reads 0 between the sweep's sends and its
    responses. Correctness under overlap comes from the correlation scoping and the markers,
    not from this wait.
  • Inherited bug: old delegates overwriting the current active delegate #253 coupling: once the sweep marks done, fire_legacy_migration_request
    stops firing, so the walk stops retrying Unresponsive predecessors.

Mutation campaign on the fixes

Re-run against the changed code, predictions written first; all killed, each verified
actually applied before running.

# Mutation Test that caught it
M1 scoped() dropped at the single correlation site response_handler_derives_correlation_keys_from_one_place
M2 List base drifts on the response side only request_and_response_correlation_keys_round_trip
M3 !completed gate removed (duplicate migration spawns) consumed_legacy_list_response_does_not_double_spawn_migration
M4 deferring hydrate helper restored inside the merge merge_outbound_dms_awaits_its_deferred_writes
M5 the DM-merge await dropped same test
M6 any_dispatched gate removed from the walk arming crate_walk_is_wired_behind_the_latch_and_the_253_gate
M7 prewarm reverted to join_all over request prewarm_fans_out_through_the_borrow_safe_primitive

Two of these pins are deliberately source scrapes rather than behavioural tests, and
that is the point rather than a shortcut. util::defer is setTimeout(0) on wasm32 but a
synchronous call natively, and MemorySeam merges synchronously — so on every surface a
test can actually run, the deferred write has always landed by the time flush reads. A
behavioural test for B2 would pass identically with the bug present, which is worse than
no test. The same applies to S3: a mock holds no shared WEB_API borrow, so it cannot
observe the double-borrow.

The previous correlation pin is also corrected. It checked only the request side, for
distinctness, against a base literal (b"list") that appears nowhere in the code — so a
mismatch across the seam was the only interesting failure and the only one it could not
detect. A guard at a seam must check both sides of the seam.

Withdrawn from the round-1 review

Two findings (revert contract-version.txt; split the member_info_modal.rs change) were
false and have been withdrawn — neither file is in this PR, which touches exactly six.
They came from a three-dot diff against a stale local main, which attributes main's own
newer commits to the branch. Called out here because the author pushed back with evidence
instead of complying, which is the only reason they were caught.

Review round 3 — the round-2 pins were audited, and several were defeatable

Four independent lenses reviewed the round-2 fixes. They confirmed the two blocking
fixes are correct, and found that a number of the pins protecting them would pass with
their bug reintroduced. Those are fixed. Given this PR's own thesis, a pin that cannot
fail is worse than no pin, so these are reported rather than quietly corrected.

The most serious was mine, and it is worth stating plainly. The assertion that the
single correlation site applies scoped() was written and mutation-verified, then
destroyed by a git checkout -- used to revert a mutation while it was still
uncommitted. Its loss went unnoticed because only two of the three affected edits were
re-checked afterwards. Production was always correct; the pin protecting it was simply
absent from the pushed tree
, so the round-2 mutation table's "M1 killed" did not hold
for the committed code. Restored and re-verified. The general rule — commit before
mutating — was followed for the first campaign and not the second.

Fixed in round 3:

  • The scoped() assertion is restored, and the blast radius is now recorded: an unscoped
    completion site also kills the hand-rolled sweep's per-room recovery, because
    migrate_legacy_per_room reads through the legacy-scoped send_delegate_request_to.
    Collapsing six arms into one made that single site wider than the bug it fixed.
  • request_and_response_correlation_keys_round_trip now round-trips through the real
    PENDING_REQUESTS registry
    : it registers a waiter under the scoped key and asserts a
    bare-key completion does NOT satisfy it while the scoped one does. The previous scoping
    assertions were f(a) == f(a) on a pure function and could never fail, while the
    doc-comment advertised them as covering the half B1 broke.
  • request_all loses its trait default. The obvious default is a join_all over
    request — exactly the unsafe shape — so deleting NodeDelegateChannel's override
    would have silently returned production to the overlapping-borrow fan-out with every
    test green. MockChannel now states its naive fan-out explicitly, with a note that its
    max_in_flight assertions measure the double and never production.
  • The B2 pin bounded tx.send only by the defer's opening. Placing the send after the
    whole closure compiles, satisfies every assertion, and restores B2 exactly. Now bounded
    by the closure's terminator.
  • production_request_all_enqueues_before_it_awaits checked only text order, which a
    fully sequential enqueue-and-await-in-one-loop rewrite also satisfies — borrow-safe, but
    serialising the pre-warm into 27 x the 10 s timeout. Now also requires a real fan-out
    and requires the enqueue loop to close first.
  • The duplicate-spawn pin required the guard to precede the spawn, which also passes for
    if !completed { debug!(..); } followed by an ungated spawn. Now requires containment.
  • response_correlation_base enumerates the non-storage variants instead of _ => None,
    so a new storage variant is a compile error rather than a silently uncorrelated
    response — B1's exact failure mode.
  • Both _now hydrate helpers regain the empty-input short-circuit, so a pre-archive
    predecessor no longer triggers a no-op GlobalSignal write per predecessor.
  • prewarm asserts request_all returned one outcome per request; the length contract
    was documentation only, so a short vec would have silently dropped tail predecessors.

Comment corrections, each verified against source rather than assumed:

  • There is only ONE PENDING_REQUESTS map. The non-storage responses use it under
    prefixed keys. The previous wording ("a different registry, this map is not involved")
    would leave a reader unaware that those four are unscoped for legacy targets.
  • The !completed guard's real invariant is a counting argument, not the sender-based
    one previously given. Responses bind to waiters by correlation key, so the sweep's
    response can complete the walk's waiter; safety comes from N responses meeting at most
    N-1 waiters. Documented, with the two reachable exceptions (a timed-out waiter evicted
    before its response lands; the sweep's send failing while the walk's succeeds). Both
    fail-safe, both recovered next load.
  • "No variant can be left unscoped" holds only within the storage family.
  • Removed a reference to a pin that never existed, and corrected a stand-in-base
    attribution that named the wrong test in the wrong file.
  • request_all removes the overlap within the pre-warm, not the cross-task case.
  • The N2 gate's residual is stated rather than implied away.

The repo's own rules files were also stale in a way that matters: both told an author a
new fixed storage key needs two edits when it now needs three (the walk carries
its own fixed-probe list), and nothing under .claude/ mentioned the walk or its durable
markers at all. Corrected in this PR, since that is the "next agent follows stale
instructions" failure the third site exists to prevent.

Known remaining gap, stated rather than closed: there is still no test that runs the
sweep and the walk concurrently against one mock delegate. MockChannel models per-delegate
KV stores and has no correlation map, no single-waiter slot and no displacement — so the
layer B1's blocking half lived in is still not exercised end to end. The round-trip test
now covers register-then-complete for a single driver, which is the part that was
achievable without new harness. A real two-driver test needs a registry double with
displacement semantics; that is the next piece of harness work, not something to fake.

Round-3 mutation re-verification

# Mutation Result
M1 scoped() dropped at the single site (restored pin) killed
M8 tx.send moved outside the defer closure killed
M9 List base drifts on the response side killed

Each verified actually applied (non-empty diff) before running, and reverted after.
Suite: 921 passed / 0 failed on cargo test -p river-ui --bins; full cargo make test
green; wasm32 and riverctl clean; no WASM binary touched.

Deployment

No publish in this PR. The UI republish will be rehearsed against a throwaway container
key on a local-mode node before it goes to the production contract. Release 2 (retiring
the hand-rolled sweep) only happens after this release field-validates.

Part of freenet-core#2776 (A3) / #398 phase 3. Follows PR #616.

Full-tier review requested (state migration surface).

[AI-assisted - Claude]

claude and others added 2 commits August 11, 2026 18:12
Preserved from an agent session that hit the account usage limit
mid-implementation. Committing so the work survives worktree cleanup;
this is NOT reviewed, NOT tested, and NOT ready.

What is here:
- freenet-migrate bumped 0.3 -> 0.5 in ui/ and cli/ (contract-side call
  sites in backward_probe.rs / cli api.rs may still need API migration --
  UNVERIFIED).
- ui/src/components/app/freenet_api/delegate_migration.rs, ~955 lines,
  the PredecessorSecretsIo/SuccessorSecretsIo adapter.
- chat_delegate.rs changes exposing what the walk needs.

What is NOT done (from the plan):
- Module not confirmed wired into freenet_api/mod.rs.
- No tests, no mutation evidence, no differential.
- Delegate WASM hash pin not added; b3 must stay
  a44c64014d60fd245fe8fb5172f8fd7397039b248b3b6a8e95e89a0bb539fb5e.
- Compilation unverified.

Design constraints for whoever resumes (see the plan and the
freenet-app-migration skill):
- Markers go in the CURRENT delegate's KV store, hex-encoded predecessor
  key -- durable here, UNLIKE ghostkeys which bans durable markers. River
  legacy data is frozen after re-key so sealing is safe, and durability is
  what fixes the iframe re-probe waste.
- fetch_secrets must UNION ListRequest with fixed rooms_data/outbound_dms
  probes; the legacy WASM swallows index decode errors into an empty list.
- write_secret must route through Rooms::merge_from_source with the
  generation as source rank, never a raw slot write.
- Every fault is UNKNOWN -> retry, never write, never mark. A node-side
  DelegateError is uncorrelated, so "error" and "no reply" are
  indistinguishable.
- Keep the #253 gate; add a TESTED once-per-page-load latch.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BEgtjegwuJPWSaAnVJ3z4e
Completes the WIP adoption: declares the module, fixes the prewarm borrow
bug, adds the once-per-page-load latch (tested) wired into
fire_legacy_migration_request behind the #253-inherited gates, refactors
the entry point through a transport/seam-generic walk_with core, pins the
delegate WASM b3 hash, and adds 17 native tests driving the real
freenet_migrate driver via a scripted transport + in-memory seam (real
Rooms/MergeRanks, so the #590/#527 machinery is exercised, not mocked).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BEgtjegwuJPWSaAnVJ3z4e
@sanity

sanity commented Aug 12, 2026

Copy link
Copy Markdown
Contributor Author

Review: 2 blocking findings — do not merge as-is

Adversarial review by an independent reviewer (Fable 5) that did not write the code, reading the
full new module, the wiring in chat_delegate.rs / response_handler.rs, and the
freenet-migrate 0.5.0 driver source. Suite re-run independently: 915/915.

Both blocking findings sit in the class the author's 13-mutation campaign structurally could
not reach
. That campaign was good work — predictions written first, all 13 killed, plus a
canary that panicked the crate itself to prove the harness was armed. It missed these because
MockChannel implements RiverDelegateChannel directly and bypasses PENDING_REQUESTS and the
response handler, which is exactly the layer B1 lives in; and MemorySeam's synchronous merge
cannot represent B2's async defect.


B1 (blocking) — the walk's ListRequests can never be answered in production

ui/src/components/app/freenet_api/response_handler.rs:243-250

The request and response sides build different correlation keys, so the waiter is never
completed. Verified on both sides:

  • Request: enqueue_delegate_request_to (chat_delegate.rs:6349) registers under
    legacy_scoped_correlation(&delegate_key, &get_request_key(&request)). For ListRequest the
    base is b"__list_request__" (chat_delegate.rs:6191), so the key is delegate-scoped.
  • Response: the ListResponse arm builds a fresh bare b"__list_request__". It is the
    only variant that does not go through the scoped() closure (response_handler.rs:194-203) —
    Get/Store/Delete/GetVersioned/CasStore all do.
  • complete_pending_request is exact-key removal, so the two never meet.

Failure scenario: every prewarm probe and every key_index retry times out after 10 s even
when the legacy delegate answered → Prewarmed::Silentprobe_executableOk(false) for
all 27 generations → every predecessor Unresponsive, permanently. Nothing recovered, nothing
sealed. It fails in the safe direction, but the shipped behaviour is: burn 10 s, log 27 warnings,
recover nothing, every page load.

The regression risk is worse than the dead feature. Each prewarm ListRequest reaching an
installed legacy delegate still produces a ListResponse, which response_handler.rs:485-489
routes unconditionally into migrate_legacy_per_room — a duplicate concurrent instance beside
the one the sweep's own fire-and-forget list already spawned (chat_delegate.rs:7518). Both GET
the same room:<vk> keys under identical scoped correlation keys; single-waiter displacement
resolves one instance's awaits as Cancelledmark_fetch_failure → possible false
LoadFailed/Retry UI for a user whose migration actually succeeded. The walk manufactures this
deterministically, and the quiescence loop cannot prevent it (see 7d).

The pin at this seam cannot fail. walk_is_sequential_and_prewarm_keys_are_distinct
(delegate_migration.rs:2062-2067) checks request-side distinctness using base b"list" — not
even the real __list_request__ base — and asserts nothing about the response side rebuilding
the same key.

Fix: complete ListResponse under scoped(b"__list_request__"); pin the response side
against the real base; and gate the duplicate migrate_legacy_per_room spawn on !completed (or
add an in-flight-per-delegate guard), since the processing match at response_handler.rs:306
runs unconditionally once routing is fixed.


B2 (blocking) — flush can seal Done before recovered DM data is durably saved

ui/src/components/app/freenet_api/delegate_migration.rs:716-723

merge_outbound_dms calls hydrate_hidden_dm_threads / hydrate_outbound_dms_cache, which
defer their signal writes via util::defer = setTimeout(0) — a macrotask — and returns
Ok immediately. flushsave_outbound_dms_to_delegatedo_save_outbound_dms_to_delegate
reads OUTBOUND_DMS synchronously at its start (chat_delegate.rs:5879-5881). The
merge→flush→snapshot chain runs on microtasks, which drain before any setTimeout(0).

Failure scenario: a DM-only predecessor (no rooms, so no intervening network round-trip
between the last item and flush). The save deterministically serializes the pre-merge cache,
StoreResponse returns Ok, and the driver seals Done { had_data: true }. The recovered
plaintext reaches memory but is never durably persisted, and the marker guarantees the
predecessor is never re-fetched.

This is worst in exactly the environment that motivates durable markers: in the gateway iframe
localStorage is unavailable, so the sweep's flag never persists and the marker is the only
authority. And outbound_dms is a whole-blob plain StoreRequest, replace-not-merge
(chat_delegate.rs:5915-5920) — the one store in River with Delta's dangerous whole-list shape —
so a save whose snapshot misses a still-deferred hydrate can also overwrite already-persisted
entries.

Fix: merge_outbound_dms should await its deferred writes via oneshot exactly as merge_rooms
already does (delegate_migration.rs:685-714, which is correct). MemorySeam::merge_outbound_dms
is synchronous, which is why no current test can see this — make the seam able to represent what
was actually merged at flush time.


Check 7 — sweep and walk running concurrently

Requested specifically, because release 1 keeps both mechanisms live.

Verdict: they are NOT independent — they deliberately share every dangerous resource, and that
sharing is what makes the data path safe.
No interleaving could be constructed that loses a
room or seals a predecessor wrongly. Recording the reasons, since they are load-bearing and were
undocumented:

  • 7a — clobber: no, for rooms. One correction to the premise: nothing writes a whole-list
    rooms_data blob any more. Since Multi-tab room loss: chat delegate rooms_data is a blind full-blob overwrite (last-write-wins across tabs) #345 the current delegate is per-room CAS
    (do_save_rooms_to_delegate, chat_delegate.rs:4933-5097); the legacy blob is a read-only
    rollback fallback. Both mechanisms funnel through one in-memory merge point (ROOMS via
    Rooms::merge_from_source, at ranks on the same scale, pinned against the real registry by
    lineage_generation_matches_the_sweep_merge_rank_scale), so any interleaving commutes to the
    same ranked outcome; and one save chokepoint (shared coalesce mutex, snapshot at save
    execution, per-room CAS merge). The single exception is outbound_dms — B2.
  • 7b — walk seals while sweep mid-import: no. Predecessor stores are frozen (both
    mechanisms only read them; markers and imports go to the current delegate), so the walk's fetch
    sees a complete immutable set regardless of sweep state, and the crate seals Done only after
    its own fetch+merge+flush succeeded (verified in 0.5.0 source: permanent rejections also
    yield Incomplete, never Done). The two done-mechanisms don't read each other.
  • 7c — correlation with the sweep in flight: the author's distinctness claim holds between
    walk probes but not against the sweep. migrate_legacy_per_room registers waiters under the
    same scoped keys the walk's fetch uses. Mutual displacement is real, but degrades safely: walk
    side → CancelledErrUnresponsive, no seal, retry next load; sweep side →
    mark_fetch_failure → possible false UI. Nobody ever receives another key's data. B1 makes
    these collisions deterministic rather than rare.
  • 7d — ordering: none. The quiescence loop (delegate_migration.rs:937-940) is
    load-shedding, not a safety mechanism: PENDING_LOADS counts only response-processing
    workers, so it reads 0 between the sweep's sends and its responses, and the walk routinely
    starts mid-flight. Correctness rests on 7a-7c, which is the right design — but the PR body
    should describe it accurately.

On "Delta shipped this shape": the staging strategy transfers, the store shape does not. Delta's
StoreKnownSites is replace-whole-list; River's rooms are per-room CAS (safer), while River's
outbound_dms is the one surface with Delta's shape — exactly where B2 lives.


Should-fix

  • S1published-contract/contract-version.txt bumped 30000381→30000382 in a PR whose body
    says "No publish in this PR". That counter is written by sign-webapp during a publish; it
    looks like a leaked local-rehearsal artifact. Revert.
  • S2member_info_modal.rs carries an unrelated UI: a deputy sees an enabled "Ban User" button on their own Member Info, and self-banning cascades to their whole invite subtree #478 layout change. Mention or split.
  • S3 — prewarm's join_all is the first caller that can overlap
    enqueue_delegate_request_inner's WEB_API.write() borrow held across api.send(...).await
    (chat_delegate.rs:6402-6410). If send ever returns Pending under backpressure, the next
    probe's WEB_API.write() is a RefCell double-borrow panic in single-threaded WASM. The safe
    shape is free: enqueue sequentially, join_all only the await halves.

Nits

  • N1 — marker Get/Store responses fall through to warn!("Unexpected key in GetResponse")
    (response_handler.rs:466-471), ~54 warns per run. Swallow is_migration_marker_key keys.
  • N2 — if the first fire happens with the connection down, the sweep resets
    LEGACY_MIGRATION_ATTEMPTED but CRATE_WALK_ARMED stays set, so the walk burns its
    once-per-page-load shot on a dead transport. Arm after a successful dispatch.

Verified sound (tried to break, could not)

Three-way absence throughout (get_key / get_marker / key_index map silence, mismatch and
unexpected replies to Err; the only Ok(None)-as-absent is a delegate that executed and said
value: None; UndecodableItemWrite::permanent yields Incomplete, so an unparseable blob
is retried, not stranded). Marker ordering (WIP before first item, flush before Done,
driver-enforced; a partial rooms flush withholds the seal). #253 gate inheritance, including that
stored markers cannot flip the current-delegate-empty observation. The latch. Hex marker encoding
vs from_utf8_lossy aliasing. Delegate WASM untouched (byte-identity pin passes;
check-delegate-migration green). The scenario tests genuinely drive the real crate driver and
real Rooms::merge_from_source — not vacuous for the decision logic.

Could not verify

Actual browser/WASM runtime behaviour (no runtime test exists for NodeDelegateChannel, which is
what B1 exploits — everything above is from code reading); coalesce_save's last-result
propagation (took the documented contract at face value); the 0.5.0 driver line-by-line (read for
seal conditions, marker ordering, permanent-vs-retryable only); whether WebApi::send can yield
for small payloads (bears on S3).


B1 and B2 both have contained fixes. The invariant architecture around them held up under
adversarial reading, and the check-7 independence argument is sound once B2's defer-await fix
lands.

[AI-assisted - Claude]

@sanity
sanity marked this pull request as draft August 12, 2026 01:00
@sanity

sanity commented Aug 12, 2026

Copy link
Copy Markdown
Contributor Author

Correction: S1 and S2 in the review above are WRONG — withdrawn

Both cite files that are not in this PR. gh pr view 617 --json files lists exactly six:

Cargo.lock
cli/Cargo.toml
ui/Cargo.toml
ui/src/components/app/chat_delegate.rs
ui/src/components/app/freenet_api.rs
ui/src/components/app/freenet_api/delegate_migration.rs

Credit to the PR author for pushing back on both with evidence rather than complying.

Cause, and it is worth naming because it is a trap in this repo. A three-dot diff against a
stale local main ref shows every commit that has landed on origin/main since your local ref
was last updated, attributed as if it were part of the branch. I hit this exact trap myself on this
same PR about an hour before the review ran, noticed it, and corrected to the real six-file diff —
and then failed to apply that knowledge when relaying someone else's findings. git fetch first,
or diff against origin/main explicitly, and confirm against gh pr view --json files before
asserting a file is in a PR.

B1 and B2 are unaffected. Both were verified independently against the actual source, by me and
subsequently by the PR author, and both stand. So do S3, N1, N2, the check-7 analysis, and the
verified-sound list. Only S1 and S2 are withdrawn.

[AI-assisted - Claude]

sanity added 5 commits August 11, 2026 20:52
Two blocking bugs found by review on #617, both of which survived the
PR's 19 tests and 13-mutation campaign because they live in layers that
harness structurally could not reach.

B1 — the walk's ListRequests could never be answered, so the feature was
dead code. `enqueue_delegate_request_to` registers a legacy ListRequest's
waiter under `legacy_scoped_correlation(delegate, base)`, but the
response side rebuilt a BARE `b"__list_request__"` key. It was the only
storage variant that skipped the `scoped()` closure. Since
`complete_pending_request` is an exact-key removal, every prewarm probe
and key_index retry timed out after 10s, leaving all 27 generations
permanently `Unresponsive`.

Worse, each such ListResponse still routed unconditionally into
`migrate_legacy_per_room`, spawning a duplicate concurrent migration
beside the sweep's. Identical scoped correlation keys, so single-waiter
displacement produced `Cancelled` reads and a possible false LoadFailed
for a user whose migration actually succeeded. Deterministic, not a race.

Root-cause the shape rather than the instance: the correlation base now
comes from one shared `list_request_correlation_key()`, the response side
derives every storage key through a single `response_correlation_base()`
mirroring `get_request_key`, and the legacy scoping is applied in exactly
ONE place instead of six hand-written per-variant arms.

B2 — `flush` could seal `Done` over DM data that was never saved.
`merge_outbound_dms` called hydrate helpers that defer their signal
writes through `util::defer` (`setTimeout(0)`, a macrotask) and returned
`Ok` immediately, while `flush` reads `OUTBOUND_DMS` synchronously on a
microtask chain that drains first. For a DM-only predecessor the save
serialised the PRE-merge cache and the driver sealed
`Done { had_data: true }`; the marker then guarantees the predecessor is
never re-fetched. Silent permanent loss, worst in the gateway iframe
where no localStorage means the marker is the only authority.

It now runs both writes inside one `defer` and awaits a oneshot signalled
from within it, exactly as `merge_rooms` already did.

Tests: a round-trip pin over BOTH sides of the correlation seam for every
variant that uses it (the previous pin checked only the request side, and
against a `b"list"` base that appears nowhere in the code); a pin that the
response side derives keys from one place; a pin that the duplicate
migrate spawn stays gated on `!completed`; and a source pin on the DM
merge await. The DM pin is a source scrape deliberately — `defer` is
synchronous natively and `MemorySeam` merges synchronously, so no test
that can run here is able to observe the ordering, and a behavioural test
would pass identically with the bug present.

Refs #617
… walk latch

Follow-ups from the same #617 review round.

S3 — the pre-warm's `join_all` could overlap the single `WEB_API` write
borrow. `NodeDelegateChannel::request` holds that borrow across
`api.send().await`, so polling a second probe into its send while the
first is suspended there is a `RefCell` double-borrow, which in
single-threaded WASM is a panic rather than contention. It was survived
only by `WebApi::send` happening not to yield for small payloads — a
property of the transport, not of this code, and free to stop being true.

The channel trait gains `request_all`. Its default is today's naive
fan-out, which is correct for test doubles (no shared borrow) and keeps
their observed concurrency unchanged. `NodeDelegateChannel` overrides it
with the safe shape the startup per-room fan-out already uses: enqueue
sequentially so each borrow is confined to its own synchronous send, then
await all replies together. Every request is still in flight before any
reply is awaited, so wire behaviour is unchanged.

N1 — a walk logged ~54 `Unexpected key in GetResponse` warnings per page
load, one per marker read. Markers are consumed by the awaiting walk via
the pending-request registry while the processing match still runs for
every response, exactly like the per-room keys immediately above, so they
are swallowed the same way via the existing `is_migration_marker_key`.
Noise at that volume trains the reader to ignore a warning meant to flag
a real gap.

N2 — the walk was armed before its sends, so a page load whose every send
failed burned the once-per-page-load latch on a dead transport. The sweep
already resets its own per-session guard in that case so a reconnect
re-probes, but the reconnect then found the latch armed and never ran the
walk for the rest of the load. Arming now happens after `any_dispatched`
is final and is gated on it, matching the guard-reset precedent directly
above it. Still at most one arm per page load, so a reconnect cannot
stack a second concurrent walk — the property the latch exists for.

Tests: a source pin that prewarm fans out through `request_all` and not
`join_all` (a mock holds no shared borrow, so it cannot catch this), and
the existing walk-wiring pin extended to require the `any_dispatched`
gate and to require it be read after the value is final.

Refs #617
Review found that several pins from the previous commit could pass with
their bug reintroduced, and that several comments claimed more than the
code supports. Both are worse than no pin and no comment respectively,
given this PR's whole thesis is that green signals were trusted when
they said nothing.

The most serious: the assertion that the single correlation site applies
`scoped()` was NOT present in the pushed tree. It was written and
mutation-verified, then destroyed by a `git checkout --` used to revert
a mutation while it was still uncommitted, and its loss went unnoticed
because only two of the three affected edits were re-checked. Production
was always correct; the pin protecting it was absent. Restored, and the
blast radius is now stated: an unscoped completion site kills the
hand-rolled sweep's per-room recovery too (`migrate_legacy_per_room`
reads via the legacy-scoped `send_delegate_request_to`), so consolidating
six arms into one made that single site wider than the bug it fixed.

- `request_and_response_correlation_keys_round_trip` now round-trips
  through the REAL `PENDING_REQUESTS` registry: it registers a waiter
  under the scoped key and asserts a bare-key completion does NOT satisfy
  it while the scoped one does. The previous scoping assertions were
  `f(a) == f(a)` on a pure function and could never fail, while the
  doc-comment advertised them as covering the half B1 broke.
- `request_all` loses its trait default. The obvious default is a
  `join_all` over `request` — exactly the unsafe shape — so deleting
  `NodeDelegateChannel`'s override would have silently returned
  production to the overlapping-borrow fan-out with every test green.
  Each implementor must now state its choice. `MockChannel` keeps the
  naive fan-out explicitly, with a note that its `max_in_flight`
  assertions measure the double and never production.
- New `production_request_all_enqueues_before_it_awaits` pins the
  override's shape; nothing exercised it before.
- The B2 pin now asserts `tx.send` happens INSIDE the defer, after the
  writes. Signalling before the defer satisfied every previous assertion
  while restoring the bug exactly.
- The duplicate-spawn pin now requires the spawn to be the guarded
  block's first statement, not merely to follow the guard.
- `hydrate_hidden_dm_threads_now` regains the empty-input short-circuit
  the wrapper kept. `hidden_threads` is `#[serde(default)]`, so without
  it every pre-archive predecessor did a no-op GlobalSignal write, once
  per predecessor, for no reason.
- `response_correlation_base` enumerates the non-storage variants
  instead of `_ => None`, so a new storage variant is a compile error
  rather than a silently uncorrelated response — B1's exact failure mode.

Comment corrections, each verified against source:
- There is only ONE `PENDING_REQUESTS` map; the non-storage responses use
  it under prefixed keys. The previous wording ("a different registry,
  this map is not involved") would leave a reader unaware that those four
  are unscoped for legacy targets.
- The `!completed` guard's real invariant is a COUNTING argument, not the
  sender-based one previously given: responses bind to waiters by
  correlation key, so the sweep's response can complete the walk's
  waiter. Documented, with the two reachable exceptions (a timed-out
  waiter evicted before its response lands; the sweep's send failing
  while the walk's succeeds). Both fail-safe, both recovered next load.
- "no variant can be left unscoped" holds only within the storage family.
- Removed a reference to a pin that never existed, and corrected the
  `b"list"` stand-in attribution — it was in a different test, in a
  different file, from the one named.
- The `b"list"` -> real-base swap is a readability fix that detects
  nothing new; it no longer implies otherwise.

Refs #617
Skeptical review found the pins added in 9e1dab6 defeatable by the
exact edits they exist to stop.

- The B2 pin bounded `tx.send` only by the defer's OPENING. Placing the
  send after the whole closure compiles (tx is simply not captured),
  satisfies every assertion, and restores B2 exactly — the await returns
  before the macrotask runs. Now bounded by the closure's terminator, and
  both hydrate calls must sit between the opening and the signal.
- `production_request_all_enqueues_before_it_awaits` checked only text
  order, which a fully sequential enqueue-and-await-in-one-loop rewrite
  also satisfies. That stays borrow-safe but serialises the pre-warm into
  27 x the 10 s timeout — the exact cost prewarm exists to remove. Now
  also requires a real fan-out in the await phase and requires the
  enqueue loop to CLOSE before the first await.

Also from the same review:
- `prewarm` asserts `request_all` returned one outcome per request. The
  ordering/length contract was documentation only, so a short vec would
  have silently dropped tail predecessors from the cache and degraded
  them to full live probes instead of failing loudly.
- `hydrate_outbound_dms_cache_now` gains the empty-input short-circuit
  its sibling already had; the justification (no gratuitous GlobalSignal
  write per predecessor) applies identically to OUTBOUND_DMS.

Refs #617
Review found the repo's own rules describe a codebase that no longer
exists — the exact "next agent follows stale instructions" failure the
third probe site was added to prevent.

- `river-publish.md` and `direct-messages.md` both told an author that a
  new fixed storage key needs TWO edits. It needs THREE: the crate walk
  carries its own fixed-probe list. An author following the old text
  would leave the new key invisible to the walk.
- Nothing under `.claude/` mentioned freenet-migrate, the walk, or its
  durable `__migrate_pred_done__:` / `__migrate_pred_wip__:` markers — a
  second recovery mechanism with its own persistent state in the current
  delegate's KV store, absent from the file that claims to be the
  canonical description of delegate migration. Added, including why the
  markers are hex-encoded and why the ListResponse routing is now
  conditional. Matters before release 2 retires the sweep.
- `direct-messages.md` now records the second hydration path and the
  `hydrate_*_now` contract (no internal defer; callers MUST supply one).

Two code comments bounded to what is actually true:
- `request_all` removes the overlap WITHIN the pre-warm, not the
  cross-task case, where a `migrate_legacy_per_room` task can still take
  the `WEB_API` borrow while the walk is suspended in `send().await`.
  That remains survived only by `send` not yielding for small payloads.
- The N2 gate's residual is stated: if every send fails AND a reconnect
  bumped the attempt, nothing retries this page load. Not arming on a
  dead transport is the intended half; the guard-reset half is
  pre-existing.

Refs #617
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants