feat(wallet-toolbox): horizontally scale Storage with shared async sessions

[ty-everett]

Headline

Storage can now run behind a non-sticky load balancer with a shared, durable BRC-103 session store instead of process-local authentication state.

This draft deliberately consolidates and supersedes #279, #280, and #281 so maintainers have one review surface for the complete production follow-up.

Shared async sessions

• add exported KnexSessionManager, implementing the SDK's existing AsyncSessionManager contract
• add the auth_sessions Knex migration and indexed nonce, identity, and expiry lookup paths
• let StorageServer inject a shared session manager while retaining the in-memory default for compatibility
• make cross-replica writes monotonic by lastUpdate
• merge same-millisecond authentication/certificate progress so equal Date.now() values cannot downgrade a session
• make established identity and peer nonces immutable and prevent stale/equal-state removals from deleting stronger state
• exclude expired sessions on reads and provide explicit indexed pruning
• document the required migrate-before-start sequence, shared database/TTL requirement, and maintenance pruning

The focused test uses two independent Knex connections to one SQLite database, verifies lookup by nonce and identity across connections, exercises stale/equal-timestamp races, expiry, pruning, and safe removal. The StorageClient integration test performs a real authenticated RPC through StorageServer using the Knex manager.

Correctness and production hardening

• repair proof fan-out for every local transaction row sharing a proven txid, including already-proven self-healing after concurrent internalizeAction notification drift
• add logRpcRequests: false so saturated deployments can skip both the unconditional POST log and request-parameter serialization
• add the measured missing monitor_events.created_at index used for production proof-latency correlation
• declare missing runtime imports (mime-types, body-parser, dotenv) so built packages load under strict package-manager resolution; prepare auth middleware 2.1.1
• keep wallet-toolbox, client, and mobile at 2.4.2 so release checks stay in lockstep

Production evidence for the fan-out repair: txid 2cdf4bb89c765d5b9fadef1c2bc4aceb2258ac607d0471e9df5f833bb08e6250 had transaction 444915 omitted while 444912 and 444918 completed.

E2E and throughput methodology

The production harness now:

• bounds individual Storage and Arcade observations so one hung request cannot freeze proof polling
• reads Storage action state once per wallet per poll and observes Arcade txids with bounded concurrency
• records first Arcade proof, first Storage completion, and convergence time
• sizes and batches independently mined BRC-29 funding instead of conflating parent propagation with throughput
• preserves partial no-send evidence immediately for deterministic recovery
• separates raw HTTP, authenticated reads, preparation, Arcade submission, Storage reconciliation, pre-proof client E2E, block wait, and proof convergence
• requires both an Arcade merkle path and originating Storage status completed for every tracked tx

The pre-change production baseline (wallet-toolbox 2.4.1, one Storage Node process) remains the comparison point, not a claim for this un-deployed branch:

Boundary	Measured baseline
warm public raw GET	up to ~4,464 req/s
sustained authenticated reads, 8 identities	~95-103 req/s
long authenticated run	100.1 req/s over 2,000 requests
Storage CPU during long run	1.034 cores
preparation, 3-tx single identity	5.0 tx/s
Arcade submission	12.9 tx/s
Storage reconciliation	5.1 tx/s
submission through Storage	3.7 tx/s
client E2E before proof	2.1 tx/s
full createAction, 3 identities	7.5 tx/s
Monitor block proof drain	25 proofs in 3.073 s (~8.1 proofs/s)
final proof convergence	28/28 Arcade proofs and 28/28 Storage completed

Current bottleneck interpretation

1. The demonstrated authenticated-read ceiling is one Node main thread at about 100 req/s; the two-core pod limit, ingress, TLS, and Knex pool were not the ceiling.
2. Shared SQL sessions add a small indexed DB operation to authenticated traffic, but remove the process-local state barrier so capacity can scale with replicas. The next production benchmark must compare one shared-session replica with multiple replicas before claiming a gain.
3. MySQL work is material (about 7.0 ms main output select and 4.8 ms count in the retained sample) but the pool was not saturated.
4. The unconditional RPC log remained on the saturated path; this branch supplies a benchmarkable opt-out without changing the default.
5. Arcade acceptance, Storage reconciliation, and proof ingestion are separate ceilings. Monitor's 25-proof write drain was ~3 seconds; the larger convergence delay came from header discovery cadence and the fixed five-second delay.
6. SSE cursor persistence still requires an after-success checkpoint with replay-safe handlers; persisting on receive would introduce an at-most-once loss window and is intentionally not bundled here.

Release behavior

Single-package publishes no longer wait five minutes or run global workspace synchronization. Cascade releases retain the existing propagation wait and synchronization path. This prevents a successful package publish from being reported failed after consumers are rewritten to unpublished sibling versions.

Because this PR now prepares both auth middleware 2.1.1 and wallet-toolbox 2.4.2, a cascade release is the clean path after merge; the single-package fix remains useful for future isolated releases.

Validation

• wallet-toolbox full non-manual suite: 126 passed suites, 1,142 passed tests; 5 suites / 27 tests skipped
• focused shared-session, migration, authenticated StorageClient, and proof-fan-out suites: 4 suites, 15 tests passed
• auth-express-middleware: 2 suites, 20 tests passed
• payment-express-middleware: 1 suite, 1 test passed
• SDK -> auth middleware -> payment middleware -> wallet-toolbox build chain passed
• built wallet-toolbox root import and KnexSessionManager export load under strict pnpm resolution
• cross-package version and SDK peer-dependency checks passed
• production E2E suite loads with all 13 mainnet tests skipped behind the safety gate
• release.yaml parses and git diff --check passes
• new session files pass ts-standard; full wallet-toolbox lint still reports only pre-existing WalletPermissionsManager test warnings outside this diff

Draft gates

• maintainer review of the session schema/atomic merge semantics and release grouping
• CI on this consolidated branch
• after merge and package release: wire every production Storage replica to one shared manager/database and TTL, designate one expiry-pruning job, then rerun the retained single-vs-multi-replica workload and full proof convergence suite before enabling HPA

[sonarqubecloud]

Quality Gate passed

Issues
5 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud