fix(processing): retry publish with exponential backoff, dead-letter on exhaustion

[irefavor15-dotcom]

Summary

Closes #343

StreamSession.pump() previously called this.fail(error) on the very first publish() failure, which cleared the entire queue and moved the session to errored. A single transient API hiccup caused permanent data loss for every event behind the failed one.

Root Cause

// Before (session.ts pump())
} catch (err) {
  this.queue.unshift(next)  // put event back
  this.fail(error)           // → clears queue, errors session
  return
}

Fix

Each event publish is now wrapped in an exponential-backoff retry loop. After exhausting the retry budget the individual event is dead-lettered and the loop continues — the session stays running and the rest of the queue is processed normally.

attempt 1 → fail → wait 100ms
attempt 2 → fail → wait 200ms
attempt 3 → fail → wait 400ms
attempt 4 → DEAD-LETTER (emit 'dead-letter', log ERROR, continue queue)

The backoff is capped at 5 s per attempt. fail() is still available for coordinator-level failures (e.g. lock errors) but is no longer called from inside the publish loop.

Changes

xstreamroll-processing/src/session.ts

• New maxPublishRetries constructor param (default 3)
• pump() inner loop replaced with backoff-retry logic
• Emits 'dead-letter'(event: StreamEvent, err: Error) on exhaustion
• Session remains in running state after dead-lettering

xstreamroll-processing/src/session-registry.ts

• Added maxPublishRetries?: number to SessionRegistryOptions
• Passes options.maxPublishRetries to new StreamSession() in spawn()

xstreamroll-processing/src/config.ts

Added PROCESSING_PUBLISH_MAX_RETRIES to the zod env schema (default '3', min 0).

xstreamroll-processing/src/worker.ts

Reads env.PROCESSING_PUBLISH_MAX_RETRIES and passes it as maxPublishRetries to the SessionRegistry.

xstreamroll-processing/.env.example

Documents PROCESSING_PUBLISH_MAX_RETRIES with notes on the fail-fast mode (=0).

xstreamroll-processing/__tests__/session.test.ts

• Updated existing test: session no longer errors on a single failure; replaced with a maxPublishRetries=0 dead-letter test and a fail()-is-still-wired test
• 4 new tests in a dedicated describe block:
  1. Retries on transient failure and eventually publishes successfully
  2. Dead-letters after retry exhaustion and continues with the rest of the queue
  3. dead-letter event carries the original StreamEvent and the Error
  4. Session stays running and accepts new events after dead-lettering

Test Results

PASS __tests__/session.test.ts (5.07 s)
  StreamSession
    ✓ starts in the idle state
    ✓ transitions to running on start()
    ✓ enqueues and publishes events in order
    ✓ publishes null latency for invalid event timestamps
    ✓ rejects new events when not running
    ✓ transitions to stopped via stop()
    ✓ emits state and processed events
    ✓ dead-letters an event and stays running when maxPublishRetries=0
    ✓ explicit fail() still transitions to errored and emits error event
    ✓ is a no-op when stop() is called on a stopped session
  StreamSession — publish retry and dead-letter (issue #343)
    ✓ retries on publish failure and eventually succeeds
    ✓ dead-letters after exhausting retries and continues processing remaining queue
    ✓ emits dead-letter event with the original StreamEvent and the error
    ✓ session stays running after dead-lettering and accepts new events

Tests: 19 passed