Implement a multi-tier circuit breaker for Soroban RPC and Horizon calls with per-dependency health tracking and automatic recovery probes

[Chucks1093]

Summary

When Soroban RPC or Horizon becomes degraded, the server continues sending requests until connection pool exhaustion, causing cascading failures across all endpoints. A per-dependency circuit breaker should open after a configurable failure threshold, reject all calls to the degraded dependency with a fast 503, and automatically send low-frequency recovery probes to detect when the dependency is healthy again before closing the circuit.

Scope

• Implement three circuit states per dependency: CLOSED (normal), OPEN (failing fast), HALF_OPEN (probing)
• CLOSED → OPEN: open after N consecutive failures or failure rate > X% in a rolling 60-second window (both thresholds configurable per dependency)
• OPEN → HALF_OPEN: after a configurable cooldown (default 30 seconds), allow one probe request through
• HALF_OPEN → CLOSED: close on probe success; HALF_OPEN → OPEN: reopen on probe failure
• Expose circuit state and metrics (failure count, last failure, last probe) via a GET /internal/health/circuits endpoint (internal network only)
• All endpoints that depend on an OPEN circuit should return 503 dependency_unavailable with a Retry-After header set to the remaining cooldown
• Emit structured logs on every state transition: circuit_opened, circuit_half_opened, circuit_closed with dependency name, failure count, and transition timestamp
• Maintain separate circuit breaker instances for: Soroban RPC, Horizon, Redis, and the database connection pool
• Add integration tests: failures trigger OPEN state, OPEN state fast-fails subsequent calls, probe success closes the circuit, probe failure re-opens, health endpoint reflects current state

Acceptance Criteria

• Circuit opens after N failures or X% failure rate in 60-second window
• OPEN circuit fast-fails all calls with 503 and Retry-After
• Recovery probe sent after cooldown; success closes, failure re-opens
• Separate circuit instances for Soroban RPC, Horizon, Redis, and database
• /internal/health/circuits endpoint exposes state and metrics
• Structured log emitted on every state transition
• All thresholds and cooldowns configurable per dependency without code changes

ETA: 24 hours

---

Coordinate on Telegram

[GazzyLee]

I’m keen to take this on. I’ll implement configurable, per-dependency circuit breakers for Soroban RPC, Horizon, Redis, and the database, including CLOSED/OPEN/HALF_OPEN transitions, fast 503 responses with Retry-After, recovery probes, structured transition logs, and the internal circuit health endpoint. I’ll also add integration coverage for failure thresholds, fast-fail behavior, recovery, and state reporting.