fix(server): bound alert webhook/SMTP delivery timeouts (JEF-497)

[thejefflarson]

JEF-497 — alert.notify p99 ~11.5s: webhook/SMTP sinks have no timeout

Symptom

alert.notify spans cluster at ~11.5s (11,494 / 11,604 / 11,750ms) with error_count=0 — slow, not failing: a fixed connect/handshake stall in the alert delivery path (not ingest).

Root cause

notify() delivered to two sinks with no timeout:

• Webhook — reqwest::Client::new() has neither a request nor a connect timeout, so a slow/unresponsive receiver blocked send() up to the OS TCP timeout.
• Email — the lettre AsyncSmtpTransport was built without .timeout(...), so a slow SMTP relay handshake stalled just as long.

notify() is awaited inside the eval tick (run → evaluate → notify), so a stalled sink delayed the whole tick and backed up other rules' notifications.

Fix

• New shared build_webhook_client() helper builds the reqwest client once with explicit bounds — 10s total request / 5s connect — reused by both run and evaluate_once so both delivery paths get identical config. A failed build() logs and disables the webhook rather than panicking, matching how a bad SMTP config disables email.
• Mailer::build now sets .timeout(Some(Duration::from_secs(10))) on the SMTP transport.
• Values chosen short enough that a dead sink can't stall the loop, long enough for a legitimately slow-but-alive receiver. One-line comments explain each.

Re-architecting the loop (spawning notify off the tick / a delivery queue) is a noted follow-up and explicitly out of scope here — the timeouts are the fix.

Test

notify_gives_up_on_a_hung_webhook_within_timeout: a local tokio::net::TcpListener that accepts but never responds (the exact hung-receiver case). notify() gives up in ~10s (measured 10.01s), asserted < 15s — bounded by the request timeout, not the multi-second OS TCP timeout. Deterministic, no real-network dependency.

Gates

cargo fmt / cargo check / cargo clippy --all-targets -- -D warnings / cargo test --locked — all green (57 tests incl. the new one).

🤖 Generated with Claude Code

https://claude.ai/code/session_01JbrmfzHsTMMzPaSrUkZgWo