feat(api,web): webhook receiver and delivery pull queue

[marcorivm]

External systems (GitHub, alerting, generic senders) can now trigger agents through OneCLI.

A provider POSTs to a public per-endpoint URL; OneCLI verifies the signature, stores the delivery, renders a text template, and queues it. The agent runtime long-polls to drain the queue and acks each delivery.

GitHub ──▶ POST /v1/hooks/<publicId>
              verify → store → render → queue
                          ▲
  agent runtime ──────────┘   GET  /v1/hooks/pending   (outbound, held ~25s)
                              POST /v1/hooks/ack

Why pull

The machine running the agent needs no public port, no tunnel, and no inbound firewall rule. It also makes retries free: an unacked delivery becomes claimable again when its lease lapses, so a consumer that dies mid-dispatch loses nothing.

This supersedes an earlier design that put the whole surface — verification, templating, dedup, retry, replay, and a second admin UI — in a companion app beside the agent. That needed a public port on the agent's VM (colliding with exe.dev's one-public-port limit), a second database, and it only ever served one runtime. Putting the receiver here reuses auth, projects, audit, encryption, the dashboard, and an existing public HTTPS origin — and makes it usable by any runtime.

The interesting parts

The claim is race-free without a transaction. Candidates are selected, a guarded updateMany repeats the same availability predicate, then the claimed set is read back by a per-claim uuid. Under READ COMMITTED the loser of a race re-evaluates its WHERE against the committed row version, finds availableAt in the future, and matches nothing — so no SELECT … FOR UPDATE SKIP LOCKED and no long-lived connection. Correctness rests on exactly two things, both commented in place: the guard repeats the predicate, and claimId is fresh per claim.

Leases instead of a reaper. A poller that dies mid-batch leaves rows that become claimable when the lease passes. attempts was already incremented at claim time, so a consumer crashlooping on one delivery walks up to the cap and falls out of the predicate permanently.

The wait holds nothing. pollPending sleeps between queries, never inside one — no claim, no transaction, no pool connection. A SIGTERM mid-wait loses nothing and N parked pollers cost zero connections.

Verification runs over the raw bytes, before any parse — a re-serialized body changes the digest. Dedup catches P2002 rather than pre-reading, which is what makes it correct against a provider retrying concurrently.

retryable: false on a nack terminates a delivery immediately. A routing blob the consumer cannot interpret will fail identically forever; the reason lands in lastError and renders in the delivery dialog, so a misconfigured endpoint is diagnosable from the dashboard without SSH access to the box.

Decisions worth a second look

• A muted endpoint answers 200, not 403. GitHub disables a webhook that fails continuously, so a local mute must stay invisible to the sender — otherwise a week-long mute silently breaks the integration and needs re-enabling on their side.
• An unknown endpoint answers 404, not a silent 202. Enumeration isn't a real threat against a 128-bit path, whereas a mistyped URL answered 202 debugs for hours while the provider's delivery list stays green.
• Secrets are re-readable, matching every other credential here (the project API key on the overview page, an agent's access token in its list row). A webhook secret gets re-pasted into a provider's config months later; show-once turns a copy slip into a forced rotation on their side.
• Mutations use recordAuditEvent, not withAudit. withAudit also flushes the gateway's CONNECT cache — right for secrets and policy, waste for webhook config. Renaming a webhook shouldn't make every agent re-resolve its credentials. (This surfaced as a 500 in tests before it surfaced in production.)
• routes/hooks.ts has no use("*"). Hono.route() flattens a sub-app's routes into the parent, so a sibling router's wildcard middleware would 401 the public ingest POST. The cost is that a route added later without inline middleware is silently public — hooks-admin.test.ts enumerates every route and asserts 401 without credentials. Keep that test green.
• routing is opaque. OneCLI stores it, never parses it, and passes it through verbatim. Growing an opinion about its shape would couple the receiver to one consumer's semantics.

Verification

• 1328 tests pass, 0 skipped, with every *.pg.test.ts suite enabled against real Postgres.
• prisma migrate diff --exit-code → No difference detected.
• The pg proof that matters: 20 deliveries × 4 concurrent claimPending calls → every delivery claimed exactly once, union complete, intersection empty. Plus lease expiry → reclaim → the first poller's ack returns stale, and attempts walking out of the predicate.
• Manual end-to-end against a running instance: 202 queued · 200 duplicate (no second row) · 401 + a rejected row on a bad signature · 200 handshake on ping · 404 unknown · long-poll returning in 0.4s with work and holding exactly wait when empty · ack → delivered, re-ack → 409 · replay after a template edit rendering with the new template.

Notes for the reviewer

• The migration was generated by diffing the pre-change schema against the new one (Docker wasn't available at the time), then verified with the real drift check and applied cleanly. Plain tables and indexes only — no partial or expression indexes, which CI's check would reject.
• Retention is opportunistic (throttled, off the ingest path) because there's no scheduler in this deployment, plus POST /v1/internal/webhooks/sweep behind the existing shared-secret guard.
• GET /v1/hooks/pending gets its own Next segment with maxDuration = 60 so the long poll can park past the platform default without touching any other /v1 route.
• Deliberately deferred: provider-native subscription (the seams — WebhookVerifier.subscription, WebhookEndpoint.subscriptionState — are in place and inert, so adding it is code-only), push destinations, and lease extension.
• docs/webhooks.md carries the full consumer contract so a third-party runtime can be written from that page alone. The nanoclaw-side poller adapter is a separate repo and not in this PR.

https://claude.ai/code/session_012yM2sscojX6zzDUaVJXo6m