Outpost builds a separate HTTP client for every destination and never configures connection pooling on it. Two problems compound:
Each client keeps only 2 idle connections. That's Go's default, inherited because nothing sets it. Above two concurrent deliveries to a destination, connections are closed after use instead of reused.
Each client is thrown away every 60 seconds. The client is owned by a cached publisher whose entries expire after a minute, so even a well-sized pool is discarded and rebuilt empty. The abandoned connections outlive the publisher that made them, because the idle timeout is longer than the cache TTL.
Under sustained load, connection reuse collapses to roughly one new connection per delivery.
Impact
The cost lands differently depending on how fast the destination responds, and both ends are bad:
| destination |
what it costs |
| fast (tens of ms) |
latency — the TCP and TLS handshake is several times the request itself |
| slow (seconds) |
ephemeral ports — closed connections accumulate in TIME_WAIT on the sender |
At the slow end this is the more serious one. A single destination responding near the delivery timeout can hold several thousand ports; a handful of such destinations can exhaust the sender's ephemeral range, after which deliveries start failing at the connection level. Those failures are currently recorded as a generic network error with the underlying cause discarded, which makes them hard to attribute — see the companion issue on transport-error diagnostics.
When it triggers
The threshold is more than two concurrent deliveries to a single destination, which is rate × response time. Roughly 100 events/s to one destination at typical response times, or far less if that destination is slow.
DELIVERY_MAX_CONCURRENCY defaults to 1, so a default deployment never reaches it. This affects deployments that have raised it.
Why the per-destination client exists
No reason, on inspection. Nothing in the client configuration varies per destination — user agent, proxy settings, and the transport wrapper all come from provider-level options fixed at registration, and there's no per-destination TLS configuration anywhere in the registry. We build many structurally identical clients and give each its own small, short-lived pool.
Proposed change
Build the client once per distinct transport configuration — today, once per provider at registration, rather than per destination in CreatePublisher.
This makes the per-host idle limit meaningful, gives a real ceiling on total idle connections, and makes publisher cache eviction irrelevant to connection reuse. There's precedent: the hookdeck provider already accepts an injected shared client.
The rule worth preserving: only connection-level concerns justify a separate client. Proxy settings and TLS material qualify. Headers, auth, timeouts and response body limits are per-request and belong on the request. If per-destination proxy or client certificates are needed later, transports should be keyed by configuration and shared within a key — not built per destination.
Sizing the pool
Two limits matter, and they mean different things:
- per-host — how many warm connections a single destination keeps. This is depth, bounded by concurrent deliveries to that destination.
- total — how many destinations can keep a warm connection at all. This is breadth. If it's too small, destinations evict each other's connections and reuse never happens regardless of the per-host value.
Per-host derives cleanly from DELIVERY_MAX_CONCURRENCY: the worker pool caps how many deliveries can be in flight, so it caps how many connections one destination could need.
Total does not derive from concurrency. A deployment with 600 destinations each receiving one delivery a minute has almost no concurrency but needs ~600 warm connections; with a smaller ceiling, every delivery evicts some other destination's connection and reuse stays near zero. Fan-out is the normal shape for this product, so Go's default of 100 is the wrong thing to inherit.
What actually bounds the total is the host's file descriptor limit — each idle connection holds one FD for up to 90 seconds. So derive it from RLIMIT_NOFILE, clamped:
| host FD limit |
derived total |
| 1024 (bare metal, systemd) |
256 |
| 65536+ (typical container) |
4096 (capped) |
A quarter of the soft limit, floored at 100 and capped at 4096, leaves room for the database, queue, Redis and listener FDs that share the same per-process budget.
The hookdeck provider should get a small fixed value instead — it talks to one host, so it needs depth, not breadth.
No new configuration
Deliberately not exposing these as env vars. The correct total depends on active destination count and the host's FD limit, which an operator would have to know both of to set sensibly — a knob that invites confident wrong values. The FD limit is readable at startup, so the process can size itself.
Adding a knob later is backward-compatible; removing one isn't. If the derivation proves wrong, the escape hatch can be added then.
Instead:
- log the resolved value at startup, alongside the FD limit it came from
- add a connection reuse metric — connections opened against deliveries attempted. This is the signal that the ceiling is binding, and it's currently the thing we have no way to observe in production.
Verification
A test asserting connections opened stays near concurrency, not near request count, swept across concurrency levels and against both fast and slow destinations, since the two regimes fail differently.
The test must run longer than the publisher cache TTL — a shorter test cannot observe the eviction half of the bug by construction.
Tradeoffs worth stating
Cross-tenant pooling. One provider serves all tenants, so the pool becomes shared. Connections are per-host so there's no correctness concern, but a busy tenant can evict a quiet tenant's idle connections. The per-host limit bounds the blast radius.
Connections held on the destination. Idle connections occupy a slot on the receiver's server. Those connections were opened anyway to serve the traffic; pooling changes whether they're held afterward, for up to 90 seconds. Bounded, but a real behavior change from the receiver's side.
Platform. Reading RLIMIT_NOFILE is Unix-specific and needs a fallback on Windows.
Outpost builds a separate HTTP client for every destination and never configures connection pooling on it. Two problems compound:
Each client keeps only 2 idle connections. That's Go's default, inherited because nothing sets it. Above two concurrent deliveries to a destination, connections are closed after use instead of reused.
Each client is thrown away every 60 seconds. The client is owned by a cached publisher whose entries expire after a minute, so even a well-sized pool is discarded and rebuilt empty. The abandoned connections outlive the publisher that made them, because the idle timeout is longer than the cache TTL.
Under sustained load, connection reuse collapses to roughly one new connection per delivery.
Impact
The cost lands differently depending on how fast the destination responds, and both ends are bad:
TIME_WAITon the senderAt the slow end this is the more serious one. A single destination responding near the delivery timeout can hold several thousand ports; a handful of such destinations can exhaust the sender's ephemeral range, after which deliveries start failing at the connection level. Those failures are currently recorded as a generic network error with the underlying cause discarded, which makes them hard to attribute — see the companion issue on transport-error diagnostics.
When it triggers
The threshold is more than two concurrent deliveries to a single destination, which is
rate × response time. Roughly 100 events/s to one destination at typical response times, or far less if that destination is slow.DELIVERY_MAX_CONCURRENCYdefaults to 1, so a default deployment never reaches it. This affects deployments that have raised it.Why the per-destination client exists
No reason, on inspection. Nothing in the client configuration varies per destination — user agent, proxy settings, and the transport wrapper all come from provider-level options fixed at registration, and there's no per-destination TLS configuration anywhere in the registry. We build many structurally identical clients and give each its own small, short-lived pool.
Proposed change
Build the client once per distinct transport configuration — today, once per provider at registration, rather than per destination in
CreatePublisher.This makes the per-host idle limit meaningful, gives a real ceiling on total idle connections, and makes publisher cache eviction irrelevant to connection reuse. There's precedent: the
hookdeckprovider already accepts an injected shared client.The rule worth preserving: only connection-level concerns justify a separate client. Proxy settings and TLS material qualify. Headers, auth, timeouts and response body limits are per-request and belong on the request. If per-destination proxy or client certificates are needed later, transports should be keyed by configuration and shared within a key — not built per destination.
Sizing the pool
Two limits matter, and they mean different things:
Per-host derives cleanly from
DELIVERY_MAX_CONCURRENCY: the worker pool caps how many deliveries can be in flight, so it caps how many connections one destination could need.Total does not derive from concurrency. A deployment with 600 destinations each receiving one delivery a minute has almost no concurrency but needs ~600 warm connections; with a smaller ceiling, every delivery evicts some other destination's connection and reuse stays near zero. Fan-out is the normal shape for this product, so Go's default of 100 is the wrong thing to inherit.
What actually bounds the total is the host's file descriptor limit — each idle connection holds one FD for up to 90 seconds. So derive it from
RLIMIT_NOFILE, clamped:A quarter of the soft limit, floored at 100 and capped at 4096, leaves room for the database, queue, Redis and listener FDs that share the same per-process budget.
The
hookdeckprovider should get a small fixed value instead — it talks to one host, so it needs depth, not breadth.No new configuration
Deliberately not exposing these as env vars. The correct total depends on active destination count and the host's FD limit, which an operator would have to know both of to set sensibly — a knob that invites confident wrong values. The FD limit is readable at startup, so the process can size itself.
Adding a knob later is backward-compatible; removing one isn't. If the derivation proves wrong, the escape hatch can be added then.
Instead:
Verification
A test asserting connections opened stays near concurrency, not near request count, swept across concurrency levels and against both fast and slow destinations, since the two regimes fail differently.
The test must run longer than the publisher cache TTL — a shorter test cannot observe the eviction half of the bug by construction.
Tradeoffs worth stating
Cross-tenant pooling. One provider serves all tenants, so the pool becomes shared. Connections are per-host so there's no correctness concern, but a busy tenant can evict a quiet tenant's idle connections. The per-host limit bounds the blast radius.
Connections held on the destination. Idle connections occupy a slot on the receiver's server. Those connections were opened anyway to serve the traffic; pooling changes whether they're held afterward, for up to 90 seconds. Bounded, but a real behavior change from the receiver's side.
Platform. Reading
RLIMIT_NOFILEis Unix-specific and needs a fallback on Windows.