fix(destregistry): record the cause of a transport-level delivery failure

[alexluong]

Problem

When a webhook delivery fails before receiving an HTTP response — DNS failure, connection refused, TLS error, dial timeout — the attempt records that it failed and nothing about why. response_data is written NULL.

To be precise about which failures these are: a destination that replies 503 is not this path — that's an HTTP response, it goes down the 4xx/5xx branch, and its body is already recorded. This path is the case where no response ever arrives.

It's the only delivery failure path through destwebhook that behaves this way:

failure	code	response_data
destination returns 4xx/5xx	"503"	status, body
payload format failure	"ERR"	the error message
publisher resolution failure	"ERR"	the error message
connection-level failure	connection_refused	absent

A destination that returns a 500 with an empty body leaves more diagnostic trace than a destination we could not reach at all.

What the end user sees

GET /destinations/:id/attempts?include=response_data, for a delivery to a host that isn't accepting connections.

Before:

{
  "id": "att_...",
  "status": "failed",
  "code": "connection_refused",
  "attempt_number": 1
}

response_data is omitted entirely — it was NULL, and the field is omitempty. The user knows the category and nothing else: which host, which port, whether it was one endpoint or all of them.

After:

{
  "id": "att_...",
  "status": "failed",
  "code": "connection_refused",
  "response_data": {
    "error": "Post \"https://api.example.com/hooks/abc\": dial tcp 203.0.113.10:443: connect: connection refused"
  },
  "attempt_number": 1
}

The code is unchanged. What's new is the response_data.error string, which names the URL, the resolved address, and the failing syscall — enough to tell "my DNS points at a stale IP" from "my service is down" from "my TLS cert expired" without contacting support.

Worth noting the string is Go's raw transport error, so it reads like a Go error rather than a written message. It's the same trade the 4xx/5xx path already makes by storing the destination's response body verbatim. Normalizing it is a reasonable follow-up; recording it at all is the change here.

The catch-all case is where this matters most. dns_error and connection_refused at least name a category; network_error is everything unrecognized, and before this change an attempt with that code carried no information whatsoever.

Both webhook providers get this — destwebhook and destwebhookstandard share the same helper. desthookdeck already records the cause on its own error path and is unaffected.

Open question: is response_data the right field for this?

There's a reasonable objection to the change above: the connection failed, so there is no response — leaving response_data empty is arguably correct, and putting an error string there is a misuse of the field.

Raising it because the answer isn't obvious, and because the field already has this problem independently of this PR.

response_data is not "the HTTP response" today. NewFormatError fires when an event can't be turned into a request at all — an unparseable payload, an invalid partition-key or object-key template. Nothing is sent, no connection is opened, and it still fills the field:

func NewFormatError(provider, message string, err error) (*Delivery, error) {
	if message == "" {
		message = "could not format event for delivery"
	}
	return &Delivery{
		Status:   "failed",
		Code:     "ERR",
		Response: map[string]interface{}{"error": message},
	}, ...
}

Eight providers call it — destwebhook, destwebhookstandard, destkafka, destawss3, destawssqs, destawskinesis, destgcppubsub, destazureservicebus. Every one passes an empty message, so what the customer actually receives is the literal default:

{
  "id": "att_...",
  "status": "failed",
  "code": "ERR",
  "response_data": {
    "error": "could not format event for delivery"
  },
  "attempt_number": 1
}

A response_data for an attempt where no request ever left the process. Validation and publisher-resolution failures do the same, storing {"error": err.Error()} under Code: "ERR".

So the field carries three distinct shapes:

when	shape
response received (2xx, 4xx, 5xx)	{"status": 503, "body": "..."}
never sent — format failure	{"error": "could not format event for delivery"}
never sent — validation / resolution failure	{"error": "<err.Error()>"}

And the doc comment on NewFormatError states the intent outright: "message is the customer-facing string persisted on the attempt (ResponseData)." Not "the response" — the customer-facing string.

That leaves two positions:

1. response_data is the attempt's explanation field, and the name is historical. This PR follows that, making the transport path consistent with the two other no-response failures that already populate it.
2. response_data should strictly mean a response, and the explanation belongs in a separate error / failure_reason field. Cleaner, but it touches models.Attempt, both log store schemas, the API type, and the SDKs — plus the {"error": ...} values already sitting in customers' attempt history.

This PR takes the first, since it's what the surrounding code does. If we'd rather have the second, this becomes one more call site to migrate rather than an obstacle — but worth deciding, because the inconsistency is customer-visible either way.

Also: source-side failures are no longer charged to the destination

Ephemeral port exhaustion on the sending host previously fell into the network_error catch-all, beside genuinely remote failures. It now records address_unavailable.

This matters to the user because the two need different responses: a destination that's unreachable is theirs to fix, while running out of local ports is ours, and retrying against the destination won't help. Under the old classification the two were indistinguishable on the attempt.

It does change a customer-visible value — an attempt that would have recorded network_error now records address_unavailable, so anything matching on that string sees new behavior. Happy to split this out if you'd rather it land separately.

Testing

go test -short ./... passes. Adds coverage asserting a connection-level failure records a non-empty cause, plus the errno classification.

🤖 Generated with Claude Code

[alexluong]

@alexbouchardd @leggetter would appreciate your input here. Please see the PR description for context. I'm not 100% sold on this PR itself but I think it highlights a potential problem in Outpost behavior when it comes to surfacing error to end users.