Skip to content

driver/kubernetes: enable gRPC keepalive to avoid infinite hangs#3967

Open
pgimalac wants to merge 1 commit into
docker:masterfrom
pgimalac:pgimalac/kubernetes-driver-grpc-keepalive
Open

driver/kubernetes: enable gRPC keepalive to avoid infinite hangs#3967
pgimalac wants to merge 1 commit into
docker:masterfrom
pgimalac:pgimalac/kubernetes-driver-grpc-keepalive

Conversation

@pgimalac

@pgimalac pgimalac commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Add a gRPC keepalive in the kubernetes driver to detect when the remote doesn't respond for too long and stop the connection, which avoids blocking the process forever.

This is related to #556 but fixes a wider bug on the kubernetes driver: the issue is about the hanging when the remote container/pod is killed (NB: this is addressed directly by #3966), the keepalive fixes this but also avoids issues when the connection stays open but the remote is unresponsive (eg. suspended).

The values for ClientParameters have been picked arbitrarily and I'm very open to suggestions to change them.

I didn't add unit tests for this because the keepalive package enforces a minimum of 10 seconds for Time, so any unit test exercising this would take at least 10 seconds.

Repro / testing

Setup:

# create kind cluster
kind create cluster --name buildkit-repro

# create a builder
docker buildx create --driver kubernetes --name kindbuilder --driver-opt namespace=default,replicas=1 --bootstrap

Create a Dockerfile which takes some time, eg.:

FROM alpine:latest
RUN echo "step1 starting" && sleep 120 && echo "step1 done"

and build it:

docker buildx build --builder kindbuilder --progress=plain -f Dockerfile .

Wait for the log to show that it's running the sleep step, then open a new shell and suspend the buildkit process:

# Find the PID
docker exec buildkit-repro-control-plane ps aux | grep buildkitd

# Suspend it
docker exec buildkit-repro-control-plane kill -STOP <PID>

The build command will then hang forever (in particular won't finish after 2 minutes), even with #3966.

Now build cmd/buildx from this PR, and re-run the same steps using this buildx binary instead of docker buildx, you should observe that the build command fails with the following error around 40 seconds after SIGSTOP is sent:

ERROR: failed to build: failed to receive status: rpc error: code = Unavailable desc = keepalive ping failed to receive ACK within timeout

Signed-off-by: Pierre Gimalac <23154723+pgimalac@users.noreply.github.com>
@pgimalac pgimalac changed the title driver/kubernetes: enable gRPC keepalive as a dead-peer backstop driver/kubernetes: enable gRPC keepalive to avoid infinite hangs Jul 22, 2026
@pgimalac
pgimalac marked this pull request as ready for review July 22, 2026 17:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant