driver/kubernetes: enable gRPC keepalive to avoid infinite hangs#3967
Open
pgimalac wants to merge 1 commit into
Open
driver/kubernetes: enable gRPC keepalive to avoid infinite hangs#3967pgimalac wants to merge 1 commit into
pgimalac wants to merge 1 commit into
Conversation
Signed-off-by: Pierre Gimalac <23154723+pgimalac@users.noreply.github.com>
pgimalac
marked this pull request as ready for review
July 22, 2026 17:22
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add a gRPC keepalive in the kubernetes driver to detect when the remote doesn't respond for too long and stop the connection, which avoids blocking the process forever.
This is related to #556 but fixes a wider bug on the kubernetes driver: the issue is about the hanging when the remote container/pod is killed (NB: this is addressed directly by #3966), the keepalive fixes this but also avoids issues when the connection stays open but the remote is unresponsive (eg. suspended).
The values for ClientParameters have been picked arbitrarily and I'm very open to suggestions to change them.
I didn't add unit tests for this because the
keepalivepackage enforces a minimum of 10 seconds forTime, so any unit test exercising this would take at least 10 seconds.Repro / testing
Setup:
Create a Dockerfile which takes some time, eg.:
and build it:
docker buildx build --builder kindbuilder --progress=plain -f Dockerfile .Wait for the log to show that it's running the sleep step, then open a new shell and suspend the buildkit process:
The
buildcommand will then hang forever (in particular won't finish after 2 minutes), even with #3966.Now build
cmd/buildxfrom this PR, and re-run the same steps using thisbuildxbinary instead ofdocker buildx, you should observe that the build command fails with the following error around 40 seconds after SIGSTOP is sent: