fix(chart): agent DaemonSet updateStrategy maxSurge:1 so a stuck node can't stall the roll (JEF-423)#241
Merged
thejefflarson merged 1 commit intoJul 18, 2026
Conversation
… can't stall the roll (JEF-423) The in-repo agent DaemonSet declared no updateStrategy, inheriting k8s's stall-prone default (maxUnavailable:1, maxSurge:0): each node's old pod must terminate before the next starts, so a single stuck/cordoned node (disk-pressure, a wedged kernel that won't unload the old eBPF probes) blocks the whole roll and drops runtime coverage to zero fleet-wide until an operator intervenes. Adds spec.updateStrategy.rollingUpdate.maxSurge, values-driven via a new agent.maxSurge (default 1). A replacement pod now comes up on healthy nodes WITHOUT first tearing down the old one, so the roll proceeds past a bad node. Surge — not a higher maxUnavailable — keeps every reachable node observed throughout the upgrade, so coverage never dips below the running fleet. Set agent.maxSurge=0 to restore the in-place default. Chart YAML — untestable beyond template render + lint: helm template charts/protector --set agent.enabled=true → renders maxSurge: 1 ...--set agent.maxSurge=2 → renders maxSurge: 2 (override honoured) helm lint charts/protector → clean Claude-Session: https://claude.ai/code/session_01VtjoJttCvBY4dzCoE4f9vP Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
thejefflarson
deleted the
thejefflarson/jef-423-agent-daemonset-add-updatestrategy-maxsurge1-so-a
branch
July 18, 2026 06:42
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes JEF-423
What
The in-repo agent DaemonSet (
charts/protector/templates/agent-daemonset.yaml) declared noupdateStrategy, so it inherited Kubernetes' stall-prone default (maxUnavailable: 1, maxSurge: 0): each node's old pod must terminate before the next node's new pod starts. One stuck node — cordoned, disk-pressure, a wedged kernel that won't unload the old eBPF probes — blocks the entire roll and drops runtime coverage to zero fleet-wide until an operator intervenes.This adds
spec.updateStrategy.rollingUpdate.maxSurge, values-driven via a newagent.maxSurge(default1). A replacement pod now comes up on healthy nodes without first tearing down the old one, so the roll proceeds past a bad node. Surge — deliberately, not a highermaxUnavailable— keeps every reachable node observed throughout the upgrade, so coverage never dips below the running fleet. Setagent.maxSurge: 0to restore the in-place default.Verification
Chart YAML — untestable beyond template render + lint (stated in the ticket):
Required infra-repo follow-up (human)
This PR changes ONLY the in-repo reference chart. The separate cluster/infra repo (the deployed
charts/protector) is a flagged human follow-up and was intentionally not touched here — mirrorspec.updateStrategy.rollingUpdate.maxSurge: 1into that repo's agent DaemonSet so the running fleet gets the same non-stalling roll. Until that lands, deployed agents still inherit the stall-prone default.🤖 Generated with Claude Code
https://claude.ai/code/session_01VtjoJttCvBY4dzCoE4f9vP