flannel migration: document removing leftover flannel iptables rules#2768
flannel migration: document removing leftover flannel iptables rules#2768stitrace wants to merge 2 commits into
Conversation
The live-migration controller removes the flannel daemonset and deletes the flannel network devices, but it does not remove the iptables chains flannel programs (FLANNEL-POSTRTG in nat, FLANNEL-FWD in filter). These survive the migration and the FLANNEL-POSTRTG masquerade rule keeps SNAT-ing cross-node pod-to-pod traffic to the node tunnel IP, which silently breaks NetworkPolicy after migration. Add a cleanup step (with a reboot alternative) so operators can remove the leftover rules.
✅ Deploy Preview for calico-docs-preview-next ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
❌ Deploy Preview for tigera failed. Why did it fail? →Built without sensitive environment variables
|
|
@stitrace Thanks for the submission. Someone will take a look at this together with projectcalico/calico#12920 |
|
Is this an alternative to projectcalico/calico#12920 ? With the automated removal in the linked PR, presumably we don't need manual steps? |
|
Not quite an alternative — they're complementary. #12920 (projectcalico/calico#12920) automates the cleanup for migrations performed with a Calico version that includes the fix, so going forward the manual steps won't be needed. However, the leftover chains don't clean themselves up: any cluster that already migrated with an earlier version still has the stale Flannel iptables rules sitting there, and anyone migrating on an older release will keep hitting this. For those users the docs are the only discoverable fix. I'd suggest trimming this PR down to a short note along the lines of: "migrations performed with Calico vX.Y+ remove these rules automatically; if you migrated with an earlier version, clean them up manually as follows" — with the version filled in once the fix ships in a release. Happy to rework the PR that way if it sounds good. |
|
Got it. Also, just a thought - for a lot of folks simply doing a rolling update of cluster nodes will be an easier way to handle this rather than scripting running these commands across a whole cluster. Might be worth taking that approach? |
|
Good point, agreed. Since flannel is already gone after the migration, nothing re-creates the chains, so a rolling reboot (or node replacement in immutable/managed node setups) fully clears them. The current text mentions rebooting only as a one-line "alternatively" at the end. I'll flip the emphasis: recommend a rolling reboot/replacement of nodes as the primary approach, and keep the script as an option for clusters where cycling every node is impractical, or when you need to restore cross-node traffic immediately without waiting for a full node roll (the SNAT breakage affects live traffic until the rules are gone). I'll push an update shortly. |
|
|
ac8072b to
c123eca
Compare
4 participants
Description
The live-migration guide tells you to delete the migration controller once migration completes, but it does not mention that flannel leaves iptables rules behind on every node.
The migration controller removes the flannel daemonset and deletes the flannel network devices (
flannel.<vni>,cni0), but it does not remove the iptables chains flannel programs:FLANNEL-POSTRTG(nat) andFLANNEL-FWD(filter). These survive the migration.The masquerade rule in
FLANNEL-POSTRTGkeeps SNAT-ing cross-node pod-to-pod traffic to the node's tunnel IP. This is invisible until you useNetworkPolicy: the SNAT'd source no longer matches pod-selector rules, so Calico's default-deny drops the traffic. Symptom after an otherwise successful migration: cross-node connections to policy-selected pods silently time out, while same-node traffic keeps working.This PR adds a cleanup step after "Delete the migration controller" with an idempotent flush command (legacy + nft backends) and a reboot alternative.
Notes
calico/...); maintainers may want to backport to the versioned snapshots.Reproduction
After migrating, on any node:
nft list chain ip nat POSTROUTINGstill showsjump FLANNEL-POSTRTGwith non-zero counters; a cross-node listener sees the client's source as the sender node's tunnel IP (<block>.0) rather than the pod IP, and NetworkPolicy-selected pods become unreachable cross-node.