chore: Upgrade OVN chart/images to 2026.1.3 and fix chassis-claiming regression#2143
Draft
skrobul wants to merge 5 commits into
Draft
chore: Upgrade OVN chart/images to 2026.1.3 and fix chassis-claiming regression#2143skrobul wants to merge 5 commits into
skrobul wants to merge 5 commits into
Conversation
Cleans up stale HA_Chassis rows left behind by decommissioned/replaced chassis, and repopulates per-network unified HA_Chassis_Groups that were left empty as a result — avoiding the need to manually detach/reattach subnets per router to re-trigger neutron_understack's link_vxlan_network_ha_chassis_group workaround. Dry-run by default.
…ork HCG link_vxlan_network_ha_chassis_group inferred the gateway chassis from every HA_Chassis row in the NB database, requiring them all to share one chassis_name. A single row left behind by a decommissioned/replaced chassis made that check ambiguous and silently blocked HA_Chassis_Group population for every vxlan network in the fleet. Now filters to chassis still present in the Southbound DB before checking for uniqueness.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Upgrades the OVN chart to
2026.1.3+a659ab8a2and points it at the renamedopenstack-helmrepo (wasopenstack-helm-infra, which no longer publishes this chart).Updates the OVN images (
ovn_ovsdb_nb,ovn_ovsdb_sb,ovn_northd,ovn_controller) fromdocker.io/openstackhelm/ovn:ubuntu_jammy-20250111toquay.io/airshipit/ovn:ubuntu_noble-20260711.While validating the upgrade, found that a baremetal/external port on a est network wasn't being claimed by any chassis. Root cause turned out to be a pre-existing (not upgrade-related) bug in
neutron_understack.routers.link_vxlan_network_ha_chassis_group: it infers the gateway chassis by requiring everyHA_Chassisrow in the whole NB database to share onechassis_name. A single stale row left behind by a decommissioned/replaced chassis made that check ambiguous, silently blockingHA_Chassis_Grouppopulation for every vxlan network in the fleet, not just the one being tested.Fixed the check to ignore
HA_Chassisrows whose chassis no longer exists in the Southbound DB, so a stale row elsewhere doesn't block inference.Added
scripts/cleanup_dead_ovn_ha_chassis.py: a dry-run-by-default operator script that (1) removes staleHA_Chassisrows pointing at chassis no longer in the SB DB, and (2) repopulates any per-networkHA_Chassis_Groupleft empty as a result, deriving the chassis from the router's own group — no more manually detaching/reattaching subnets per router to work around it.Warning
Deploying this requires coordinated updated of the Network Node's OVS.
Deployment
Backup both databases
SB Database:
NB Database:
If you are not copying the commands verbatim, note that missing
-tfromkubectl execis intentional, otherwise the backup does not work.Steps
Specifically, the ovn-controller must use the compatible version of OVS libraries to talk to the openvswitch on the host.
On each network node, upgrade OVS from 2.17 to 3.3.4:
Then restart all of the OVN related pods.
Manual schema migration might be needed too (check the northd logs):
Following the restart, monitor the logs for any unusual errors (especially "OVNSB commit failed"). You may need to delete records in
Chassis,Chassis_Privatetables. Ideally start withovn-sbctl --db=tcp:ovn-ovsdb-sb-0.ovn-ovsdb-sb.openstack.svc.cluster.local:6642 chassis-del <stale-chassis-name>if stale chassis is found.If after that the routing still does not work, check the cleanup script included in this PR.