Skip to content

chore: Upgrade OVN chart/images to 2026.1.3 and fix chassis-claiming regression#2143

Draft
skrobul wants to merge 5 commits into
mainfrom
understack-ovn
Draft

chore: Upgrade OVN chart/images to 2026.1.3 and fix chassis-claiming regression#2143
skrobul wants to merge 5 commits into
mainfrom
understack-ovn

Conversation

@skrobul

@skrobul skrobul commented Jul 15, 2026

Copy link
Copy Markdown
Collaborator
  • Upgrades the OVN chart to 2026.1.3+a659ab8a2 and points it at the renamed openstack-helm repo (was openstack-helm-infra, which no longer publishes this chart).

  • Updates the OVN images (ovn_ovsdb_nb, ovn_ovsdb_sb, ovn_northd, ovn_controller) from docker.io/openstackhelm/ovn:ubuntu_jammy-20250111 to quay.io/airshipit/ovn:ubuntu_noble-20260711.

  • While validating the upgrade, found that a baremetal/external port on a est network wasn't being claimed by any chassis. Root cause turned out to be a pre-existing (not upgrade-related) bug in neutron_understack.routers.link_vxlan_network_ha_chassis_group: it infers the gateway chassis by requiring every HA_Chassis row in the whole NB database to share one chassis_name. A single stale row left behind by a decommissioned/replaced chassis made that check ambiguous, silently blocking HA_Chassis_Group population for every vxlan network in the fleet, not just the one being tested.

  • Fixed the check to ignore HA_Chassis rows whose chassis no longer exists in the Southbound DB, so a stale row elsewhere doesn't block inference.

  • Added scripts/cleanup_dead_ovn_ha_chassis.py: a dry-run-by-default operator script that (1) removes stale HA_Chassis rows pointing at chassis no longer in the SB DB, and (2) repopulates any per-network HA_Chassis_Group left empty as a result, deriving the chassis from the router's own group — no more manually detaching/reattaching subnets per router to work around it.

Warning

Deploying this requires coordinated updated of the Network Node's OVS.

Deployment

Backup both databases

SB Database:

kubectl exec -i ovn-ovsdb-sb-0 -c ovsdb -- ovsdb-client backup unix:/var/run/ovn/ovnsb_db.sock > ovnsb_backup_$(date +%s).db

NB Database:

kubectl exec -i ovn-ovsdb-nb-0 -c ovsdb -- ovsdb-client backup unix:/var/run/ovn/ovnnb_db.sock > ovnnb_backup_$(date +%s).db

If you are not copying the commands verbatim, note that missing -t from kubectl exec is intentional, otherwise the backup does not work.

Steps

Specifically, the ovn-controller must use the compatible version of OVS libraries to talk to the openvswitch on the host.

On each network node, upgrade OVS from 2.17 to 3.3.4:

apt-mark unhold openvswitch-common openvswitch-switch python3-openvswitch
apt-get install openvswitch-common openvswitch-switch python3-openvswitch
apt-mark hold openvswitch-common openvswitch-switch python3-openvswitch

Then restart all of the OVN related pods.

Manual schema migration might be needed too (check the northd logs):

ovsdb-client convert tcp:ovn-ovsdb-nb-0.ovn-ovsdb-nb.openstack.svc.cluster.local:6641 /usr/share/ovn/ovn-nb.ovsschema
ovsdb-client convert tcp:ovn-ovsdb-sb-0.ovn-ovsdb-sb.openstack.svc.cluster.local:6642 /usr/share/ovn/ovn-sb.ovsschema

Following the restart, monitor the logs for any unusual errors (especially "OVNSB commit failed"). You may need to delete records in Chassis, Chassis_Private tables. Ideally start with ovn-sbctl --db=tcp:ovn-ovsdb-sb-0.ovn-ovsdb-sb.openstack.svc.cluster.local:6642 chassis-del <stale-chassis-name> if stale chassis is found.

If after that the routing still does not work, check the cleanup script included in this PR.

skrobul added 5 commits July 14, 2026 18:04
Cleans up stale HA_Chassis rows left behind by decommissioned/replaced
chassis, and repopulates per-network unified HA_Chassis_Groups that were
left empty as a result — avoiding the need to manually detach/reattach
subnets per router to re-trigger neutron_understack's
link_vxlan_network_ha_chassis_group workaround. Dry-run by default.
…ork HCG

link_vxlan_network_ha_chassis_group inferred the gateway chassis from every
HA_Chassis row in the NB database, requiring them all to share one
chassis_name. A single row left behind by a decommissioned/replaced chassis
made that check ambiguous and silently blocked HA_Chassis_Group population
for every vxlan network in the fleet. Now filters to chassis still present
in the Southbound DB before checking for uniqueness.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant