From 0f01d6a2c695f4c41c322002acb8b2af79ab91aa Mon Sep 17 00:00:00 2001 From: Yan Sun Date: Tue, 14 Jul 2026 23:34:46 -0700 Subject: [PATCH 1/2] fix(olm): raise controller-manager memory limit to 1Gi (KUBE-49) (#1603) The OpenShift OLM bundle shipped the kubebuilder scaffold defaults (384Mi limit / 64Mi request) for the controller-manager pod, while the Helm chart was tuned to 1Gi / 256Mi. The two install paths drifted out of sync, and the under-provisioned OLM limit intermittently OOMKills the controller-manager under load (KMM builds, ANR remediation, DRA reconciles). Reported by an IBM customer on AMD GPU Operator 1.5.0 / OpenShift 4.20. Align the OLM bundle with the Helm chart at 1Gi limit / 256Mi request by updating the manifest source of truth and regenerating the CSV. Co-authored-by: Claude (cherry picked from commit 00cce5b676c7a2158cba1844a8cef931321c8750) --- ...md-gpu-operator.clusterserviceversion.yaml | 8 +- config/manager-base/manager.yaml | 14 +-- ...6-07-14-olm-bundle-manager-memory-limit.md | 87 +++++++++++++++++++ 3 files changed, 99 insertions(+), 10 deletions(-) create mode 100644 docs-internal/knowledge/plans/2026-07-14-olm-bundle-manager-memory-limit.md diff --git a/bundle/manifests/amd-gpu-operator.clusterserviceversion.yaml b/bundle/manifests/amd-gpu-operator.clusterserviceversion.yaml index 15b56dbce..3267e3158 100644 --- a/bundle/manifests/amd-gpu-operator.clusterserviceversion.yaml +++ b/bundle/manifests/amd-gpu-operator.clusterserviceversion.yaml @@ -1660,11 +1660,11 @@ spec: periodSeconds: 10 resources: limits: - cpu: 500m - memory: 384Mi + cpu: "1" + memory: 1Gi requests: - cpu: 10m - memory: 64Mi + cpu: 100m + memory: 256Mi securityContext: allowPrivilegeEscalation: false volumeMounts: diff --git a/config/manager-base/manager.yaml b/config/manager-base/manager.yaml index 4095a85c6..93a1dfea4 100644 --- a/config/manager-base/manager.yaml +++ b/config/manager-base/manager.yaml @@ -64,15 +64,17 @@ spec: port: 8081 initialDelaySeconds: 5 periodSeconds: 10 - # TODO(user): Configure the resources accordingly based on the project requirements. - # More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ + # Keep in sync with the Helm chart controller-manager resources + # (helm-charts-k8s/values.yaml). The previous 384Mi limit was the + # kubebuilder scaffold default and caused intermittent OOMKilled + # events on OpenShift under load. resources: limits: - cpu: 500m - memory: 384Mi + cpu: 1000m + memory: 1Gi requests: - cpu: 10m - memory: 64Mi + cpu: 100m + memory: 256Mi serviceAccountName: controller-manager terminationGracePeriodSeconds: 10 tolerations: diff --git a/docs-internal/knowledge/plans/2026-07-14-olm-bundle-manager-memory-limit.md b/docs-internal/knowledge/plans/2026-07-14-olm-bundle-manager-memory-limit.md new file mode 100644 index 000000000..24f7b93f3 --- /dev/null +++ b/docs-internal/knowledge/plans/2026-07-14-olm-bundle-manager-memory-limit.md @@ -0,0 +1,87 @@ +# Align OLM bundle controller-manager resources with Helm chart + +- **Date:** 2026-07-14 +- **Author:** Yan Sun +- **Related PR(s):** # +- **Related issue(s) / JIRA:** [KUBE-49](https://amd.atlassian.net/browse/KUBE-49) + — IBM customer report: intermittent OOMKilled of the controller-manager pod, + AMD GPU Operator 1.5.0 on OpenShift 4.20 + +## Context + +An IBM customer running the AMD GPU Operator 1.5.0 on OpenShift 4.20 (installed +via the OLM bundle) reported intermittent `OOMKilled` events on the +controller-manager pod. Raising the memory limit to ~600Mi resolved it. IBM +asked whether a bundle update is planned. + +Root cause is a resource-limit inconsistency between install methods: + +| Install path | mem limit | mem request | source of truth | +|---|---|---|---| +| Helm chart (k8s) | **1Gi** | 256Mi | `helm-charts-k8s/values.yaml` | +| OLM bundle (OpenShift) | **384Mi** | 64Mi | `config/manager-base/manager.yaml` → CSV | + +The OLM bundle's `384Mi`/`64Mi` values are the original kubebuilder scaffold +defaults (the block still carried the `# TODO(user): Configure the resources +accordingly` comment) and were never tuned to the operator's real footprint. +The Helm chart was tuned to `1Gi`/`256Mi` at some point but the OLM manifest +source was not kept in sync. OpenShift OLM installs therefore shipped an +under-provisioned limit, which OOMKills under load (KMM builds, ANR +remediation, DRA reconciles all drive transient memory spikes). + +## Approach + +Update the OLM manifest source of truth and regenerate the bundle so the +OpenShift install matches the Helm chart: + +- `config/manager-base/manager.yaml` — set controller-manager `resources` to + `limits: cpu 1000m / memory 1Gi`, `requests: cpu 100m / memory 256Mi`. + Replace the scaffold `TODO` comment with a note to keep in sync with the + Helm chart and why the bump was made. +- Regenerate `bundle/manifests/amd-gpu-operator.clusterserviceversion.yaml` + (the CSV embeds the manager Deployment). Generated via the bundle tooling, + not hand-edited — only the four resource values change in the CSV. The + regenerated `createdAt` timestamp was reverted to keep the diff focused. + +Choosing `1Gi` (rather than the customer's minimal `600Mi`) deliberately +aligns the two install paths on a single, already-validated value and leaves +generous headroom over the observed failure point. + +### Alternatives considered + +- **Set the OLM limit to 600Mi** (match the customer's fix exactly) — rejected: + introduces a third distinct value and leaves the Helm/OLM inconsistency in + place; only marginally above the observed OOM threshold. +- **Lower the Helm chart to match the OLM bundle** — rejected: wrong direction; + it would regress the tuned Helm value and reintroduce the OOM risk everywhere. +- **Make the limit configurable in the bundle** — out of scope; OLM installs + can already override via the Subscription/CSV, and the immediate need is a + sane default. + +## Scope + +- **In scope:** controller-manager pod `resources` (limits + requests) in the + OLM bundle manifest source and the regenerated CSV. +- **Out of scope:** Helm chart values (already at 1Gi/256Mi); resource limits + of any other operand pod (device-plugin, node-labeller, metrics-exporter, + etc.); making the limit user-configurable via CRD/values. + +## Validation + +- `operator-sdk bundle validate ./bundle` — passes (only a pre-existing + unrelated example-annotation warning). +- Diff review: exactly two files change — `config/manager-base/manager.yaml` + and the CSV. CSV delta is only the four resource values + (`cpu 500m→"1"`, `memory 384Mi→1Gi`, `cpu 10m→100m`, `memory 64Mi→256Mi`; + `1000m` normalizes to `"1"` in the CSV, functionally identical). +- Manual / hardware: deploy the regenerated bundle on OpenShift and confirm + the controller-manager pod comes up with `limits.memory: 1Gi` and no longer + OOMKills under the customer's workload. + +## Risks and rollback + +- **Risks:** higher memory *request* (64Mi → 256Mi) marginally increases + scheduling reservation on the control-plane node; negligible on any real + cluster. No behavioral/API change. +- **Rollback:** revert the two files and regenerate the bundle; or override the + limit at install time via the Subscription/CSV `config.resources`. From bc8912686180a1c03c87b7e567977225df5eaa75 Mon Sep 17 00:00:00 2001 From: yansun1996 Date: Wed, 15 Jul 2026 07:44:52 +0000 Subject: [PATCH 2/2] chore: remove plan file from PR --- ...6-07-14-olm-bundle-manager-memory-limit.md | 87 ------------------- 1 file changed, 87 deletions(-) delete mode 100644 docs-internal/knowledge/plans/2026-07-14-olm-bundle-manager-memory-limit.md diff --git a/docs-internal/knowledge/plans/2026-07-14-olm-bundle-manager-memory-limit.md b/docs-internal/knowledge/plans/2026-07-14-olm-bundle-manager-memory-limit.md deleted file mode 100644 index 24f7b93f3..000000000 --- a/docs-internal/knowledge/plans/2026-07-14-olm-bundle-manager-memory-limit.md +++ /dev/null @@ -1,87 +0,0 @@ -# Align OLM bundle controller-manager resources with Helm chart - -- **Date:** 2026-07-14 -- **Author:** Yan Sun -- **Related PR(s):** # -- **Related issue(s) / JIRA:** [KUBE-49](https://amd.atlassian.net/browse/KUBE-49) - — IBM customer report: intermittent OOMKilled of the controller-manager pod, - AMD GPU Operator 1.5.0 on OpenShift 4.20 - -## Context - -An IBM customer running the AMD GPU Operator 1.5.0 on OpenShift 4.20 (installed -via the OLM bundle) reported intermittent `OOMKilled` events on the -controller-manager pod. Raising the memory limit to ~600Mi resolved it. IBM -asked whether a bundle update is planned. - -Root cause is a resource-limit inconsistency between install methods: - -| Install path | mem limit | mem request | source of truth | -|---|---|---|---| -| Helm chart (k8s) | **1Gi** | 256Mi | `helm-charts-k8s/values.yaml` | -| OLM bundle (OpenShift) | **384Mi** | 64Mi | `config/manager-base/manager.yaml` → CSV | - -The OLM bundle's `384Mi`/`64Mi` values are the original kubebuilder scaffold -defaults (the block still carried the `# TODO(user): Configure the resources -accordingly` comment) and were never tuned to the operator's real footprint. -The Helm chart was tuned to `1Gi`/`256Mi` at some point but the OLM manifest -source was not kept in sync. OpenShift OLM installs therefore shipped an -under-provisioned limit, which OOMKills under load (KMM builds, ANR -remediation, DRA reconciles all drive transient memory spikes). - -## Approach - -Update the OLM manifest source of truth and regenerate the bundle so the -OpenShift install matches the Helm chart: - -- `config/manager-base/manager.yaml` — set controller-manager `resources` to - `limits: cpu 1000m / memory 1Gi`, `requests: cpu 100m / memory 256Mi`. - Replace the scaffold `TODO` comment with a note to keep in sync with the - Helm chart and why the bump was made. -- Regenerate `bundle/manifests/amd-gpu-operator.clusterserviceversion.yaml` - (the CSV embeds the manager Deployment). Generated via the bundle tooling, - not hand-edited — only the four resource values change in the CSV. The - regenerated `createdAt` timestamp was reverted to keep the diff focused. - -Choosing `1Gi` (rather than the customer's minimal `600Mi`) deliberately -aligns the two install paths on a single, already-validated value and leaves -generous headroom over the observed failure point. - -### Alternatives considered - -- **Set the OLM limit to 600Mi** (match the customer's fix exactly) — rejected: - introduces a third distinct value and leaves the Helm/OLM inconsistency in - place; only marginally above the observed OOM threshold. -- **Lower the Helm chart to match the OLM bundle** — rejected: wrong direction; - it would regress the tuned Helm value and reintroduce the OOM risk everywhere. -- **Make the limit configurable in the bundle** — out of scope; OLM installs - can already override via the Subscription/CSV, and the immediate need is a - sane default. - -## Scope - -- **In scope:** controller-manager pod `resources` (limits + requests) in the - OLM bundle manifest source and the regenerated CSV. -- **Out of scope:** Helm chart values (already at 1Gi/256Mi); resource limits - of any other operand pod (device-plugin, node-labeller, metrics-exporter, - etc.); making the limit user-configurable via CRD/values. - -## Validation - -- `operator-sdk bundle validate ./bundle` — passes (only a pre-existing - unrelated example-annotation warning). -- Diff review: exactly two files change — `config/manager-base/manager.yaml` - and the CSV. CSV delta is only the four resource values - (`cpu 500m→"1"`, `memory 384Mi→1Gi`, `cpu 10m→100m`, `memory 64Mi→256Mi`; - `1000m` normalizes to `"1"` in the CSV, functionally identical). -- Manual / hardware: deploy the regenerated bundle on OpenShift and confirm - the controller-manager pod comes up with `limits.memory: 1Gi` and no longer - OOMKills under the customer's workload. - -## Risks and rollback - -- **Risks:** higher memory *request* (64Mi → 256Mi) marginally increases - scheduling reservation on the control-plane node; negligible on any real - cluster. No behavioral/API change. -- **Rollback:** revert the two files and regenerate the bundle; or override the - limit at install time via the Subscription/CSV `config.resources`.