From 1cbba5d7463272467aa0a18c9774314f98f07d5f Mon Sep 17 00:00:00 2001 From: 1008covingtonlane <42551186+1008covingtonlane@users.noreply.github.com> Date: Sat, 4 Jul 2026 12:58:31 -0400 Subject: [PATCH 1/3] TSG: SBEHealth Test-Installed-SBE-Env-Vars (Environment Validator remediation guide) New troubleshooting guide for the AzStackHci_SBEHealth_Test-Installed-SBE-Env-Vars pre-update SBE health check. Documents the Warning 'Inconsistent SBE ENV vars' state (a partial/interrupted SBE stage that leaves SBEInstalledContent/SBEInstalledMetadata mismatched), where it surfaces (portal Updates + EventID 17205 + the two machine env vars), and the remediation (re-run the Solution Builder Extension update to restore a consistent state). Adds the README index entry. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- TSG/EnvironmentValidator/README.md | 1 + ...g-SBEHealth-Test-Installed-SBE-Env-Vars.md | 238 ++++++++++++++++++ 2 files changed, 239 insertions(+) create mode 100644 TSG/EnvironmentValidator/Troubleshooting-SBEHealth-Test-Installed-SBE-Env-Vars.md diff --git a/TSG/EnvironmentValidator/README.md b/TSG/EnvironmentValidator/README.md index e023546a..f7e14f7c 100644 --- a/TSG/EnvironmentValidator/README.md +++ b/TSG/EnvironmentValidator/README.md @@ -8,6 +8,7 @@ This folder contains the TSG's related to Environment Validators. * [Troubleshooting TestPowerShell Module Version](./Troubleshooting-Test-PowerShell-Module-Version.md) * [Troubleshooting Module Versions](Troubleshooting-Module-Versions.md) * [Troubleshooting MSI Does Not Have Access to Subscription](Troubleshooting-MSI-Does-Not-Have-Access-To-Subscription.md) +* [Troubleshooting SBE Health: Installed SBE Environment Variables Consistency](./Troubleshooting-SBEHealth-Test-Installed-SBE-Env-Vars.md) * [Troubleshooting Services Version Failure](../Update/Test-ServicesVersion-Failure-Mitigation-In-HealthCheck.md) * [Known Issue: High Disk Space Usage in TEMP](Known-Issue-High-Disk-Space-usage-in-TEMP.md) * [Known Issue: WinRM cannot process the configuration request](Known-Issue-WinRM-cannot-process-the-configuration-request.md) diff --git a/TSG/EnvironmentValidator/Troubleshooting-SBEHealth-Test-Installed-SBE-Env-Vars.md b/TSG/EnvironmentValidator/Troubleshooting-SBEHealth-Test-Installed-SBE-Env-Vars.md new file mode 100644 index 00000000..dc7e75c1 --- /dev/null +++ b/TSG/EnvironmentValidator/Troubleshooting-SBEHealth-Test-Installed-SBE-Env-Vars.md @@ -0,0 +1,238 @@ +# AzStackHci_SBEHealth_Test-Installed-SBE-Env-Vars + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NameAzStackHci_SBEHealth_Test-Installed-SBE-Env-Vars
Display nameInstalled Solution Builder Extension environment variables consistency ("Validate Installed SBE Env Vars")
Validator / testTest-Installed-SBE-Env-Vars (an SBE health check emitted during pre-update validation)
ComponentSBEHealth (Environment Validator / Environment Checker)
SeverityWarning: the check flags an inconsistent installed-SBE state so you can reconcile it. It does not fail the check or block the operation, but the mismatch should be cleared before the next update.
RequirementThe machine environment variables that record the installed Solution Builder Extension (SBEInstalledContent and SBEInstalledMetadata), together with the SBE version read from oemMetadata.xml, must be internally consistent: either a complete installed SBE (both set), or no SBE at all (both unset).
Applicable ScenariosPre-update SBE health validation (update readiness), on solutions that ship a Solution Builder Extension.
Affected VersionsAzure Local, version 23H2 and later.
+ +## Quick fix + +If you just want the short version: this warning means the machine environment variables +that record the **installed** Solution Builder Extension (SBE) are in a half-set, +mismatched state on this node, usually left behind by an interrupted or partial SBE +update. Reconcile it by re-running the Solution Builder Extension update to completion so +the platform re-populates those variables consistently (**"Update to latest available +Solution Builder Extension to restore consistent SBE state"**), then re-run the pre-update +check. Full detail and how to verify the fix are below. + +## Overview + +A **Solution Builder Extension (SBE)** is the hardware partner (OEM) content that ships +alongside the Azure Local solution: drivers, firmware, and a partner module that can +contribute health tests during deployment and updates. When an SBE is installed, the +platform records where it lives on the node in two **machine environment variables**: + +- `SBEInstalledContent`: the path to the installed SBE content. +- `SBEInstalledMetadata`: the path to the installed SBE metadata (which contains + `oemMetadata.xml`, the file the SBE **version** is read from). + +During **pre-update** validation, the SBE health check reads those two variables and the +derived version, and reports the installed-SBE state as one of three outcomes: + +- **Installed (SUCCESS).** Both variables are set: that pairing is the installed signal. The + node has a complete installed SBE. The SBE version shown in the detail is read from + `oemMetadata.xml` under the metadata path and is informational; it defaults to `1.0` when + that file is absent, so a missing version does not by itself change this outcome. The detail + reads *"Detected SBE `` is installed."* +- **No SBE (SUCCESS).** Both variables are unset (or no version resolves). There is no + installed SBE, which is a valid state; SBE health checks are simply skipped. The detail + reads *"No SBE installed."* +- **Inconsistent (WARNING).** The variables are in a mismatched combination that is neither + a complete install nor a clean "no SBE" (for example the content path is missing while + the metadata path still points at a real SBE version). The detail reads *"Inconsistent + SBE ENV vars!!"* and lists the content path, metadata path, and version it saw. The + remediation is *"Update to latest available Solution Builder Extension to restore + consistent SBE state."* + +Only the **Inconsistent** outcome is actionable, and it is a **Warning**, not a hard +failure: the check does not block the deployment or update. It is an early guard. An +inconsistent installed-SBE state most often comes from an SBE stage or update that was +interrupted or only partially applied, leaving one variable updated and the other stale. If +that mismatch is carried into the next update it can cause downstream "new plus old" SBE +path problems, so the check surfaces it now so you can reconcile it first. + +## Before you start: who should do this, and is it safe? + +- **Who owns this.** This is a Solution Builder Extension / update task, owned by the person + running the update, together with the **hardware partner (OEM)** whose SBE is in use if + the SBE itself needs to be re-staged. It is **not** a generic Windows task and **not** a + networking task, so do not route it to the network team. +- **This is safe to investigate read-only.** Reading the check result, the event log, and + the two environment variables changes nothing. +- **It does not restart nodes or bounce running workloads.** This is a pre-update validation + signal, not a runtime operation. Reading the check and re-running the SBE update do not + restart cluster nodes or move running VMs. +- **Do not hand-edit these environment variables to "make the warning go away."** They are + meant to be a faithful record of the installed SBE. Set them by hand and you can hide a + real half-staged SBE and cause a later update to run against the wrong content. The + supported fix is to reconcile the installed SBE itself (re-run the SBE update), which sets + the variables correctly as a side effect. + +## Where this failure appears + +You can see this warning in two places, the Azure portal and the node itself. + +### In the Azure portal + +When you run update readiness (or update validation) from the portal, the validation phase +runs the Environment Checker and surfaces SBE health results on the cluster's **Updates** +view. An inconsistent `Test-Installed-SBE-Env-Vars` appears there as a warning under the SBE +health checks, with the "Inconsistent SBE ENV vars" detail. + +### On the node + +The Environment Checker writes each check result to the `AzStackHciEnvironmentChecker` event +log as the JSON body of an **Event ID 17205** entry, and to the cluster-wide +`HealthCheckResult.*.json` on the infrastructure share. Read this check's most recent result +on a node with: + +```powershell +Get-WinEvent -LogName AzStackHciEnvironmentChecker -FilterXPath '*[System[(EventID=17205)]]' -MaxEvents 2000 | + ForEach-Object { $_.Message | ConvertFrom-Json } | + Where-Object { $_.Name -like '*Test-Installed-SBE-Env-Vars*' } | + Select-Object -First 1 Name, Status, Severity, Description, @{n='Detail';e={$_.AdditionalData.Detail}} +``` + +The `Name` on the node carries a domain prefix (`AzStackHci_SBEHealth_`) and can carry a node +suffix, so the query uses `-like '*Test-Installed-SBE-Env-Vars*'` (leading and trailing +wildcard) to match it. When the state is inconsistent, `Severity` is `WARNING` and `Detail` +reads *"Inconsistent SBE ENV vars!! content: [...], metadata: [...], sbeVersion [...]"*, which +tells you exactly which paths and version the check saw. + +You can also read the two environment variables directly on the node to see the mismatch: + +```powershell +[pscustomobject]@{ + SBEInstalledContent = [System.Environment]::GetEnvironmentVariable('SBEInstalledContent','Machine') + SBEInstalledMetadata = [System.Environment]::GetEnvironmentVariable('SBEInstalledMetadata','Machine') +} +``` + +A consistent node shows **both** values populated (an installed SBE) or **both** empty (no +SBE). Any other combination is the inconsistency this check warns about. + +These are **per-node** machine variables, so the state is evaluated on each node +independently: an inconsistency can exist on one node while the others are fine. Read the two +variables on each node to see which nodes are affected. You do not reconcile them node by node +by hand; re-running the Solution Builder Extension update (step 2) is a cluster-level operation +that re-stages the SBE and re-populates these variables consistently across the nodes. + +## Requirements + +A consistent installed-SBE state is one of: + +- **A complete installed SBE:** `SBEInstalledContent` and `SBEInstalledMetadata` are both set + (that pairing is what marks the SBE installed). Normally both paths exist and `oemMetadata.xml` + under the metadata path supplies the SBE version, though the version is informational and + defaults to `1.0` when that file is absent. +- **No SBE at all:** `SBEInstalledContent` and `SBEInstalledMetadata` are both unset. + +Anything else (one set and the other missing, or a metadata path with a real version but no +content path) is inconsistent and raises the warning. + +## Troubleshooting Steps + +### 1. Read the warning detail and the two variables + +Run the Event ID 17205 query above (or open the `HealthCheckResult.*.json`) and read the +`Detail`, then read the two environment variables directly (the second snippet above). +Confirm the mismatch and note which side is stale: for example `SBEInstalledContent` empty +while `SBEInstalledMetadata` still points at a metadata directory that has a real +`oemMetadata.xml` version, or a content or metadata path that no longer exists on disk. + +### 2. Reconcile the installed SBE by re-running the update + +Do not edit the environment variables by hand. Instead, re-run the **Solution Builder +Extension update** to completion so the platform re-stages the SBE and re-populates +`SBEInstalledContent` and `SBEInstalledMetadata` consistently. This is the verbatim +remediation the check gives: *"Update to latest available Solution Builder Extension to +restore consistent SBE state."* + +- Re-add or re-download the SBE through the same channel you used to add it (the Azure Local + update / Solution Builder Extension flow), and let the update finish rather than cancelling + it partway. +- Confirm the SBE version matches what the cluster expects, and that the transfer completed + (no partial extraction). +- If the referenced content or metadata **path does not exist** on disk, the earlier stage + was incomplete. Re-stage the correct partner (OEM) package from your source, then re-run + the update. If you did not stage this SBE yourself (most customers do not; the update or + partner engineer, or the OEM, does), hand this off to them along with the two environment + variable values and the SBE version. For the exact per-solution steps, see the Solution + Builder Extension guidance under **Related**. + +### 3. Re-run the pre-update check + +Re-run the same **update readiness** check that first surfaced this warning, and let it +re-evaluate SBE health. If you are not sure what that means: in the Azure portal, open the +cluster's **Updates** page and run the update readiness (validation) check again; or on a node, +an administrator can run `Invoke-SolutionUpdatePrecheck`. Either way it re-reads the two +environment variables and re-classifies the installed-SBE state. See the Azure Local update +troubleshooting guidance under **Related** for more on the readiness / precheck step. + +### 4. Verify the fix + +Re-read the Event ID 17205 result (step 1). A reconciled node reports `Test-Installed-SBE-Env-Vars` +with **no `WARNING` severity** and a detail of either *"Detected SBE `` is installed."* +(both variables now set) or *"No SBE installed."* (both now cleared). Re-reading the two +environment variables shows them **both set** or **both empty**, never one without the other. +In the portal, the SBE health warning clears on the next validation pass. + +## When to escalate + +The operator owns the first fix: re-running the SBE update to completion (step 2). It becomes +the **hardware partner (OEM)'s** problem only when that clean, completed re-run cannot produce +a consistent state. Specifically: + +- A **completed** SBE update still leaves the variables inconsistent. That points at a + staging or update-engine problem rather than an interrupted run; escalate with the two + environment variable values, the SBE version, and the Event ID 17205 detail. +- The content or metadata path is set but points at a location that does not exist and cannot + be re-created by re-running the update. Escalate to the hardware partner (OEM) to re-supply + the correct SBE package. The package must install so that, on every node, the end-state is + consistent: `SBEInstalledContent` and `SBEInstalledMetadata` are both set to paths that + exist, and the metadata path carries an `oemMetadata.xml` with a resolvable version. That is + the exact end-state the OEM should confirm before handing the package back. +- The sibling SBE health checks also warn or fail (see **Related**), which can indicate a + broader SBE configuration problem rather than just a stale environment variable. + +## Related + +- **Rerun a deployment / update after fixing prerequisites** (Azure Local deployment + troubleshooting): https://learn.microsoft.com/en-us/azure-stack/hci/deploy/deployment-tool-troubleshoot#rerun-deployment +- **Solution Builder Extension** overview and partner content: + https://learn.microsoft.com/en-us/azure-stack/hci/update/solution-builder-extension +- Sibling SBE health checks that validate other parts of the same SBE: + `Test-SolutionExtensionModule` (the staged SBE `SolutionExtension` module is present, + integrity-intact, and signed), `Test-SBEPropertiesValid` (partner property values match the + SBE manifest), and `Test-SBECredentialsValid` (SBE credentials in the secret store match the + SBE manifest). From abf65b9d5b960f3024dd21f284c0f018b0198247 Mon Sep 17 00:00:00 2001 From: 1008covingtonlane <42551186+1008covingtonlane@users.noreply.github.com> Date: Sat, 4 Jul 2026 13:16:21 -0400 Subject: [PATCH 2/3] TSG: use Invoke-SolutionUpdatePrecheck -SystemHealth to re-run the check (PR review) Address the Copilot review comment: step 3 now uses 'Invoke-SolutionUpdatePrecheck -SystemHealth' (with a Get-SolutionUpdateEnvironment health-state verify), matching the repo convention in the two other pre-update readiness guides. A bare Invoke-SolutionUpdatePrecheck does not re-run the health checks; -SystemHealth is what actually re-evaluates SBE health. Also adds the HealthState check to step 4. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- ...g-SBEHealth-Test-Installed-SBE-Env-Vars.md | 23 +++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) diff --git a/TSG/EnvironmentValidator/Troubleshooting-SBEHealth-Test-Installed-SBE-Env-Vars.md b/TSG/EnvironmentValidator/Troubleshooting-SBEHealth-Test-Installed-SBE-Env-Vars.md index dc7e75c1..97703805 100644 --- a/TSG/EnvironmentValidator/Troubleshooting-SBEHealth-Test-Installed-SBE-Env-Vars.md +++ b/TSG/EnvironmentValidator/Troubleshooting-SBEHealth-Test-Installed-SBE-Env-Vars.md @@ -195,9 +195,21 @@ restore consistent SBE state."* Re-run the same **update readiness** check that first surfaced this warning, and let it re-evaluate SBE health. If you are not sure what that means: in the Azure portal, open the cluster's **Updates** page and run the update readiness (validation) check again; or on a node, -an administrator can run `Invoke-SolutionUpdatePrecheck`. Either way it re-reads the two -environment variables and re-classifies the installed-SBE state. See the Azure Local update -troubleshooting guidance under **Related** for more on the readiness / precheck step. +an administrator can trigger a fresh system health check with `Invoke-SolutionUpdatePrecheck +-SystemHealth`: + +```powershell +# Trigger a fresh system health check (this re-runs the SBE health checks) +Invoke-SolutionUpdatePrecheck -SystemHealth + +# Wait a few minutes, then check the health state +Get-SolutionUpdateEnvironment | Format-List HealthState, HealthCheckDate +``` + +The `-SystemHealth` switch is what actually re-runs the health checks (a bare +`Invoke-SolutionUpdatePrecheck` does not re-evaluate them); it re-reads the two environment +variables and re-classifies the installed-SBE state. See the Azure Local update troubleshooting +guidance under **Related** for more on the readiness / precheck step. ### 4. Verify the fix @@ -205,7 +217,10 @@ Re-read the Event ID 17205 result (step 1). A reconciled node reports `Test-Inst with **no `WARNING` severity** and a detail of either *"Detected SBE `` is installed."* (both variables now set) or *"No SBE installed."* (both now cleared). Re-reading the two environment variables shows them **both set** or **both empty**, never one without the other. -In the portal, the SBE health warning clears on the next validation pass. +If you re-ran the check with `-SystemHealth` (step 3), confirm the overall result with +`Get-SolutionUpdateEnvironment | Format-List HealthState, HealthCheckDate` and check that +`HealthState` is `Success` (not `Failure`). In the portal, the SBE health warning clears on the +next validation pass. ## When to escalate From 5652e59153c0ce92e3601dab1d2c78965e28d7ef Mon Sep 17 00:00:00 2001 From: 1008covingtonlane <42551186+1008covingtonlane@users.noreply.github.com> Date: Sat, 4 Jul 2026 15:55:09 -0400 Subject: [PATCH 3/3] TSG: read the human-readable check result from AdditionalData in the 17205 query (PR review parity) Same fix as the Endpoint-Matches-ModelSKU TSG (Copilot review on #317), adapted for this WARNING-severity check: in the EventID 17205 JSON the top-level Status and Severity are numeric enums and Description is generic; there is no AdditionalData.Severity (verified on a live node: AdditionalData keys have Status + Detail but no Severity). This check keeps AdditionalData.Status = SUCCESS even when inconsistent and flags via the numeric Severity + the detail, so the reliable signal is AdditionalData.Detail reading 'Inconsistent SBE ENV vars'. The query now projects AdditionalData.Status + Severity + AdditionalData.Detail and the prose keys on the Detail text. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- ...g-SBEHealth-Test-Installed-SBE-Env-Vars.md | 20 +++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/TSG/EnvironmentValidator/Troubleshooting-SBEHealth-Test-Installed-SBE-Env-Vars.md b/TSG/EnvironmentValidator/Troubleshooting-SBEHealth-Test-Installed-SBE-Env-Vars.md index 97703805..4c857074 100644 --- a/TSG/EnvironmentValidator/Troubleshooting-SBEHealth-Test-Installed-SBE-Env-Vars.md +++ b/TSG/EnvironmentValidator/Troubleshooting-SBEHealth-Test-Installed-SBE-Env-Vars.md @@ -120,13 +120,20 @@ on a node with: Get-WinEvent -LogName AzStackHciEnvironmentChecker -FilterXPath '*[System[(EventID=17205)]]' -MaxEvents 2000 | ForEach-Object { $_.Message | ConvertFrom-Json } | Where-Object { $_.Name -like '*Test-Installed-SBE-Env-Vars*' } | - Select-Object -First 1 Name, Status, Severity, Description, @{n='Detail';e={$_.AdditionalData.Detail}} + Select-Object -First 1 Name, + @{n='Status';e={$_.AdditionalData.Status}}, + Severity, + @{n='Detail';e={$_.AdditionalData.Detail}} ``` The `Name` on the node carries a domain prefix (`AzStackHci_SBEHealth_`) and can carry a node suffix, so the query uses `-like '*Test-Installed-SBE-Env-Vars*'` (leading and trailing -wildcard) to match it. When the state is inconsistent, `Severity` is `WARNING` and `Detail` -reads *"Inconsistent SBE ENV vars!! content: [...], metadata: [...], sbeVersion [...]"*, which +wildcard) to match it. In this JSON the human-readable status and message live under +`AdditionalData` (the top-level `Status` and `Severity` are numeric enums, and the top-level +`Description` is a generic check description). This check keeps `AdditionalData.Status` = +`SUCCESS` even when the state is inconsistent, flagging the problem through the numeric `Severity` +(a non-zero Warning value) and the detail, so the reliable signal is `AdditionalData.Detail` +reading *"Inconsistent SBE ENV vars!! content: [...], metadata: [...], sbeVersion [...]"*, which tells you exactly which paths and version the check saw. You can also read the two environment variables directly on the node to see the mismatch: @@ -214,9 +221,10 @@ guidance under **Related** for more on the readiness / precheck step. ### 4. Verify the fix Re-read the Event ID 17205 result (step 1). A reconciled node reports `Test-Installed-SBE-Env-Vars` -with **no `WARNING` severity** and a detail of either *"Detected SBE `` is installed."* -(both variables now set) or *"No SBE installed."* (both now cleared). Re-reading the two -environment variables shows them **both set** or **both empty**, never one without the other. +with an `AdditionalData.Detail` of either *"Detected SBE `` is installed."* (both variables +now set) or *"No SBE installed."* (both now cleared), and no longer the *"Inconsistent SBE ENV vars"* +detail (the numeric `Severity` returns to its non-warning value). Re-reading the two environment +variables shows them **both set** or **both empty**, never one without the other. If you re-ran the check with `-SystemHealth` (step 3), confirm the overall result with `Get-SolutionUpdateEnvironment | Format-List HealthState, HealthCheckDate` and check that `HealthState` is `Success` (not `Failure`). In the portal, the SBE health warning clears on the