From 281f143959a7bd16dc8c6b412140cfa37d843877 Mon Sep 17 00:00:00 2001 From: 1008covingtonlane <42551186+1008covingtonlane@users.noreply.github.com> Date: Fri, 3 Jul 2026 20:45:06 -0400 Subject: [PATCH 1/5] TSG: SBEHealth Test-SolutionExtensionModule (Environment Validator) Add a troubleshooting guide for the SBEHealth Test-SolutionExtensionModule Environment Validator check (validator Invoke-AzStackHciSBEHealthValidation). The check validates the staged Solution Builder Extension (SBE) SolutionExtension module (content integrity, partner certificate, HealthServiceIntegration tag); a FAILURE means "the SolutionExtension module could not be validated" and blocks deployment or update at SBE health validation. Covers what the check validates and its SUCCESS / skip / FAILURE outcomes; where the failure appears (portal Updates view, EventID 17205, HealthCheckResult JSON, and the SBEContentIntegrityErrors__.txt integrity report); how to classify the cause (integrity mismatch vs certificate vs missing metadata); re-staging the correct partner SBE package and re-running the precheck; verification; and when to escalate to the hardware partner (OEM). Indexed in the EnvironmentValidator README. ADO 38357441. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- TSG/EnvironmentValidator/README.md | 1 + ...-SBEHealth-Test-SolutionExtensionModule.md | 177 ++++++++++++++++++ 2 files changed, 178 insertions(+) create mode 100644 TSG/EnvironmentValidator/Troubleshooting-SBEHealth-Test-SolutionExtensionModule.md diff --git a/TSG/EnvironmentValidator/README.md b/TSG/EnvironmentValidator/README.md index 9aa3fbd0..3d76215b 100644 --- a/TSG/EnvironmentValidator/README.md +++ b/TSG/EnvironmentValidator/README.md @@ -7,6 +7,7 @@ This folder contains the TSG's related to Environment Validators. * [Troubleshooting Test PhysicalDisk API Failure](./Troubleshooting-Test-PhysicalDisk-API.md) * [Troubleshooting Test System Drive Free Space](./Troubleshooting-Test-SystemDrive-Free-Space.md) * [Troubleshooting Domain Membership (Software IsNotPartofDomain)](./Troubleshooting-Software-IsNotPartofDomain.md) +* [Troubleshooting SBE Health Solution Builder Extension Module](./Troubleshooting-SBEHealth-Test-SolutionExtensionModule.md) * [Troubleshooting TestPowerShell Module Version](./Troubleshooting-Test-PowerShell-Module-Version.md) * [Troubleshooting Module Versions](Troubleshooting-Module-Versions.md) * [Troubleshooting MSI Does Not Have Access to Subscription](Troubleshooting-MSI-Does-Not-Have-Access-To-Subscription.md) diff --git a/TSG/EnvironmentValidator/Troubleshooting-SBEHealth-Test-SolutionExtensionModule.md b/TSG/EnvironmentValidator/Troubleshooting-SBEHealth-Test-SolutionExtensionModule.md new file mode 100644 index 00000000..b76452a0 --- /dev/null +++ b/TSG/EnvironmentValidator/Troubleshooting-SBEHealth-Test-SolutionExtensionModule.md @@ -0,0 +1,177 @@ +# AzStackHci_SBEHealth_Test-SolutionExtensionModule + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NameAzStackHci_SBEHealth_Test-SolutionExtensionModule
Display nameSolution Builder Extension module health
Validator / testTest-SolutionExtensionModule (run with Invoke-AzStackHciSBEHealthValidation)
ComponentSBEHealth (Environment Validator / Environment Checker)
SeverityCritical: when the module cannot be validated the check fails and the operation (deployment or update) is blocked at SBE health validation.
RequirementThe staged Solution Builder Extension (SBE) package must contain a valid, integrity-intact, correctly signed SolutionExtension module.
Applicable ScenariosDeployment and Update (SBE health validation), on solutions that ship a Solution Builder Extension.
Affected VersionsAzure Local, version 23H2 and later.
+ +## Overview + +A **Solution Builder Extension (SBE)** is the hardware partner (OEM) content that ships +alongside the Azure Local solution: drivers, firmware, and a partner **SolutionExtension** +PowerShell module that can contribute health tests during deployment and updates. This +validator checks that the staged SBE package contains a `SolutionExtension` module that can +be **validated** for health testing. + +The check runs `Test-SolutionExtensionModule` against the SBE package path. It looks for the +module at `\Configuration\SolutionExtension`, runs an **SBE content integrity +check** (the staged content must match the SBE manifest), confirms the module is signed with a +valid partner certificate, and confirms it carries the `HealthServiceIntegration` tag. The +outcome is one of: + +- **SUCCESS (validated).** The module is present, intact, signed, and health-integration + capable, so SBE health testing proceeds. +- **SUCCESS (skipped).** There is no SBE, or the SBE does not implement health tests / does + not carry the `HealthServiceIntegration` tag. This is a benign skip, not a failure. +- **FAILURE (Critical).** The module **could not be validated**: the staged SBE content + failed its integrity check, the module is not correctly signed, or the SBE metadata expected + for the installed SBE version could not be found. The detail reads *"The SolutionExtension + module could not be validated"*. + +A FAILURE blocks the operation at SBE health validation. This is almost always a problem with +the **staged SBE content** (corrupt, incomplete, hand-edited, or mismatched against the +manifest), not with the cluster hardware itself. + +## Before you start: who should do this, and is it safe? + +- **Who owns this.** This is a Solution Builder Extension / deployment task, owned by the + person running the deployment or update together with the **hardware partner (OEM)** whose + SBE is in use. It is **not** a generic Windows task and **not** a networking task, so do not + route it to the network team. +- **Do not hand-edit the staged SBE content.** The integrity check compares the staged + content against the SBE manifest, so editing, adding, or removing files under the SBE + package will itself cause this check to fail. The fix is to re-stage the correct partner + package, never to patch files inside it. +- **This is safe to investigate read-only.** Reading the validator result, the event log, and + the integrity error report changes nothing. The remediation (re-staging the SBE and + re-running the precheck) is a normal deployment/update action. + +## Where this failure appears + +You can see this failure in two places, the Azure portal and the node itself. + +### In the Azure portal + +When you deploy or update from the portal, the validation phase runs the Environment Checker +and surfaces failed checks on the cluster's **Updates** (or deployment **Validation**) view. +A failed `Test-SolutionExtensionModule` appears there in red, under the SBE health checks, +with the "could not be validated" detail. + +### On the node + +The Environment Checker writes each check result to the `AzStackHciEnvironmentChecker` event +log as the JSON body of an **Event ID 17205** entry, and to the cluster-wide +`HealthCheckResult.*.json` on the infrastructure share. Read this check's most recent result +on a node with: + +```powershell +Get-WinEvent -LogName AzStackHciEnvironmentChecker -FilterXPath '*[System[(EventID=17205)]]' -MaxEvents 2000 | + ForEach-Object { $_.Message | ConvertFrom-Json } | + Where-Object { $_.Name -like 'Test-SolutionExtensionModule*' } | + Select-Object -First 1 Name, Status, Severity, Description, @{n='Detail';e={$_.AdditionalData.Detail}} +``` + +When the failure is an **integrity** mismatch, the check also writes a detailed report next to +the SBE metadata, named `SBEContentIntegrityErrors__.txt`. The failure detail +points at that file and summarizes what it found (for example files with hash mismatches, or +extra or missing files). Read that report to see exactly which staged files diverged from the +manifest, which is the fastest way to tell a corrupt or hand-edited stage from an incomplete +one. + +## Requirements + +- The staged SBE package contains `Configuration\SolutionExtension\SolutionExtension.psd1` and + `SolutionExtension.psm1`. +- The staged SBE content matches the SBE manifest (passes the content integrity check). +- The `SolutionExtension` module is signed with a valid partner (OEM) certificate. +- The SBE metadata for the installed SBE version is present and reachable. + +## Troubleshooting Steps + +### 1. Read the failure detail and classify it + +Run the Event ID 17205 query above (or open the `HealthCheckResult.*.json`) and read the +`Detail`. Then classify the cause: + +- **"SBE content integrity check failed"** (with a `SBEContentIntegrityErrors_*.txt` reference): + the staged content does not match the manifest. Open the referenced report to see whether it + is *hash mismatches* (files were changed or corrupted), *extra files* (content was added, for + example a hand copied or partially extracted stage), or *missing files* (an incomplete stage). +- **A certificate or module-load error:** the `SolutionExtension` module is present but is not + correctly signed, or its manifest is malformed. +- **"could not find the SBE Metadata directory"** for an installed SBE version: the SBE is + registered as installed but its staged metadata is missing. + +### 2. Re-stage the correct partner SBE package + +Do not patch the staged files. Instead, obtain the exact SBE package your solution expects from +the hardware partner (OEM) and re-stage it through the normal path (the deployment/update SBE +source your solution uses). Confirm the SBE version matches what the cluster expects, and that +the transfer completed (no partial extraction, no added files). This replaces the corrupt or +incomplete stage with content that matches the manifest. + +### 3. Re-run the check + +Re-run SBE health validation and confirm the check now passes. During an update you can drive +this with the update precheck; during deployment, re-run the deployment validation step (see +the Azure Local deployment troubleshooting guidance under **Related**). Confirm +`Test-SolutionExtensionModule` returns **SUCCESS** (validated, or a benign skip if this SBE +does not implement health tests). + +### 4. Verify the fix + +Re-read the Event ID 17205 result (step 1). A fixed check reports `Status = SUCCESS` for +`Test-SolutionExtensionModule`, and no new `SBEContentIntegrityErrors_*.txt` is written on the +next run. In the portal, the SBE health check clears from red on the next validation pass. + +## When to escalate + +- The **OEM-provided** SBE package fails the integrity check even after a clean re-stage from + the partner source. That points at a bad partner package rather than a staging problem; + escalate to the hardware partner (OEM) with the `SBEContentIntegrityErrors_*.txt` report and + the SBE version. +- The module is present and intact but fails certificate validation. That is a partner signing + issue; escalate to the OEM. +- The sibling SBE health checks also fail (see **Related**), which can indicate a broader SBE + configuration or credential problem rather than a content-integrity one. + +## Related + +- **Rerun a deployment / update after fixing prerequisites** (Azure Local deployment + troubleshooting): https://learn.microsoft.com/en-us/azure-stack/hci/deploy/deployment-tool-troubleshoot#rerun-deployment +- Sibling SBE health checks that validate other parts of the same SBE: + `Test-SBEPropertiesValid` (partner property values match the SBE manifest; remediated with + `Set-SolutionExtensionProperty`) and `Test-SBECredentialsValid` (SBE credentials in the secret + store match the SBE manifest). +- **Solution Builder Extension** overview and partner content: + https://learn.microsoft.com/en-us/azure-stack/hci/update/solution-builder-extension From 85ff7fc058e40111ca3c2ddae1f1d867f9ef5a0f Mon Sep 17 00:00:00 2001 From: 1008covingtonlane <42551186+1008covingtonlane@users.noreply.github.com> Date: Fri, 3 Jul 2026 20:53:10 -0400 Subject: [PATCH 2/5] TSG: fix the Event ID 17205 query filter to match the prefixed check Name The diagnostic query anchored the Name filter to the start ('Test-SolutionExtensionModule*'), but the on-box check Name is prefixed (AzStackHci_SBEHealth_...) and node-suffixed, so it matched nothing and returned no results. Use a leading + trailing wildcard ('*Test-SolutionExtensionModule*'), consistent with the existing SBE content-integrity TSG. Addresses the Copilot PR review on #314. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../Troubleshooting-SBEHealth-Test-SolutionExtensionModule.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/TSG/EnvironmentValidator/Troubleshooting-SBEHealth-Test-SolutionExtensionModule.md b/TSG/EnvironmentValidator/Troubleshooting-SBEHealth-Test-SolutionExtensionModule.md index b76452a0..2e956f97 100644 --- a/TSG/EnvironmentValidator/Troubleshooting-SBEHealth-Test-SolutionExtensionModule.md +++ b/TSG/EnvironmentValidator/Troubleshooting-SBEHealth-Test-SolutionExtensionModule.md @@ -97,7 +97,7 @@ on a node with: ```powershell Get-WinEvent -LogName AzStackHciEnvironmentChecker -FilterXPath '*[System[(EventID=17205)]]' -MaxEvents 2000 | ForEach-Object { $_.Message | ConvertFrom-Json } | - Where-Object { $_.Name -like 'Test-SolutionExtensionModule*' } | + Where-Object { $_.Name -like '*Test-SolutionExtensionModule*' } | Select-Object -First 1 Name, Status, Severity, Description, @{n='Detail';e={$_.AdditionalData.Detail}} ``` From bf87ccb02d86f99bdef7e8d2baf3ed973fac41c8 Mon Sep 17 00:00:00 2001 From: 1008covingtonlane <42551186+1008covingtonlane@users.noreply.github.com> Date: Sat, 4 Jul 2026 10:52:47 -0400 Subject: [PATCH 3/5] TSG: address grader persona work items (raises the usability panel to 5/5) Applies the three sub-5 persona findings from the tsg-forge grader: - concrete-remediation (5 personas): step 2 now gives concrete re-stage steps split by operation (update: re-add/re-download the SBE update, then Invoke-SolutionUpdatePrecheck; deployment: replace the SBE content in the deployment source, then re-run validation), plus a concrete hand-off action (give the integrity report + SBE version to the deployment/partner engineer or OEM) for readers who did not stage the SBE themselves. - workload-impact-note (App/VM Engineer): "Before you start" now states outright that the check and its fix do not restart nodes or bounce running VM workloads. - oem-action-clarity (OEM Vendor): the escalation now spells out the exact end-state the returned package must meet: manifest integrity, a valid partner certificate, and the HealthServiceIntegration tag in SolutionExtension.psd1. Static lint stays A; the persona usability panel goes 4.3/5 -> 5/5 (no remaining work items). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- ...-SBEHealth-Test-SolutionExtensionModule.md | 32 +++++++++++++++---- 1 file changed, 26 insertions(+), 6 deletions(-) diff --git a/TSG/EnvironmentValidator/Troubleshooting-SBEHealth-Test-SolutionExtensionModule.md b/TSG/EnvironmentValidator/Troubleshooting-SBEHealth-Test-SolutionExtensionModule.md index 2e956f97..85ba0476 100644 --- a/TSG/EnvironmentValidator/Troubleshooting-SBEHealth-Test-SolutionExtensionModule.md +++ b/TSG/EnvironmentValidator/Troubleshooting-SBEHealth-Test-SolutionExtensionModule.md @@ -75,6 +75,10 @@ manifest), not with the cluster hardware itself. - **This is safe to investigate read-only.** Reading the validator result, the event log, and the integrity error report changes nothing. The remediation (re-staging the SBE and re-running the precheck) is a normal deployment/update action. +- **It does not restart nodes or bounce running workloads.** This is a deployment/update + validation gate, not a runtime operation. Reading the check and re-staging the SBE content do + not restart cluster nodes or move running VMs. On an already-deployed cluster a failure blocks + the in-progress update from proceeding, but it does not by itself disrupt running workloads. ## Where this failure appears @@ -134,11 +138,23 @@ Run the Event ID 17205 query above (or open the `HealthCheckResult.*.json`) and ### 2. Re-stage the correct partner SBE package -Do not patch the staged files. Instead, obtain the exact SBE package your solution expects from -the hardware partner (OEM) and re-stage it through the normal path (the deployment/update SBE -source your solution uses). Confirm the SBE version matches what the cluster expects, and that -the transfer completed (no partial extraction, no added files). This replaces the corrupt or -incomplete stage with content that matches the manifest. +Do not patch the staged files. Replace the **whole** SBE package with the exact one your solution +expects from the hardware partner (OEM), staged through the same path the operation uses. Which +path depends on the operation: + +- **During an update** (the SBE arrives as a solution update): re-add or re-download the SBE + update through the same channel you used to add it (the Azure Local update / Solution Builder + Extension flow), so the staged copy is replaced with the correct partner content. Then re-run + the readiness check with `Invoke-SolutionUpdatePrecheck`. +- **During deployment** (the SBE comes from your deployment media / source): replace the SBE + content in that deployment source with the OEM-provided package, then re-run the deployment + validation step. + +In both cases, confirm the SBE version matches what the cluster expects and that the transfer +completed (no partial extraction, no added files). If you did not stage this SBE yourself (most +customers do not; the deployment or partner engineer, or the OEM, does), hand this off to them +along with the `SBEContentIntegrityErrors_*.txt` report and the SBE version. For the exact +per-solution steps, see the Solution Builder Extension guidance under **Related**. ### 3. Re-run the check @@ -159,7 +175,11 @@ next run. In the portal, the SBE health check clears from red on the next valida - The **OEM-provided** SBE package fails the integrity check even after a clean re-stage from the partner source. That points at a bad partner package rather than a staging problem; escalate to the hardware partner (OEM) with the `SBEContentIntegrityErrors_*.txt` report and - the SBE version. + the SBE version. The package the OEM returns must satisfy all three of the module's validation + requirements, which the OEM can confirm before handing it back: its content matches the SBE + manifest (passes the integrity check), the `SolutionExtension` module is signed with a valid + partner (OEM) certificate, and the module manifest (`SolutionExtension.psd1`) declares the + `HealthServiceIntegration` tag. - The module is present and intact but fails certificate validation. That is a partner signing issue; escalate to the OEM. - The sibling SBE health checks also fail (see **Related**), which can indicate a broader SBE From b8788d4b1ddeebc6f94f1167719bab23e6f17597 Mon Sep 17 00:00:00 2001 From: 1008covingtonlane <42551186+1008covingtonlane@users.noreply.github.com> Date: Sat, 4 Jul 2026 15:56:16 -0400 Subject: [PATCH 4/5] TSG: read the human-readable check result from AdditionalData in the 17205 query (PR review parity) Same fix as the Endpoint-Matches-ModelSKU TSG (Copilot review on #317): in the EventID 17205 JSON the top-level Status and Severity are numeric enums and Description is generic; the human-readable FAILURE/SUCCESS is AdditionalData.Status and the specific message is AdditionalData.Detail. The on-box query now projects AdditionalData.Status + AdditionalData.Detail and the verify-fix step reads AdditionalData.Status. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- ...oting-SBEHealth-Test-SolutionExtensionModule.md | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/TSG/EnvironmentValidator/Troubleshooting-SBEHealth-Test-SolutionExtensionModule.md b/TSG/EnvironmentValidator/Troubleshooting-SBEHealth-Test-SolutionExtensionModule.md index 85ba0476..88f94634 100644 --- a/TSG/EnvironmentValidator/Troubleshooting-SBEHealth-Test-SolutionExtensionModule.md +++ b/TSG/EnvironmentValidator/Troubleshooting-SBEHealth-Test-SolutionExtensionModule.md @@ -102,9 +102,17 @@ on a node with: Get-WinEvent -LogName AzStackHciEnvironmentChecker -FilterXPath '*[System[(EventID=17205)]]' -MaxEvents 2000 | ForEach-Object { $_.Message | ConvertFrom-Json } | Where-Object { $_.Name -like '*Test-SolutionExtensionModule*' } | - Select-Object -First 1 Name, Status, Severity, Description, @{n='Detail';e={$_.AdditionalData.Detail}} + Select-Object -First 1 Name, + @{n='Status';e={$_.AdditionalData.Status}}, + @{n='Detail';e={$_.AdditionalData.Detail}} ``` +In this JSON the human-readable status and message live under `AdditionalData` (the top-level +`Status` and `Severity` are numeric enums, and the top-level `Description` is a generic check +description), so the query projects `AdditionalData.Status` and `AdditionalData.Detail`. When the +module cannot be validated, `AdditionalData.Status` is `FAILURE` and `AdditionalData.Detail` reads +*"The SolutionExtension module could not be validated ..."*. + When the failure is an **integrity** mismatch, the check also writes a detailed report next to the SBE metadata, named `SBEContentIntegrityErrors__.txt`. The failure detail points at that file and summarizes what it found (for example files with hash mismatches, or @@ -166,8 +174,8 @@ does not implement health tests). ### 4. Verify the fix -Re-read the Event ID 17205 result (step 1). A fixed check reports `Status = SUCCESS` for -`Test-SolutionExtensionModule`, and no new `SBEContentIntegrityErrors_*.txt` is written on the +Re-read the Event ID 17205 result (step 1). A fixed check reports `AdditionalData.Status = SUCCESS` +for `Test-SolutionExtensionModule`, and no new `SBEContentIntegrityErrors_*.txt` is written on the next run. In the portal, the SBE health check clears from red on the next validation pass. ## When to escalate From 47a247be1ec35764c79491ef6f159a7370f3516e Mon Sep 17 00:00:00 2001 From: 1008covingtonlane <42551186+1008covingtonlane@users.noreply.github.com> Date: Mon, 6 Jul 2026 12:57:16 -0400 Subject: [PATCH 5/5] Fix Related links: retired deployment-troubleshoot page + legacy azure-stack/hci base Per TJ's review. Verified with curl (effective URL + status): - The "Rerun a deployment" link pointed at the retired azure-stack/hci/deploy/deployment-tool-troubleshoot#rerun-deployment, which now 301-redirects to the generic azure-local/whats-new page (the #rerun-deployment anchor no longer resolves). Repointed to the canonical page azure/azure-local/manage/troubleshoot-deployment#restart-the-deployment-via-azure-portal (verified 200, anchor present). - The Solution Builder Extension link used the legacy en-us/azure-stack/hci base (301 to azure/azure-local). Switched to the canonical azure/azure-local form, locale-less, matching the house style in #304/#305. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../Troubleshooting-SBEHealth-Test-SolutionExtensionModule.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/TSG/EnvironmentValidator/Troubleshooting-SBEHealth-Test-SolutionExtensionModule.md b/TSG/EnvironmentValidator/Troubleshooting-SBEHealth-Test-SolutionExtensionModule.md index 88f94634..e236fb0e 100644 --- a/TSG/EnvironmentValidator/Troubleshooting-SBEHealth-Test-SolutionExtensionModule.md +++ b/TSG/EnvironmentValidator/Troubleshooting-SBEHealth-Test-SolutionExtensionModule.md @@ -196,10 +196,10 @@ next run. In the portal, the SBE health check clears from red on the next valida ## Related - **Rerun a deployment / update after fixing prerequisites** (Azure Local deployment - troubleshooting): https://learn.microsoft.com/en-us/azure-stack/hci/deploy/deployment-tool-troubleshoot#rerun-deployment + troubleshooting): https://learn.microsoft.com/azure/azure-local/manage/troubleshoot-deployment#restart-the-deployment-via-azure-portal - Sibling SBE health checks that validate other parts of the same SBE: `Test-SBEPropertiesValid` (partner property values match the SBE manifest; remediated with `Set-SolutionExtensionProperty`) and `Test-SBECredentialsValid` (SBE credentials in the secret store match the SBE manifest). - **Solution Builder Extension** overview and partner content: - https://learn.microsoft.com/en-us/azure-stack/hci/update/solution-builder-extension + https://learn.microsoft.com/azure/azure-local/update/solution-builder-extension