From 1185482a47f7e15dfd039ddcf617f346277c5a49 Mon Sep 17 00:00:00 2001 From: 1008covingtonlane <42551186+1008covingtonlane@users.noreply.github.com> Date: Sat, 4 Jul 2026 19:47:02 -0400 Subject: [PATCH 1/2] Add Environment Validator TSG: AzStackHci_DNS_ExternalDnsResolution (External DNS Resolution) Remediation guide for the dedicated external-DNS Environment Validator check AzStackHci_DNS_ExternalDnsResolution (also reported as AzStackHci_DNS_Test_External_Hostname_Resolution on current builds). This validator confirms each node can resolve an external public DNS name (management.azure.com), retries up to three times, and lists each failing node as a bullet. It is the successor of the legacy connectivity test AzStackHci_Connectivity_Test_Dns; the root cause and fix are identical, so this focused guide documents what differs (the queried hostname, the retry/Attempt suffix, the per-node bullet layout, and the result-name filter) and cross-links the Connectivity Test DNS TSG for the shared full walkthrough and DNS glossary. Adds the TSG markdown and a README index entry. --- TSG/EnvironmentValidator/README.md | 1 + ...bleshooting-DNS-External-DNS-Resolution.md | 251 ++++++++++++++++++ 2 files changed, 252 insertions(+) create mode 100644 TSG/EnvironmentValidator/Troubleshooting-DNS-External-DNS-Resolution.md diff --git a/TSG/EnvironmentValidator/README.md b/TSG/EnvironmentValidator/README.md index 9aa3fbd0..c0d21bfb 100644 --- a/TSG/EnvironmentValidator/README.md +++ b/TSG/EnvironmentValidator/README.md @@ -10,6 +10,7 @@ This folder contains the TSG's related to Environment Validators. * [Troubleshooting TestPowerShell Module Version](./Troubleshooting-Test-PowerShell-Module-Version.md) * [Troubleshooting Module Versions](Troubleshooting-Module-Versions.md) * [Troubleshooting MSI Does Not Have Access to Subscription](Troubleshooting-MSI-Does-Not-Have-Access-To-Subscription.md) +* [Troubleshooting DNS External DNS Resolution (AzStackHci_DNS_ExternalDnsResolution)](./Troubleshooting-DNS-External-DNS-Resolution.md) * [Troubleshooting Services Version Failure](../Update/Test-ServicesVersion-Failure-Mitigation-In-HealthCheck.md) * [Known Issue: High Disk Space Usage in TEMP](Known-Issue-High-Disk-Space-usage-in-TEMP.md) * [Known Issue: WinRM cannot process the configuration request](Known-Issue-WinRM-cannot-process-the-configuration-request.md) diff --git a/TSG/EnvironmentValidator/Troubleshooting-DNS-External-DNS-Resolution.md b/TSG/EnvironmentValidator/Troubleshooting-DNS-External-DNS-Resolution.md new file mode 100644 index 00000000..5d6e16d0 --- /dev/null +++ b/TSG/EnvironmentValidator/Troubleshooting-DNS-External-DNS-Resolution.md @@ -0,0 +1,251 @@ +# AzStackHci_DNS_ExternalDnsResolution + +> **At a glance** +> - **Owner:** the customer's network or DNS administrator. This is not a Microsoft software defect and not an OEM hardware or firmware issue. +> - **Impact:** Critical. It blocks Azure Local deployment and updates until external DNS resolution works on every node. +> - **Effort and downtime:** small. A per-node DNS change applies immediately, with no reboot, no cluster drain, and no impact to running VMs or live migration. +> - **Typical time to resolve:** about 15 to 30 minutes per affected node for a DNS-client fix once you have the correct DNS server addresses. Allow longer if the fix is an upstream DNS server change (a forwarder or firewall rule) that must be coordinated with whoever owns that server. +> - **Before you change anything:** do not guess DNS server IP addresses. Get the cluster's intended DNS servers from your network or DNS administrator first. + +## Overview + +This Environment Validator check confirms that each Azure Local node can resolve an +external (public) DNS name. On every node, for each DNS server configured on every +network adapter that is up, the dedicated DNS validator resolves the public name +`management.azure.com` and expects at least one A record back. It retries up to three +times before it fails, and it lists each failing node as its own bullet. If any +configured DNS server returns no records (or no DNS server is configured at all), the +check fails for that node. + +- **Severity:** Critical. When this check fails on a node and no proxy is in use, the + validator stops the remaining connectivity tests for that node, so a single DNS + failure can also hide other connectivity findings. +- **When it runs:** pre-deployment readiness, deployment, add-node, and the pre-update + health check. In practice you will most often see it block a pending Azure Local + update. +- **Result names.** This is the dedicated DNS validator that resolves + `management.azure.com`. On current builds the same external-DNS test is reported under + one of two result names, so search the health-check results for either: + `AzStackHci_DNS_ExternalDnsResolution` or + `AzStackHci_DNS_Test_External_Hostname_Resolution`. +- **Same check, older name.** This validator is the successor of the legacy + connectivity test `AzStackHci_Connectivity_Test_Dns`, which resolved `microsoft.com` + instead of `management.azure.com` and reported a single flat `Detail` line rather than + per-node bullets. The cause and the fix are identical; only the validator name and the + queried hostname differ. + +**Who owns this fix.** This is a customer network and DNS configuration check. The owner +is the customer's network or DNS administrator. It is not a Microsoft software defect, +and it is not an OEM hardware or firmware issue, so it does not require a hardware vendor +or a Microsoft product fix. A Microsoft support engineer can guide the customer through +it, but the change itself is made in the customer's DNS infrastructure or in a node's +network configuration. + +**Where a node's DNS comes from.** A node's DNS servers are set at deployment time from +the deployment configuration's management network settings and applied to the management +network adapter; they are not baked into the OEM factory image. If you are an OEM or +field engineer checking your own imaging process, confirm the image does not pin DNS +servers and leaves them to be set by deployment, so each cluster picks up the customer's +intended DNS rather than a stale value carried over from imaging. + +> **Full companion guide.** This guide is focused on the `management.azure.com` dedicated +> DNS validator. The root cause, the complete multi-entry-point discovery options, the +> per-node fan-out script, and a plain-language DNS glossary are shared with the legacy +> connectivity test and are documented once, in depth, in the companion TSG +> [Troubleshooting AzStackHci_Connectivity_Test_Dns](https://github.com/Azure/AzureLocal-Supportability/blob/main/TSG/EnvironmentValidator/Troubleshooting-Connectivity-Test-Dns.md). +> Use this guide for the `management.azure.com` specifics below; use the companion for the +> full walkthrough. (The companion link resolves once that guide is merged.) + +## Requirements + +- Administrative (local administrator) access to each Azure Local node, or a remote + PowerShell session to the nodes. +- The list of DNS servers the cluster is supposed to use, from the deployment's network + configuration. +- Access to, or coordination with, whoever administers those DNS servers, in case an + upstream server needs a forwarder or an external-resolution fix. +- No maintenance window is required. A node DNS-client change applies immediately and + does not need a reboot or a cluster drain. + +## Where this failure appears + +The same failure surfaces in several places; they all converge on the same `Detail` +string. The health-check result file is the recommended entry point; for the event-log +and portal alternatives, and a `Get-SolutionUpdate` "is an update being blocked?" check, +see the companion TSG linked above. + +Read the newest health-check result file on the cluster's infrastructure share and +filter to this check. The on-box result stores the human-readable status and message +under `AdditionalData`, so project those (the top-level `Status` is a numeric enum and +the top-level `Description` is generic): + +```powershell +$base = 'C:\ClusterStorage\Infrastructure_1\Shares\SU1_Infrastructure_1\Updates\HealthCheck\System' +if (-not (Test-Path $base)) { + # Fallback: walk all ClusterStorage volumes for the HealthCheck folder. + $base = Get-ChildItem 'C:\ClusterStorage' -Directory -ErrorAction SilentlyContinue | + ForEach-Object { Join-Path $_.FullName 'Shares\SU1_Infrastructure_1\Updates\HealthCheck\System' } | + Where-Object { Test-Path $_ } | Select-Object -First 1 +} + +$latest = $null +if ($base) { + $latest = Get-ChildItem $base -Filter 'HealthCheckResult.EnvironmentChecker.*.json' -ErrorAction SilentlyContinue | + Sort-Object LastWriteTime -Descending | Select-Object -First 1 +} + +if (-not $latest) { + Write-Warning "No HealthCheck result file found on this node (the folder is missing or no health check has run yet). Read the AzStackHciEnvironmentChecker event log (Event ID 17205) instead, or run this on a different node." +} +else { + Write-Host "Reading: $($latest.FullName)" + Get-Content $latest.FullName -Raw | ConvertFrom-Json | + Where-Object { $_.Name -like '*ExternalDnsResolution*' -or $_.Name -like '*Test_External_Hostname_Resolution*' } | + Where-Object { $_.AdditionalData.Status -eq 'FAILURE' } | + Select-Object Severity, + @{ n = 'Status'; e = { $_.AdditionalData.Status } }, + @{ n = 'Detail'; e = { $_.AdditionalData.Detail } }, + Remediation +} +``` + +Each row is one currently-failing DNS server on one node. On the Windows event log the +same record is written to `AzStackHciEnvironmentChecker` as Event ID 17205; filter its +`Name` the same way (`-like '*ExternalDnsResolution*'`) and read `AdditionalData.Detail`. +In the Azure portal, open the Azure Local cluster then the **Updates** tab; a failing +pre-update health check names the failing validator there. + +### What it looks like: example failure signature + +The dedicated validator lists each failing node as its own bullet and adds an +`(Attempt: n/3)` retry suffix, for example: + +``` +- AzL-Node-01 + - Queried dns server 10.0.0.10 for management.azure.com on AzL-Node-01 (Attempt: 3/3). Result returned 0 A records. Expected at least 1. Error: +``` + +This means the node reached the DNS server at that IP, but the server returned no A +records for `management.azure.com` after three attempts. A `No DNS server configured` +message instead means the node's up adapters have no DNS server configured at all. A +passing node reports a count of one or more and lists the resolved addresses. + +> If a proxy is configured on a node (WinHTTP proxy), the check is skipped on that node +> and reported as success, with a `Detail` of `Skipping DNS resolution test on +> because a proxy is configured.` That is expected behavior, not a failure. + +## Consequences if you do not fix this + +This check is Critical. When it fails on a node and no proxy is configured, the validator +stops the remaining connectivity tests for that node, so one DNS failure can mask other +connectivity problems and fails the pre-update health check overall. A pending Azure +Local update or a deployment that runs this readiness check will not proceed until +external DNS resolution succeeds on every node, and the cluster's cloud-managed lifecycle +(Arc, updates, billing, telemetry) is impaired until external DNS works, even though +local workloads keep running. + +## Remediation + +The check fails when a DNS server configured on a node cannot resolve the external name +`management.azure.com`. The fix is a customer-side DNS change, either on the node's +DNS-client configuration or on the upstream DNS server. The essential steps are below; +the companion TSG carries the full per-node fan-out and the option-by-option decision +tree. + +**Most common fix (start here).** The usual cause is a node pointed at a DNS server that +cannot resolve external names. Re-point that node's management adapter at a DNS server +that can (step 3, first option), or add an external-resolving forwarder on the current +server (step 3, second option). The numbered steps confirm which applies; most failures +are resolved by one of those two. + +_New to any DNS term used here (A record, forwarder, split-horizon, WinHTTP proxy)? See +the Glossary in the companion TSG linked in the overview._ + +1. List the DNS servers currently configured on the affected node: + + ```powershell + Get-DnsClientServerAddress -AddressFamily IPv4 | + Where-Object ServerAddresses | + Select-Object InterfaceAlias, @{ n = 'DnsServers'; e = { $_.ServerAddresses -join ', ' } } + ``` + +2. Test each configured DNS server the way the validator does, resolving + `management.azure.com` directly against that server: + + ```powershell + foreach ($dns in ((Get-DnsClientServerAddress -AddressFamily IPv4).ServerAddresses | Sort-Object -Unique)) { + $count = (Resolve-DnsName -Name management.azure.com -Server $dns -Type A -DnsOnly -QuickTimeout -ErrorAction SilentlyContinue).Count + '{0}: {1} A record(s)' -f $dns, ([int]$count) + } + ``` + + A server reporting `0 A record(s)` is the failing one: the node reaches it, but it + cannot resolve the external name. + +3. Fix the failing DNS server, choosing the option that matches the environment: + + - If the configured server is wrong or stale, re-point the node's management adapter + at a DNS server that can resolve external names. First identify the management + adapter (the up adapter whose IPv4 address is the node's management IP), so the + `` placeholder is concrete: + + ```powershell + Get-NetIPConfiguration | Where-Object { $_.IPv4Address } | + Select-Object InterfaceAlias, @{ n = 'IPv4'; e = { $_.IPv4Address.IPAddress -join ', ' } } + ``` + + Record the original DNS values first so the change can be rolled back, then set the + correct servers on that adapter (the correct ``,`` are your deployment's + documented management DNS servers; per node, applies immediately, no reboot): + + ```powershell + Set-DnsClientServerAddress -InterfaceAlias '' -ServerAddresses '','' + ``` + + - If the configured server is correct but internal-only, add a forwarder on that DNS + server to a resolver that can answer external queries. This change is made on the DNS + server, not on the Azure Local node, so coordinate with whoever owns that server. + - If the server resolves internal names but returns nothing for the external name, an + internal-only or split-horizon DNS zone may be shadowing external resolution; add a + forwarder or otherwise enable external resolution as above. + - Confirm that DNS traffic on port 53 from the nodes to the DNS servers is not blocked + by a firewall. + +4. If the cluster intentionally has no direct outbound name resolution and uses a proxy + for all outbound traffic, configure the WinHTTP proxy on each node. When a proxy is + present, this check self-skips and reports success. Only do this if a proxy is + genuinely part of the design. + +Re-pointing a node's DNS client is a [LOW RISK] change: it is per-node, immediate, and +reversible by restoring the previous servers. Changing an upstream DNS server is a +[MEDIUM RISK] change, because it can affect other systems that use it, so coordinate with +its owner. No node drain or reboot is required for DNS-client changes. + +## Verify the fix + +Re-run the pre-update health check so a fresh result file and fresh event-log entries are +written on every node: + +```powershell +Invoke-SolutionUpdatePrecheck -SystemHealth +Get-SolutionUpdateEnvironment | Format-List HealthState, HealthCheckDate +``` + +Confirm `HealthState` is `Success` (not `Failure`) with a current `HealthCheckDate`, then +re-read the health-check result (the query in "Where this failure appears" above): the +check should return no failing rows. You can also confirm the underlying resolution +directly on each node: + +```powershell +Resolve-DnsName -Name management.azure.com -Type A +``` + +A result containing one or more A records means external DNS resolution is working. If +failures remain, re-read the new `Detail` text: the failure may have moved to a different +DNS server or a different node that needs the same fix. + +> **Note:** the Azure portal readiness view and the cluster-wide `HealthCheckResult` file +> refresh only when a full health check or `Invoke-SolutionUpdatePrecheck` runs, not on a +> targeted per-node re-test. The portal can therefore lag a just-applied fix until the +> next precheck or the periodic (roughly daily) health check, so confirm the fix on-node +> with `Resolve-DnsName` rather than waiting on the portal. From d3e70afed5bc753a70b09e5353a218686bea9a61 Mon Sep 17 00:00:00 2001 From: 1008covingtonlane <42551186+1008covingtonlane@users.noreply.github.com> Date: Sat, 4 Jul 2026 19:58:19 -0400 Subject: [PATCH 2/2] Resolve PR review: make the TSG self-sufficient + fix event-log filter Addresses the two Copilot review findings on PR #319: 1. Broken/pending companion links + delegated core content. The guide no longer depends on the not-yet-merged Connectivity Test DNS guide: the per-node fan-out and a full DNS glossary are now inline, and the event-log / portal discovery alternatives are covered inline. The companion is reduced to a single optional "Related guide" pointer for the legacy connectivity check, honestly labelled as a separate in-progress PR that this guide does not depend on. 2. Event-log filter gap. The event-log guidance and the per-node fan-out now match BOTH result names (*ExternalDnsResolution* OR *Test_External_Hostname_Resolution*), consistent with the health-check-file query, so a build reporting the second name is not missed. Lint A; prose has no spaced clause separators; fences balanced. --- ...bleshooting-DNS-External-DNS-Resolution.md | 89 +++++++++++++++---- 1 file changed, 73 insertions(+), 16 deletions(-) diff --git a/TSG/EnvironmentValidator/Troubleshooting-DNS-External-DNS-Resolution.md b/TSG/EnvironmentValidator/Troubleshooting-DNS-External-DNS-Resolution.md index 5d6e16d0..c7753964 100644 --- a/TSG/EnvironmentValidator/Troubleshooting-DNS-External-DNS-Resolution.md +++ b/TSG/EnvironmentValidator/Troubleshooting-DNS-External-DNS-Resolution.md @@ -48,13 +48,13 @@ field engineer checking your own imaging process, confirm the image does not pin servers and leaves them to be set by deployment, so each cluster picks up the customer's intended DNS rather than a stale value carried over from imaging. -> **Full companion guide.** This guide is focused on the `management.azure.com` dedicated -> DNS validator. The root cause, the complete multi-entry-point discovery options, the -> per-node fan-out script, and a plain-language DNS glossary are shared with the legacy -> connectivity test and are documented once, in depth, in the companion TSG +> **Related guide.** This guide is self-contained for the dedicated `management.azure.com` +> DNS validator: the discovery, per-node fan-out, remediation, verification, and a DNS +> glossary are all below. A related guide covers the legacy connectivity DNS test +> `AzStackHci_Connectivity_Test_Dns` (which resolves `microsoft.com`); the root cause and +> fix are the same, so consult it only if you also see that older check. It is a separate, +> in-progress supportability PR, so this guide does not depend on it: > [Troubleshooting AzStackHci_Connectivity_Test_Dns](https://github.com/Azure/AzureLocal-Supportability/blob/main/TSG/EnvironmentValidator/Troubleshooting-Connectivity-Test-Dns.md). -> Use this guide for the `management.azure.com` specifics below; use the companion for the -> full walkthrough. (The companion link resolves once that guide is merged.) ## Requirements @@ -70,9 +70,9 @@ intended DNS rather than a stale value carried over from imaging. ## Where this failure appears The same failure surfaces in several places; they all converge on the same `Detail` -string. The health-check result file is the recommended entry point; for the event-log -and portal alternatives, and a `Get-SolutionUpdate` "is an update being blocked?" check, -see the companion TSG linked above. +string. The health-check result file is the recommended entry point (below); the Windows +event log (Event ID 17205) and the Azure portal **Updates** tab show the same failure and +are covered right after it. Read the newest health-check result file on the cluster's infrastructure share and filter to this check. The on-box result stores the human-readable status and message @@ -111,9 +111,10 @@ else { Each row is one currently-failing DNS server on one node. On the Windows event log the same record is written to `AzStackHciEnvironmentChecker` as Event ID 17205; filter its -`Name` the same way (`-like '*ExternalDnsResolution*'`) and read `AdditionalData.Detail`. -In the Azure portal, open the Azure Local cluster then the **Updates** tab; a failing -pre-update health check names the failing validator there. +`Name` the same way, matching BOTH result names +(`$_.Name -like '*ExternalDnsResolution*' -or $_.Name -like '*Test_External_Hostname_Resolution*'`), +and read `AdditionalData.Detail`. In the Azure portal, open the Azure Local cluster then +the **Updates** tab; a failing pre-update health check names the failing validator there. ### What it looks like: example failure signature @@ -134,6 +135,39 @@ passing node reports a count of one or more and lists the resolved addresses. > and reported as success, with a `Detail` of `Skipping DNS resolution test on > because a proxy is configured.` That is expected behavior, not a failure. +### Identify every affected node + +Run across all cluster nodes so you see exactly which ones are failing and against which +DNS server, matching both result names: + +```powershell +Invoke-Command -ComputerName (Get-ClusterNode).Name -ScriptBlock { + $base = 'C:\ClusterStorage\Infrastructure_1\Shares\SU1_Infrastructure_1\Updates\HealthCheck\System' + if (-not (Test-Path $base)) { + $base = Get-ChildItem 'C:\ClusterStorage' -Directory -ErrorAction SilentlyContinue | + ForEach-Object { Join-Path $_.FullName 'Shares\SU1_Infrastructure_1\Updates\HealthCheck\System' } | + Where-Object { Test-Path $_ } | Select-Object -First 1 + } + $latest = $null + if ($base) { + $latest = Get-ChildItem $base -Filter 'HealthCheckResult.EnvironmentChecker.*.json' -ErrorAction SilentlyContinue | + Sort-Object LastWriteTime -Descending | Select-Object -First 1 + } + if (-not $latest) { + # Emit an explicit NO DATA row so this node is never silently treated as passing. + return [pscustomobject]@{ Detail = 'NO DATA: no HealthCheck result file on this node (read the event log instead)' } + } + $failing = Get-Content $latest.FullName -Raw | ConvertFrom-Json | + Where-Object { ($_.Name -like '*ExternalDnsResolution*' -or $_.Name -like '*Test_External_Hostname_Resolution*') -and $_.AdditionalData.Status -eq 'FAILURE' } | + Select-Object @{ n = 'Detail'; e = { $_.AdditionalData.Detail } } + if ($failing) { $failing } else { [pscustomobject]@{ Detail = 'PASS: no failing DNS result in the latest health check' } } +} | Sort-Object PSComputerName | Select-Object PSComputerName, Detail +``` + +Every node reports one of three things: a failing `Detail` (fix it), `PASS` (the check +passed there), or `NO DATA` (the result could not be read, so confirm it with the event +log rather than assuming it passed). + ## Consequences if you do not fix this This check is Critical. When it fails on a node and no proxy is configured, the validator @@ -148,9 +182,9 @@ local workloads keep running. The check fails when a DNS server configured on a node cannot resolve the external name `management.azure.com`. The fix is a customer-side DNS change, either on the node's -DNS-client configuration or on the upstream DNS server. The essential steps are below; -the companion TSG carries the full per-node fan-out and the option-by-option decision -tree. +DNS-client configuration or on the upstream DNS server. The essential steps are below, +including a per-node fan-out to find every affected node and an option-by-option decision +tree for the fix. **Most common fix (start here).** The usual cause is a node pointed at a DNS server that cannot resolve external names. Re-point that node's management adapter at a DNS server @@ -159,7 +193,7 @@ server (step 3, second option). The numbered steps confirm which applies; most f are resolved by one of those two. _New to any DNS term used here (A record, forwarder, split-horizon, WinHTTP proxy)? See -the Glossary in the companion TSG linked in the overview._ +the [Glossary](#glossary) at the end of this guide._ 1. List the DNS servers currently configured on the affected node: @@ -249,3 +283,26 @@ DNS server or a different node that needs the same fix. > targeted per-node re-test. The portal can therefore lag a just-applied fix until the > next precheck or the periodic (roughly daily) health check, so confirm the fix on-node > with `Resolve-DnsName` rather than waiting on the portal. + +## Glossary + +Plain-language definitions of the DNS terms used in this guide. Experienced readers can +skip this section. + +- **A record:** the basic DNS record that maps a name (such as `management.azure.com`) to + an IPv4 address. This check passes only when a configured DNS server returns at least one + A record for the external name. +- **DNS server / resolver:** the server a node asks to turn a name into an address. Each + node lists one or more on its network adapters; this check tests each one. +- **Forwarder:** a setting on a DNS server that hands off queries it cannot answer itself + (such as external or public names) to another resolver that can. An internal-only DNS + server usually needs a forwarder to resolve external names. +- **Conditional forwarder:** a forwarder that applies only to a specific domain, so a + server can send just some queries (for example external names) to a particular resolver. +- **Split-horizon (split-brain) DNS:** a setup where the same DNS name resolves differently + for internal versus external clients. An internal-only zone can shadow an external name, + so the server answers internal lookups but returns nothing for the public name this check + asks for. +- **WinHTTP proxy:** a system-level outbound proxy configured on a node. When one is set, + the node routes outbound traffic through it, and this DNS check self-skips on that node + and reports success.