[App Service] az webapp troubleshoot commands: status/config commands that help customers troubleshoot failing apps#33722
Conversation
Cherry-picks the entire origin/developer/amberamos/troubleshootcommand branch on top of the config command changes so both 'az webapp troubleshoot config' and 'az webapp troubleshoot status' preview commands can be tested together. Conflict resolution: additive merge — both commands, their help entries, param contexts, table transformers, and tests coexist.
…nfig failure message Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…t rendered Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add extra blank line above SITE RUNTIME ERROR RECOMMENDATION. - Render 'No runtime error reported.' in primary (white) instead of green. - Reword SCM failure message to lead with retry before restart. - Drop trailing horizontal rule at end of report. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Combined update to the troubleshoot config and status commands. troubleshoot config: - Retry SCM config fetch on transient 5xx / connection errors (3 attempts, 1.5s backoff). - Read KuduLite's new DetailsLevel field (info/warning/error) with legacy IssueDetected fallback; map to SUCCESS/WARNING/ERROR styles. - Restyle the built-in checks table: HIGHLIGHT header, SECONDARY underlines, per-row Details colored by level; wrap Details column so long values hang-indent under the Details column. - Runtime error section: hanging-indent wrapping for State / Last Error / Last Error Details / Last Error Timestamp, plus relative age next to the timestamp. - Gate runtime section on last-error-within-15-min OR config-check-failed (never on stale flagged settings alone). - Add '▶ Hint:' footer with az webapp config appsettings set / az webapp config set / az webapp log tail commands when any check is warning/error. - _help.py: drop KuduLite / ARM implementation details from long-summary. - Tests: transient-5xx retry cases, config-check-failed keeps runtime section case, refresh stale-timestamp fixtures. troubleshoot status: - Correlate ARM /siteStatus items with KuduLite startup summaries by machineName first, then fall back to cardinality pairing for the common single-instance case; leftover SCM entries surface as an orphanStartups bucket. - When ARM's machineName and SCM's InstanceId disagree, per-instance header shows 'Instance <machine> Full Status Report (SCM: <scm-id>)'. - _emit_labeled hanging-indent helper wraps long values under the value column. - Switch overview table header + underline to Style.PRIMARY; '- * 76' separators around instance headers; upgrade Hint: label to '▶ Hint:'. - Suppress the 'machine name could not be determined' fallback when cardinality pairing succeeded. - _help.py: reword status long-summary to match config phrasing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: da04b3ec-d5e4-4ab8-aeea-58dd7010ac84
KuduLite now always emits DetailsLevel (info|warning|error) for every check, so the IssueDetected string flag is no longer part of the contract. Removing the legacy fallback in _details_level to keep the mapping single-sourced. - custom.py: _details_level reads only DetailsLevel/detailsLevel (defaults to 'info'); dropped the IssueDetected/issueDetected fallback branch. - custom.py: refreshed the stale comment above the runtime section to match the current show_runtime gating (fresh ARM error OR config fetch failed). - tests: migrated three test fixtures from IssueDetected: 'False'/'True' to DetailsLevel: 'info'/'warning' so they exercise the current contract. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: da04b3ec-d5e4-4ab8-aeea-58dd7010ac84
Previously the SITE RUNTIME ERROR RECOMMENDATION section would render whenever the built-in config check fetch failed, regardless of how old the ARM lastErrorTimestamp was. This produced inconsistent output: a 3-hour-old runtime error would surface on the attempt where SCM failed but not on adjacent successful attempts against the same site. The section now renders if and only if lastErrorTimestamp is within the last 15 minutes, applied uniformly whether the built-in checks succeeded, failed, or reported no issues. A stale error should not be resurrected just because the SCM fetch timed out. - custom.py: show_runtime = _last_error_within(15); dropped the 'or config_check_failed' short-circuit. - custom.py: refreshed comments on both the assignment and the section header. - tests: rewrote test_troubleshoot_config_report_shows_runtime_when_config_check_failed to use a fresh timestamp; added test_troubleshoot_config_report_suppresses_runtime_when_config_check_failed_and_error_is_stale as its negative counterpart. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: da04b3ec-d5e4-4ab8-aeea-58dd7010ac84
… 404 When the SCM built-in-check endpoint returns 404 the site is on a stamp where the KuduLite config-check route hasn't rolled out yet -- there is no actionable remediation the user can take, so the verbose 'retry-and-restart' guidance is misleading. Replace it with a compact, definitive message. - custom.py: added 'configCheckStatus' to the payload so the renderer can distinguish 404 from other failure modes. - custom.py: in --report mode, when configCheck is None and configCheckStatus == 404, print 'Feature is currently unavailable.'; otherwise keep the existing 'Failed to retrieve...' guidance. - custom.py: in JSON mode, collapsed both 404 warning branches (with-body and without-body) to the same 'Feature is currently unavailable.' logger.warning call. - tests: extended test_troubleshoot_config_scm_404_returns_empty_settings to assert the exact warning text; added test_troubleshoot_config_report_shows_feature_unavailable_on_404 for the styled report path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: da04b3ec-d5e4-4ab8-aeea-58dd7010ac84
- Remove duplicate/unused top-level 'import os' and 'import sys' (avoid W0404/W0611).
- Remove _log_webapp_status_tip helper and all cross-promotion call sites in
create_webapp, webapp_up, and _poll_deployment_runtime_status; the tip
suggested 'az webapp status' which is out of scope for this PR and (with
the group nesting) would not resolve.
- Pin troubleshoot_status api-version to a validated literal ('2024-11-01')
instead of reading protected client._config.api_version, so the URL doesn't
drift if the SDK bumps its default.
- Update mock tests to assert 2024-11-01 URLs.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: da04b3ec-d5e4-4ab8-aeea-58dd7010ac84
- Drop group scaffolding collision with Azure#33673: shared argument_context('webapp troubleshoot') is folded into the webapp-troubleshoot-config argument block (name / resource_group_name / slot / report). Group help + group-level scaffolding belong to the status PR. - Unify webapp-troubleshoot group short-summary to match Azure#33673. - Fix payload/report asymmetry on runtime-error freshness: annotate the runtimeError dict with 'isRecent' (bool) and 'freshnessWindowMinutes' (int) so structured-payload consumers can apply the same 15-minute gate the --report view uses. Move the freshness helper to module scope and drive both the payload annotation and the report gate off the shared _RUNTIME_ERROR_FRESHNESS_MINUTES constant. - Pin troubleshoot_config api-version to literal '2024-11-01' (both the /siteStatus fetch and the /instances retry-on-404 fallback) instead of reading protected client._config.api_version. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: da04b3ec-d5e4-4ab8-aeea-58dd7010ac84
|
Validation for Azure CLI Full Test Starting...
Thanks for your contribution! |
|
Validation for Breaking Change Starting...
Thanks for your contribution! |
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
|
Thank you for your contribution @aamos-company! We will review the pull request and get back to you soon. |
Move both --report renderers (status ~340 lines, config ~150 lines) out of custom.py into dedicated helper modules with a single public entry point, render_report(payload): - _troubleshoot_status_report.py - _troubleshoot_config_report.py custom.py now lazily imports each module only on the --report path. Tests updated to patch the extracted modules where the names are looked up. No behavioral change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: da04b3ec-d5e4-4ab8-aeea-58dd7010ac84
az webapp troubleshoot commands: status/config commands that help customers troubleshoot failing apps
transform_troubleshoot_status_output imports _format_dt, _most_recent_startup, and _startup_fetch_failed which have moved from custom.py to _troubleshoot_status_report.py. Point the import at the new module so 'az webapp troubleshoot status -o table' stops failing with 'Table output unavailable.' Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: da04b3ec-d5e4-4ab8-aeea-58dd7010ac84
…o table Match the --report view: use _short_id() for the InstanceId column, format LastErrorTimestamp via _format_dt(), and surface the lastErrorDetails field as its own column whenever the LastError columns are shown. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: da04b3ec-d5e4-4ab8-aeea-58dd7010ac84
Any None/empty value in a shown column now renders as '-' so a healthy instance's LastError/LastErrorDetails or a Kudu fetch failure's Succeeded/Failed/Updated are visually distinct from an accidentally blank row. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: da04b3ec-d5e4-4ab8-aeea-58dd7010ac84
Adds a per-instance `Details` column (from `inst.details`) so the table surfaces the same short state description the report shows. When at least one instance reports a visible error, registers an `atexit` callback that writes the recommended follow-up `az webapp log tail` / `az webapp log startup show` commands to stderr AFTER the table has flushed to stdout, matching the --report hint footer. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: da04b3ec-d5e4-4ab8-aeea-58dd7010ac84
Rename Succeeded/Failed/Updated to Succeeded (last 24h) / Failed (last 24h) / Updated (last 24h) so the columns' 24-hour window is obvious without cross-referencing the report. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: da04b3ec-d5e4-4ab8-aeea-58dd7010ac84
The most-recent-attempt timestamp is redundant on the summary table; keep it in --report / structured output. Also drop the now-unused _format_dt / _most_recent_startup imports. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: da04b3ec-d5e4-4ab8-aeea-58dd7010ac84
… cell Removing the Updated column dropped _format_dt from the transformer's import list, but the LastErrorTimestamp cell still calls _format_dt inside the show_errors branch. When any instance reported a visible error, the transformer raised NameError which knack surfaces as the opaque 'Table output unavailable' message. Re-import _format_dt and add a regression test that drives the transformer with an errored instance so this path is covered. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: da04b3ec-d5e4-4ab8-aeea-58dd7010ac84
…ection The site runtime error recommendation section reports the freshest error seen on any single instance, but until now the report didn't identify which instance or how many were failing. Users seeing 'BootFailure' on a 3-instance app couldn't tell whether it was a stamp-wide outage or a single stuck worker. Tag the runtime error block returned by _extract_runtime_error with failingInstanceCount (the number of instances that reported lastError), and render a new 'Instance' row above 'State' in the report showing the shortened ARM instanceId plus '(1 out of N failing instances)'. The count is also included in the structured JSON payload for scripting consumers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: da04b3ec-d5e4-4ab8-aeea-58dd7010ac84
| ]) for s in settings if isinstance(s, dict)] | ||
|
|
||
|
|
||
| def transform_troubleshoot_status_output(result): |
| # Licensed under the MIT License. See License.txt in the project root for license information. | ||
| # -------------------------------------------------------------------------------------------- | ||
|
|
||
| """Human-readable report rendering for 'az webapp troubleshoot status --report'. |
| # Licensed under the MIT License. See License.txt in the project root for license information. | ||
| # -------------------------------------------------------------------------------------------- | ||
|
|
||
| """Human-readable report rendering for 'az webapp troubleshoot config --report'. |
There was a problem hiding this comment.
Examples:
Note: there is a PR out that will improve the suggestions for the last error details
- Built in checks is unavailable but runtime status still shows (if the error time is within the last 15 mins)
- Built in checks available (**test data, not actual value. Corresponding Json from Kudu below). If any of the details level is error, hints will be shown
- Built in checks + site runtime recommendation
| ]) for r in result] | ||
|
|
||
|
|
||
| def transform_troubleshoot_config_output(result): |
…ate in help Users have been asking why the SITE RUNTIME ERROR RECOMMENDATION section sometimes disappears even though the JSON payload still contains a runtimeError block. Add a note to 'az webapp troubleshoot config --help' explaining that the recommendation section is only rendered for errors that occurred within the last 15 minutes, while the structured payload carries older errors tagged with isRecent=false. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: da04b3ec-d5e4-4ab8-aeea-58dd7010ac84
The structured payload leaked three implementation details: - top-level configCheckStatus (HTTP status of the last SCM attempt) - runtimeError.freshnessWindowMinutes (the 15-minute constant) - runtimeError.isRecent (freshness gate used by --report) These were only useful to the --report renderer, not to script consumers, and made the JSON output noisier than necessary. Compute them internally, pass them to the renderer as before, then pop them from the payload before it is returned to the CLI framework. Update tests to assert the fields are absent, and update the config command help so it no longer promises an isRecent flag on the payload. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: da04b3ec-d5e4-4ab8-aeea-58dd7010ac84
Changed the SCM-404 warning from the generic "Feature is currently unavailable." to "Configuration check feature is currently unavailable." so the message stands on its own without surrounding context — e.g. in a CI log tail or a support ticket paste. Applied to both the logger.warning() emitted in JSON/YAML/table mode and the Style.WARNING row rendered in --report mode. Tests updated to assert against the new string. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: da04b3ec-d5e4-4ab8-aeea-58dd7010ac84
| (2) The most recent site runtime status error reported by App Service. | ||
| The runtime error recommendation section is only surfaced when the | ||
| error occurred within the last 15 minutes; older errors are still | ||
| included in the structured payload but are hidden from the `--report` |
There was a problem hiding this comment.
I added this because the output (see example images) deviates from the standard table/tsv/yaml/etc. Open to suggestions/thoughts if you think it should be handled a different way.
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
There was a problem hiding this comment.
Pull request overview
Adds a new preview az webapp troubleshoot command group under the App Service module to help diagnose failing Linux web apps by combining ARM runtime status with SCM (KuduLite) diagnostics, with both structured output and a --report human-readable mode.
Changes:
- Introduces
az webapp troubleshoot statusandaz webapp troubleshoot config, including ARM/siteStatus+/instancescalls and SCM endpoints for startup/config diagnostics. - Adds dedicated report-rendering helper modules and table transformers for
-o table. - Adds extensive mocked unit tests plus wiring updates (help, params, command table, changelog).
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| src/azure-cli/HISTORY.rst | Adds changelog entries for the new troubleshoot commands. |
| src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_commands_thru_mock.py | Adds mocked unit tests for troubleshoot status/config and table output transformers. |
| src/azure-cli/azure/cli/command_modules/appservice/custom.py | Implements troubleshoot command logic, SCM/ARM calls, correlation, and report mode plumbing. |
| src/azure-cli/azure/cli/command_modules/appservice/commands.py | Registers the new command group and adds -o table transformers. |
| src/azure-cli/azure/cli/command_modules/appservice/_troubleshoot_status_report.py | New human-readable status --report renderer and formatting helpers. |
| src/azure-cli/azure/cli/command_modules/appservice/_troubleshoot_config_report.py | New human-readable config --report renderer and formatting helpers. |
| src/azure-cli/azure/cli/command_modules/appservice/_params.py | Adds arguments for webapp troubleshoot config/status (slot/instance/report). |
| src/azure-cli/azure/cli/command_modules/appservice/_help.py | Adds help + examples + parameter docs for the new troubleshoot commands. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if parsed.tzinfo is None: | ||
| parsed = parsed.replace(tzinfo=timezone.utc) | ||
| except (ValueError, TypeError): | ||
| return False | ||
| return (datetime.now(timezone.utc) - parsed) <= timedelta(minutes=minutes) |
| try: | ||
| summary_response = requests.get(summary_url, headers=headers, timeout=30) | ||
| except requests.RequestException as ex: | ||
| logger.warning("Failed to call '%s': %s", summary_url, ex) | ||
| summary_response = None | ||
|
|
There was a problem hiding this comment.
Will look into this -- good suggestion
| # per-instance until one worker responds with data. | ||
| scm_url = _get_scm_url(cmd, resource_group_name, name, slot) | ||
| headers = get_scm_site_headers(cmd.cli_ctx, name, resource_group_name, slot) | ||
| config_url = '{}/api/troubleshoot/config'.format(scm_url) |
| # Strip internal plumbing fields from the structured payload so the | ||
| # JSON/YAML/table output stays focused on user-visible data. | ||
| payload.pop('configCheckStatus', None) | ||
| if runtime_error is not None: | ||
| runtime_error.pop('freshnessWindowMinutes', None) | ||
| runtime_error.pop('isRecent', None) |
There was a problem hiding this comment.
Will update the PR description/tests. I don't think those fields are necessary for the json output that the Cx will see
| * `az webapp list-runtimes`: Add `--runtime` and `--support` filter parameters (#32903) | ||
| * [BREAKING CHANGE] `az webapp list-runtimes`: Remove deprecated `--linux` and `--show-runtime-details` parameters (#32903) | ||
| * `az webapp log startup`: Add commands to list and view Linux container startup logs (#33256) | ||
| * `az webapp troubleshoot config`: Add preview command that runs built-in configuration checks against a Linux web app and surfaces the last runtime error from ARM |
There was a problem hiding this comment.
Need to update both PR numbers to this PR
| with self.argument_context('webapp troubleshoot config') as c: | ||
| c.argument('name', arg_type=webapp_name_arg_type, id_part=None) | ||
| c.argument('resource_group_name', arg_type=resource_group_name_type) | ||
| c.argument('slot', options_list=['--slot', '-s'], help="the name of the slot. Default to the production slot if not specified") |
| with self.argument_context('webapp troubleshoot status') as c: | ||
| c.argument('name', arg_type=webapp_name_arg_type, id_part=None) | ||
| c.argument('resource_group', arg_type=resource_group_name_type) | ||
| c.argument('slot', options_list=['--slot', '-s'], help="the name of the slot. Default to the production slot if not specified") |
|
App Service |
- _runtime_error_is_recent: reject future timestamps (clock skew) - troubleshoot status: surface SCM transport errors via SummaryFetchStatus - HISTORY.rst: add tracking PR number - _params.py: grammar Default to -> Defaults to (config + status) Adds two regression tests covering the freshness gate and the SCM RequestException path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: da04b3ec-d5e4-4ab8-aeea-58dd7010ac84
Drop the SCM error details and endpoint URL from the fallback message so the report reads cleanly. The report renderer no longer prefixes with 'Startup summary unavailable:' since the new message is self-describing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: da04b3ec-d5e4-4ab8-aeea-58dd7010ac84
…n ARM is empty When /siteStatus returns no instances the report now explains that this is normal for a healthy app and links the two standard follow-up commands. If SCM returned startup summaries anyway, still render them under the orphan block instead of bailing early. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: da04b3ec-d5e4-4ab8-aeea-58dd7010ac84
…ic label Use portal-oriented guidance rather than exposing internal ARM/SCM plumbing. Also drop the '(no matching ARM instance)' qualifier from the orphan startup header. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: da04b3ec-d5e4-4ab8-aeea-58dd7010ac84







Related command
az webapp troubleshoot configaz webapp troubleshoot statusDescription
Introduces the az webapp troubleshoot command group (preview) with two commands for diagnosing Linux web apps without leaving the terminal.
az webapp troubleshoot statusReports per-instance runtime health and recent startup outcomes for a Linux web app.
Data sources
• Site Runtime Status — ARM
GET .../sites/{name}[/slots/{slot}]/siteStatus[/{instanceId}]?api-version=2024-11-01• Per-instance startup summary — KuduLite
GET https://{scm-host}/api/startuplogs/summary[?instance={id}]• Machine-name ↔ ARM hex id mapping — ARM /instances , so --instance accepts either form.
What it surfaces
• Per-instance State + Details, LastError / LastErrorDetails / LastErrorTimestamp (hidden for Started instances or when there have been no failed startups in the report window), and the last 24h Succeeded / Failed / most-recent-attempt counts.
• Handles KuduLite SummaryFetchStatus failures gracefully (missing startup summary is reported per instance, not silently zeroed).
• Correlates orphan startup entries (instances the log endpoint knows about but ARM no longer lists) so nothing is dropped.
az webapp troubleshoot configRuns the KuduLite built-in configuration checks and pairs them with the most recent runtime error from ARM.
Data sources
• Configuration diagnostics — KuduLite
GET https://{scm-host}/api/troubleshoot/config• Site runtime error — ARM
GET .../sites/{name}[/slots/{slot}]/siteStatus?api-version=2024-11-01What it surfaces
• Every setting KuduLite evaluated with its value, details message, and severity (info/warning/error).
• A separate "Site runtime error recommendation" section that appears only when the last ARM error is within the last 15 minutes, so stale errors don't pollute the output.
• Prints a single-line "Configuration check feature is currently unavailable." message when the SCM /api/troubleshoot /config endpoint returns 404 (older KuduLite builds), then still renders the runtime-error section if a fresh error is available.
Output modes (both commands)
• Default: structured JSON/YAML/TSV payload — the standard CLI formatters handle it.
• -o table : focused per-instance / per-setting projection with - for empty cells; the status table also prints an "▶ Hint:" trailer with the recommended az webapp log tail / az webapp log startup show follow-ups when at least one instance shows a visible error.
• --report : full color-coded human-readable report, rendered by the new _troubleshoot_status_report.py / _troubleshoot_config_report.py helper modules so custom.py stays focused on the command's control flow.
Testing Guide
This checklist is used to make sure that common guidelines for a pull request are followed.
The PR title and description has followed the guideline in Submitting Pull Requests.
I adhere to the Command Guidelines.
I adhere to the Error Handling Guidelines.