-
Notifications
You must be signed in to change notification settings - Fork 3.4k
[App Service] az webapp troubleshoot commands: status/config commands that help customers troubleshoot failing apps
#33722
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
aamos-company
wants to merge
29
commits into
Azure:dev
Choose a base branch
from
aamos-company:developer/amberamos/commands
base: dev
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from all commits
Commits
Show all changes
29 commits
Select commit
Hold shift + click to select a range
1eeeedb
first push
aamos-company 839b4b7
output updates, nest config check
aamos-company 4ebcb50
Merge status branch into combined test branch
aamos-company cf78e84
troubleshoot: dash for null most-recent success/failure + friendly co…
aamos-company 6c6fab4
troubleshoot config: shorter failure message
aamos-company f293fb1
troubleshoot config: drop trailing blank line when runtime section no…
aamos-company c092b87
troubleshoot config: report polish
aamos-company c8a8448
troubleshoot config + status: retries, correlation, wrapping, hints
aamos-company f1551a5
troubleshoot config: drop legacy IssueDetected fallback
aamos-company 96b8c20
troubleshoot config: gate runtime section on timestamp only
aamos-company 85282ec
troubleshoot config: print 'Feature is currently unavailable.' on SCM…
aamos-company d2102c6
troubleshoot status: address PR review
aamos-company 4d84d21
troubleshoot config: address PR review
aamos-company 4a309c6
troubleshoot status/config: extract --report rendering into helpers
aamos-company 921a176
troubleshoot status: fix -o table after report-helper extraction
aamos-company 541409a
troubleshoot status: shorten InstanceId and add LastErrorDetails to -…
aamos-company ad7c493
troubleshoot status: render '-' for empty cells in -o table
aamos-company 961a724
troubleshoot status: add Details column and hint trailer to -o table
aamos-company e1a34ed
troubleshoot status: qualify -o table windowed columns with '(last 24h)'
aamos-company 0e1c2a0
troubleshoot status: drop 'Updated' column from -o table
aamos-company a7e4a10
troubleshoot status: restore _format_dt import for LastErrorTimestamp…
aamos-company 18ceb12
troubleshoot config: show Instance + failing count in runtime error s…
aamos-company b302c77
troubleshoot config: document the 15-minute runtime error freshness g…
aamos-company e26b47c
troubleshoot config: strip internal plumbing fields from JSON payload
aamos-company 7e2d7e3
troubleshoot config: rename SCM 404 message to name the specific feature
aamos-company 30786b8
troubleshoot: address PR #33722 review feedback
aamos-company ddbad9d
troubleshoot status: shorten SummaryFetchStatus message
aamos-company d29b428
troubleshoot status: clearer message + render orphan SCM startups whe…
aamos-company d4e4c66
troubleshoot status: generic empty-instance message + drop ARM-specif…
aamos-company File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
245 changes: 245 additions & 0 deletions
245
src/azure-cli/azure/cli/command_modules/appservice/_troubleshoot_config_report.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,245 @@ | ||
| # -------------------------------------------------------------------------------------------- | ||
| # Copyright (c) Microsoft Corporation. All rights reserved. | ||
| # Licensed under the MIT License. See License.txt in the project root for license information. | ||
| # -------------------------------------------------------------------------------------------- | ||
|
|
||
| """Human-readable report rendering for 'az webapp troubleshoot config --report'. | ||
|
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Examples:
|
||
|
|
||
| Extracted from ``custom.py`` to keep the command's control flow separate from | ||
| its presentation layer. The command builds a structured payload; this module | ||
| renders it. ``render_report(payload)`` is the sole public entry point. | ||
| """ | ||
|
|
||
| import shutil | ||
| import sys | ||
| import textwrap | ||
| from datetime import datetime, timezone | ||
|
|
||
| from azure.cli.core.style import Style, print_styled_text | ||
|
|
||
|
|
||
| def _format_dt(value): | ||
| """Human-readable timestamp: 'YYYY-MM-DD HH:MM:SS UTC' (or best-effort).""" | ||
| if not value: | ||
| return None | ||
| if isinstance(value, str): | ||
| v = value.replace('T', ' ') | ||
| is_utc = v.endswith('Z') | ||
| if '.' in v: | ||
| v = v.split('.', 1)[0] | ||
| if is_utc: | ||
| if v.endswith('Z'): | ||
| v = v[:-1] | ||
| v = v + ' UTC' | ||
| elif '+' in v: | ||
| v = v.split('+', 1)[0] | ||
| return v | ||
| return str(value) | ||
|
|
||
|
|
||
| def _short_id(instance_id): | ||
| """Truncate a long hex ARM instanceId to 10 characters for display.""" | ||
| if not instance_id: | ||
| return None | ||
| if len(instance_id) > 12: | ||
| return instance_id[:10] | ||
| return instance_id | ||
|
|
||
|
|
||
| def _relative_age(iso_value): | ||
| """Return a short 'Nh Mm ago' / 'Nm ago' / 'just now' / 'in the future' string | ||
| for an ISO-8601 UTC timestamp, or None if the input is unparseable/missing.""" | ||
| if not iso_value or not isinstance(iso_value, str): | ||
| return None | ||
| v = iso_value | ||
| if '.' in v: | ||
| head, _, tail = v.partition('.') | ||
| tz = '' | ||
| for suffix in ('Z', '+', '-'): | ||
| if suffix in tail: | ||
| idx = tail.find(suffix) | ||
| tz = tail[idx:] | ||
| break | ||
| v = head + tz | ||
| v = v.replace('Z', '+00:00') | ||
| try: | ||
| dt = datetime.fromisoformat(v) | ||
| except ValueError: | ||
| return None | ||
| if dt.tzinfo is None: | ||
| dt = dt.replace(tzinfo=timezone.utc) | ||
| delta = datetime.now(timezone.utc) - dt | ||
| total_seconds = int(delta.total_seconds()) | ||
| if total_seconds < 0: | ||
| return 'in the future' | ||
| if total_seconds < 60: | ||
| return 'just now' | ||
| minutes = total_seconds // 60 | ||
| if minutes < 60: | ||
| return '{}m ago'.format(minutes) | ||
| hours = minutes // 60 | ||
| rem_min = minutes % 60 | ||
| if hours < 24: | ||
| return '{}h {}m ago'.format(hours, rem_min) if rem_min else '{}h ago'.format(hours) | ||
| days = hours // 24 | ||
| rem_hours = hours % 24 | ||
| return '{}d {}h ago'.format(days, rem_hours) if rem_hours else '{}d ago'.format(days) | ||
|
|
||
|
|
||
| def render_report(payload): | ||
| """Print the human-readable report: BUILT-IN CHECKS + SITE RUNTIME ERROR | ||
| RECOMMENDATION. Invoked when ``--report`` is passed.""" | ||
|
|
||
| def _out(*objs): | ||
| print_styled_text(*objs, file=sys.stdout) | ||
|
|
||
| def _row(*objs): | ||
| _out(list(objs)) | ||
|
|
||
| def _labeled(label, value): | ||
| """Emit '<label><value>' with hanging indent so wrapped continuation | ||
| lines align under the value column.""" | ||
| text = '' if value is None else str(value) | ||
| term_w = shutil.get_terminal_size(fallback=(120, 40)).columns | ||
| indent = ' ' * len(label) | ||
| body_w = max(20, term_w - len(label)) | ||
| lines = textwrap.wrap(text, width=body_w) or [text] | ||
| _row((Style.PRIMARY, label), (Style.PRIMARY, lines[0])) | ||
| for cont in lines[1:]: | ||
| _row((Style.PRIMARY, indent), (Style.PRIMARY, cont)) | ||
|
|
||
| config_check = payload.get('configCheck') or {} | ||
| config_check_failed = payload.get('configCheck') is None | ||
| config_check_status = payload.get('configCheckStatus') | ||
| settings = config_check.get('Settings') or config_check.get('settings') or [] | ||
| if not isinstance(settings, list): | ||
| settings = [] | ||
| settings = [s for s in settings if isinstance(s, dict)] | ||
| runtime_error = payload.get('runtimeError') | ||
|
|
||
| def _details_level(setting): | ||
| # KuduLite tags each check with DetailsLevel: 'info' | 'warning' | 'error'. | ||
| raw = setting.get('DetailsLevel') | ||
| if raw is None: | ||
| raw = setting.get('detailsLevel') | ||
| if isinstance(raw, str): | ||
| level = raw.strip().lower() | ||
| if level in ('info', 'warning', 'error'): | ||
| return level | ||
| return 'info' | ||
|
|
||
| def _style_for_level(level): | ||
| if level == 'error': | ||
| return Style.ERROR | ||
| if level == 'warning': | ||
| return Style.WARNING | ||
| return Style.SUCCESS | ||
|
|
||
| def _is_issue(level): | ||
| return level in ('warning', 'error') | ||
|
|
||
| any_issue = any(_is_issue(_details_level(s)) for s in settings) | ||
|
|
||
| # Show the runtime error section only when the ARM lastErrorTimestamp is | ||
| # within the freshness window. The payload already carries the pre-computed | ||
| # 'isRecent' signal so structured-payload consumers can apply the same gate | ||
| # (see _RUNTIME_ERROR_FRESHNESS_MINUTES / _runtime_error_is_recent in custom.py). | ||
| show_runtime = bool(runtime_error and runtime_error.get('isRecent')) | ||
|
|
||
| # ---- Section 1: Built-in checks ---- | ||
| _out() | ||
| _row((Style.HIGHLIGHT, '═══ BUILT-IN CHECKS ' + '═' * 55)) | ||
| _out() | ||
| if config_check_failed: | ||
| if config_check_status == 404: | ||
| _row((Style.WARNING, 'Configuration check feature is currently unavailable.')) | ||
| else: | ||
| _row((Style.WARNING, | ||
| 'Failed to retrieve built-in configuration checks. Please try again. ' | ||
| 'If the issue persists, restart the application (\'az webapp restart\') and confirm the SCM (Kudu) ' | ||
| 'is running and reachable.')) | ||
| elif not settings: | ||
| _row((Style.WARNING, 'No built-in configuration checks reported.')) | ||
| else: | ||
| term_w = shutil.get_terminal_size(fallback=(120, 40)).columns | ||
| setting_w = max(20, min(40, max(len(str(s.get('Setting') or '')) for s in settings) + 2)) | ||
| value_w = max(15, min(30, max(len(str(s.get('Value') or '')) for s in settings) + 2)) | ||
|
|
||
| header = '{sname:<{sw}}{vname:<{vw}}{dname}'.format( | ||
| sname='Setting', sw=setting_w, vname='Value', vw=value_w, dname='Details') | ||
| _row((Style.HIGHLIGHT, header)) | ||
| _row((Style.SECONDARY, '{s}{v}{d}'.format( | ||
| s=('─' * (setting_w - 2)).ljust(setting_w), | ||
| v=('─' * (value_w - 2)).ljust(value_w), | ||
| d='─' * 40))) | ||
| for setting in settings: | ||
| name_v = str(setting.get('Setting') or '') | ||
| value_v = str(setting.get('Value') if setting.get('Value') is not None else '') | ||
| details_v = str(setting.get('Details') or '') | ||
| details_style = _style_for_level(_details_level(setting)) | ||
| prefix = '{s:<{sw}}{v:<{vw}}'.format( | ||
| s=name_v, sw=setting_w, v=value_v, vw=value_w) | ||
| details_w = max(20, term_w - len(prefix)) | ||
| detail_lines = textwrap.wrap(details_v, width=details_w) or [details_v] | ||
| _row((Style.PRIMARY, prefix), (details_style, detail_lines[0])) | ||
| cont_indent = ' ' * len(prefix) | ||
| for cont in detail_lines[1:]: | ||
| _row((Style.PRIMARY, cont_indent), (details_style, cont)) | ||
|
|
||
| # ---- Section 2: Site runtime error recommendation ---- | ||
| # Rendered only when the ARM lastErrorTimestamp is within the last 15 | ||
| # minutes. Applied consistently regardless of whether the built-in | ||
| # checks succeeded, failed, or reported no issues. | ||
| if show_runtime: | ||
| _out() | ||
| _out() | ||
| _row((Style.HIGHLIGHT, '═══ SITE RUNTIME ERROR RECOMMENDATION ' + '═' * 37)) | ||
| _out() | ||
| if runtime_error is None: | ||
| _row((Style.PRIMARY, 'No runtime error reported.')) | ||
| else: | ||
| state = str(runtime_error.get('state') or '') | ||
| last_error = str(runtime_error.get('lastError') or '') | ||
| details = str(runtime_error.get('lastErrorDetails') or runtime_error.get('details') or '') | ||
| timestamp_raw = runtime_error.get('lastErrorTimestamp') | ||
| timestamp = _format_dt(timestamp_raw) or str(timestamp_raw or '') | ||
| if timestamp: | ||
| age = _relative_age(timestamp_raw if isinstance(timestamp_raw, str) else None) | ||
| if age: | ||
| timestamp = '{} ({})'.format(timestamp, age) | ||
|
|
||
| instance_short = _short_id(runtime_error.get('instanceId')) | ||
| failing_count = runtime_error.get('failingInstanceCount') | ||
| if instance_short: | ||
| if isinstance(failing_count, int) and failing_count >= 1: | ||
| instance_value = '{} (1 out of {} failing instances)'.format( | ||
| instance_short, failing_count) | ||
| else: | ||
| instance_value = instance_short | ||
| _labeled('Instance ', instance_value) | ||
| if state: | ||
| _labeled('State ', state) | ||
| if last_error: | ||
| _labeled('Last Error ', last_error) | ||
| if details: | ||
| _labeled('Last Error Details ', details) | ||
| if timestamp: | ||
| _labeled('Last Error Timestamp ', timestamp) | ||
|
|
||
| _out() | ||
|
|
||
| # ---- Section 3: Suggested next steps ---- | ||
| # When any built-in check flagged a warning/error, point the user at the | ||
| # az commands they'll actually need to remediate: update the app setting | ||
| # and check application logs. Rendered in the same "Hint:" style used by | ||
| # the status command's failure footer. | ||
| if any_issue: | ||
| rg = payload.get('resourceGroup') or '<resource-group>' | ||
| site_name = payload.get('name') or '<site-name>' | ||
| _out() | ||
| _out((Style.WARNING, '▶ Hint:')) | ||
| _out(' Update flagged app setting: az webapp config appsettings set -n {} -g {} ' | ||
| '--settings KEY=VALUE'.format(site_name, rg)) | ||
| _out(' Update flagged config: az webapp config set -n {} -g {} ' | ||
| '--settings KEY=VALUE'.format(site_name, rg)) | ||
| _out(' Check application logs: az webapp log tail -n {} -g {}'.format(site_name, rg)) | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.




There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I added this because the output (see example images) deviates from the standard table/tsv/yaml/etc. Open to suggestions/thoughts if you think it should be handled a different way.