feat(metrics-endpoint): count /metrics scrape failures by outcome

[hatamzad-nv]

Closes #3546. Adds a carbide_metrics_scrape_failures_total counter labeled by a bounded ScrapeOutcome enum, emitted at each of the five /metrics failure branches alongside the existing logs. Includes a catalogue row in docs/observability/core_metrics.md and a unit test.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: f597ad14-e696-4eef-bc76-bcb46fbb907e

📥 Commits

Reviewing files that changed from the base of the PR and between 5ae90e8 and 6348f12.

⛔ Files ignored due to path filters (1)

• Cargo.lock is excluded by !**/*.lock

📒 Files selected for processing (3)

• crates/metrics-endpoint/Cargo.toml
• crates/metrics-endpoint/src/lib.rs
• docs/observability/core_metrics.md

🚧 Files skipped from review as they are similar to previous changes (3)

• crates/metrics-endpoint/Cargo.toml
• docs/observability/core_metrics.md
• crates/metrics-endpoint/src/lib.rs

---

Summary by CodeRabbit

• New Features
  • Added carbide_metrics_scrape_failures_total, a Prometheus counter that tracks /metrics scrape failures broken down by failure outcome.
  • The service now records a scrape-failure metric immediately for each distinct error condition.
• Documentation
  • Documented carbide_metrics_scrape_failures_total in core observability metrics.
• Tests
  • Updated and expanded tests to confirm the counter increments correctly for each outcome (including panic-related and dropped-reply scenarios).

Walkthrough

The metrics endpoint now emits carbide_metrics_scrape_failures_total, labeled by scrape failure outcome. All failure branches report an outcome, tests verify label-specific counter deltas, and the core metrics documentation lists the new counter.

Changes

Metrics scrape observability

Layer / File(s)	Summary
Failure metric contract and instrumentation wiring crates/metrics-endpoint/Cargo.toml, crates/metrics-endpoint/src/lib.rs	Adds carbide-instrument dependencies and defines the labeled MetricsScrapeFailed event.
Failure-path event emission crates/metrics-endpoint/src/lib.rs	Records queue, encoder, encoding, panic, and dropped-reply failures before returning HTTP errors.
Metric validation and documentation crates/metrics-endpoint/src/lib.rs, docs/observability/core_metrics.md	Tests outcome-specific counter deltas and documents carbide_metrics_scrape_failures_total.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant ScrapeHandler as Metrics scrape handler
  participant FailureEvent as MetricsScrapeFailed
  participant Counter as Failure counter
  ScrapeHandler->>FailureEvent: emit labeled failure outcome
  FailureEvent->>Counter: increment outcome counter
  ScrapeHandler-->>ScrapeHandler: return HTTP error response

Loading

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main change: counting /metrics scrape failures by outcome.
Description check	✅ Passed	The description is directly related to the PR and accurately describes the metric, docs, and test additions.
Linked Issues check	✅ Passed	The changes implement the five failure outcomes, emit the counter at each branch, and add the required documentation and tests.
Out of Scope Changes check	✅ Passed	The added docs, tests, and Cargo dependency updates support the stated metric work and do not appear unrelated.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pull-request-3686.docs.buildwithfern.com/infra-controller

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/metrics-endpoint/src/lib.rs`:
- Around line 591-628: Update the test scrape_failures_count_per_outcome to
invoke handle_metrics_request for each failure scenario instead of emitting
MetricsScrapeFailed directly. Use a table-driven set of cases that exercises all
five handler failure outcomes, then assert carbide_metrics_scrape_failures_total
deltas for every outcome label, including zero where appropriate.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 03dceb13-5362-4cd2-b120-b965825b69a2

📥 Commits

Reviewing files that changed from the base of the PR and between 0dae3ca and b95f711.

📒 Files selected for processing (3)

• crates/metrics-endpoint/Cargo.toml
• crates/metrics-endpoint/src/lib.rs
• docs/observability/core_metrics.md

[hatamzad-nv]

Good call — reworked the tests to drive the handler rather than emit directly. test_handle_metrics_request_failure_outcomes is now table-driven and asserts both the status and the per-outcome counter delta for encoder_busy, encoder_gone, and reply_dropped through handle_metrics_request; test_encoder_survives_collector_panic now also asserts the encoder_panicked counter. The encode_failed branch (a TextEncoder error) can't be triggered deterministically — the text encoder doesn't error on a valid registry and there's no seam to inject a failure — so it keeps a documented direct-emit check. That covers 4/5 outcomes through the real handler.