Skip to content

fix(ops): gate collector health on mount and disk runway - #804

Open
proerror77 wants to merge 1 commit into
mainfrom
codex/collector-health-mount-runway
Open

fix(ops): gate collector health on mount and disk runway#804
proerror77 wants to merge 1 commit into
mainfrom
codex/collector-health-mount-runway

Conversation

@proerror77

@proerror77 proerror77 commented Aug 10, 2026

Copy link
Copy Markdown
Owner

Change contract

Promote the existing /data mount check and disk exhaustion thresholds into hard collector-host health gates, with a minimum two-sample runway calculation below 25% free space.

Issue relationship

None

Out of scope

Collector release/install/cutover, Fee credentials, raw-ops Gate cleanup, CI cache shaping, and trading Live.

Dependencies and merge order

Depends only on main@2fb07d94, which contains PR #800 hard gates 1-4.

Focused validation

  • deployment/aliyun/test-monday-collector-health.sh: 106 passed, 0 failed
  • shellcheck: both health script and focused contract test passed
  • dash -n: both scripts passed
  • git diff --check passed
  • Counterexamples: unmounted /data does not sample or persist the root filesystem; dry-run does not read persistent runway state; first/stable low-space samples remain warnings; under-10% and measured runway under six hours breach.
  • Local agent-worktree-preflight cannot complete because its existing pipefail pipeline exits 141 with this repository's 190 worktrees; branch, worktree, base SHA, owner record, and allowed files were read back manually and match.

Rollout and rollback

Code only. After merge, install the exact merged health script as a separate runtime unit after Fee and raw-ops prerequisites are healthy. Roll back by restoring the prior script identity; no collector service or data format changes are included.

Scope exception

None.

Summary by CodeRabbit

  • New Features

    • Added validation that the /data mount is available and has measurable free space.
    • Added disk-capacity sampling and estimated exhaustion runway reporting.
    • Health checks now flag disks expected to run out within six hours.
  • Bug Fixes

    • Disk and mount failures are now reported as critical health breaches instead of warnings.
    • Health output provides clearer disk capacity, warning, critical, and runway details.

@coderabbitai

coderabbitai Bot commented Aug 10, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@proerror77, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 29 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: fdccea7d-73fa-4288-84ae-72834aad60aa

📥 Commits

Reviewing files that changed from the base of the PR and between 6f2b689 and 2900c8b.

📒 Files selected for processing (3)
  • deployment/aliyun/README.md
  • deployment/aliyun/monday-collector-health.sh
  • deployment/aliyun/test-monday-collector-health.sh
📝 Walkthrough

Walkthrough

The collector health contract now has six hard gates. The monitor validates /data mounting, disk measurement, free space, and projected exhaustion runway. It persists disk samples, reports runway metrics, and updates contract tests and worktree metadata.

Changes

Collector health mount and runway

Layer / File(s) Summary
Health contract and thresholds
agent-worktree.yml, deployment/aliyun/README.md, deployment/aliyun/monday-collector-health.sh
The contract defines six hard gates. /data mounting and disk runway checks are blocking conditions. Disk usage below 25% remains a warning until projected exhaustion falls below six hours.
Mount and runway enforcement
deployment/aliyun/monday-collector-health.sh
The monitor breaches on unmounted /data, unavailable disk data, critical free space, or short runway. It persists disk samples and emits capacity, consumption, runway, warning, and critical fields.
Contract and scenario validation
deployment/aliyun/test-monday-collector-health.sh
Tests cover disk samples, nullable runway output, warning-only low space, critical disk failures, unavailable measurements, short runway, and unmounted /data.
Estimated code review effort: 3 (Moderate) ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant CollectorHealthScript
  participant DataMount
  participant Df
  participant StateFile
  participant JsonOutput
  CollectorHealthScript->>DataMount: Check /data mount
  CollectorHealthScript->>Df: Read disk capacity and free space
  CollectorHealthScript->>StateFile: Persist disk sample
  StateFile-->>CollectorHealthScript: Return prior sample
  CollectorHealthScript->>CollectorHealthScript: Calculate consumption and runway
  CollectorHealthScript->>JsonOutput: Emit disk status and gate result
Loading

Possibly related PRs

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly and concisely describes the main change: gating collector health on the /data mount and disk runway.
Description check ✅ Passed The description includes all required sections and provides clear behavior, scope, dependencies, validation, rollout, and rollback details.
✨ Finishing Touches 💡 2
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🛠️ Fix failing CI checks 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/collector-health-mount-runway

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@deployment/aliyun/monday-collector-health.sh`:
- Around line 266-312: In deployment/aliyun/monday-collector-health.sh lines
266-312, update check_disk to track parse validity for both disk_total_kib and
disk_avail_kib, and use the existing unavailable-measurement breach whenever
either field is invalid before calculating free-space metrics or persisting a
baseline. In deployment/aliyun/test-monday-collector-health.sh lines 645-652,
add coverage for numeric total with invalid or missing available-space data and
assert the unavailable-measurement breach without a critical-free-space breach.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: cc29e6fd-cb62-41b0-bfa7-af4e9b6ee870

📥 Commits

Reviewing files that changed from the base of the PR and between 2fb07d9 and 6f2b689.

📒 Files selected for processing (4)
  • agent-worktree.yml
  • deployment/aliyun/README.md
  • deployment/aliyun/monday-collector-health.sh
  • deployment/aliyun/test-monday-collector-health.sh

Comment thread deployment/aliyun/monday-collector-health.sh

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6f2b689335

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread deployment/aliyun/monday-collector-health.sh Outdated
Comment thread deployment/aliyun/monday-collector-health.sh
@proerror77
proerror77 force-pushed the codex/collector-health-mount-runway branch 3 times, most recently from a3eca37 to 0c2a4ff Compare August 10, 2026 13:37
@proerror77
proerror77 force-pushed the codex/collector-health-mount-runway branch from 0c2a4ff to 2900c8b Compare August 13, 2026 06:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant