feat(health): add skip_initial_history flag to periodic log collector. False by default#3694
Open
cdraman wants to merge 1 commit into
Open
feat(health): add skip_initial_history flag to periodic log collector. False by default#3694cdraman wants to merge 1 commit into
cdraman wants to merge 1 commit into
Conversation
When `forge-hw-health` runs in periodic log collection mode, the first poll against a LogService with no saved cursor drains the full entry history. For clusters that have accumulated thousands of historical entries — including past XID faults or CPER records — this creates noise on startup and makes it hard to distinguish pre-existing events from new real-time faults. Add `skip_initial_history` to `collectors.logs.periodic` (and the auto-mode fallback config). When `true`, the collector fetches all current entries on the first encounter of an unseen LogService, records the highest entry ID as the cursor, and discards the entries without emitting them. Subsequent polls collect only new entries from that point forward, matching the real-time-only behaviour of SSE mode. Closes NVIDIA#3636 - [x] **Add** - New feature or capability - [ ] **Change** - Changes in existing functionality - [ ] **Fix** - Bug fixes - [ ] **Remove** - Removed features or deprecated functionality - [ ] **Internal** - Internal changes (refactoring, tests, docs, etc.) - [ ] **This PR contains breaking changes** - [ ] Unit tests added/updated - [ ] Integration tests added/updated - [x] Manual testing performed - [ ] No testing required (docs, internal refactor, etc.) Validated on `vr-nvl72-ts2-l11-032-c01` (10.71.23.88) against a BMC with 4 GPU XID log services containing historical XID 149 entries: - Current (skip_initial_history=false): collector drained all historical entries on first poll, emitting existing XID events. - With the changes (skip_initial_history=true): collector logged `skip_initial_history: anchored at current log position` for all 4 services, emitting zero historical log events. Signed-off-by: Dasa Chandramouli <dchandramoul@nvidia.com>
Contributor
Summary by CodeRabbit
WalkthroughAdds ChangesPeriodic log history handling
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant PeriodicLogConfig
participant spawn_periodic_logs
participant LogsCollector
participant LogService
participant PersistentState
PeriodicLogConfig->>spawn_periodic_logs: provide skip_initial_history
spawn_periodic_logs->>LogsCollector: construct collector configuration
LogsCollector->>LogService: fetch entries
LogsCollector->>PersistentState: store highest entry ID
LogsCollector-->>PersistentState: emit no historical records on first poll
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/health/src/collectors/logs/periodic.rs`:
- Around line 331-364: Update the skip_initial_history initialization flow
around last_seen_ids so services are durably marked initialized even when
all_entries is empty or contains no parseable numeric IDs; distinguish this
marker from a missing last_seen_id so subsequent entries are collected rather
than discarded as initial history. Add table-driven coverage for empty,
non-parseable, and numeric initial histories, preserving the existing numeric
anchor behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: bcb88826-4d76-49fc-8281-f0300655f4db
📒 Files selected for processing (4)
crates/health/example/config.example.tomlcrates/health/src/collectors/logs/periodic.rscrates/health/src/config.rscrates/health/src/discovery/spawn.rs
Comment on lines
+331
to
364
| None => { | ||
| let all_entries = match service.entries().await { | ||
| Ok(Some(v)) => v, | ||
| Ok(None) => continue, | ||
| Err(error) => { | ||
| fetch_failures += 1; | ||
| tracing::warn!( | ||
| %service_id, | ||
| ?error, | ||
| "Failed to fetch log entries" | ||
| ); | ||
| continue; | ||
| } | ||
| }; | ||
|
|
||
| if self.skip_initial_history { | ||
| // Anchor at the current highest entry ID without emitting | ||
| // historical entries, so the next poll collects only new | ||
| // entries. Matches the real-time-only behaviour of SSE. | ||
| let anchor = all_entries | ||
| .iter() | ||
| .filter_map(|e| e.base.id.parse::<i32>().ok()) | ||
| .max(); | ||
| if let Some(id) = anchor { | ||
| state.last_seen_ids.insert(service.odata_id().clone(), id); | ||
| tracing::info!( | ||
| %service_id, | ||
| anchor_id = id, | ||
| "skip_initial_history: anchored at current log position, \ | ||
| skipping historical entries" | ||
| ); | ||
| } | ||
| continue; | ||
| } |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift
Persist initialization for services without a numeric anchor.
At Line 354, state is updated only when an entry ID parses. An initially empty service therefore remains unseen; if an entry arrives before the next poll, it is anchored and discarded as “initial history.” The same applies when the initial entries have no parseable numeric IDs.
Persist a separate durable initialized-service marker, distinguish it from a missing last_seen_id, and add table-driven coverage for empty, non-parseable, and numeric initial histories.
As per coding guidelines, “Use table-driven tests for functions mapping inputs to outputs or errors.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/health/src/collectors/logs/periodic.rs` around lines 331 - 364, Update
the skip_initial_history initialization flow around last_seen_ids so services
are durably marked initialized even when all_entries is empty or contains no
parseable numeric IDs; distinguish this marker from a missing last_seen_id so
subsequent entries are collected rather than discarded as initial history. Add
table-driven coverage for empty, non-parseable, and numeric initial histories,
preserving the existing numeric anchor behavior.
Source: Coding guidelines
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When
forge-hw-healthruns in periodic log collection mode, the first poll against a LogService with no saved cursor drains the full entry history. For clusters that have accumulated thousands of historical entries including past XID faults or CPER records, this creates noise on startup and makes it hard to distinguish pre-existing events from new real-time faults.Add
skip_initial_historytocollectors.logs.periodic(and the auto-mode fallback config). Whentrue, the collector fetches all current entries on the first encounter of an unseen LogService, records the highest entry ID as the cursor, and discards the entries without emitting them. Subsequent polls collect only new entries from that point forward, matching the real-time-only behavior of SSE mode.skip_initial_historyis set to false by default and hence there is no change in behaviorRelated issues
Closes #3636
Type of Change
Breaking Changes
Testing
Additional Notes
Validated against a BMC with 4 GPU XID log services containing historical XID 149 entries:
skip_initial_history: anchored at current log positionfor all 4 services, emitting zero historical log events.