fix(infinifi): re-arm liquid reserves alert when cache is stale - #331
Draft
spalen0 wants to merge 1 commit into
Draft
fix(infinifi): re-arm liquid reserves alert when cache is stale#331spalen0 wants to merge 1 commit into
spalen0 wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The Infinifi "Low Liquid Reserves" alert only fires on a downward crossing of the $8M threshold (
last >= 8Mandcurrent < 8M). If the script doesn't run for an extended period, the cachedlast_reservesbecomes stale and the alert can stay suppressed indefinitely, even when reserves remain well below threshold for the entire outage.Fix
In
protocols/infinifi/main.py, the Liquid Reserves alert block now also writes a*_tscache key on every run. On read, if the cached value is older thanLIQUID_RESERVES_STALE_AFTER_SECONDS(12h), the previous reading is treated as "at threshold", which re-arms the crossing detection so the alert fires again on the next successful run.A
logger.inforecords the re-arm event so it is visible in logs.Audit of other monitors
A repo-wide audit (full report in the PR conversation) found ~35 other cache keys with the same shape (delta/change alerts gated on a
last > 0check or anif previous is not Noneinitialisation). Highlights:protocols/3jane/main.py(PPS, TVL, debt cap, insurance fund),protocols/strata/main.py(sUSDe/srUSDe rate, jr drain, strategy ratio, total deposits),protocols/ustb/main.py(Chainlink NAV, supply),protocols/stables/oracles.py(rate-oracle monotonicity, roundId regression),protocols/maple/main.py(PPS, TVL, delegate cover),protocols/lrt-pegs/origin_protocol.py(redeem value),protocols/apyusd/main.py(rate),protocols/rtoken/monitor_rtoken.py(StRSR rate),protocols/morpho/governance_v2.py(owner/curator/sentinels/allocators/adapters).protocols/3jane/main.pyinsurance fund outflow —previous_sharesis treated as 0, so the entire fund balance looks like a one-step outflow on the first post-outage run.Suggested follow-up (separate PR)
Add a tiny pair of helpers to
utils/cache.py:Then convert each of the 35 wrong-delta sites in small protocol-scoped PRs, picking a per-monitor
max_age_seconds(4h for CRITICAL signals, 6–8h for MEDIUM/LOW). Happy to do that as a follow-up if this approach is approved.Checklist