Skip to content

fix(infinifi): re-arm liquid reserves alert when cache is stale - #331

Draft
spalen0 wants to merge 1 commit into
mainfrom
fix/infinifi-liquid-reserves-stale-cache
Draft

fix(infinifi): re-arm liquid reserves alert when cache is stale#331
spalen0 wants to merge 1 commit into
mainfrom
fix/infinifi-liquid-reserves-stale-cache

Conversation

@spalen0

@spalen0 spalen0 commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator

Problem

The Infinifi "Low Liquid Reserves" alert only fires on a downward crossing of the $8M threshold (last >= 8M and current < 8M). If the script doesn't run for an extended period, the cached last_reserves becomes stale and the alert can stay suppressed indefinitely, even when reserves remain well below threshold for the entire outage.

Fix

In protocols/infinifi/main.py, the Liquid Reserves alert block now also writes a *_ts cache key on every run. On read, if the cached value is older than LIQUID_RESERVES_STALE_AFTER_SECONDS (12h), the previous reading is treated as "at threshold", which re-arms the crossing detection so the alert fires again on the next successful run.

A logger.info records the re-arm event so it is visible in logs.

Audit of other monitors

A repo-wide audit (full report in the PR conversation) found ~35 other cache keys with the same shape (delta/change alerts gated on a last > 0 check or an if previous is not None initialisation). Highlights:

  • CRITICAL/HIGH wrong-delta after outage: protocols/3jane/main.py (PPS, TVL, debt cap, insurance fund), protocols/strata/main.py (sUSDe/srUSDe rate, jr drain, strategy ratio, total deposits), protocols/ustb/main.py (Chainlink NAV, supply), protocols/stables/oracles.py (rate-oracle monotonicity, roundId regression), protocols/maple/main.py (PPS, TVL, delegate cover), protocols/lrt-pegs/origin_protocol.py (redeem value), protocols/apyusd/main.py (rate), protocols/rtoken/monitor_rtoken.py (StRSR rate), protocols/morpho/governance_v2.py (owner/curator/sentinels/allocators/adapters).
  • False-positive risk on cache loss: protocols/3jane/main.py insurance fund outflow — previous_shares is treated as 0, so the entire fund balance looks like a one-step outflow on the first post-outage run.
  • The only true STUCK_AFTER_OUTAGE bug is the Infinifi reserves alert fixed here; every other finding is a wrong-delta / cadence issue where the first run after cache loss silently rebaselines.

Suggested follow-up (separate PR)

Add a tiny pair of helpers to utils/cache.py:

def read_cache_fresh(key: str, max_age_seconds: int, filename: str = cache_filename):
    """Returns the cached value, or None if missing/stale (>max_age_seconds old)."""

def write_cache_fresh(key: str, value, filename: str = cache_filename):
    """Writes value + a current-timestamp marker for read_cache_fresh()."""

Then convert each of the 35 wrong-delta sites in small protocol-scoped PRs, picking a per-monitor max_age_seconds (4h for CRITICAL signals, 6–8h for MEDIUM/LOW). Happy to do that as a follow-up if this approach is approved.

Checklist

  • ruff format
  • ruff check
  • mypy (ran locally clean)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant