Skip to content

fix(axle): warn and continue when Axle component is unhealthy#4236

Merged
springfall2008 merged 3 commits into
mainfrom
fix/axle-unhealthy-warn-continue
Jul 12, 2026
Merged

fix(axle): warn and continue when Axle component is unhealthy#4236
springfall2008 merged 3 commits into
mainfrom
fix/axle-unhealthy-warn-continue

Conversation

@mgazza

@mgazza mgazza commented Jul 11, 2026

Copy link
Copy Markdown
Collaborator

Summary

Make an unhealthy / erroring Axle Energy VPP component warn and continue instead of influencing (and thrashing) battery control.

Problem

When the Axle component errors on every run — e.g. Axle rotated their API keys and a customer's key expired — fetch.py still trusted Axle:

  • fetch_axle_sessions() injected Axle VPP sessions as forced import/export slots into the plan (load_axle_slot).
  • fetch_axle_active() could flip PredBat into read-only mode, handing battery control to the (broken) VPP.

Neither path checked component health, so stale / inconsistent Axle state flip-flopped read-only mode and injected phantom forced slots — thrashing charge/discharge. Observed on a live Intelligent Octopus Go customer: ~5-minutely battery charge↔discharge, SOC 98%→49%, and peak-rate grid import during an active IOG car session, for ~2 days until the API key was regenerated.

Fix

  • Add Fetch.axle_component_healthy(), using the same "active but not alive" definition already used for the components_healthy dashboard sensor (predbat.py).
  • Gate both Axle influence paths on it:
    • Skip loading Axle sessions (self.axle_sessions = []) when unhealthy.
    • Never enter Axle read-only control when unhealthy.
  • Logs a single Warn: and continues with the normal tariff plan — the integration degrades to absent rather than actively wrong (fail safe).

A healthy or entirely absent Axle component is unaffected.

Tests

  • Extends tests/test_axle.py with unit tests for the predicate: healthy / not-configured / active-and-alive → left untouched; active-but-not-alive → ignored and warns.
  • unit_test.py --test axle29/29 pass; -k fetch_config passes; Black / flake8 (250) / cspell clean.

🤖 Generated with Claude Code

mgazza and others added 2 commits July 11, 2026 23:23
A failed Axle Energy VPP integration (e.g. an expired/rotated API key that errors on every run) previously still influenced the plan: fetch.py trusted fetch_axle_sessions() and the fetch_axle_active() read-only handoff even when the component was in an error state. With stale/inconsistent Axle state this flip-flopped read-only mode and injected phantom forced import/export slots, thrashing battery charge/discharge on IOG tariffs (observed live: ~5-minutely charge<->discharge, battery 98%->49%, peak-rate import during an active IOG car session).

Add Fetch.axle_component_healthy() (same 'active but not alive' definition as the components_healthy dashboard sensor) and gate both Axle influence paths on it: skip loading Axle sessions and never enter Axle read-only control when the component is unhealthy. The integration degrades to absent (warn and continue) and PredBat plans normally on the tariff. Adds unit tests for the predicate.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…th guard

test_fetch_config_options runs against the shared my_predbat instance, whose
`components` was left as a bare _MockComponents by an earlier test (no is_active).
The new axle_control path now calls axle_component_healthy() → components.is_active
("axle"), so Test 2 crashed with AttributeError.

Inject a minimal _FakeComponents (is_active/is_alive) for the duration of the test
and restore the original in teardown. Axle is present+healthy here so the control
path is exercised; the unhealthy warn-and-continue path is covered in test_axle.py.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@mgazza mgazza requested a review from springfall2008 July 11, 2026 22:47
@springfall2008

Copy link
Copy Markdown
Owner

I don't think this does anything useful, all you are doing is preventing a previous scheduled event from running if the API key becomes invalid.

…n health

The real bug behind the stale-session thrashing was in axle.py itself:
publish_axle_event() only ran from inside the fetch functions' success paths
(or a now-unreachable "not fetch_due" branch), so once fetching got stuck
failing (e.g. an expired API key), the on/off sensor state froze forever -
including a since-ended event still being reported as active - because
nothing kept re-deriving it against the current time.

Move publish_axle_event() out of _fetch_byok_event()/_fetch_managed_price_curve()
and call it unconditionally at the end of run(), after every fetch attempt
regardless of outcome. This makes the state self-correcting: a frozen cached
event naturally reports "off" once its own end_time passes, with no need to
know whether Axle's backend is still reachable.

With that fixed at the source, the fetch.py-level axle_component_healthy()
gate (and its "active but not alive" duplication of the components_healthy
sensor's health formula) is no longer needed, so it's removed along with its
dedicated test scaffolding.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@springfall2008

Copy link
Copy Markdown
Owner

Update: moved the fix to the actual root cause

Thanks for the diagnosis and repro on this one — the thrashing symptom is real. Digging into why fetch.py was seeing stale/frozen Axle data, the root cause turned out to be in axle.py itself, so I've replaced the fetch.py-level health gate with a fix at the source.

What was actually happening

AxleAPI.run() only called publish_axle_event() (the code that compares the cached event's start_time/end_time against now to derive on/off state) from inside the fetch functions' success paths, or from an elif (seconds % 60) == 0 branch that's unreachable whenever a fetch is due. Once fetching started failing continuously (e.g. an expired API key), self.updated_at never got refreshed, so fetch_due stayed True forever — meaning publish_axle_event() never ran again at all. The sensor state simply froze at whatever it was the moment the key broke, with no path left to notice that a cached event had since ended.

Same issue on the event_current/event_history side used by fetch_axle_sessions()/load_axle_slot(): since nothing kept re-evaluating the cached data against real time, a frozen "current" session kept being re-applied as long as its own recorded window was still notionally in range.

The fix (axle.py)

  • Removed the publish_axle_event() calls from inside _fetch_byok_event() and _fetch_managed_price_curve().
  • run() now calls publish_axle_event() unconditionally after every fetch attempt, regardless of success/failure/not-due. This is safe and equivalent in frequency — component_base.py's scheduler already only ever invokes run() when seconds is an exact multiple of 60, so there's no extra overhead, just no more gap.
  • This makes the on/off state self-correcting: a stale cached event naturally flips to "off" once its own end_time passes, purely from data already in memory, independent of whether Axle's backend is reachable.

Why the fetch.py gate came out

With state correctly self-expiring at the source, axle_component_healthy() was solving a symptom rather than the cause, and it duplicated the "active but not alive" health formula that already exists independently in predbat.py's components_healthy sensor and web.py's component status page (three copies of the same logic with no shared helper). Removed it, fetch.py's two call sites, and the dedicated test scaffolding in test_fetch_config_options.py.

Tests

  • Added _test_axle_publishes_state_after_fetch_failure (TDD — confirmed it fails against the old code with the state never getting published, then verified the fix): covers both an expired cached event correctly flipping to "off" and a still-active one correctly staying "on", in each case with the underlying fetch actively raising.
  • test_axle: 30/30 passing.
  • Full --quick suite and run_pre_commit (ruff/black/cspell) clean.

Happy to talk through the tradeoff if there's a reason to keep a belt-and-braces health check as well, but wanted to fix the actual freeze first rather than paper over it per-consumer.

@springfall2008 springfall2008 merged commit c78c134 into main Jul 12, 2026
2 checks passed
@springfall2008 springfall2008 deleted the fix/axle-unhealthy-warn-continue branch July 12, 2026 18:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants