[WIP] Gaia charts#362
Open
mrosseel wants to merge 457 commits into
Open
Conversation
mrosseel
force-pushed
the
deepchart
branch
5 times, most recently
from
June 23, 2026 11:03
f0fb516 to
b31666e
Compare
…ld/cache) + point channels at fork during transition
The Lint & Test workflow had been red on every push since 2026-02-24 (all three jobs failing). Root causes, one per job: lint — ruff format drift: 25 files were unformatted against the dev-shell ruff. Reformatted them (no logic changes). type-check — two problems. (1) The tetra3/cedar-solve git submodule was never checked out, so the python/tetra3 symlink dangled and mypy died with "Cannot read file 'tetra3'" before checking anything; CI now checks out submodules. (2) mypy was bumped to 2.x (flake.lock), whose stricter class-vs-instance check surfaced 28 real annotation bugs: UIModule.display_class was typed Type[DisplayBase] but holds a DisplayBase instance, and MenuManager.stack was list[type[UIModule]] but holds instances. Fixed the annotations (and dropped the now-redundant type: ignore/cast workarounds they required). Also ignore the pynput / waitress / google.protobuf stubs and skip following tetra3 imports, so 'mypy --install-types' no longer tries to pip-install stubs in the pip-less Nix dev env. test — same missing submodule (pytest 'No module named tetra3'), plus the Steinicke .xls parse needs xlrd, which the uv migration dropped. Added xlrd to the dev group (uv.lock regenerated) and check out submodules. Verified locally in the nixos-lint workspace: ruff check + format, mypy, and pytest -m smoke/-m unit all pass.
…inent unstable trunk row; fix CI nix PATH on self-hosted runner; add NixOS deploy docs
…ion-tarball runner
… on-device UI reports live WiFi mode PiFinder-AP had autoconnect=true, so NetworkManager started the device's own AP instantly at boot and stayed on it, beating a saved client network (e.g. apollo) that still had to scan+associate. Set autoconnect=false so the AP comes up only on demand via pifinder-wifi-fallback. The on-device WiFi-mode menu read a static wifi_status.txt (always 'Client'); it now reads the live NetworkManager mode like the web UI.
…iFi mode/IP The device status + web UI showed an empty/AP value when reachable over ethernet. Add Network.is_wired_connected()/get_active_label(); local_ip() no longer hard-returns 10.10.10.1 in AP mode when a wired link is up. The status screen drops its stale wifi_status.txt read and uses live values.
The size-based progress query ran under set -euo pipefail; nix path-info against the release cache (which lacks the closure) returned non-zero and killed the whole upgrade before it downloaded anything. Wrap the query in set +e with timeouts and jq fallbacks so it can never abort — the only gate is the nix build, and the profile changes only after it succeeds. The 'no longer available' check moves to a post-failure diagnostic.
… code-quality, i18n, tests)
The previous size query ran nix path-info -r over the whole closure (tens of thousands of narinfo lookups) — so slow it hung the bar at 0/0. Query only the delta: nix-store --realise --dry-run lists the to-fetch paths, then size just those. First-boot (whole-system) uses a path-count splash since per-path sizing the full system is inherently too slow. Still fully advisory (set +e + timeouts); never aborts the upgrade.
…weep instrumentation)
…mat stragglers mypy: the B-V cache globals are Optional, so assignments cannot narrow them across statements; parse into locals and use those for savez/len. format: 4 files the CI format session was silently reformatting on every run (timeentry, three tests).
Telemetry captured only IMU and solve streams — blind to the radiometric side. A third event type records the camera-side radiometer sample at ~1 Hz: t (capture epoch, s), exp (driver-reported exposure, s), bg (background median, ADU), mad (median absolute deviation, ADU), grad (quadrant gradient, ADU), seq (frame sequence). Dedupes on sequence; replay players skip unknown event types, so old builds read new files. A full-night radio stream answers what sweep snapshots cannot: how auto-exposure actually behaves over a session (lever-arm availability for the black-level fit), a continuous sky-steadiness timeline, and real-night input for replaying pedestal estimators.
The marking menu down slot is the one readable at the eyepiece without moving your head — it goes to SWEEP, the everyday reference-meter action. SQM Correct is deleted rather than kept: a magnitude-additive session knob silently absorbs ADU-space, brightness-dependent errors (tonight: a stuck pedestal), masking exactly the fault class it appears to fix, and staling whenever the layers beneath it change. Its legitimate output — the unit-vs-reference-meter difference — is already recorded as evidence by every sweep run with a reference reading. The feature never shipped upstream (fork-only), so now is the cheapest moment to remove it. Removed: UISQMCorrect screen, menu entry and corrected indicator, _apply_sqm_correct and both solver call sites, the sqm_correct_delta shared state, and all test references. CONTEXT.md records the decision.
python/result and python/result-lib are nix build output symlinks that were committed before .gitignore's result/result-* rule existed, so they stayed tracked and kept getting re-snapshotted into unrelated commits. Untrack them; the existing ignore rule keeps them out from now on. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXDxJkU5CzAegv3FcMEwEq
The NixOS migration UI flow (UIMigrationConfirm/UIMigrationProgress) was removed from ui/software.py in 'refactor(software): delete the NixOS migration UI flow', but a later sqm-fix merge reintroduced the test's import and spec builders for those deleted classes, breaking test collection. Drop the stale references. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXDxJkU5CzAegv3FcMEwEq
test_radec_entry exercised an older dependency-injection API (BlinkingCursor time_provider; CoordinateConverter/CoordinateEntryLogic calc_utils_provider; LayoutConfig no-arg) that radec_entry dropped for module-level calc_utils/time. Patch calc_utils/time at the module level and pass a display stub to LayoutConfig so the suite matches the implementation. 6 stale tests were failing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXDxJkU5CzAegv3FcMEwEq
test_radec_entry exercised an older dependency-injection API (BlinkingCursor time_provider; CoordinateConverter/CoordinateEntryLogic calc_utils_provider; LayoutConfig no-arg) that radec_entry dropped for module-level calc_utils/time. Patch calc_utils/time at the module level and pass a display stub to LayoutConfig so the suite matches the implementation. 6 stale tests were failing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXDxJkU5CzAegv3FcMEwEq
Surface the ten shielded optical-black rows the IMX290/462 sensor already
transmits and use their trimmed mean as the complete per-frame pedestal for
radiometric SQM. This makes IMX462 SQM zero-touch: no lens cap, dark frames,
or calibration wizard from the user.
Kernel: a second sensor source pad (MEDIA_BUS_FMT_SENSOR_DATA) routes the OB
rows to the receiver as a separate metadata stream without altering the
1920x1080 image matrix.
libcamera: the imx290 cam-helper unpacks the shielded pixels, trims the outer
5% each side and averages the central 90% (preserving sub-ADU resolution on
the 16-bit SensorBlackLevels scale), publishing {l,l,l,l+1}. The +1 marker
lets Python tell a measured value from a static tuning tuple.
Python: camera_pi reads SensorBlackLevels for imx290/imx462 and passes the
native-ADU value as the radiometer sample's optical_black_pedestal. A valid OB
value is the complete pedestal; calibrated then profile pedestals remain the
fallback when OB is missing or unmarked.
Validated on mr2 (IMX462, gain 30) by a same-frame cupboard test: active-green
vs OB dark-accumulation slopes agree to well under 1 ADU/s, so OB is a valid
complete pedestal. Kernel is built via nixos-hardware; pifinder-fast/
pifinder-kernel-cross give a fast x86_64 cross build to seed the binary cache
so CI substitutes the kernel instead of compiling it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015EWb2JET2JE1t4ya3HZPNL
Add the OB pedestal to the SQM architecture note and glossary, and record the decision in ADR-0023. The justification is the same-frame cupboard test on mr2's IMX462 (active-green vs OB slopes agree to <1 ADU/s), scoped to IMX462; IMX296/IMX477 OB remain open. Defines 'optical black' as a glossary term and sets the pedestal precedence (valid OB > calibrated > profile bias). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015EWb2JET2JE1t4ya3HZPNL
python/.venv, mypy/pytest/ruff caches and __pycache__ directories are not needed at runtime and inflated the source output to ~935 MiB, making Attic cache pushes absurdly slow. Removing them drops it to ~19 MiB (cache push ~10-15 s, device upgrade fetch ~30-43 MiB). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015EWb2JET2JE1t4ya3HZPNL
Bring the per-frame optical-black pedestal (IMX290/462), its docs/ADR, and the runtime source-closure slimming onto deepchart. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015EWb2JET2JE1t4ya3HZPNL
…stal Gate BlackLevelTracker creation behind SQM_BLACK_LEVEL_TRACKER_ENABLED (default False) so the radiometer pedestal is OB when present, else the static profile bias_offset — no tracker in between. Lets us observe the optical-black pedestal's effect in isolation during rollout/validation. Reversible: flip the flag (or revert) to restore the tracker fallback. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015EWb2JET2JE1t4ya3HZPNL
All aperture/annulus/wing radii are defined for the ~1.0-scale Bayer-green photometry images (imx462 green: 490px vs the 512 solve image). The mono imx296 does photometry on its full-res 1088px frame (scale 2.125), where the fixed radii sit inside the star profile: the r=5 aperture holds only 83-94% of a star's flux (measured growth curves, 2026-07-18 sweeps), every local background annulus lands on the PSF itself, and the wing estimator's sky ring eats the wings so it measures f~1 and stays permanently inert. Convert all radii from solve-image pixels to the photometry image's pitch (_scaled_photometry_radii) and give WingEstimator a set_scale() that rescales its patch geometry and clears the rolling window on change. Both the production path (solver.update_sqm) and the calibration wizard's SQM preview use the shared helper. Effect by sensor: - imx462/imx290: radii round to (5, 10, 17) - essentially unchanged. - imx296: radii become (11, 21, 38); on the 2026-07-18 reference sweeps this moves the median error vs a hand-held meter from -0.20 to +0.00 and lets the estimator track real focus drift. - hq: also gets larger radii; its sqm_band_offset (like imx296's) should be refit against reference sweeps after this change. Validated: 121 tests pass across test_sqm, test_solver_sqm, test_sqm_calibration, test_sweep_frame_record, test_solver_shmem (9 new tests cover the scaling).
The sensor's OB clamp pins raw black to a target that moves with sensor state: on the 2026-07-18 imx296 reference sweeps the delivered level was 55.9-56.3 ADU while the device's wizard calibration (bias 58) and the profile constant (60) both over-subtracted. At dark-site signal levels a 2-4 ADU over-subtraction produces a strong SQM-vs-exposure slope (-0.9..-1.5 mag/decade) and kills short exposures outright (background - pedestal <= 0, 'background unresolved'). The BlackLevelTracker already measures the in-session intercept but was suppressed whenever a wizard calibration existed - precisely backwards: a stored bias is one moment's measurement of a moving level. Flip the priority in both SQM paths (radiometric and stellar): a confidently fitted tracked intercept wins for bias; the wizard's dark-current rate remains authoritative and is added on top (the intercept fit cannot separate dark from sky). The tracker's own stderr / deviation-band / lease gates bound how far it can stray, and an accepted fit >2 ADU from the static bias now logs at info. Replayed against the 2026-07-18 imx296 sweeps (radiometric feed, wizard anchor 58): tracker conditions on all four (stderr 0.26-0.32), fits 55.9-56.3, and the recomputed SQM exposure slopes collapse from -0.9..-1.5 to -0.01..+0.52 with the residual offset a tight constant (sd 0.04) for a later band-offset refit. Also: the published black_level_tracked flag now reflects the leased pedestal actually used, not the raw last fit. Validated: 136 tests pass across the SQM suites (4 updated/new).
Rich's 2026-07-18 calibration reports (imx296 + imx462, wizard run interleaved with the reference sweeps) show two capture defects: 1. The first ~3 frames of every stage report actual_exposure_us at the PREVIOUS exposure (req=1us, actual=999999us) yet pass the frame gate, because _capture_and_wait checks the exposure_time field, which echoes the committed setting rather than what the sensor delivered. Stale full-length frames land in the bias/dark stacks. 2. After the exposure change lands, the sensor's optical-black clamp re-settles to a new level over ~3 more frames (imx296: 56 -> 59 ADU). Frames in that window measure neither the old nor the new black level. (On the imx296 the settled clamp is genuinely exposure-dependent: ~56 ADU at 1s vs ~59 ADU at the 29us minimum - a min-exposure bias is only valid for its own regime, which is why the tracked in-session pedestal takes priority in operation.) Together these explain the wizard bias of 58 on a sensor whose 1s clamp level is 56: a median over stale-1s frames at 56, settling frames, and settled-29us frames at 59. Fix: _capture_and_wait gates on actual_exposure_us (exposure_time only as fallback), re-requests a capture when a stale frame arrives, and uses an exposure-proportional tolerance; bias and dark stages discard CLAMP_SETTLE_DISCARD_FRAMES frames after every exposure change (the dark ramp changes exposure per step). Adds ~30-40s to a wizard run. Evidence: dumps/logs/rich-imx296/sqm_calibration_report_imx296_*.json. Validated: 141 tests pass across the SQM suites (5 new).
feat(sqm): per-frame optical-black pedestal for IMX290/IMX462 (zero-touch)
Telemetry menu under Settings > Telemetry: - Record: inline On/Off toggle (no submenu), same pattern as Test Mode. - Sections: one multi-select checklist with inline checkmarks (like the Catalogs filter) for what to record — IMU, SQM, Solves, Targets and Images. Toggling applies live to an in-progress recording. - Max Size: session cap (250 MB / 500 MB / 1 GB / 2 GB / Unlimited). - Load: unchanged. Images ships OFF by default, preserving the old telemetry_images=false: one 512x512 PNG is written per solve, so at ~1 solve/s it costs roughly 300 MB/hour against a few MB/hour for every other section combined. Session size cap. Frames are written by the camera process, so the recorder measures the session directory from its flush loop rather than counting its own writes. On reaching the cap, frame capture is suspended (the one unbounded consumer) while the event log keeps running, so a capped session stays useful instead of going dark. Default 1 GB. Also logs the SQM *ingredients*, not just the end products, so a session stays recomputable if the SQM/airglow maths changes in code later: - radio event gains the raw per-frame inputs — red/blue Bayer backgrounds, optical-black pedestal, photometry image size (green bg and exposure were already there). - the session header snapshots the FULL camera profile constants plus the airglow calibration and camera_type — the constants those ingredients were produced under. The derived published 'sqm' and applied 'floor' are still logged, but explicitly as a what-the-device-showed audit trail, not ground truth. The airglow import is optional: the model is part of the SQM stack only on branches that carry it (deepchart has the radiometer but not airglow), and there a session simply records no airglow constants.
Telemetry menu under Settings > Telemetry: - Record: inline On/Off toggle (no submenu), same pattern as Test Mode. - Sections: one multi-select checklist with inline checkmarks (like the Catalogs filter) for what to record — IMU, SQM, Solves, Targets and Images. Toggling applies live to an in-progress recording. - Max Size: session cap (250 MB / 500 MB / 1 GB / 2 GB / Unlimited). - Load: unchanged. Images ships OFF by default, preserving the old telemetry_images=false: one 512x512 PNG is written per solve, so at ~1 solve/s it costs roughly 300 MB/hour against a few MB/hour for every other section combined. Session size cap. Frames are written by the camera process, so the recorder measures the session directory from its flush loop rather than counting its own writes. On reaching the cap, frame capture is suspended (the one unbounded consumer) while the event log keeps running, so a capped session stays useful instead of going dark. Default 1 GB. Also logs the SQM *ingredients*, not just the end products, so a session stays recomputable if the SQM/airglow maths changes in code later: - radio event gains the raw per-frame inputs — red/blue Bayer backgrounds, optical-black pedestal, photometry image size (green bg and exposure were already there). - the session header snapshots the FULL camera profile constants plus the airglow calibration and camera_type — the constants those ingredients were produced under. The derived published 'sqm' and applied 'floor' are still logged, but explicitly as a what-the-device-showed audit trail, not ground truth. The airglow import is optional: the model is part of the SQM stack only on branches that carry it (deepchart has the radiometer but not airglow), and there a session simply records no airglow constants.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add a gaia achart view to all objects, down to mag 17 but with an auto limiting magnitude taking into account SQM and equipment. This auto limit is overridable.
TODO: