NixOS support#379
Open
mrosseel wants to merge 396 commits into
Open
Conversation
mrosseel
added
testable
Merges upstream PiFinder changes including: - Bottle → Flask/Jinja webserver migration (.tpl → .html) - Harris and Lynga catalog loaders + data + DB update - Selenium-based webserver test suite NixOS-specific decisions: - Keep deletion of requirements.txt, requirements_dev.txt, noxfile.py, version.txt (managed via flake.nix / pifinder-build.json) - Keep Python 3.13 target (was reset to py39 by upstream) - Keep stub sys_utils_fake (uses sys_utils_base) - Replace bottle/cheroot with flask/flask-babel/waitress in nixos/pkgs/python-packages.nix; add selenium to devPackages - Reimplement Bottle-era network status message in Jinja syntax
Includes nixos/pkgs/pifinder-src.nix fix: drop fetchurl for hip_main.dat, which upstream now ships as a committed file in astro_data/.
- nixos/RELEASE.md: document version flow + release/dev pipelines - software.py: MIN_NIXOS_VERSION 2.5.0 → 3.0.0 - python-packages.nix: add pyerfa (used by calc_utils since upstream brickbots#423, silently dropped during upstream merge because requirements.txt is not mirrored into the Nix env) - python-packages.nix: include hardwarePackages in devEnv so nix develop matches the runtime import surface - python-packages.nix: select simplejpeg wheel by host arch (was hard-pinned to aarch64; failed to import on x86_64 dev shells) - flake.nix: apply libcamera -Dpycamera=enabled overlay to the x86_64 devShell and export PYTHONPATH so picamera2 finds the python bindings Verified: nix develop --command python -c 'import …' on x86_64 succeeds for all 34 imports (erfa, picamera2, libcamera, PyHotKey, pynput, hardware packages, etc.). RPi.GPIO still raises its own "only on a Raspberry Pi" RuntimeError at import time — expected, matches upstream pip behavior on non-Pi hardware. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1. gpsd-add-uart: rename /dev/ttyAMA3 → /dev/ttyAMA1 (6 sites). The uart3 overlay surfaces as ttyAMA1, matching hardware.nix's udev rule and the Debian image's gpsd.conf. 2. /etc/default/gpsd: drop the custom USBAUTO+GPSD_SOCKET pair, write upstream pi_config_files/gpsd.conf's three lines verbatim. DEVICES now opens the on-board UART at startup. gpsd-add-uart kept as the boot-time socket-activation kick; can retire after on-Pi confirmation. 3. pifinder-upgrade: replace fragile `nix build --dry-run | grep` progress with `nix --log-format internal-json build … --max-jobs 0` parsed by gawk, counting type=100 (actCopyPath) start/stop events. Stable across Nix ≥ 2.4. Validated against a real cache.nixos.org substitute (5/5). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
build.yml was defaulting VERSION=2.5.0 on push triggers (and the workflow_dispatch default also read 2.5.0), so this branch's auto-build was publishing v2.5.0-migration tarballs while the migration branch's downloader (software.py _MIGRATION_VERSION_INFO and the brickbots/PiFinder release branch's migration_gate.json) points at v3.0.0-migration. Bump both the workflow_dispatch default and the push-trigger fallback to 3.0.0 so a normal push to nixos publishes the artifact at the URL the migration branch actually downloads. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Lint & Test workflow was using DeterminateSystems/magic-nix-cache-action,
which is backed by GitHub's Actions Cache and gets HTTP-418 rate-limited
under sustained traffic — exactly the failure mode that just broke
type-check ("--install-types failed: substituter disabled, rate limit
exceeded"). The Nix substituter is then disabled mid-run and dependent
commands like mypy --install-types fall over.
Replace it with cachix/cachix-action@v17 pointed at the pifinder cache
(read-only, no auth token needed). Same backing as build.yml, so dev-shell
substitutes hit the same store paths the system closure was built against.
cache.nixos.org remains the default fallback.
Also bump actions/checkout@v4 → @v6 in this file to align with the Node 24
migration in build.yml/release.yml.
This is a stop-gap. The real fix is standing up Attic with an S3 backend
so both build.yml and lint.yml can retire cachix.org and MNC together —
tracked separately.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…rong) Previous commit on this branch swapped DeterminateSystems/magic-nix-cache-action for cachix/cachix-action@v17 thinking the MNC HTTP-418 rate-limit was the root cause of the failed lint/type-check. That swap made things worse: the pifinder Cachix only contains the NixOS *system closure*, not the *dev shell* (cedar-detect-server's Rust crate builds). With MNC removed, the dev shell had to rebuild from source, which fetched crate tarballs from a crates.io mirror and hit 403s. MNC was carrying real weight by caching locally-built derivations between runs. Restoring it. The original MNC rate-limit was a transient flake — re-runs work around it. Real fix is standing up Attic with S3-backed storage so both build.yml and lint.yml can retire MNC and cachix.org together. The checkout@v4 → @v6 bump from the swap commit is preserved. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Nix derivation was overwriting pifinder-build.json with
"nix-${gitRev}" at build time, so even released devices reported a
random short-sha instead of the release version. Three writers became
two, with consistent semantics everywhere:
- pifinder-src.nix: drop the cat > pifinder-build.json block and the
gitRev arg — the derivation now copies the source file through
verbatim, no version invention.
- flake.nix: drop the pifinderGitRev _module.args plumbing.
- services.nix: drop pifinderGitRev / gitRev from the pifinder-src
import.
- release.yml: reorder so the version stamp is written into the
working tree BEFORE the nix build (so the store path bakes in the
release version, not the previous stamp), then re-stamp with the
resulting store_path after the build, commit, push, tag.
Result: SD image, cachix closure, and committed JSON all agree on
the released version. Matches the flow already documented in
nixos/RELEASE.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Captures the decision to self-host Attic at cache.pifinder.eu, backed by SQLite + local disk initially with Cloudflare R2 as the eventual chunk store. Covers considered alternatives (cachix.org, Magic Nix Cache, nix-casync, harmonia) and the operational consequences for CI publishing, on-device updates, and failure fall-through to cache.nixos.org. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces pifinder.cachix.org with the self-hosted Attic instance at https://cache.pifinder.eu/pifinder (ADR 0004) on both the device-side substituter list and the CI publish path. cachix.org is removed entirely with no fallback, so this is the first build that proves Attic stands on its own. services.nix: substituters = ["https://cache.pifinder.eu/pifinder" "https://cache.nixos.org"] trusted-public-keys = ["pifinder:8UU/O3oLkaJHHUyqEcPGl+9F1m4MqDca39Ewl49jBmE=" "cache.nixos.org-1:..."] (pifinder.cachix.org and its key removed.) build.yml — build-native, build-emulated, build-migration-tarball: - remove cachix/cachix-action steps - remove `cachix push` (replaced by `attic push pifinder:pifinder`) - add a "Setup Attic substituter" step that runs nix profile install nixpkgs#attic-client attic login pifinder https://cache.pifinder.eu \"\$ATTIC_TOKEN\" attic use pifinder:pifinder before the build, so the build itself substitutes from cache.pifinder.eu. build-emulated swaps cachix/install-nix-action for DeterminateSystems/nix-installer-action — no cachix dependencies left. First post-cachix build is expected to be slow: pifinder.cachix.org's warmed-up paths are gone. Once it completes and `attic push` lands, subsequent builds substitute from cache.pifinder.eu. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ection; centroid-excluded annuli Three defects found live on-sky (mr2, 2026-07-17), each verified against the raw frames before and after: 1. ROTATION (the 3-magnitude bug): the camera process rotates the solve/ display image (90 or 270 deg CCW depending on screen_direction / camera_rotation) but stores the raw frame unrotated. The raw-photometry SQM path mapped solve-image centroids straight onto the raw, so every star aperture landed on the wrong sky and SQM read ~15-16 under an ~18 sky on every device with a rotated display (which is all of them). The camera now publishes its rotation (SharedState.solve_image_rotation) and update_sqm counter-rotates matched + detected centroids onto the raw grid (_derotate_centroids, validated against PIL rotation for 0/90/180/ 270 and arbitrary angles). Measured live: matched-star flux recovered ~4x; SQM 15.4 -> 17.6 agreeing with the offline pipeline to 0.05. 2. Wing correction was an artifact: the stacked curve of growth of 21 bright live stars shows the imx462 PSF fully enclosed by r~4 (FWHM 2.6px) -- no wings. The per-star boundary search integrated noise and reported f~0.72-0.75, a spurious +0.3-0.5 mag correction whose variance polluted the calibration. WingEstimator now median-stacks aperture-normalized bright stars per frame and reads the enclosed fraction off the stack's growth-curve plateau: wingless optics measure f=1.0 (correction 0); hq's mild focus-dependent wings (f 0.87-1.0) are measured honestly. min_wing_radius and CameraProfile.wing_min_radius_px are gone. 3. Background annuli in dense fields: every *detected* centroid (cedar finds them all; the parameter was previously unused) is now masked out of each star's background annulus, with a 3-sigma clip as backstop for undetected sources. A neighbour star can no longer inflate the local sky estimate. Band offsets recalibrated against all referenced clear-night sweeps with this pipeline: imx462/imx290 0.64 (residuals +/-0.06 over 6 sweeps), hq 0.63 (3 independent references). Cloudy-night sweeps excluded and documented: star-calibrated SQM under bright clouds reads ~0.7-0.9 low vs a radiometric meter because the zero point is measured through cloud extinction while the glow largely is not. Analysis artifacts: myPiFinder/support/dumps/analysis/20260716.
The mono sweep (2025-10-31) rerun through the final pipeline (growth-curve aperture correction: f=0.985, wingless like its siblings; centroid-excluded annuli) computes 17.45 against the recorded 17.8-17.9 hand-held reference. Refit: -0.46 -> -0.06. The +0.40 shift matches the phantom-wing retirement on imx462/hq (+0.33/+0.31), and a near-zero offset is physically consistent with the Pregius mono passband being the closest of the three sensors to the meter's. Single moonlit reference night: confidence +/-0.2; re-anchor with Session Correct or a reference sweep when this camera flies again.
…mp logging The titlebar SQM runs under PID auto-exposure (10 ms - 1 s), and the sweep ramps showed the frame zero point drifting ~-0.2 mag/decade of exposure. Per-star tracking across the ramps (860 star tracks) exonerates the sensor: response is linear to ±1% below 0.2 full-scale, no compression. The drift is population composition: matched-star sets slide ~4 magnitudes across the exposure range, and per-star zero points carry a mild magnitude dependence, so every exposure-dependent selection (all-star median, top-K, brighter half) inherits drift. Three changes: - mzero is now the median over stars in a fixed catalog-magnitude band (3.5-6.5, fallback to all stars below 5 in band): the same physical stars vote at every exposure. Measured drift on the imx462 ramps: -0.10 -> -0.01 mag/decade. hq keeps a ~-0.15 residual consistent with its real focus-dependent wings (tracked per session by the growth estimator). - Background annulus moved 6-14 -> 10-18, outside the PSF-wing zone that inflated bright stars' local sky at long exposure (~35% of the drift); safe against neighbours now that all detected centroids are masked out of annuli. - Sensor die temperature (picamera2 SensorTemperature) recorded into last_image_metadata and sweep metadata, diagnostic only: the residual black level wanders ±2 ADU night-to-night and temperature is the prime suspect; normal sweeps now collect the correlation data. Band offsets refit against all referenced clear-night sweeps with this pipeline (they absorb the annulus/selection level shifts): imx462/imx290 0.61, hq 0.60, imx296 -0.10. Clear-night residuals: imx462 ±0.08 (6 refs), hq -0.16..+0.18 (3 refs, the shared 2025-11-16 reading remains the outlier). Cloud bias unchanged (-0.4..-0.6, physics).
…tion Publish SQM as a fixed-calibration radiometer (factory zero point x measured sky flux) at 1 Hz from every camera frame, no plate solve required. Stellar photometry becomes a 10-second transmission diagnostic that establishes, maintains and audits the calibration: - sqm/radiometer.py: camera-side sparse-central-median sample; 12-sample/15 s accumulator; factory radiometric zero point + FOV per sensor profile. - sqm/clouds.py: cloud/dew discrimination. Cloud dims stars AND brightens a light-polluted sky (radiometer immune, no correction); dew/optics dim stars and sky together (stellar deficit compensates the radiometer once a clear session baseline exists). Asymmetric median baselines, factory-seeded. - sqm/black_level.py: sensor pedestal tracked from the sky-vs-exposure intercept of ordinary frames (the true black level wanders +/-2 ADU night to night); fed at 1 Hz, gated on transmission stability, zero-touch only. - sqm/gaia_ref.py + astro_data/hip_gaia_g.npz: Gaia G reference band with BP-RP trim for bare sensors (24-29% less star scatter than Hipparcos V). - Robust stellar zero point: fixed catalog-magnitude band + 3-MAD rejection; centroid-excluded background annuli; growth-curve aperture correction. - Calibration wizard rework: temporal read noise, multi-exposure dark fit, exact frame/solve pairing, typed reload command, transient exposures. Archive replay (16 sweeps, 800 frames): imx462 MAE 0.05 mag over six SQM-L-referenced clear nights; bright-cloud errors 0.35-0.84 collapse to under 0.10 with zero corrections on clear skies; 800/800 frames published including 318 without usable stars.
…, cloud/dew guard)
Each sweep image pair now gets img_NNN_XXms_metadata.json carrying the driver-applied exposure/gain, sensor die temperature, sensor black levels, Lux, colour gains, and raw-frame ADU statistics (mean/median/std/percentiles, saturated fraction). frame_metadata.json aggregates the same records. Also gate the sweep UI COMPLETE transition on sweep_metadata.json existing: the camera process writes it a few seconds after the last image, and enriching before that silently skipped the SQM/solve/noise-floor details on every sweep.
Each tracker (BlackLevelTracker, CloudEstimator, WingEstimator, RadiometerAccumulator) gains dump(): config, derived stats, and the full window sample contents. The solver publishes these into sqm_details as window_black_level / window_clouds / window_wings / window_radiometer on every update, and each sweep image JSON snapshots the whole sqm_details, so a sweep records the pipeline state frame by frame.
…ot the old base activate_system compared the persisted camera-type against --default-camera, i.e. the RUNNING generation base camera. On a device whose build had that camera as its base (mr: imx477-base build), upgrading onto an imx462-base build concluded camera is the base, nothing to do and rebooted into the wrong DTB, killing the camera. Ask the new store path for a matching specialisation instead, and run the NEW build set-extlinux-default so the camera-to-entry mapping matches the entries just written.
…erization Plan B for the stuck-pedestal incident (2026-07-17): - BlackLevelTracker: an accepted pedestal is now a lease (max_age_seconds, default 900 s) instead of a latch — without a fresh accepting refit it expires and callers fall back to the profile constant, the same state every session boots in. Regression test replays the incident. - max_intercept_stderr tightened 1.0 -> 0.6 ADU: honest ramp fits on the 27-sweep archive run 0.10-0.69; the cloud-poisoned fit passed at 0.97. Calibration wizard (dark-frame path the imx296 tester will run): - persists a per-frame report JSON (median/MAD/percentiles, requested AND driver-reported exposure, sensor temp) beside the fitted scalars — the ramp shape is where short-exposure pedestal elevation shows, and three fitted numbers cannot preserve it - dark-current fit uses driver-reported exposures (the imx477 delivers intermittent half-exposure frames; a requested-exposure fit is silently corrupted) - dark ladder extended from one decade to near sensor minimum, covering the elevation zone (imx462: +2.9 ADU at 25 ms, 0 at ~300 ms) Sweep instrumentation: - per-image records now carry the full driver metadata dict (new sensors report different keys) and the live camera-side radiometer sample (the sqm_details snapshot freezes while the solver starves mid-sweep) - camera_pi keeps last_frame_metadata; both image_metadata sites publish actual_exposure_us
…weep instrumentation)
…mat stragglers mypy: the B-V cache globals are Optional, so assignments cannot narrow them across statements; parse into locals and use those for savez/len. format: 4 files the CI format session was silently reformatting on every run (timeentry, three tests).
Telemetry captured only IMU and solve streams — blind to the radiometric side. A third event type records the camera-side radiometer sample at ~1 Hz: t (capture epoch, s), exp (driver-reported exposure, s), bg (background median, ADU), mad (median absolute deviation, ADU), grad (quadrant gradient, ADU), seq (frame sequence). Dedupes on sequence; replay players skip unknown event types, so old builds read new files. A full-night radio stream answers what sweep snapshots cannot: how auto-exposure actually behaves over a session (lever-arm availability for the black-level fit), a continuous sky-steadiness timeline, and real-night input for replaying pedestal estimators.
The marking menu down slot is the one readable at the eyepiece without moving your head — it goes to SWEEP, the everyday reference-meter action. SQM Correct is deleted rather than kept: a magnitude-additive session knob silently absorbs ADU-space, brightness-dependent errors (tonight: a stuck pedestal), masking exactly the fault class it appears to fix, and staling whenever the layers beneath it change. Its legitimate output — the unit-vs-reference-meter difference — is already recorded as evidence by every sweep run with a reference reading. The feature never shipped upstream (fork-only), so now is the cheapest moment to remove it. Removed: UISQMCorrect screen, menu entry and corrected indicator, _apply_sqm_correct and both solver call sites, the sqm_correct_delta shared state, and all test references. CONTEXT.md records the decision.
python/result and python/result-lib are nix build output symlinks that were committed before .gitignore's result/result-* rule existed, so they stayed tracked and kept getting re-snapshotted into unrelated commits. Untrack them; the existing ignore rule keeps them out from now on. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXDxJkU5CzAegv3FcMEwEq
The NixOS migration UI flow (UIMigrationConfirm/UIMigrationProgress) was removed from ui/software.py in 'refactor(software): delete the NixOS migration UI flow', but a later sqm-fix merge reintroduced the test's import and spec builders for those deleted classes, breaking test collection. Drop the stale references. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXDxJkU5CzAegv3FcMEwEq
test_radec_entry exercised an older dependency-injection API (BlinkingCursor time_provider; CoordinateConverter/CoordinateEntryLogic calc_utils_provider; LayoutConfig no-arg) that radec_entry dropped for module-level calc_utils/time. Patch calc_utils/time at the module level and pass a display stub to LayoutConfig so the suite matches the implementation. 6 stale tests were failing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXDxJkU5CzAegv3FcMEwEq
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test plan
🤖 Generated with Claude Code