feat: Add historical time slider with live/historical mode switching and Grafana sync - #61
Merged
Conversation
cchwala
changed the title
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #61 +/- ##
==========================================
- Coverage 86.10% 85.39% -0.71%
==========================================
Files 39 41 +2
Lines 3418 3678 +260
==========================================
+ Hits 2943 3141 +198
- Misses 475 537 +62
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
cchwala
force-pushed
the
feat/time-slider
branch
from
July 25, 2026 17:56
3d85638 to
1748b4b
Compare
Add new hypertable to store hourly snapshots of CML statistics, enabling historical queries without scanning raw data. Changes: - Create cml_stats_history table with compression policy (7 days) - Add materialize_cml_stats_snapshot() function to write definitive snapshots from cml_data_1h aggregates - Add get_cml_stats_at() function for efficient historical lookups - Grant template comments for per-user isolation via generate_config.py Key features: - Idempotent upserts via ON CONFLICT (snapshot_time, cml_id, user_id) - Supports provisional (current partial hour) and definitive rows - Hour-truncated snapshot times for consistent querying - Compression enabled for snapshots older than 7 days
Implement background jobs to populate cml_stats_history with hourly snapshots, enabling fast historical queries without raw data scans. Changes: - Add DBWriter.write_stats_snapshot(at_time) - materializes definitive snapshot from cml_data_1h for completed hours - Add DBWriter.write_provisional_snapshot() - copies live cml_stats every 60s for current partial hour (is_provisional=TRUE) - Wire both methods into stats background thread in sftp_push.py - Add backfill_stats_history.py script for populating historical data Architecture: - Provisional snapshots: written every 60s, at most 60s stale - Definitive snapshots: written when hour turns, sourced from aggregates - ON CONFLICT DO UPDATE ensures idempotent re-materialization - Backfill script processes all available hours from cml_data_1h
Extend API to support historical queries for the time slider feature. Changes: - Extend /api/cml-stats with optional ?at= parameter for historical lookups - Parses ISO 8601 timestamp, queries get_cml_stats_at() function - Returns identical response shape with added is_provisional flag - Invalid timestamps return HTTP 400 - Add /api/cml-stats-time-range endpoint - Returns min(snapshot_time) and max (current hour) from cml_stats_history - Enables frontend to configure slider min/max bounds - Update generate_config.py to grant permissions for new table/function Key properties: - Live mode unchanged when ?at= omitted - Historical reads use index lookups, no raw data scans - Response includes is_provisional for current partial hour
Implement interactive time slider for historical data exploration with Grafana dashboard synchronization. Changes: - Add time slider section to Leaflet ColorControl widget - Range input with 1-hour steps, enabled after loading time range - Live button toggles between live/history modes - Visual display shows selected time or 'Live' - Debounced input (300ms) to avoid excessive API calls - Implement fetchAndApplyStats(atIso) function - Fetches stats with optional ?at= parameter - Displays '(partial)' label for provisional snapshots - Updates map path colors based on historical stats - Add syncGrafanaTime(epochSec) function - Syncs iframe time range via contentWindow.location - Live mode: relative 'now-1h' to 'now' - Historical mode: ±30 min window around selection - Sets var-marker_time variable for annotation pin - Update Grafana dashboard cml-realtime.json - Add hidden marker_time template variable (textbox, epoch ms) - Add 'Selected time' annotation pinned to marker_time - Auto-refresh in live mode (30s), paused when viewing history User experience: - Slider drag shows preview timestamp immediately - After 300ms debounce, fetches historical stats and syncs Grafana - Clicking 'Live' button restores auto-refresh and current time - Partial hours clearly labeled to indicate incomplete data
Test coverage: - parser/tests/test_stats_snapshot.py: Tests for DBWriter snapshot methods - write_stats_snapshot() materialization from aggregates - write_provisional_snapshot() copying live stats - Error handling and rollback behavior - Timezone handling for naive datetimes - webserver/tests/test_time_slider_api.py: API endpoint tests - Historical queries with ?at= parameter - Invalid timestamp handling (HTTP 400) - Time range endpoint responses - Empty history scenarios Documentation: - docs/plan-time-slider-2026-07-24.md: Complete implementation plan - Architecture overview and data flow - Database schema and functions - Parser background job design - Webserver API extensions - Frontend UI components - Grafana dashboard integration - Testing strategy and deployment notes
Upstream/main only has migrations up to 009. The deploy fork's ifu/main branch contains additional migrations 010-016 that are either deployment-specific (not intended for upstream) or generic performance improvements not yet submitted as upstream PRs. Neither set has SQL dependencies on cml_stats_history, so the correct next number for this feature in the upstream sequence is 010. Updated all migration number references in docs and backfill script.
perf/db-improvements-batch now claims migration numbers 010-013 upstream, so cml_stats_history must move to 014 to avoid a collision once both PRs are merged.
cchwala
force-pushed
the
feat/time-slider
branch
from
July 25, 2026 21:08
9ae331f to
3de23a8
Compare
…bugs migration 014 adds record_count to the cml_data_1h continuous aggregate, a column materialize_cml_stats_snapshot() (015, renumbered from 014) has always depended on but which never existed anywhere (neither upstream nor in the original plan's schema). While testing against a throwaway DB, found and fixed two further bugs in materialize_cml_stats_snapshot() that only surfaced once record_count existed: - the 1h LATERAL subquery never computed rsl_count (valid-record count), unlike the 6h subquery - the outer query wrapped the 6h LATERAL subquery's already-aggregated columns in a second layer of SUM/AVG with no GROUP BY, which is invalid SQL (the LATERAL join already produces one aggregated row per cml_id) Verified end-to-end against a throwaway container: inserted sample cml_data, refreshed cml_data_1h, called materialize_cml_stats_snapshot() and get_cml_stats_at(), confirmed correct completeness percentages and record counts.
A fresh DB (docker compose up with an empty volume) only runs init.sql, never the migrations/ files. Folds in migrations 014 (record_count on cml_data_1h) and 015 (cml_stats_history table + materialize_cml_stats_snapshot/get_cml_stats_at) so a fresh install matches a fully-migrated one and the time slider feature works out of the box. Verified by initializing a throwaway container from a clean volume, inserting sample cml_data, and confirming materialize_cml_stats_snapshot() and get_cml_stats_at() return correct results.
…ml-stats The invalid-timestamp check ran inside user_db_scope(), so any DB/user error (e.g. Unknown user_id) raised first would return 200 with a fallback instead of the intended 400. Validate 'at' before touching the DB so a malformed timestamp always yields 400 regardless of DB state. Fixes CI failure in test_api_cml_stats_invalid_at_parameter.
- database/init.sql: Apply Row Level Security properly in get_cml_stats_at() by filtering cml_metadata on user_id before joining. This ensures users only see their own CML data in historical queries. - parser/db_writer.py: Add explicit refresh_continuous_aggregate() call for cml_data_1h before materializing snapshots. This ensures the 1h stats subquery finds current data when building hourly snapshots.
- webserver/main.py:
- Unify column ordering between live and historical stats queries
- Add mean_rsl_1h column to live query (previously missing)
- Standardize response shape with consistent is_provisional field
- Add /api/coloring-annotation endpoint (POST/DELETE) for managing
Grafana region annotations that show the 1h coloring window
- Add /api/coloring-annotation-cleanup endpoint for removing stale
annotations from previous sessions
These annotation endpoints use admin credentials internally so Viewer-role
users can drive annotations. Rapid slider dragging is handled gracefully
with AbortController-style cleanup of orphaned annotations.
- webserver/templates/realtime.html:
- Change default coloring option from 'completeness' to 'std_dev'
(more relevant for real-time RSL stability monitoring)
- Improve slider event handling: 'input' event for live updates during
dragging, 'change' event to commit selection and stop live refresh
- Add AbortController to cancel in-flight requests during rapid slider
movement, preventing race conditions
- Enhance syncGrafanaTime(): preserve user's zoom level by reading
current from/to params, use replaceState+popstate for seamless in-place
updates instead of full iframe reloads
- Add updateColoringAnnotation(): creates gray shaded region annotation
showing the 6h coloring window used for map coloring
- Add cleanupColoringAnnotations(): removes stale annotations on page load
- grafana/provisio- grafana/provisio- grafana/provisio- grafana/provisio- grafana/provisio- grafana/provisio- grafana/provisio- grafana/provisio- grafana/provisio- grafana/provisio- grafana/provisio- grles- grafana/provisio- grafana/pret defa- grafana/provisio- grafana/provisio- grafana/provisio- grafanze marker_time variable to '0' as placeholder
Changed the default halfWindow from ±30 min (1h total) to ±3h (6h total) in syncGrafanaTime(). This provides better temporal context when the user first moves the time slider, while still preserving manual zoom adjustments for subsequent interactions.
…-red progression - Changed from green-yellow-orange-red to blue-yellow-orange-red - Fixed thresholds: ≤2 blue, 2-4 yellow, 4-6 orange, 6-8 deep orange, 8+ red - Colorblind-friendly: removes green-red confusion while maintaining intuitive meaning - Updated missing data color to slate gray (#94a3b8) for consistency
…al data
- webserver/templates/realtime.html: Reset time slider to max position when
clicking Live button, ensuring visual feedback matches live mode state
- parser/db_writer.py: Add CALL refresh_continuous_aggregate('cml_stats', ...)
in write_provisional_snapshot() to ensure the 6-hour rolling view is
refreshed every 60 seconds alongside provisional snapshot writes
This ensures both live view and historical slider have access to fresh stats
data, fixing grey colors for recent hours.
When clicking Live, manually dispatch input event after resetting slider position to ensure map colors and Grafana zoom update immediately.
Ensure map colors are refreshed when clicking Live button by calling updateMapColors() in the promise chain after fetchAndApplyStats().
- Fixed JavaScript ReferenceError (sliderDebounce, sliderFetchCtrl not defined) - Removed premature updateMapColors() call before stats fetch completed - Added debug console.log statements for troubleshooting - Added no-cache headers to /realtime endpoint to prevent browser caching - Live button now correctly: * Fetches live stats and applies them to map lines via applyStatsToLine() * Resets Grafana to 'now-1h/now' mode with full reload * Removes coloring annotation * Starts auto-refresh timer
…napshot() cml_stats is a regular table updated by update_cml_stats(), not a continuous aggregate. Attempting to refresh it caused 'relation is not a continuous aggregate' errors, preventing provisional snapshots from being written every 60 seconds. This fix allows the parser to continuously populate cml_stats_history with current-hour data, eliminating the 3-4 hour gap of grey/NULL values on the map.
…ng annotation API tests - Fixed test_write_stats_snapshot_success to expect 2 execute() calls (CAGG refresh + materialize) - Fixed test_write_stats_snapshot_rollback_on_error to only fail on materialize step - Added test_write_provisional_snapshot_does_not_refresh_cml_stats to verify no invalid CAGG refresh - Added new test_coloring_annotation_api.py with 5 tests for /api/coloring-annotation endpoints
cchwala
changed the title
- test_api_cml_stats: Updated mock data to 11 columns (added mean_rsl_1h, is_provisional) - test_create_annotation_success: Use call_args.kwargs instead of deprecated indexing - test_cleanup_removes_all_tagged_annotations: Fix params assertion to use .get() method
…leans up duplicates first)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements a fully functional historical time slider for the GMDI realtime dashboard, allowing users to explore past data quality states with synchronized map coloring and Grafana chart updates.
Key Features
Time Slider (
/realtime)Grafana Integration
replaceState + popstatefor in-place time range updates (no full reload)now-1h)/api/coloring-annotationendpoint creates stored Grafana annotations server-side (bypasses Viewer role restrictions)Data Quality Improvements
Bug Fixes
updateMapColors()call before async stats fetch completedreplaceStatetolocation.replace()for proper relative time resetwrite_provisional_snapshot()attempting to refreshcml_statsas CAGG (it's a regular table)sliderDebounceandsliderFetchCtrldeclarations/realtimeendpointTechnical Changes
Backend (
webserver/main.py)/api/coloring-annotationPOST/DELETE endpoint (creates Grafana annotations with admin credentials)/api/coloring-annotation-cleanupendpoint (removes stale annotations)/realtimerouteFrontend (
webserver/templates/realtime.html)AbortControllerfor canceling in-flight slider requestsstd_devParser (
parser/db_writer.py)refresh_continuous_aggregate('cml_stats', ...)call fromwrite_provisional_snapshot()cml_statsis a regular table updated byupdate_cml_stats(), not a continuous aggregateDatabase (
database/init.sql)materialize_cml_stats_snapshot()usesNULLIF(val, 'NaN'::float)to prevent NaN poisoning aggregatesTesting
New Tests
webserver/tests/test_coloring_annotation_api.py(5 tests)parser/tests/test_stats_snapshot.py(updated + 1 new test)test_write_provisional_snapshot_does_not_refresh_cml_statsTest Results
99 passed, 0 failed
Deployment Notes
materialize_cml_stats_snapshot()function should be updated with NaN filtering (already ininit.sql)db_writer.pyfix for provisional snapshotsScreenshots
With time slider at an archive time stamp
In "live" mode showing most recent data
