feat(backend): adopt climate_ref.results.Reader for metric values

[lewisjared]

Summary

Migrates the backend's two /values endpoints off hand-rolled SQLAlchemy queries and onto the new climate_ref.results.Reader read-layer facade — the consumer that read layer was explicitly designed for. This is the first, keystone slice of a phased adoption (plan tracked separately); it targets the biggest duplication: metric-value filtering, source-id-aware IQR outlier detection, facet collection, and the load-all-then-paginate rule now all live upstream.

• GET /executions/{group_id}/values and GET /diagnostics/{provider}/{diagnostic}/values now query app_context.reader.values.{scalar,series}_values(...) via MetricValueFilter + OutlierPolicy.
• New get_reader dependency on AppContext (additive; the raw session dependency is untouched and still used by the not-yet-migrated routes).
• New core/reader_values.py for CSV rendering from reader collections and CV-dimension query-param gating; MetricValueCollection.build_{scalar,series}_from_reader adapters in models.py.

Response shapes are unchanged. promoted_only=False and include_retracted=True are set explicitly to preserve the previous unfiltered behaviour rather than adopt the reader's stricter defaults — so this is behaviour-preserving, which the existing route tests confirm.

Dependency / environment changes (please review)

• Requires climate-ref >= 0.16.0 (>=0.16.0,<0.17). v0.16.0 is tagged and GitHub-released but is not yet on PyPI (PyPI currently tops out at 0.14.6 — 0.14.7/0.15.0/0.16.0 all have green Release runs but no published artifact). It is therefore sourced from the v0.16.0 git tag via [tool.uv.sources]; the version constraint is already correct, so that source line can simply be deleted once 0.16.0 lands on PyPI.
• Python floor moves 3.11 → 3.12, because climate-ref 0.16 requires >=3.12.
• The decimated test-fixture DB is migrated to the current schema (it was 4 alembic revisions behind; diagnostic.promoted_version was missing). This was done with climate-ref's own migrations — the same path a deployed 0.13-era database will take. Deployed REF databases must be alembic-migrated to head before a reader-based backend can read them.

Testing

• Full backend suite green: 185 passed, 1 xfailed (unchanged from the pre-migration baseline), against climate-ref v0.16.0.
• mypy (strict) and ruff clean.

Follow-ups (later phases)

• Migrate the executions/diagnostics list routes, datasets, and artifacts endpoints.
• Delete the now-dead core/outliers.py and the query half of core/metric_values.py once nothing references them.
• Harden the scalar adapter against null-valued scalars (float(item.value) would raise; none in the current fixture).

Summary by CodeRabbit

• Breaking Changes

  • The backend now requires Python 3.12 or newer.
  • Existing databases must be upgraded before use.

• New Features

  • Metric values are now served through a shared reading layer, improving consistency across API responses.
  • CSV exports for metric values continue to be available with the same column layout.

• Bug Fixes

  • Improved handling of filtering, pagination, and outlier detection for metric value results.

[netlify]

✅ Deploy Preview for climate-ref canceled.

Name	Link
🔨 Latest commit	e9a6ca3
🔍 Latest deploy log	https://app.netlify.com/projects/climate-ref/deploys/6a4e523773cf870008cfdb73

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR bumps the backend to require Python 3.12+ and climate-ref 0.16, pinned via a Git tag source. A new Reader dependency is wired into the app context, and the diagnostics/executions metric-values endpoints are refactored to use reader-based filtering, outlier detection, pagination, and CSV export instead of direct SQLAlchemy queries.

Changes

Reader layer migration

Layer / File(s)	Summary
Dependency pin and Python version bump backend/pyproject.toml	Raises requires-python to >=3.12, updates the climate-ref constraint to >=0.16.0,<0.17, and adds a Git source pinned to tag v0.16.0.
Reader dependency wiring backend/src/ref_backend/api/deps.py	Adds a Reader import, a _get_reader_dependency factory exposed as ReaderDep, extends AppContext with a reader field, and threads the injected reader into get_app_context.
Reader value filter, CSV export and model constructors backend/src/ref_backend/core/reader_values.py, backend/src/ref_backend/models.py	Adds parse_dimension_filters, generate_csv_response_scalar, and generate_csv_response_series helpers, plus MetricValueCollection.build_scalar_from_reader and build_series_from_reader static methods for mapping reader collections into API responses.
Diagnostics metric-values endpoint refactor backend/src/ref_backend/api/routes/diagnostics.py	Replaces manual SQL filtering, outlier processing, and pagination in list_metric_values with MetricValueFilter/OutlierPolicy and app_context.reader.values.* calls for CSV and JSON responses.
Executions metric-values endpoint refactor backend/src/ref_backend/api/routes/executions.py	Removes _EXECUTION_NON_FILTER_PARAMS and rewrites the /executions/{group_id}/values endpoint to use MetricValueFilter-based reader retrieval, reader-driven CSV export, and reader-based response construction.
Changelog updates changelog/39.breaking.md, changelog/39.trivial.md	Documents the breaking climate-ref 0.16/Python 3.12 requirement, the need for ref db migrate, and the internal switch to the shared read layer with unchanged API responses.

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant Route as diagnostics/executions route
  participant Filter as MetricValueFilter/OutlierPolicy
  participant Reader as app_context.reader.values
  participant Model as MetricValueCollection

  Client->>Route: GET .../values?query_params
  Route->>Filter: build filter (dimensions, isolate/exclude ids)
  Route->>Reader: scalar_values(filter, outlier_policy) / series_values(filter)
  Reader-->>Route: ScalarValueCollection / SeriesValueCollection
  alt format=csv
    Route->>Route: generate_csv_response_scalar/series(collection)
    Route-->>Client: StreamingResponse (CSV)
  else JSON
    Route->>Model: build_scalar_from_reader / build_series_from_reader(collection)
    Model-->>Route: MetricValueCollection
    Route-->>Client: MetricValueCollection (JSON)
  end

Loading

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	Clearly summarizes the main change: backend metric-value endpoints now use climate_ref.results.Reader.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/adopt-results-reader

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (2)

backend/src/ref_backend/api/routes/executions.py (1)

341-375: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Consider extracting the shared scalar/series/CSV branching.

This block is essentially identical to diagnostics.py's list_metric_values (filter build aside). A shared helper taking a MetricValueFilter plus the value-type/format/pagination/outlier args would remove the duplication and keep the two endpoints' response behaviour in lock-step as the reader layer evolves.

backend/src/ref_backend/core/reader_values.py (1)

37-62: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win

CSV generators buffer the entire output in memory before yielding.

Both generate_csv closures write all rows into a single io.StringIO and yield once at the end. Since CSV export deliberately returns all results without pagination, the full collection is already in memory, and the buffer roughly doubles that footprint. For large diagnostics this can cause significant memory pressure on the worker.

Consider yielding rows (or batches of rows) as they are written so StreamingResponse can flush incrementally:

♻️ Proposed refactor: yield per-row instead of buffering

     def generate_csv() -> Generator[str]:
         output = io.StringIO()
         writer = csv.writer(output)

         items = collection.items
         if not items:
             yield ""
             return

         dimensions = sorted(items[0].dimensions.keys())
         header = [*dimensions, "value", "type"]
         if detection_ran:
             header.extend(["is_outlier", "verification_status"])
         writer.writerow(header)
+        yield output.getvalue()
+        output.seek(0)
+        output.truncate(0)

         for item in items:
             row = [item.dimensions.get(d) for d in dimensions] + [
                 sanitize_float_value(item.value),
                 "scalar",
             ]
             if detection_ran:
                 row.extend([item.is_outlier, item.verification_status])
             writer.writerow(row)
+            yield output.getvalue()
+            output.seek(0)
+            output.truncate(0)
-
-        output.seek(0)
-        yield output.read()

The same pattern applies to generate_csv_response_series (lines 87-112). For fewer, larger chunks, accumulate N rows before yielding.

Also applies to: 87-112

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 62c6e7e3-e968-4cd4-a930-25a867d6132a

📥 Commits

Reviewing files that changed from the base of the PR and between e501f1d and e9a6ca3.

⛔ Files ignored due to path filters (2)

• backend/tests/test-data/tests.integration.test_cmip7_aft/test_solve_cmip7_aft/db/climate_ref.db is excluded by !**/*.db
• backend/uv.lock is excluded by !**/*.lock

📒 Files selected for processing (8)

• backend/pyproject.toml
• backend/src/ref_backend/api/deps.py
• backend/src/ref_backend/api/routes/diagnostics.py
• backend/src/ref_backend/api/routes/executions.py
• backend/src/ref_backend/core/reader_values.py
• backend/src/ref_backend/models.py
• changelog/39.breaking.md
• changelog/39.trivial.md