diff --git a/BACKEND_REDESIGN_PLAN.md b/BACKEND_REDESIGN_PLAN.md new file mode 100644 index 0000000..0e9de91 --- /dev/null +++ b/BACKEND_REDESIGN_PLAN.md @@ -0,0 +1,466 @@ +# PyRCN Backend Redesign — Living Plan + +> Status: **Complete through Phase B.** Torch backend behind ESN*/ELM* (fast +> path + NumPy fallback), public `pyrcn.nn`, gradient and fully-trainable +> training (readout / reservoir / input), examples updated, and the Sphinx +> docs build clean. On branch `backend-redesign` (pushed to origin; no PR +> yet). Edited incrementally as decisions are made. + +## Status summary + +**Completed** +- Phase 1 modernization (branch `modernize-sklearn`; PR #62 to `dev`). +- Phase A (A0-A6) + A4b: torch backend behind ESN*/ELM* (fast path + NumPy + fallback for arbitrary sub-estimators); `washout`; + `predict(initial_state=, return_state=)`. +- P1: public, documented `pyrcn.nn`; "implementation detail" disclaimer + removed. +- Phase B: B1 (gradient readout solver + `random_state` reproducibility + + tight closed-form consistency demo), B2 (trainable reservoir, BPTT), + B2+ (`trainable_input`, BPTT mini-batching over sequences). Optimizers + {adam, adamw, sgd, rmsprop, adagrad}, losses {mse, mae, huber}. +- Examples updated to the current API (no torch usage introduced). +- Documentation: complete API reference incl. `pyrcn.nn`; Sphinx build clean + (0 warnings); tutorials refreshed. +- Activation set unified to the five supported by both backends (dropped the + NumPy-only `softmax`/`softplus`). +- GitHub issues addressed locally: #54 (activation validation), #61 + (estimators run; notebooks fixed). + +**Open / deferred** +- B3 demonstrations: gradient/trainable example notebooks (part of the + deferred post-upgrade demonstrations batch). Not started. +- Issue #53 (multiple reservoirs, one readout): under discussion. +- Merge / PR of `backend-redesign` (target `dev`, not `main`); none opened. +- GitHub issue comments (#54, #61) held until merge. +- Audio examples (`f0_extraction`, `multipitch_tracking`) remain + non-functional: dataset loaders (`fetch_ptdb_tug_dataset`, + `fetch_maps_piano_dataset`) intentionally not restored. + +## Goal +Keep the **frontend** (the scikit-learn-compatible API) unchanged; replace the +**backend** so that sequential data is a first-class concern rather than a set +of workarounds bolted onto scikit-learn's stateless, 2-D-array contract. + +## Hard constraints +- Public API preserved: `ESNRegressor` / `ESNClassifier` / `ELMRegressor` / + `ELMClassifier` with `fit` / `predict` / `predict_proba` / `get_params` / + `set_params`; `InputToNode` / `NodeToNode` blocks as configuration; + `IncrementalRegression`; `SequentialSearchCV`; `metrics`; `datasets`. +- Runs on the modern stack (scikit-learn 1.9 / NumPy 2 / SciPy / Python 3.10+). +- **INV-1 (2-D behavior — structural, not bit-exact):** a 2-D `(T, F)` input + keeps today's *semantics* — one continuous reservoir pass (no mid-stream + reset) and `predict` returns a plain `(T, n_targets)` ndarray, **not** a + length-1 object array. Numeric outputs need only match the historical + float64 result **within a tolerance** (relaxed per D3; float32 drift allowed). + +## Current-state audit (summary; full detail in memory `pyrcn-architecture`) +- Sequences smuggled as 1-D `object` arrays; sequence branch bypasses + `validate_data`. Detection via the fragile `X.ndim == 1`; equal-length 3-D + input is rejected. +- Reservoir is a pure-Python per-timestep loop; batching is Python-level + (per-sequence AND per-timestep); parallelism only via heavyweight joblib-loky. +- Transformers are stateless: every `transform` resets state to zero → no + washout, no initial-state carry, no streaming inference; the start transient + pollutes the readout fit. +- `predict` returns object arrays → the whole custom `metrics/` module exists to + cope. `ESN/ELM` hand-roll `get_params`/`set_params` via private + `_get_param_names`. +- Worth preserving: incremental normal-equations readout (`K`, `xTy`, single + solve, mergeable via `__add__`); the block-composition model. + +## Design axes (to decide, incrementally) +1. **Sequence data model** — **PUBLIC CONTRACT DECIDED** (see D1 + section below); + internal canonical form still open (coupled to axis #2). +2. **Reservoir engine** — **DECIDED: PyTorch** (see D2 + section below). +3. **State semantics** — **DECIDED (D6 washout, D7 state carry).** +4. **Params / `clone`** — **DECIDED (D8): nested params + `**kwargs` alias.** +5. **Numerics** — **DECIDED (D3): float32 allowed, INV-1 relaxed to tolerance.** + +## Decisions log +- **D1 (public sequence contract): accept-all + normalize.** `fit`/`predict` + accept several input forms and normalize internally to one canonical form. + 2-D-means-one-sequence **confirmed**, conditional on INV-1 (no change to the + current 2-D behavior, incl. plain-array output). +- **DECIDED:** `(k,)`-target ambiguity → auto-detect (length-based) by default, + with an explicit estimator param `task` in + `{'auto','sequence-to-sequence','sequence-to-value'}` to force it; raise a + clear error only on an unresolved collision (`k == Lᵢ == n_targets`). +- **D2 (reservoir engine): PyTorch.** The reservoir/`NodeToNode` (and the ELM + feature map) run as batched PyTorch tensor ops; reservoir weights are fixed + buffers (`requires_grad=False`), so autograd is unused but batching + optional + CUDA are the win. Reintroduces torch as a backend dependency (reverses the + Phase-1 removal — this is the deliberate post-upgrade `nn` plan; build clean, + not the old draft). + +## Reservoir engine (PyTorch) — from D2 + +### What it settles +- Internal canonical form → **padded batch `(N, L_max, F)` + `lengths`** (and/or + `torch.nn.utils.rnn.pack_padded_sequence`), which is the natural PyTorch shape + for batched recurrence. (Resolves the axis-1 "internal form" open item.) +- Time loop becomes a batched tensor recurrence over `L_max` (mask/packed to + respect `lengths`), replacing the per-sample + per-sequence Python loops. + +### Sub-decisions +- **D4 — torch is a HARD dependency.** Single code path, no NumPy fallback + engine. Re-add torch to `pyproject` core `dependencies` (reverses the Phase-1 + removal). `device=` still selects cpu/cuda. +- **D3 — allow float32; INV-1 relaxed to a tolerance.** Backend may run in + float32 (default torch dtype / GPU-friendly); historical float64 parity is not + required bit-exact, only within tolerance. Expect a `dtype=` knob (float32/64). + - **Verification strategy (honest despite chaotic float drift):** a reservoir + is a dynamical system, so float32-vs-float64 states can *diverge* over long + sequences — a raw-state tolerance is unreliable. Validate the port by + comparing **torch-float64 vs the old NumPy-float64** backend (expect tight + agreement) to prove correctness; treat float32 as a separate speed/precision + mode judged on task-level metrics, not state equality. +- **Resolved:** + - **D10 — dense `Parameter` weights.** Store input/recurrent weights as dense + torch `Parameter`s (zeros where `k_in`/`k_rec` init made them sparse). Dense + matmul is GPU-ideal and gradient-friendly (sparse autograd is limited); + `predefined_*_weights` load into dense `Parameter`s. Sparse storage may be + added later as an opt-in for very large reservoirs. + - Readout location and device placement are resolved by **D5** below. + +## Sequence data model (from D1) + +### Accepted public input forms → normalization +- **2-D `(T, F)`** → **one** sequence of length `T` (continuous series). + *Preserves* the current single-sequence/time-series behavior (e.g. + `mackey_glass`): the reservoir runs over all rows with no mid-stream reset. + Rows are NOT independent tabular samples for the recurrent path. +- **list / tuple / 1-D `object` array of 2-D `(Lᵢ, F)`** → **N** sequences + (ragged). Object-array form kept for back-compat. +- **3-D `(N, L, F)`** → **N** equal-length sequences (newly allowed). +- Everything collapses to a canonical *"batch of N≥1 sequences"*. + +### Targets `y` → task kind (replace the length heuristic) +- `y[i]` shaped `(Lᵢ,)` or `(Lᵢ, n_targets)` ⇒ **sequence-to-sequence**. +- `y[i]` scalar or `(n_targets,)` (one label per sequence) ⇒ + **sequence-to-value**. Determined by rank/shape, not by + `not np.any(len_X == len_y)`. + +### Implementation note +- A single validating normalizer (working name `check_sequences`) replaces + `concatenate_sequences` + `_check_if_sequence` + `_check_if_sequence_to_value`. + It returns the canonical batch plus per-sequence lengths and the task kind, + and is the one place object/ragged input is handled (so estimators no longer + bypass validation ad hoc). +- Must stay an sklearn-indexable of length `N` (for `SequentialSearchCV`/CV): + `len` = N, `X[idx]` selects sequences. + +### Still open (couple to axis #2 — reservoir engine) +- Canonical **internal** form: concatenated `(ΣLᵢ, F)` + boundaries (memory-lean, + loop/streaming-friendly) vs. padded `(N, Lmax, F)` + mask (vectorized-batch + friendly) vs. list-of-arrays kept as-is (bucketed batching). + +## Frontend / Backend architecture (D5) + +**D5 — boundary at the estimator edge (thin frontend).** NumPy↔torch conversion +happens once, at estimator `fit`/`predict` entry/exit; everything internal is +torch and stays on `device`. + +**Frontend (NumPy, sklearn — the kept public API)** +- Estimators `ESN*/ELM*`: sklearn protocol, `get_params`/`set_params`, + validation + `check_sequences` normalization, orchestration, and the single + NumPy→torch (entry) / torch→NumPy (exit) conversion. +- Blocks `InputToNode`/`NodeToNode` (+variants) as configuration + (hyperparameters, param round-trip). Standalone `block.transform(X_np)` still + accepts/returns NumPy by converting at its own edge (back-compat). +- `model_selection`, `metrics`, `datasets`, `preprocessing` (Coates), + `projection` — remain NumPy/sklearn. + +**Backend (PyTorch, tensors — never imported by users)** +- Reservoir engine (batched recurrence), input feature-map compute, weight-init + buffers, and the **readout** (ridge / incremental normal equations) — all in + torch, on-device. +- `device`/`dtype` owned here (configured via estimator params). + +**Consequences** +- Readout is torch by default (implements the `K`/`xTy` incremental normal + equations + solve on-device, preserving the mergeable `__add__` semantics). +- The `regressor=` extension point: a torch-native readout is the default; + passing a NumPy sklearn regressor is still allowed but forces a torch→NumPy + round-trip at the readout (documented performance caveat). *(refine later)* +- Only two host↔device transfers per `fit`/`predict` call. + +## State semantics (D6, D7) + +**D6 — washout (training-only).** New estimator param `washout: int = 0`, +applied **per sequence**: during `fit`, skip the first `w` reservoir states and +targets from the readout least-squares accumulation. `predict` runs the full +length and returns all `T` steps (early steps warm-influenced but present, so +seq-to-seq length alignment is preserved). Default `0` = current behavior +(INV-1). Validate `w < min(Lᵢ)`. ELM has no temporal transient → washout N/A +(ignored / rejected for ELM). + +**D7 — initial-state carry (backend primitive + modest frontend hook).** +- Backend reservoir primitive: + `reservoir(X_t, initial_state=None) -> (states, final_state)` (tensors, + on-device). `initial_state=None` ⇒ zero init (default, INV-1). +- Frontend: `predict(X, initial_state=None, return_state=False)`. With + `return_state=True` returns `(y, final_state)`; `initial_state` seeds the + reservoir to continue a series across calls. State crosses the estimator edge + as NumPy (converted per D5); internal state is a device tensor. +- **Streaming deferred** (no `partial_predict`/`reset_state` yet) but the + primitive above is the foundation, so it can be layered on without redesign. +- Additive, back-compatible API change (defaults preserve current behavior). + +## Params / clone (D8) + +**D8 — standard nested params, with a `**kwargs` convenience alias.** +- **Canonical params** are the sub-estimators (`input_to_node`, `node_to_node`, + `regressor`) plus estimator-level params (`requires_sequence`, + `decision_strategy`, `verbose`, `washout`, `device`, `dtype`, …). Addressed + nested: `input_to_node__spectral_radius`. `get_params`/`set_params`/`clone` + come from `BaseEstimator` recursing through each block's **public** + `get_params` — we no longer *call* `_get_param_names` on sub-objects (the + Phase-2 concern is gone). +- **`**kwargs` stays** as pure construction sugar so `ESNRegressor( + hidden_layer_size=100)` keeps working; kwargs populate the default + sub-estimators at build time and are NOT part of the param set. +- **Implementation note:** `**kwargs` in `__init__` makes sklearn's default + `_get_param_names` raise (it forbids `VAR_KEYWORD`). So override our own + `_get_param_names` (classmethod) to return the explicit param list; everything + else (nested get/set, clone, GridSearchCV `input_to_node__…`) then works by + standard sklearn machinery. Fixes the flat-namespace / bare-name `set_params` + mismatch that caused the GridSearchCV friction. +- **Fallback** (if the override proves fragile): drop `**kwargs` entirely → + pure nested, zero overrides, but loses the flat constructor shortcut. + +## Training modes (D9) + +**D9 — `nn.Module` architecture + two training paths (phased).** The backend is +built as PyTorch `nn.Module`s so both closed-form and gradient-based training of +RCNs are supported from one design. +- **Reservoir** = an `nn.Module`; its input/recurrent weights are `Parameter`s + with **`requires_grad=False` by default** (classic fixed reservoir). The RC + initialization strategy (spectral radius, `k_in`/`k_rec` sparsity, predefined + weights) is preserved — it just fills these parameters. An opt-in toggle + (`requires_grad=True`) makes the reservoir trainable end-to-end. +- **Readout** = an `nn.Linear`, populated one of two ways via a `solver`: + - `closed_form` (default): analytically set weights via the ridge / + normal-equations solve (the D5/D7 path). Requires a fixed reservoir. + - `gradient`: train weights with an optimizer loop (works with a fixed or a + trainable reservoir). +- **Valid configurations:** (1) fixed reservoir + closed-form (**default**, + behavior-preserving); (2) fixed reservoir + gradient (converges to #1 for a + linear readout under MSE — a built-in consistency check); (3) trainable + reservoir + gradient. "trainable + closed-form" is invalid → validated out. +- **Gradient mode adds opt-in hyperparams:** optimizer, learning rate, epochs, + loss, batch size. All default-off; closed-form users see no new required args. +- **Phasing:** + - **Phase A** — torch backend, fixed reservoir + closed-form readout (exactly + D1–D8, behavior-preserving), built on the `nn.Module` foundation so + parameters / `requires_grad` exist from the start. + - **Phase B** — add gradient training (trainable reservoir + optimizer loop); + a new `fit` branch, no rewrite. +- Refines D2 (engine = `nn.Module`s), D5/D7 (readout is an `nn.Linear`). + +**D11 — closed-form readout = `IncrementalRegression`, torch-native.** The +existing default readout class is reimplemented to run on torch tensors +(dual-mode: still accepts NumPy for standalone/public use, unchanged), so it +stays the default closed-form solver and computes on-device (filling the readout +`nn.Linear`'s weights via the ridge / normal-equations solve; mergeable `__add__` +preserved). A user-supplied external sklearn regressor (Ridge, …) still works in +the fixed-reservoir/closed-form path via a documented torch→NumPy round-trip; +it is not offered in gradient/trainable mode. (Gradient-mode readout = the +`nn.Linear` trained by the optimizer loop.) + +## Block ↔ backend module, and dual usage (D12) + +**D12 — companion pattern (not inheritance).** Frontend blocks +(`InputToNode`/`NodeToNode` + variants) stay sklearn `TransformerMixin` +**configuration** objects (hyperparameters, `get_params`/`set_params`). At +`fit`, each builds its backend torch `nn.Module` counterpart (dense +`Parameter`s + `forward`) that performs the computation. Blocks are **not** +`nn.Module` subclasses — that avoids the `nn.Module.__setattr__` vs sklearn +`get_params`/`clone` clash that made the old draft unworkable. + +**Dual usage (enabled by D12) — an explicit goal.** Because the compute lives +in standalone torch `nn.Module`s, one codebase serves two audiences: +- **scikit-learn users** — the `ESN*/ELM*` estimators (NumPy I/O, full sklearn + ecosystem), unchanged. +- **PyTorch users** — the backend `nn.Module`s used directly (tensors, custom + training loops, composition into larger torch models). + +**D13 — public torch API, staged, as `pyrcn.nn`.** The backend modules become a +*supported public* torch API **after Phase-A parity** (not before — Phase A +stays focused on correctness vs. the NumPy oracle). Namespace: **`pyrcn.nn`**, +built **clean from the companion modules**. ⚠️ Do **not** resurrect or port the +old `pyrcn.nn` draft (removed in commit `3e41df2`) — it was messy; the new +`pyrcn.nn` is a fresh, minimal surface over the Phase-A backend `nn.Module`s. + +## Torch module implementation & reuse (D14) + +**D14 — maximize reuse of `nn.RNNCell` / `nn.Linear`; add only what torch +lacks (leaky integration + extra activations).** No duplication of torch's +weight / init / `state_dict` / bidirectional machinery. + +**Block → torch module mapping** +- **`InputToNode` → subclass `nn.Linear`** (`weight` = input weights, `bias`), + plus `input_scaling`/`input_shift`/`bias_scaling`/`bias_shift` and the input + activation applied in `forward`. This is the real feature map for **ELM** + (ELM = these + readout) and the input stage for **ESN**. +- **`NodeToNode` → subclass `nn.RNNCell`** (recurrence). Reuse its fused + single-step op for `tanh`/`relu` via `(1-λ)h + λ·super().forward(x, h)` + (verified parity 1.1e-16). For `logistic`/`identity`/`bounded_relu` (not in + the fused op) compute the affine with `F.linear` + apply the activation, still + `+ leaky`. `weight_ih` = identity (input is added directly — the input weights + belong to `InputToNode`); `spectral_radius` is folded into `weight_hh`. A thin + layer loops the cell over time (lengths/mask, `initial_state`→`final_state`) + and does bidirectional (reuse `nn.RNN` structure: tie reverse=forward+flip for + the parity check, independent scaled weights otherwise). + +**Two-activation handling.** PyRCN applies an *input* activation (`InputToNode`) +AND a *reservoir* activation (`NodeToNode`); a single RNN cell has one. So for a +nonlinear `input_activation` (e.g. the `ESNClassifier` default `tanh`) +`InputToNode` stays a separate stage feeding the cell (cell `weight_ih` = +identity). For `input_activation='identity'` the input weights may be folded into +the cell's `weight_ih` (a fused `ESNCell`). + +**Efficiency notes / obstacles (all minor, none blocking).** +- `nn.RNN`'s *whole-sequence* fused kernel supports only non-leaky `tanh`/`relu`; + leaky needs per-step control → per-step loop that still reuses the fused + *single-step* cell op. An optional whole-sequence fast-path (delegating to + `nn.RNN`) for the non-leaky `tanh`/`relu` case can be added later. +- The three extra activations use `affine + activation` rather than the fused + cell op. + +**Deferred:** a fused single-cell `ESNCell` (input+recurrence, one activation) +for `input_activation='identity'` as the idiomatic/fast path in the public +`pyrcn.nn`. Refactor note: the A2 custom `Reservoir` becomes a thin layer over +the `nn.RNNCell` subclass; its behavioral parity tests carry over unchanged. + +## Roadmap / workstreams + +Two phases. **Phase A** delivers the full torch backend with a fixed reservoir +and the closed-form readout — behavior-preserving (D1–D8, D10, D11), built on +the `nn.Module` foundation. **Phase B** adds gradient-based training (D9) as new +`fit` branches, no rewrite. The legacy NumPy path is kept importable until +Phase-A parity is proven, then retired. + +### Phase A — torch backend, fixed reservoir + closed-form (behavior-preserving) +- **A0 · Scaffolding & deps. — done.** Add torch as a hard dependency (D4); create a + private `pyrcn.backend` (torch) package; `device`/`dtype` helpers; stand up + the parity harness (torch-float64 vs legacy NumPy-float64, D3). Legacy path + stays alive for comparison. +- **A1 · Sequence normalization (D1 + task=). — done.** `check_sequences`: accept + list / object-array / 2-D / 3-D → canonical padded batch `(N, L_max, F)` + + `lengths`; task auto-detect + `task=` override; INV-1 (2-D → N=1). Replaces + `concatenate_sequences` + `_check_if_sequence*`. Tests for every input form. +- **A2 · Reservoir engine (D2/D6/D7/D10). — done** (feature map, leaky / + Euler / bidirectional cells, torch-native init; parity by injection). + Reservoir `nn.Module`: dense + `Parameter` input/recurrent weights (`requires_grad=False`), **RC init + preserved** (spectral radius, `k_in`/`k_rec` sparsity, antisymmetric/Euler + variants, predefined weights); batched masked recurrence over `L_max`; leaky + integration; bidirectional; `washout`; `initial_state`→`final_state`. Input + feature-map (`InputToNode` math) as an `nn.Module` too. +- **A3 · Closed-form readout (D11). — done.** `IncrementalRegression` + reimplemented torch-native as `backend.IncrementalRidge`: incremental + `K`/`xTy`, ridge solve on-device, `fit_intercept` via ones-column, mergeable + `__add__` preserved. Parity vs the legacy readout by injecting identical + features/targets (single-batch, postpone-then-solve, merge == concat). +- **A4 · Estimator integration (D5/D8). — core done.** `ESN*/ELM*` + `fit`/`predict` route feature-map→reservoir→readout through the torch + backend (single float64 conversion at the edge) via the config→backend + bridge, on a **fast path + numpy fallback**: the fast path engages only when + every component is a torch-backable pyrcn default; arbitrary sub-estimators + (`FeatureUnion` input, external `Ridge`, `normalize=True`) and `partial_fit` + keep the legacy numpy path. Param surface kept as-is (bare names) so + model selection is unchanged (decision: keep bare, additive nested deferred). + Classifier `predict_proba` + decision strategies preserved unchanged. Full + suite green (incl. the formerly-slow chunk test, now ~13s on the fast path). + A4b done: `washout` (int, training-only, drops start transient per + sequence), `predict(initial_state=, return_state=)` (ESN-only, torch + backend; state-carry primitive verified — split-and-carry reproduces the + whole-sequence pass). Defaults behavior-preserving; numpy fallback raises + NotImplementedError for these. +- **A5 · Parity & test migration. — done.** Broadened the torch-vs-numpy + parity harness across all four estimators and both sequence/non-sequence + modes via a native-vs-forced-fallback twin (fallback wraps the input in a + single-transformer `FeatureUnion`); max observed difference ~3e-9 (float64 + round-off). Model-selection tests (GridSearchCV / RandomizedSearchCV / + SequentialSearchCV / component-swap over bare-name params) confirm + `best_estimator_._use_torch` stays True through `clone` — model selection + unchanged (bare-name surface, per the A4 decision; the planned nested-params + test is moot since we kept bare names). Dropped the joblib/loky + sequence-parallel path (`n_jobs` now a serial no-op). `metrics` unaffected + (predict still returns NumPy). Suite 194 passed / 2 skipped. +- **A6 · Finalize (legacy retirement deferred). — done.** Reframed: the numpy + reservoir loop (`NodeToNode.transform`), `concatenate_sequences`, and the + numpy weight init CANNOT be removed under fast-path + fallback — the fast + path still calls `concatenate_sequences` and injects numpy-init weights, and + the fallback path + standalone block tests + parity tests still exercise the + numpy reservoir loop. True retirement waits until the companion pattern (P1+) + makes the blocks torch-native. Done here: removed the dead+broken + `__add__`/`__radd__` merge operators (only the dropped joblib `sum(reg)` + used them; they read `regressor._K`, absent on a torch estimator) and two + dead commented lines; added `tests/test_backend_device.py` (CPU runs, CUDA + skipped when unavailable); dropped `joblib` from explicit deps (sklearn + pulls it transitively). CI commands verified locally (flake8 `src/pyrcn + tests`, mypy `src/pyrcn`, full pytest) — 196 passed / 3 skipped. User-facing + docs deferred to P1 (backend still marked private). + +### Public torch API — `pyrcn.nn` (after Phase-A parity, D13) +- **P1 · Promote the backend modules to a public, documented `pyrcn.nn`. — + done.** Renamed `pyrcn.backend` → `pyrcn.nn` (single public package, built + from the Phase-A companion modules — NOT the removed draft `3e41df2`). + Public surface: reservoir layers `Reservoir`/`EulerReservoir`, cells + `LeakyESNCell`/`EulerESNCell`, `InputFeatureMap`, `IncrementalRidge`; + weight initializers under a `pyrcn.nn.init` submodule (mirrors + `torch.nn.init`). Frontend glue (`_bridge`) and the cell base + (`_ReservoirCell`) stay private. Removed the "implementation detail / + not-public-yet" disclaimer; added a pure-PyTorch usage example + (doctest-verified) and a Sphinx API page (`docs/.../pyrcn.nn.rst`). + `pyrcn.nn` registered in the top-level package. No behavior change; suite + 195 passed / 3 skipped, flake8 + mypy clean. + +### Phase B — gradient-based training (D9) +- **B1 · Gradient readout solver. — done.** Added `pyrcn.nn.LinearReadout` + (trainable `nn.Linear` readout) + `pyrcn.nn.train_readout` (optimizer loop; + optimizer/loss by name, `weight_decay` = ridge strength). `ESN*`/`ELM*` + gain flat params `solver` (`closed_form` default / `gradient`), `optimizer`, + `learning_rate`, `epochs`, `batch_size` (validated, in `get_params` for + clone/GridSearch). In gradient mode the reservoir stays fixed: states are + computed once (dropping `washout` per sequence, concatenated in sequence + mode) and a fresh `LinearReadout` is trained; the `regressor` supplies the + readout config (`fit_intercept`, `alpha`→`weight_decay`). Gradient requires + native torch-backable components (else `NotImplementedError`). Consistency + check (D9): on well-conditioned (whitened) features a gradient-trained + readout matches the closed-form ridge to ~1e-6; through the raw reservoir / + ELM map convergence is slow, so the estimator tests assert learning + the + structural contracts. Defaults behavior-preserving; full suite green. + Gradient fits are reproducible via `random_state`: the readout init and the + shuffle are seeded from it (`pyrcn.nn.torch_generator`, threaded through + `LinearReadout(generator=)` and `train_readout(generator=)`). +- **B2 · Trainable reservoir. — done.** `ESN*` gain `trainable_reservoir` + (bool, default False). With `solver="gradient"` it unfreezes the reservoir's + recurrent weights (`Reservoir/EulerReservoir.set_recurrent_trainable`) and + optimizes them jointly with the readout by backprop through the recurrence + (BPTT): each epoch recomputes the states (the fixed input feature map is + applied once and detached), full-batch. The RC init is the starting point. + The invalid `trainable + closed_form` combo is rejected; reproducible via + `random_state`; `predict` runs under `no_grad`. ESN-only (ELM has no + reservoir); sequence + non-sequence + classifier. Full suite 218 passed. + - **B2+ (done).** `trainable_input` (bool): with `solver="gradient"`, also + train the input feature-map weights (`InputFeatureMap.set_input_trainable`) + — alone or with `trainable_reservoir` for a fully trainable RNN (input + + reservoir + readout); requires the gradient solver. BPTT mini-batching: + `batch_size` batches over sequences (seeded shuffle; `None` = full-batch), + non-sequence stays full-BPTT. Reproducible via `random_state`. Suite 222. +- **B3 · Tests, examples, docs** for the gradient modes. + +### Risks / notes +- NumPy-2 ragged/object-array handling lives only in `check_sequences`. +- float32 can diverge from float64 on long sequences → correctness proven in + float64; float32 judged on task metrics (D3). +- Keep every change behind the preserved public API; the legacy path is the + parity oracle until A5 passes. + +## Open questions +- Target performance / scale (sequence count, lengths)? +- Any external backend dependency acceptable (numba, Cython), or pure + NumPy/SciPy only? diff --git a/docs/requirements.txt b/docs/requirements.txt index 7380996..2a0c9bb 100644 --- a/docs/requirements.txt +++ b/docs/requirements.txt @@ -1,5 +1,5 @@ -docutils<0.18 -sphinx-rtd-theme==0.5.1 -sphinx-copybutton==0.3.1 +sphinx>=7 +sphinx-rtd-theme>=2.0 +sphinx-copybutton>=0.5 pyrcn scikit-learn diff --git a/docs/source/api/api.rst b/docs/source/api/api.rst index d7eac2d..e1da6b1 100644 --- a/docs/source/api/api.rst +++ b/docs/source/api/api.rst @@ -6,7 +6,14 @@ PyRCN API pyrcn pyrcn.base + pyrcn.nn pyrcn.echo_state_network pyrcn.extreme_learning_machine pyrcn.linear_model - pyrcn.cluster + pyrcn.model_selection + pyrcn.metrics + pyrcn.datasets + pyrcn.projection + pyrcn.preprocessing + pyrcn.postprocessing + pyrcn.util diff --git a/docs/source/api/pyrcn.metrics.rst b/docs/source/api/pyrcn.metrics.rst new file mode 100644 index 0000000..f97cdec --- /dev/null +++ b/docs/source/api/pyrcn.metrics.rst @@ -0,0 +1,9 @@ +.. _`pyrcn.metrics`: + +pyrcn.metrics +============= + +.. automodule:: pyrcn.metrics + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/api/pyrcn.model_selection.rst b/docs/source/api/pyrcn.model_selection.rst new file mode 100644 index 0000000..0e47bb6 --- /dev/null +++ b/docs/source/api/pyrcn.model_selection.rst @@ -0,0 +1,9 @@ +.. _`pyrcn.model_selection`: + +pyrcn.model_selection +===================== + +.. automodule:: pyrcn.model_selection + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/api/pyrcn.nn.rst b/docs/source/api/pyrcn.nn.rst new file mode 100644 index 0000000..9307e7d --- /dev/null +++ b/docs/source/api/pyrcn.nn.rst @@ -0,0 +1,64 @@ +.. _`pyrcn.nn`: + +pyrcn.nn +======== + +.. automodule:: pyrcn.nn + +Reservoir layers +---------------- + +.. autoclass:: pyrcn.nn.Reservoir + :members: + :undoc-members: + :show-inheritance: + +.. autoclass:: pyrcn.nn.EulerReservoir + :members: + :undoc-members: + :show-inheritance: + +Reservoir cells +--------------- + +.. autoclass:: pyrcn.nn.LeakyESNCell + :members: + :undoc-members: + :show-inheritance: + +.. autoclass:: pyrcn.nn.EulerESNCell + :members: + :undoc-members: + :show-inheritance: + +Feature map and readout +------------------------ + +.. autoclass:: pyrcn.nn.InputFeatureMap + :members: + :undoc-members: + :show-inheritance: + +.. autoclass:: pyrcn.nn.IncrementalRidge + :members: + :undoc-members: + :show-inheritance: + +.. autoclass:: pyrcn.nn.LinearReadout + :members: + :undoc-members: + :show-inheritance: + +Gradient training +----------------- + +.. autofunction:: pyrcn.nn.train_readout + +.. autofunction:: pyrcn.nn.torch_generator + +Weight initializers +------------------- + +.. automodule:: pyrcn.nn.init + :members: + :undoc-members: diff --git a/docs/source/api/pyrcn.postprocessing.rst b/docs/source/api/pyrcn.postprocessing.rst new file mode 100644 index 0000000..669f892 --- /dev/null +++ b/docs/source/api/pyrcn.postprocessing.rst @@ -0,0 +1,9 @@ +.. _`pyrcn.postprocessing`: + +pyrcn.postprocessing +==================== + +.. automodule:: pyrcn.postprocessing + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/api/pyrcn.preprocessing.rst b/docs/source/api/pyrcn.preprocessing.rst new file mode 100644 index 0000000..9c0a33e --- /dev/null +++ b/docs/source/api/pyrcn.preprocessing.rst @@ -0,0 +1,9 @@ +.. _`pyrcn.preprocessing`: + +pyrcn.preprocessing +=================== + +.. automodule:: pyrcn.preprocessing + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/api/pyrcn.projection.rst b/docs/source/api/pyrcn.projection.rst new file mode 100644 index 0000000..d1b8786 --- /dev/null +++ b/docs/source/api/pyrcn.projection.rst @@ -0,0 +1,9 @@ +.. _`pyrcn.projection`: + +pyrcn.projection +================ + +.. automodule:: pyrcn.projection + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/api/pyrcn.util.rst b/docs/source/api/pyrcn.util.rst new file mode 100644 index 0000000..ba40dcf --- /dev/null +++ b/docs/source/api/pyrcn.util.rst @@ -0,0 +1,9 @@ +.. _`pyrcn.util`: + +pyrcn.util +========== + +.. automodule:: pyrcn.util + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/conf.py b/docs/source/conf.py index 104a54e..187e0ca 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -40,6 +40,22 @@ ] master_doc = 'index' +# Render numpydoc "Attributes" sections as info fields rather than separate +# object descriptions (avoids duplicate-object-description warnings for +# dataclass attributes and properties). +napoleon_use_ivar = True + +# Do not document scikit-learn's dynamically added metadata-routing methods +# (set_*_request); their docstrings reference sklearn-only glossary terms and +# labels that do not resolve in this documentation. +autodoc_default_options = { + 'exclude-members': ( + 'set_fit_request,set_predict_request,set_partial_fit_request,' + 'set_score_request,set_transform_request,' + 'set_predict_proba_request,set_inverse_transform_request' + ), +} + # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] diff --git a/docs/source/getting_started.rst b/docs/source/getting_started.rst index c730498..6f9eedc 100644 --- a/docs/source/getting_started.rst +++ b/docs/source/getting_started.rst @@ -62,6 +62,7 @@ Building your first Reservoir Computing Network >>> from inspect import getmembers, isclass >>> getmembers(blocks, isclass) [('BatchIntrinsicPlasticity', ), + ('EulerNodeToNode', ), ('HebbianNodeToNode', ), ('InputToNode', ), ('NodeToNode', ), @@ -94,10 +95,12 @@ Training a RCN To train the ESN, only three steps are required: - 1. Randomly distribute the time-series to each reservoir neuron (**Input-to-Node**). - 2. Compute the state of each neuron based on the current input and the previous - state. - 2. Compute a linear regression between the reservoir states and the target output. + 1. Randomly distribute the time-series to each reservoir neuron + (**Input-to-Node**). + 2. Compute the state of each neuron based on the current input and the + previous state. + 3. Compute a linear regression between the reservoir states and the + target output. These steps are handled via :py:func:`pyrcn.echo_state_network.ESNRegressor.fit`, which is the most important function to train the ESN model: @@ -120,6 +123,6 @@ Testing and predict using the ESN .. doctest:: - >>> y_pred = esn.predict(X[:4000]) + >>> y_pred = esn.predict(X[:4000].reshape(-1, 1)) .. image:: _static/img/getting_started_mackey_glass_predicted.svg diff --git a/docs/source/index.rst b/docs/source/index.rst index ea20864..55820bd 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -33,8 +33,8 @@ It is actively developed to be extended into several directions: * Interaction with `sktime `_ * Interaction with `hmmlearn `_ -* More towards future work: Related architectures, such as Liquid State Machines (LSMs) -and Perturbative Neural Networks (PNNs) +* More towards future work: Related architectures, such as Liquid State + Machines (LSMs) and Perturbative Neural Networks (PNNs) PyRCN has successfully been used for several tasks: diff --git a/docs/source/installation.rst b/docs/source/installation.rst index d39d15b..1d1fc37 100644 --- a/docs/source/installation.rst +++ b/docs/source/installation.rst @@ -1,3 +1,5 @@ +.. _installation guide: + ================== Installation guide ================== @@ -13,10 +15,10 @@ command in a PowerShell or CommandLine in Windows, or in a shell in Linux/MacOS: python --version -As any package, **PyRCN** has several dependencies as listed in the `requirements.txt - `_. To avoid any -unexpected interaction with the basic system as installed on your computer, we highly -recommend using a virtual environment +As any package, **PyRCN** has several dependencies, declared in +`pyproject.toml `_. +To avoid any unexpected interaction with the basic system as installed on your +computer, we highly recommend using a virtual environment You can find more information about virtual environments, by checking the `Python documentation on virtual environments and packages diff --git a/docs/source/introduction.rst b/docs/source/introduction.rst index 84afd14..a3d9755 100644 --- a/docs/source/introduction.rst +++ b/docs/source/introduction.rst @@ -1,3 +1,5 @@ +.. _whats rc: + Definition of Reservoir Computing ================================= diff --git a/examples/MNIST_regressors.ipynb b/examples/MNIST_regressors.ipynb index e43407e..d4d41b2 100644 --- a/examples/MNIST_regressors.ipynb +++ b/examples/MNIST_regressors.ipynb @@ -21,7 +21,7 @@ "from sklearn.linear_model import Ridge\n", "from sklearn.preprocessing import MinMaxScaler\n", "from sklearn.model_selection import RandomizedSearchCV, GridSearchCV, StratifiedKFold, ParameterGrid, cross_validate\n", - "from sklearn.utils.fixes import loguniform\n", + "from scipy.stats import loguniform\n", "from sklearn.metrics import accuracy_score\n", "\n", "from pyrcn.model_selection import SequentialSearchCV\n", diff --git a/examples/Video_Door_state_Classification.ipynb b/examples/Video_Door_state_Classification.ipynb index 2bbebf7..11e4163 100644 --- a/examples/Video_Door_state_Classification.ipynb +++ b/examples/Video_Door_state_Classification.ipynb @@ -38,7 +38,7 @@ "import pandas as pd\n", "from tqdm import tqdm\n", "import matplotlib.pyplot as plt\n", - "from IPython.display import set_matplotlib_formats\n", + "from matplotlib_inline.backend_inline import set_matplotlib_formats\n", "set_matplotlib_formats('png', 'pdf')\n", "from mpl_toolkits.axes_grid1 import make_axes_locatable\n", "from matplotlib import ticker\n", @@ -55,7 +55,7 @@ "from sklearn.metrics import make_scorer, ConfusionMatrixDisplay\n", "from sklearn.model_selection import RandomizedSearchCV, GridSearchCV, ParameterGrid\n", "from sklearn.cluster import MiniBatchKMeans\n", - "from sklearn.utils.fixes import loguniform\n", + "from scipy.stats import loguniform\n", "from scipy.stats import uniform\n", "from joblib import dump, load\n", "\n", @@ -259,10 +259,10 @@ "step3_esn_params = {'bias_scaling': np.linspace(0.0, 1.0, 11)}\n", "step4_esn_params = {'alpha': loguniform(1e-5, 1e1)}\n", "\n", - "kwargs_step1 = {'n_iter': 200, 'random_state': 42, 'verbose': 1, 'n_jobs': -1, 'scoring': make_scorer(mean_squared_error, greater_is_better=False, needs_proba=True)}\n", - "kwargs_step2 = {'n_iter': 50, 'random_state': 42, 'verbose': 1, 'n_jobs': -1, 'scoring': make_scorer(mean_squared_error, greater_is_better=False, needs_proba=True)}\n", - "kwargs_step3 = {'verbose': 1, 'n_jobs': -1, 'scoring': make_scorer(mean_squared_error, greater_is_better=False, needs_proba=True)}\n", - "kwargs_step4 = {'n_iter': 50, 'random_state': 42, 'verbose': 1, 'n_jobs': -1, 'scoring': make_scorer(mean_squared_error, greater_is_better=False, needs_proba=True)}\n", + "kwargs_step1 = {'n_iter': 200, 'random_state': 42, 'verbose': 1, 'n_jobs': -1, 'scoring': make_scorer(mean_squared_error, greater_is_better=False, response_method='predict_proba')}\n", + "kwargs_step2 = {'n_iter': 50, 'random_state': 42, 'verbose': 1, 'n_jobs': -1, 'scoring': make_scorer(mean_squared_error, greater_is_better=False, response_method='predict_proba')}\n", + "kwargs_step3 = {'verbose': 1, 'n_jobs': -1, 'scoring': make_scorer(mean_squared_error, greater_is_better=False, response_method='predict_proba')}\n", + "kwargs_step4 = {'n_iter': 50, 'random_state': 42, 'verbose': 1, 'n_jobs': -1, 'scoring': make_scorer(mean_squared_error, greater_is_better=False, response_method='predict_proba')}\n", "\n", "# The searches are defined similarly to the steps of a sklearn.pipeline.Pipeline:\n", "searches = [('step1', RandomizedSearchCV, step1_esn_params, kwargs_step1),\n", @@ -342,7 +342,7 @@ " 'leakage': loguniform(1e-5, 1e0),\n", " 'bias_scaling': uniform(loc=0, scale=2)}\n", "\n", - "kwargs_step0 = {'n_iter': 1000, 'random_state': 42, 'verbose': 1, 'n_jobs': -1, 'scoring': make_scorer(mean_squared_error, greater_is_better=False, needs_proba=True)}\n", + "kwargs_step0 = {'n_iter': 1000, 'random_state': 42, 'verbose': 1, 'n_jobs': -1, 'scoring': make_scorer(mean_squared_error, greater_is_better=False, response_method='predict_proba')}\n", "\n", "base_esn = ESNClassifier(**initially_fixed_params)\n", "\n", diff --git a/examples/Video_Door_state_Classification.py b/examples/Video_Door_state_Classification.py index 13d24d3..c4406f5 100644 --- a/examples/Video_Door_state_Classification.py +++ b/examples/Video_Door_state_Classification.py @@ -12,7 +12,7 @@ RandomizedSearchCV, GridSearchCV, ParameterGrid) from sklearn.cluster import MiniBatchKMeans -from sklearn.utils.fixes import loguniform +from scipy.stats import loguniform from scipy.stats import uniform from joblib import dump, load diff --git a/examples/Video_Door_state_Classification_randomized_search.py b/examples/Video_Door_state_Classification_randomized_search.py index 90418bf..285f183 100644 --- a/examples/Video_Door_state_Classification_randomized_search.py +++ b/examples/Video_Door_state_Classification_randomized_search.py @@ -6,8 +6,7 @@ from sklearn.metrics import make_scorer from sklearn.model_selection import RandomizedSearchCV -from sklearn.utils.fixes import loguniform -from scipy.stats import uniform +from scipy.stats import loguniform, uniform from joblib import dump, load @@ -102,7 +101,7 @@ def read_file(fname, Nfr=-1): kwargs_step0 = {'n_iter': 1000, 'random_state': 42, 'verbose': 1, 'n_jobs': 1, 'scoring': make_scorer(mean_squared_error, greater_is_better=False, - needs_proba=True)} + response_method='predict_proba')} base_esn = ESNClassifier(**initially_fixed_params) diff --git a/examples/digits-kmeans.py b/examples/digits-kmeans.py index 0dd5990..90aa6a1 100644 --- a/examples/digits-kmeans.py +++ b/examples/digits-kmeans.py @@ -14,7 +14,7 @@ from sklearn.decomposition import PCA from sklearn.cluster import KMeans -from src.pyrcn.util import get_mnist, tud_colors +from pyrcn.util import get_mnist matplotlib.rc('image', cmap='binary') @@ -92,7 +92,7 @@ def main(): labelbottom=False) ax_barchart.bar(list(map(int, values)), cos_similarity[:, i], - tick_label=values, color=tud_colors['lightblue']) + tick_label=values, color='tab:blue') # ax_barchart.set_xlim([0, 9]) ax_barchart.grid(which='both', axis='y') ax_barchart.set_yticks([-1., 0., 1.], minor=False) diff --git a/examples/digits.ipynb b/examples/digits.ipynb index ce48633..d4ccd9a 100644 --- a/examples/digits.ipynb +++ b/examples/digits.ipynb @@ -520,56 +520,6 @@ " print(f\"{esn_cv}\\t{t_fit}\\t{t_inference}\\t{acc_score}\\t{mem_size}\")" ] }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Alternatively, we can also use a PyTorch implementation of the ESN model" - ] - }, - { - "cell_type": "code", - "execution_count": 13, - "metadata": {}, - "outputs": [], - "source": [ - "from pyrcn.nn import ESN\n", - "import torch" - ] - }, - { - "cell_type": "code", - "execution_count": 11, - "metadata": {}, - "outputs": [], - "source": [ - "base_esn = ESN(input_size=8, hidden_size=50, num_layers=1, nonlinearity='tanh',\n", - " bias=True, input_scaling=0.016952130531190705,\n", - " spectral_radius=1.0214946051551315, bias_scaling=1.500499867548985,\n", - " bias_shift=0., input_sparsity=0.1, recurrent_sparsity=0.1,\n", - " bidirectional=False)" - ] - }, - { - "cell_type": "code", - "execution_count": 16, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "torch.Size([8, 50])\n" - ] - } - ], - "source": [ - "for x, y in zip(X_train, y_train):\n", - " x = torch.Tensor(x).float()\n", - " print(base_esn(x)[0].shape)\n", - " break" - ] - }, { "cell_type": "code", "execution_count": null, diff --git a/examples/digits.py b/examples/digits.py index 869669a..746db50 100644 --- a/examples/digits.py +++ b/examples/digits.py @@ -106,7 +106,7 @@ # searches that we have defined before. It can be combined with any model # selection tool from # scikit-learn. -sequential_search = SequentialSearchCV(base_esn,searches=searches).fit( +sequential_search = SequentialSearchCV(base_esn, searches=searches).fit( X_tr, y_tr) diff --git a/examples/esn_impulse_responses.ipynb b/examples/esn_impulse_responses.ipynb index 8ce4974..1ed694d 100644 --- a/examples/esn_impulse_responses.ipynb +++ b/examples/esn_impulse_responses.ipynb @@ -34,8 +34,6 @@ "sns.set_theme(context=\"talk\")\n", "# sns.set(font=\"Times New Roman\")\n", "from pyrcn.base.blocks import InputToNode, NodeToNode\n", - "from input_to_node import SimpleInputToNode\n", - "from node_to_node import DLRBNodeToNode\n", "from sklearn.pipeline import Pipeline\n", "\n", "%matplotlib notebook" diff --git a/examples/experiments.py b/examples/experiments.py index 60fa458..bb2b73f 100644 --- a/examples/experiments.py +++ b/examples/experiments.py @@ -13,10 +13,20 @@ import matplotlib.pyplot as plt from matplotlib.colors import ListedColormap -from pyrcn.util import new_logger, argument_parser, get_mnist, tud_colors +from pyrcn.util import new_logger, argument_parser, get_mnist train_size = 60000 +# Local RGBA palette (matplotlib tab10 colors). +palette = { + 'lightblue': (0.12, 0.47, 0.71, 1.0), + 'orange': (1.0, 0.5, 0.05, 1.0), + 'lightgreen': (0.17, 0.63, 0.17, 1.0), + 'lightpurple': (0.58, 0.40, 0.74, 1.0), + 'gray': (0.5, 0.5, 0.5, 1.0), + 'red': (0.84, 0.15, 0.16, 1.0), +} + def images_filter(images, kernel, stride=1): filtered = np.zeros(images.shape) @@ -133,7 +143,7 @@ def plot_confusion(directory): color_array = np.zeros((n_colorsteps, 4)) lower_margin = 255 color_array[:lower_margin, :] += np.linspace( - start=tud_colors['lightgreen'], stop=tud_colors['red'], + start=palette['lightgreen'], stop=palette['red'], num=lower_margin) cm = ListedColormap(color_array) diff --git a/examples/f0_extraction.py b/examples/f0_extraction.py index c7e0d5c..a03babb 100644 --- a/examples/f0_extraction.py +++ b/examples/f0_extraction.py @@ -5,8 +5,7 @@ from sklearn.preprocessing import StandardScaler from sklearn.utils import shuffle -from sklearn.utils.fixes import loguniform -from scipy.stats import uniform +from scipy.stats import uniform, loguniform from sklearn.base import clone from sklearn.pipeline import Pipeline from sklearn.model_selection import (ParameterGrid, RandomizedSearchCV, diff --git a/examples/input-to-node.py b/examples/input-to-node.py index ef9c641..d926b4c 100644 --- a/examples/input-to-node.py +++ b/examples/input-to-node.py @@ -7,7 +7,7 @@ from sklearn.decomposition import PCA from pyrcn.base.blocks import InputToNode -from pyrcn.util import tud_colors, get_mnist +from pyrcn.util import get_mnist import matplotlib.pyplot as plt import seaborn as sns @@ -57,10 +57,10 @@ def input2node_distribution(directory): if activation == 'bounded_relu': ax.hist(node_out, label=activation, density=True, - bins=[.0, .1, .9, 1.], color=tud_colors['lightblue']) + bins=[.0, .1, .9, 1.], color="tab:blue") else: ax.hist(node_out, label=activation, density=True, bins=20, - color=tud_colors['lightblue']) + color="tab:blue") ax.grid(axis='y') ax.set_yscale('log') diff --git a/examples/mackey-glass-t17.py b/examples/mackey-glass-t17.py index 6577761..d1546d2 100644 --- a/examples/mackey-glass-t17.py +++ b/examples/mackey-glass-t17.py @@ -162,9 +162,10 @@ fig, axs = plt.subplots(1, 2, sharey=True) -sns.heatmap(data=esn.hidden_layer_state[:100, :].T, ax=axs[0], cbar=False) +sns.heatmap(data=esn.hidden_layer_state(X_test)[:100, :].T, ax=axs[0], + cbar=False) axs[0].set_xlabel("Time Step") axs[0].set_ylabel("Neuron Index") -sns.heatmap(data=elm.hidden_layer_state[:100, :].T, ax=axs[1]) +sns.heatmap(data=elm.hidden_layer_state(X_test)[:100, :].T, ax=axs[1]) axs[1].set_xlabel("Time Step") plt.show() diff --git a/examples/mnist_regressors.py b/examples/mnist_regressors.py index f654f6f..2f147aa 100644 --- a/examples/mnist_regressors.py +++ b/examples/mnist_regressors.py @@ -10,7 +10,7 @@ from sklearn.model_selection import ( RandomizedSearchCV, GridSearchCV, ParameterGrid, cross_validate) -from sklearn.utils.fixes import loguniform +from scipy.stats import loguniform from sklearn.metrics import accuracy_score from pyrcn.model_selection import SequentialSearchCV diff --git a/examples/musical_note_prediction.ipynb b/examples/musical_note_prediction.ipynb index 8033def..96594bd 100644 --- a/examples/musical_note_prediction.ipynb +++ b/examples/musical_note_prediction.ipynb @@ -29,8 +29,7 @@ "from sklearn.metrics import make_scorer\n", "from sklearn.preprocessing import MultiLabelBinarizer\n", "\n", - "from sklearn.utils.fixes import loguniform\n", - "from scipy.stats import uniform\n", + "from scipy.stats import uniform, loguniform\n", "\n", "from pyrcn.echo_state_network import ESNClassifier\n", "from pyrcn.model_selection import SequentialSearchCV\n", @@ -135,16 +134,16 @@ "\n", "kwargs_step1 = {'n_iter': 200, 'random_state': 42, 'verbose': 1, 'n_jobs': -1,\n", " 'scoring': make_scorer(mean_squared_error, greater_is_better=False,\n", - " needs_proba=True)}\n", + " response_method='predict_proba')}\n", "kwargs_step2 = {'n_iter': 50, 'random_state': 42, 'verbose': 1, 'n_jobs': -1,\n", " 'scoring': make_scorer(mean_squared_error, greater_is_better=False,\n", - " needs_proba=True)}\n", + " response_method='predict_proba')}\n", "kwargs_step3 = {'verbose': 1, 'n_jobs': -1,\n", - " 'scoring': make_scorer(mean_squared_error,greater_is_better=False,\n", - " needs_proba=True)}\n", + " 'scoring': make_scorer(mean_squared_error, greater_is_better=False,\n", + " response_method='predict_proba')}\n", "kwargs_step4 = {'n_iter': 50, 'random_state': 42, 'verbose': 1, 'n_jobs': -1,\n", " 'scoring': make_scorer(mean_squared_error, greater_is_better=False,\n", - " needs_proba=True)}\n", + " response_method='predict_proba')}\n", "\n", "# The searches are defined similarly to the steps of a sklearn.pipeline.Pipeline:\n", "searches = [('step1', RandomizedSearchCV, step1_esn_params, kwargs_step1),\n", diff --git a/examples/musical_note_prediction.py b/examples/musical_note_prediction.py index efde155..ba38de7 100644 --- a/examples/musical_note_prediction.py +++ b/examples/musical_note_prediction.py @@ -24,8 +24,7 @@ from sklearn.metrics import make_scorer from sklearn.preprocessing import MultiLabelBinarizer -from sklearn.utils.fixes import loguniform -from scipy.stats import uniform +from scipy.stats import uniform, loguniform from pyrcn.echo_state_network import ESNClassifier from pyrcn.model_selection import SequentialSearchCV @@ -93,22 +92,22 @@ kwargs_step1 = { 'n_iter': 200, 'random_state': 42, 'verbose': 1, 'n_jobs': -1, 'scoring': make_scorer(mean_squared_error, greater_is_better=False, - needs_proba=True) + response_method='predict_proba') } kwargs_step2 = { 'n_iter': 50, 'random_state': 42, 'verbose': 1, 'n_jobs': -1, 'scoring': make_scorer(mean_squared_error, greater_is_better=False, - needs_proba=True) + response_method='predict_proba') } kwargs_step3 = { 'verbose': 1, 'n_jobs': -1, 'scoring': make_scorer(mean_squared_error, greater_is_better=False, - needs_proba=True) + response_method='predict_proba') } kwargs_step4 = { 'n_iter': 50, 'random_state': 42, 'verbose': 1, 'n_jobs': -1, 'scoring': make_scorer(mean_squared_error, greater_is_better=False, - needs_proba=True) + response_method='predict_proba') } searches = [('step1', RandomizedSearchCV, step1_esn_params, kwargs_step1), diff --git a/examples/preprocessing-mnist.py b/examples/preprocessing-mnist.py index 9d511bc..b459f9d 100644 --- a/examples/preprocessing-mnist.py +++ b/examples/preprocessing-mnist.py @@ -12,11 +12,21 @@ from sklearn.cluster import KMeans -from pyrcn.util import tud_colors, new_logger, get_mnist +from pyrcn.util import new_logger, get_mnist import matplotlib.pyplot as plt from matplotlib.colors import Normalize +# Local RGBA palette (matplotlib tab10 colors). +palette = { + 'lightblue': (0.12, 0.47, 0.71, 1.0), + 'orange': (1.0, 0.5, 0.05, 1.0), + 'lightgreen': (0.17, 0.63, 0.17, 1.0), + 'lightpurple': (0.58, 0.40, 0.74, 1.0), + 'gray': (0.5, 0.5, 0.5, 1.0), + 'red': (0.84, 0.15, 0.16, 1.0), +} + example_image_idx = 5 min_var = 3088.6875 @@ -148,8 +158,8 @@ def plot_historgram(directory, *args, **kwargs): idx_fringe = (25, 17) idx_center = (13, 12) - example[idx_center[0], idx_center[1], :] = tud_colors['lightblue'][:-1] - example[idx_fringe[0], idx_fringe[1], :] = tud_colors['orange'][:-1] + example[idx_center[0], idx_center[1], :] = palette['lightblue'][:-1] + example[idx_fringe[0], idx_fringe[1], :] = palette['orange'][:-1] bins = np.array(range(0, 287, 32)).astype(int) @@ -171,12 +181,12 @@ def plot_historgram(directory, *args, **kwargs): axs[0].set_yticklabels([0, 27]) axs[1].bar(bins[1:] - 32, height=hist_fringe / 1000, width=16, - color=tud_colors['orange'], label='fringe', align='edge') + color=palette['orange'], label='fringe', align='edge') axs[1].bar(bins[1:] - 16, height=hist_center / 1000, width=16, - color=tud_colors['lightblue'], label='center', align='edge') + color=palette['lightblue'], label='center', align='edge') axs[1].tick_params(axis='x', labelrotation=90) - # axs[1].hist([], bins=range(0, 255, 32), color=[tud_colors['orange'], - # tud_colors['lightblue']], + # axs[1].hist([], bins=range(0, 255, 32), color=[palette['orange'], + # palette['lightblue']], # align='left') axs[1].set_xticks(bins) @@ -216,15 +226,15 @@ def plot_var(directory, *args, **kwargs): (28, 28)) / 255. # blue for idx in pos: - example[idx, idx, :] = tud_colors['orange'][:-1] + example[idx, idx, :] = palette['orange'][:-1] meanX.append(scaler.mean_[idx * 28 + idx]) varX.append(scaler.var_[idx * 28 + idx]) axs[0].imshow(example, interpolation='none') - line_var, = axs[1].plot(pos, varX, color=tud_colors['orange']) + line_var, = axs[1].plot(pos, varX, color=palette['orange']) ax_mean = axs[1].twinx() - line_mean, = ax_mean.plot(pos, meanX, color=tud_colors['lightblue']) + line_mean, = ax_mean.plot(pos, meanX, color=palette['lightblue']) axs[1].legend((line_var, line_mean), (r'$\sigma^2$', r'$\mu$'), bbox_to_anchor=(1.2, .5), loc="center left") @@ -269,11 +279,11 @@ def plot_image_min_var(directory, *args, **kwargs): var_p1_2 = 255 ** 2 * p1_2 * (1 - p1_2) example_min_var_p1_1 = np.copy(example) - example_min_var_p1_1[scaler.var_ < var_p1_1, ...] = tud_colors['orange'][ + example_min_var_p1_1[scaler.var_ < var_p1_1, ...] = palette['orange'][ :-1] example_min_var_p1_2 = np.copy(example) - example_min_var_p1_2[scaler.var_ < var_p1_2, ...] = tud_colors['orange'][ + example_min_var_p1_2[scaler.var_ < var_p1_2, ...] = palette['orange'][ :-1] fig, axs = plt.subplots(1, 3, figsize=(5, 2)) @@ -480,7 +490,7 @@ def plot_imbalance(directory): ax.set_ylim([0, 8000]) ax.set_yticks([7000], minor=True) ax.grid(which='minor', axis='y', alpha=.7, linestyle='--', - color=tud_colors['lightgreen']) + color=palette['lightgreen']) ax.set_ylabel(r'\#occurrences') ax.spines['top'].set_visible(False) @@ -605,8 +615,8 @@ def plot_img_cluster(directory, *args, **kwargs): clusterer = KMeans(n_clusters=4, random_state=42) img_clusters = clusterer.fit_predict(img.reshape((784, 1))).reshape( (28, 28)) - list_cluster_colors = [tud_colors['lightblue'], tud_colors['lightgreen'], - tud_colors['lightpurple'], tud_colors['gray']] + list_cluster_colors = [palette['lightblue'], palette['lightgreen'], + palette['lightpurple'], palette['gray']] img_cluster_colors = np.zeros((28, 28, 4)) diff --git a/examples/setup_local.ipynb b/examples/setup_local.ipynb index 0198810..ab6cc0a 100644 --- a/examples/setup_local.ipynb +++ b/examples/setup_local.ipynb @@ -90,7 +90,7 @@ "from sklearn.preprocessing import LabelBinarizer\n", "from sklearn.model_selection import train_test_split\n", "\n", - "from pyrcn.extreme_learning_machine import ELMRegressor\n", + "from pyrcn.extreme_learning_machine import ELMClassifier\n", "\n", "\n", "def test_iris():\n", @@ -132,4 +132,4 @@ }, "nbformat": 4, "nbformat_minor": 4 -} +} \ No newline at end of file diff --git a/examples/sklearn_autoencoder.ipynb b/examples/sklearn_autoencoder.ipynb index 42ba545..f5fcbf5 100644 --- a/examples/sklearn_autoencoder.ipynb +++ b/examples/sklearn_autoencoder.ipynb @@ -26,7 +26,7 @@ "import numpy as np\n", "from sklearn.metrics import make_scorer\n", "from sklearn.model_selection import GridSearchCV, RandomizedSearchCV\n", - "from sklearn.utils.fixes import loguniform\n", + "from scipy.stats import loguniform\n", "from scipy.stats import uniform\n", "from joblib import dump, load\n", "\n", @@ -47,7 +47,7 @@ "plt.rc('ytick', labelsize=8)\n", "plt.rc('axes', labelsize=8)\n", "\n", - "from IPython.display import set_matplotlib_formats\n", + "from matplotlib_inline.backend_inline import set_matplotlib_formats\n", "set_matplotlib_formats('png', 'pdf')\n", "from mpl_toolkits.axes_grid1 import make_axes_locatable" ] diff --git a/examples/spoken_digit_recognition.ipynb b/examples/spoken_digit_recognition.ipynb index a037419..8e5e01c 100644 --- a/examples/spoken_digit_recognition.ipynb +++ b/examples/spoken_digit_recognition.ipynb @@ -33,10 +33,9 @@ "from sklearn.metrics import make_scorer, ConfusionMatrixDisplay\n", "from sklearn.cluster import MiniBatchKMeans\n", "from sklearn.utils import shuffle\n", - "from sklearn.utils.fixes import loguniform\n", "from scipy.stats import uniform\n", "from joblib import Parallel, delayed, dump, load\n", - "from pyrcn.echo_state_network import SeqToLabelESNClassifier\n", + "from pyrcn.echo_state_network import ESNClassifier\n", "from pyrcn.base.blocks import PredefinedWeightsInputToNode, NodeToNode\n", "from pyrcn.metrics import accuracy_score, classification_report, confusion_matrix\n", "from pyrcn.model_selection import SequentialSearchCV\n", @@ -51,7 +50,7 @@ "plt.rc('ytick', labelsize=8)\n", "plt.rc('axes', labelsize=8)\n", "\n", - "from IPython.display import set_matplotlib_formats\n", + "from matplotlib_inline.backend_inline import set_matplotlib_formats\n", "set_matplotlib_formats('png', 'pdf')\n", "from mpl_toolkits.axes_grid1 import make_axes_locatable\n", "from matplotlib import ticker\n", @@ -317,8 +316,7 @@ " 'k_rec': 10,\n", " 'reservoir_activation': 'tanh',\n", " 'bidirectional': False,\n", - " 'wash_out': 0,\n", - " 'continuation': False,\n", + " 'washout': 0,\n", " 'alpha': 1e-3,\n", " 'random_state': 42}\n", "\n", @@ -340,7 +338,7 @@ " ('step3', GridSearchCV, step3_esn_params, kwargs_step3),\n", " ('step4', RandomizedSearchCV, step4_esn_params, kwargs_step4)]\n", "\n", - "base_esn = SeqToLabelESNClassifier(**initially_fixed_params)\n", + "base_esn = ESNClassifier(**initially_fixed_params)\n", "\n", "try:\n", " sequential_search = load(\"../sequential_search_fsdd.joblib\")\n", @@ -6285,8 +6283,7 @@ " 'k_rec': 10,\n", " 'reservoir_activation': 'tanh',\n", " 'bidirectional': False,\n", - " 'wash_out': 0,\n", - " 'continuation': False,\n", + " 'washout': 0,\n", " 'alpha': 1e-3,\n", " 'random_state': 42}\n", "\n", @@ -6308,7 +6305,7 @@ " ('step3', GridSearchCV, step3_esn_params, kwargs_step3),\n", " ('step4', RandomizedSearchCV, step4_esn_params, kwargs_step4)]\n", "\n", - "base_km_esn = SeqToLabelESNClassifier(input_to_node=PredefinedWeightsInputToNode(predefined_input_weights=w_in.T),\n", + "base_km_esn = ESNClassifier(input_to_node=PredefinedWeightsInputToNode(predefined_input_weights=w_in.T),\n", " **initially_fixed_params)\n", "\n", "try:\n", diff --git a/examples/spoken_digit_recognition_auto_encoder.ipynb b/examples/spoken_digit_recognition_auto_encoder.ipynb index 67457bd..e85f133 100644 --- a/examples/spoken_digit_recognition_auto_encoder.ipynb +++ b/examples/spoken_digit_recognition_auto_encoder.ipynb @@ -27,7 +27,7 @@ "from sklearn.cluster import MiniBatchKMeans, Birch\n", "from sklearn.utils import shuffle\n", "from joblib import Parallel, delayed, dump, load\n", - "from pyrcn.echo_state_network import SeqToLabelESNClassifier\n", + "from pyrcn.echo_state_network import ESNClassifier\n", "from pyrcn.base.blocks import PredefinedWeightsInputToNode, NodeToNode\n", "from pyrcn.metrics import accuracy_score, classification_report, confusion_matrix\n", "from pyrcn.model_selection import SequentialSearchCV\n", @@ -41,7 +41,7 @@ "plt.rc('ytick', labelsize=8)\n", "plt.rc('axes', labelsize=8)\n", "\n", - "from IPython.display import set_matplotlib_formats\n", + "from matplotlib_inline.backend_inline import set_matplotlib_formats\n", "set_matplotlib_formats('png', 'pdf')\n", "from mpl_toolkits.axes_grid1 import make_axes_locatable\n", "from matplotlib import ticker\n", @@ -293,7 +293,7 @@ " 'activation': 'relu',\n", " 'random_state': 42}\n", "\n", - "mlp_enc = MLPRegressor(hidden_layer_sizes=(50, ) activation='relu', random_state=42).fit(np.concatenate(X_train_scaled), np.concatenate(X_train_scaled))" + "mlp_enc = MLPRegressor(hidden_layer_sizes=(50, ), activation='relu', random_state=42).fit(np.concatenate(X_train_scaled), np.concatenate(X_train_scaled))" ] }, { @@ -443,8 +443,7 @@ " 'k_rec': 10,\n", " 'reservoir_activation': 'tanh',\n", " 'bidirectional': False,\n", - " 'wash_out': 0,\n", - " 'continuation': False,\n", + " 'washout': 0,\n", " 'alpha': 1e-3,\n", " 'random_state': 42}\n", "\n", @@ -465,7 +464,7 @@ "\n", "w_in = np.divide(mlp_enc.coefs_[0], np.linalg.norm(mlp_enc.coefs_[0], axis=0)[None, :])\n", "\n", - "base_esn = SeqToLabelESNClassifier(input_to_node=PredefinedWeightsInputToNode(predefined_input_weights=w_in),\n", + "base_esn = ESNClassifier(input_to_node=PredefinedWeightsInputToNode(predefined_input_weights=w_in),\n", " **initially_fixed_params)\n", "\n", "try:\n", diff --git a/pyproject.toml b/pyproject.toml index 3c584cf..4550b58 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -25,10 +25,10 @@ classifiers = [ "Operating System :: OS Independent", ] dependencies = [ + "torch>=2.0", "scikit-learn>=1.6", "numpy>=1.18.1", "scipy>=1.4.0", - "joblib>=0.13.2", "pandas>=1.0.0", ] diff --git a/src/pyrcn/__init__.py b/src/pyrcn/__init__.py index 9cb1a6c..019a8a7 100644 --- a/src/pyrcn/__init__.py +++ b/src/pyrcn/__init__.py @@ -5,8 +5,8 @@ from __future__ import annotations from . import (base, echo_state_network, extreme_learning_machine, - linear_model, model_selection, postprocessing, preprocessing, - projection, util) + linear_model, model_selection, nn, postprocessing, + preprocessing, projection, util) from ._version import __version__ __all__ = ('__version__', @@ -15,6 +15,7 @@ 'extreme_learning_machine', 'linear_model', 'model_selection', + 'nn', 'postprocessing', 'preprocessing', 'projection', diff --git a/src/pyrcn/base/__init__.py b/src/pyrcn/base/__init__.py index c9a69bf..90938d8 100644 --- a/src/pyrcn/base/__init__.py +++ b/src/pyrcn/base/__init__.py @@ -6,8 +6,8 @@ References ---------- - .. [#] P. Steiner et al., ‘PyRCN: A Toolbox for Exploration and Application - of Reservoir Computing Networks’, under review. +.. [#] P. Steiner et al., ‘PyRCN: A Toolbox for Exploration and + Application of Reservoir Computing Networks’, under review. """ from __future__ import annotations diff --git a/src/pyrcn/base/_activations.py b/src/pyrcn/base/_activations.py index e45f134..7a1bead 100644 --- a/src/pyrcn/base/_activations.py +++ b/src/pyrcn/base/_activations.py @@ -61,39 +61,6 @@ def inplace_relu(X: np.ndarray) -> None: np.maximum(X, 0, out=X) -def inplace_softplus(X: np.ndarray) -> None: - """ - Compute the softplux activation function inplace: - .. math:: - f(x) = \\mathrm{ln}(1 + e^{x}) - - Parameters - ---------- - X : numpy.ndarray - - beta : float, default=1 - Scaling factor - """ - np.log(1 + np.exp(X, out=X), out=X) - - -def inplace_softmax(X: np.ndarray, beta: float = 1) -> None: - """ - Compute the softmax activation function inplace: - .. math:: - y_k = \\frac{e^{x_k}}{\\sum_{i=1}^{n} e^{x_i}} - - Parameters - ---------- - X : numpy.ndarray - - beta : float, default=1 - Scaling factor - """ - denominator = np.sum(np.exp(beta*X)) - np.divide(np.exp(beta*X, out=X), denominator, out=X) - - def inplace_bounded_relu(X: np.ndarray) -> None: """ Compute the bounded rectified linear unit function inplace. @@ -181,8 +148,6 @@ def inplace_bounded_relu_inverse(X: np.ndarray) -> None: 'logistic': inplace_logistic, 'relu': inplace_relu, 'bounded_relu': inplace_bounded_relu, - 'softmax': inplace_softmax, - 'softplus': inplace_softplus, } ACTIVATIONS_INVERSE: dict[str, Callable] = { diff --git a/src/pyrcn/base/blocks/_input_to_node.py b/src/pyrcn/base/blocks/_input_to_node.py index e358111..36478d6 100644 --- a/src/pyrcn/base/blocks/_input_to_node.py +++ b/src/pyrcn/base/blocks/_input_to_node.py @@ -37,15 +37,17 @@ class InputToNode(TransformerMixin, BaseEstimator): input_activation : Literal['tanh', 'identity', 'logistic', 'relu', 'bounded_relu'], default = 'tanh' This element represents the activation function in the hidden layer. - - 'identity', no-op activation, useful to implement linear - bottleneck, returns f(x) = x - - 'logistic', the logistic sigmoid function, - returns f(x) = 1/(1+exp(-x)). - - 'tanh', the hyperbolic tan function, returns f(x) = tanh(x). - - 'relu', the rectified linear unit function, - returns f(x) = max(0, x) - - 'bounded_relu', the bounded rectified linear unit function, - returns f(x) = min(max(x, 0),1) + + - 'identity', no-op activation, useful to implement linear + bottleneck, returns f(x) = x + - 'logistic', the logistic sigmoid function, + returns f(x) = 1/(1+exp(-x)). + - 'tanh', the hyperbolic tan function, returns f(x) = tanh(x). + - 'relu', the rectified linear unit function, + returns f(x) = max(0, x) + - 'bounded_relu', the bounded rectified linear unit function, + returns f(x) = min(max(x, 0),1) + input_scaling : float, default = 1. Scales the input weight matrix. input_shift : float, default = 0. @@ -113,7 +115,9 @@ def fit(self, X: np.ndarray, y: None = None) -> InputToNode: validate_data(self, X) if self.k_in is not None: self.sparsity = float(self.k_in) / float(X.shape[1]) - fan_in = int(np.rint(self.hidden_layer_size * self.sparsity)) + fan_in = self.k_in # k_in inputs per node + else: + fan_in = int(np.rint(self.hidden_layer_size * self.sparsity)) if self.predefined_input_weights is not None: assert self.predefined_input_weights.shape == ( self.n_features_in_, self.hidden_layer_size) @@ -205,9 +209,10 @@ def _validate_hyperparameters(self) -> None: raise ValueError("sparsity must be between 0. and 1., got {}." .format(self.sparsity)) if self.input_activation not in ACTIVATIONS: - raise ValueError("The activation_function '{}' is not supported." - "Supported activations are {}." - .format(self.input_activation, ACTIVATIONS)) + raise ValueError( + "The activation function '{}' is not supported. Supported " + "activations are {}.".format( + self.input_activation, sorted(ACTIVATIONS))) if self.input_scaling <= 0.: raise ValueError("input_scaling must be > 0, got {}." .format(self.input_scaling)) @@ -274,15 +279,17 @@ class PredefinedWeightsInputToNode(InputToNode): input_activation : Literal['tanh', 'identity', 'logistic', 'relu', 'bounded_relu'], default = 'tanh' This element represents the activation function in the hidden layer. - - 'identity', no-op activation, useful to implement linear - bottleneck, returns f(x) = x - - 'logistic', the logistic sigmoid function, - returns f(x) = 1/(1+exp(-x)). - - 'tanh', the hyperbolic tan function, returns f(x) = tanh(x). - - 'relu', the rectified linear unit function, - returns f(x) = max(0, x) - - 'bounded_relu', the bounded rectified linear unit function, - returns f(x) = min(max(x, 0),1) + + - 'identity', no-op activation, useful to implement linear + bottleneck, returns f(x) = x + - 'logistic', the logistic sigmoid function, + returns f(x) = 1/(1+exp(-x)). + - 'tanh', the hyperbolic tan function, returns f(x) = tanh(x). + - 'relu', the rectified linear unit function, + returns f(x) = max(0, x) + - 'bounded_relu', the bounded rectified linear unit function, + returns f(x) = min(max(x, 0),1) + input_scaling : float, default = 1. Scales the input weight matrix. input_shift : float, default = 0. @@ -348,15 +355,17 @@ class BatchIntrinsicPlasticity(InputToNode): input_activation : Literal['tanh', 'identity', 'logistic', 'relu', 'bounded_relu'], default = 'tanh' This element represents the activation function in the hidden layer. - - 'identity', no-op activation, useful to implement linear - bottleneck, returns f(x) = x - - 'logistic', the logistic sigmoid function, - returns f(x) = 1/(1+exp(-x)). - - 'tanh', the hyperbolic tan function, returns f(x) = tanh(x). - - 'relu', the rectified linear unit function, - returns f(x) = max(0, x) - - 'bounded_relu', the bounded rectified linear unit function, - returns f(x) = min(max(x, 0),1) + + - 'identity', no-op activation, useful to implement linear + bottleneck, returns f(x) = x + - 'logistic', the logistic sigmoid function, + returns f(x) = 1/(1+exp(-x)). + - 'tanh', the hyperbolic tan function, returns f(x) = tanh(x). + - 'relu', the rectified linear unit function, + returns f(x) = max(0, x) + - 'bounded_relu', the bounded rectified linear unit function, + returns f(x) = min(max(x, 0),1) + hidden_layer_size : int, default=500 Sets the number of nodes in hidden layer. Equals number of output features. diff --git a/src/pyrcn/base/blocks/_node_to_node.py b/src/pyrcn/base/blocks/_node_to_node.py index 70c3938..dc475a7 100644 --- a/src/pyrcn/base/blocks/_node_to_node.py +++ b/src/pyrcn/base/blocks/_node_to_node.py @@ -35,15 +35,17 @@ class NodeToNode(TransformerMixin, BaseEstimator): reservoir_activation : Literal['tanh', 'identity', 'logistic', 'relu', 'bounded_relu'], default = 'tanh' This element represents the activation function in the hidden layer. - - 'identity', no-op activation, useful to implement linear - bottleneck, returns f(x) = x - - 'logistic', the logistic sigmoid function, - returns f(x) = 1/(1+exp(-x)). - - 'tanh', the hyperbolic tan function, returns f(x) = tanh(x). - - 'relu', the rectified linear unit function, - returns f(x) = max(0, x) - - 'bounded_relu', the bounded rectified linear unit function, - returns f(x) = min(max(x, 0),1) + + - 'identity', no-op activation, useful to implement linear + bottleneck, returns f(x) = x + - 'logistic', the logistic sigmoid function, + returns f(x) = 1/(1+exp(-x)). + - 'tanh', the hyperbolic tan function, returns f(x) = tanh(x). + - 'relu', the rectified linear unit function, + returns f(x) = max(0, x) + - 'bounded_relu', the bounded rectified linear unit function, + returns f(x) = min(max(x, 0),1) + spectral_radius : float, default = 1. Scales the recurrent weight matrix. leakage : float, default = 1. @@ -184,9 +186,10 @@ def _validate_hyperparameters(self) -> None: raise ValueError("sparsity must be between 0. and 1., got {}." .format(self.sparsity)) if self.reservoir_activation not in ACTIVATIONS: - raise ValueError("The activation_function {} is not supported. " - "Supported activations are {}." - .format(self.reservoir_activation, ACTIVATIONS)) + raise ValueError( + "The activation function '{}' is not supported. Supported " + "activations are {}.".format( + self.reservoir_activation, sorted(ACTIVATIONS))) if self.spectral_radius < 0.: raise ValueError("spectral_radius must be >= 0, got {}." .format(self.spectral_radius)) @@ -245,15 +248,17 @@ class EulerNodeToNode(NodeToNode): reservoir_activation : Literal['tanh', 'identity', 'logistic', 'relu', 'bounded_relu'], default = 'tanh' This element represents the activation function in the hidden layer. - - 'identity', no-op activation, useful to implement linear - bottleneck, returns f(x) = x - - 'logistic', the logistic sigmoid function, - returns f(x) = 1/(1+exp(-x)). - - 'tanh', the hyperbolic tan function, returns f(x) = tanh(x). - - 'relu', the rectified linear unit function, - returns f(x) = max(0, x) - - 'bounded_relu', the bounded rectified linear unit function, - returns f(x) = min(max(x, 0),1) + + - 'identity', no-op activation, useful to implement linear + bottleneck, returns f(x) = x + - 'logistic', the logistic sigmoid function, + returns f(x) = 1/(1+exp(-x)). + - 'tanh', the hyperbolic tan function, returns f(x) = tanh(x). + - 'relu', the rectified linear unit function, + returns f(x) = max(0, x) + - 'bounded_relu', the bounded rectified linear unit function, + returns f(x) = min(max(x, 0),1) + recurrent_scaling : float, default = 1. Scales the recurrent weight matrix. gamma : float, default = 0.001 @@ -375,15 +380,17 @@ class PredefinedWeightsNodeToNode(NodeToNode): reservoir_activation : Literal['tanh', 'identity', 'logistic', 'relu', 'bounded_relu'], default = 'tanh' This element represents the activation function in the hidden layer. - - 'identity', no-op activation, useful to implement linear - bottleneck, returns f(x) = x - - 'logistic', the logistic sigmoid function, - returns f(x) = 1/(1+exp(-x)). - - 'tanh', the hyperbolic tan function, returns f(x) = tanh(x). - - 'relu', the rectified linear unit function, - returns f(x) = max(0, x) - - 'bounded_relu', the bounded rectified linear unit function, - returns f(x) = min(max(x, 0),1) + + - 'identity', no-op activation, useful to implement linear + bottleneck, returns f(x) = x + - 'logistic', the logistic sigmoid function, + returns f(x) = 1/(1+exp(-x)). + - 'tanh', the hyperbolic tan function, returns f(x) = tanh(x). + - 'relu', the rectified linear unit function, + returns f(x) = max(0, x) + - 'bounded_relu', the bounded rectified linear unit function, + returns f(x) = min(max(x, 0),1) + spectral_radius : float, default = 1. Scales the recurrent weight matrix. leakage : float, default = 1. @@ -448,15 +455,17 @@ class HebbianNodeToNode(NodeToNode): reservoir_activation : Literal['tanh', 'identity', 'logistic', 'relu', 'bounded_relu'], default = 'tanh' This element represents the activation function in the hidden layer. - - 'identity', no-op activation, useful to implement linear - bottleneck, returns f(x) = x - - 'logistic', the logistic sigmoid function, - returns f(x) = 1/(1+exp(-x)). - - 'tanh', the hyperbolic tan function, returns f(x) = tanh(x). - - 'relu', the rectified linear unit function, - returns f(x) = max(0, x) - - 'bounded_relu', the bounded rectified linear unit function, - returns f(x) = min(max(x, 0),1) + + - 'identity', no-op activation, useful to implement linear + bottleneck, returns f(x) = x + - 'logistic', the logistic sigmoid function, + returns f(x) = 1/(1+exp(-x)). + - 'tanh', the hyperbolic tan function, returns f(x) = tanh(x). + - 'relu', the rectified linear unit function, + returns f(x) = max(0, x) + - 'bounded_relu', the bounded rectified linear unit function, + returns f(x) = min(max(x, 0),1) + spectral_radius : float, default = 1. Scales the recurrent weight matrix. leakage : float, default = 1. diff --git a/src/pyrcn/datasets/_base.py b/src/pyrcn/datasets/_base.py index a37b7e9..1b68192 100644 --- a/src/pyrcn/datasets/_base.py +++ b/src/pyrcn/datasets/_base.py @@ -64,6 +64,7 @@ def mackey_glass(n_timesteps: int, n_future: int = 1, tau: int = 17, Mackey-Glass timeseries [#]_ [#]_, computed from the Mackey-Glass delayed differential equation: + .. math:: \\frac{dx}{dt} = \\beta\\frac{x(t-\\tau)}{1+x(t-\\tau)^n}-\\gamma x(t) @@ -148,6 +149,7 @@ def lorenz(n_timesteps: int, n_future: int = 1, sigma: float = 10., Lorenz timeseries [#]_ [#]_, computed from the Lorenz delayed differential equation: + .. math:: \\frac{dx}{dt} = \\sigma(y - x) \\frac{dy}{dt} = x(\\rho - z) - y @@ -220,6 +222,7 @@ def load_digits(*, n_class: int | np.integer = 10, Load and return the digits dataset (classification). Each datapoint is a 8x8 image of a digit. + ================= ============== Classes 10 Samples per class ~180 @@ -227,7 +230,9 @@ def load_digits(*, n_class: int | np.integer = 10, Dimensionality 64 Features integers 0-16 ================= ============== - Read more in the :ref:`User Guide `. + + Read more in the scikit-learn User Guide (Optical recognition of + handwritten digits dataset). Parameters ---------- @@ -236,6 +241,7 @@ def load_digits(*, n_class: int | np.integer = 10, return_X_y : bool, default=False If True, returns ``(data, target)`` instead of a Bunch object. See below for more information about the `data` and `target` object. + .. versionadded:: 0.18 as_frame : bool, default=False If True, the data is a pandas DataFrame including columns with @@ -243,12 +249,14 @@ def load_digits(*, n_class: int | np.integer = 10, a pandas DataFrame or Series depending on the number of target columns. If `return_X_y` is True, then (`data`, `target`) will be pandas DataFrames or Series as described below. + .. versionadded:: 0.23 Returns ------- data : :class:`~sklearn.utils.Bunch` Dictionary-like object, with the following attributes. + data : {ndarray, dataframe} of shape (1797, 64) The flattened data matrix. If `as_frame=True`, `data` will be a pandas DataFrame. @@ -259,10 +267,12 @@ def load_digits(*, n_class: int | np.integer = 10, The names of the dataset columns. target_names: list The names of target classes. + .. versionadded:: 0.20 frame: DataFrame of shape (1797, 65) Only present when `as_frame=True`. DataFrame with `data` and `target`. + .. versionadded:: 0.23 images: {ndarray} of shape (1797, 8, 8) The raw image data. diff --git a/src/pyrcn/echo_state_network/_esn.py b/src/pyrcn/echo_state_network/_esn.py index c4d7915..57a3df7 100644 --- a/src/pyrcn/echo_state_network/_esn.py +++ b/src/pyrcn/echo_state_network/_esn.py @@ -6,16 +6,24 @@ from __future__ import annotations import sys -from typing import Any, Literal +from typing import Any, Literal, cast -from joblib import Parallel, delayed import numpy as np +import torch from sklearn.base import (BaseEstimator, ClassifierMixin, MultiOutputMixin, - RegressorMixin, clone, is_regressor) + RegressorMixin, is_regressor) from sklearn.exceptions import NotFittedError from sklearn.preprocessing import LabelBinarizer from sklearn.utils.validation import validate_data +from ..nn._bridge import ( + build_input_map, build_readout, build_reservoir, input_is_backable, + node_is_backable, regressor_is_backable) +from ..nn._input import InputFeatureMap +from ..nn._readout import IncrementalRidge, LinearReadout +from ..nn._reservoir import EulerReservoir, Reservoir +from ..nn._training import (LOSSES, OPTIMIZERS, torch_generator, + train_readout) from ..base.blocks import InputToNode, NodeToNode from ..linear_model import IncrementalRegression from ..projection import MatrixToValueProjection @@ -54,6 +62,32 @@ class ESNRegressor(RegressorMixin, MultiOutputMixin, BaseEstimator): default='winner_takes_all' Decision strategy for sequence-to-label task. Ignored if the target output is a sequence + washout : int, default=0 + Number of initial reservoir states (and matching targets) to drop + per sequence when fitting the readout, discarding the start + transient. Training-only; requires the torch backend. + solver : {"closed_form", "gradient"}, default="closed_form" + Readout training method. ``"closed_form"`` solves the ridge normal + equations (requires a fixed reservoir); ``"gradient"`` trains the + readout with an optimizer loop and enables the ``trainable_*`` flags. + optimizer : {"adam", "adamw", "sgd", "rmsprop", "adagrad"}, default="adam" + Optimizer used when ``solver="gradient"``. + learning_rate : float, default=1e-3 + Learning rate used when ``solver="gradient"``. + epochs : int, default=100 + Number of training epochs when ``solver="gradient"``. + batch_size : Optional[int], default=None + Mini-batch size when ``solver="gradient"`` (``None`` means full + batch). With a trainable reservoir/input, batches are over sequences. + loss : {"mse", "mae", "huber"}, default="mse" + Loss used when ``solver="gradient"``. + trainable_reservoir : bool, default=False + If True (requires ``solver="gradient"``), train the recurrent + reservoir weights by backpropagation through the recurrence. + trainable_input : bool, default=False + If True (requires ``solver="gradient"``), train the input + feature-map weights. Combine with ``trainable_reservoir`` for a + fully trainable RNN. verbose : bool = False Verbosity output kwargs : Any @@ -69,7 +103,16 @@ def __init__(self, *, requires_sequence: Literal["auto"] | bool = "auto", decision_strategy: Literal["winner_takes_all", "median", "last_value"] = "winner_takes_all", + washout: int = 0, verbose: bool = True, + solver: str = "closed_form", + optimizer: str = "adam", + learning_rate: float = 1e-3, + epochs: int = 100, + batch_size: int | None = None, + trainable_reservoir: bool = False, + trainable_input: bool = False, + loss: str = "mse", **kwargs: Any) -> None: """Construct the ESNRegressor.""" if input_to_node is None: @@ -104,47 +147,22 @@ def __init__(self, *, if key in reg_params}) self._regressor = self.regressor self._requires_sequence = requires_sequence + self.washout = washout self.verbose = verbose self.decision_strategy = decision_strategy - - def __add__(self, other: ESNRegressor) -> ESNRegressor: - """ - Sum up two instances of an ```ESNRegressor```. - - We always need to update the correlation matrices of the regressor. - - Parameters - ---------- - other : ESNRegressor - ```ESNRegressor``` to be added to ```self``` - - Returns - ------- - self : returns the sum of two ```ESNRegressor``` instances. - """ - self.regressor._K = self.regressor._K + other.regressor._K - self.regressor._xTy = self.regressor._xTy + other.regressor._xTy - return self - - def __radd__(self, other: ESNRegressor) -> ESNRegressor: - """ - Sum up multiple instances of an ```ESNRegressor```. - - We always need to update the correlation matrices of the regressor. - - Parameters - ---------- - other : ESNRegressor - ```ESNRegressor``` to be added to ```self``` - - Returns - ------- - self : returns the sum of two ```ESNRegressor``` instances. - """ - if other == 0: - return self - else: - return self.__add__(other) + self.solver = solver + self.optimizer = optimizer + self.learning_rate = learning_rate + self.epochs = epochs + self.batch_size = batch_size + self.trainable_reservoir = trainable_reservoir + self.trainable_input = trainable_input + self.loss = loss + self._use_torch: bool = False + self._target_1d: bool = False + self._torch_input_map: InputFeatureMap + self._torch_reservoir: Reservoir | EulerReservoir + self._torch_readout: IncrementalRidge | LinearReadout def get_params(self, deep: bool = True) -> dict: """Get all parameters of the ESNRegressor.""" @@ -156,7 +174,16 @@ def get_params(self, deep: bool = True) -> dict: return {"input_to_node": self.input_to_node, "node_to_node": self.node_to_node, "regressor": self.regressor, - "requires_sequence": self._requires_sequence} + "requires_sequence": self._requires_sequence, + "washout": self.washout, + "solver": self.solver, + "optimizer": self.optimizer, + "learning_rate": self.learning_rate, + "epochs": self.epochs, + "batch_size": self.batch_size, + "trainable_reservoir": self.trainable_reservoir, + "trainable_input": self.trainable_input, + "loss": self.loss} def set_params(self, **parameters: dict) -> ESNRegressor: """Set all possible parameters of the ESNRegressor.""" @@ -247,6 +274,7 @@ def partial_fit(self, X: np.ndarray, y: np.ndarray, self : Returns a trained ```ESNRegressor``` model. """ self._validate_hyperparameters() + self._use_torch = False validate_data(self, X=X, y=y, multi_output=True) # input_to_node @@ -295,7 +323,7 @@ def fit(self, X: np.ndarray, y: np.ndarray, n_jobs : Optional[int, np.integer], default=None The number of jobs to run in parallel. ```-1``` means using all processors. - See :term:`Glossary ` for more details. + See the scikit-learn glossary for n_jobs. transformer_weights : Optional[np.ndarray] = None ignored @@ -306,6 +334,36 @@ def fit(self, X: np.ndarray, y: np.ndarray, self._validate_hyperparameters() if self.requires_sequence == "auto": self._check_if_sequence(X, y) + self._target_1d = (np.asarray(y).ndim == 1) + backable = ( + input_is_backable(self._input_to_node) + and node_is_backable(self._node_to_node) + and regressor_is_backable(self._regressor)) + if self.solver == "gradient": + if not backable: + raise NotImplementedError( + "the gradient solver requires torch-backable " + "input_to_node, node_to_node and an " + "IncrementalRegression readout") + if self.requires_sequence: + X, y, sequence_ranges = concatenate_sequences(X, y) + self._input_to_node.fit(X) + self._node_to_node.fit(self._input_to_node.transform(X)) + else: + validate_data(self, X, y, multi_output=True) + self._input_to_node.fit(X) + self._node_to_node.fit(self._input_to_node.transform(X)) + self._build_torch_backend(torch.float64) + self._use_torch = True + ranges = sequence_ranges if self.requires_sequence else None + if self.trainable_reservoir or self.trainable_input: + return self._torch_trainable_fit(X, y, ranges) + return self._torch_gradient_fit(X, y, ranges) + self._use_torch = backable + if self.washout > 0 and not self._use_torch: + raise NotImplementedError( + "washout > 0 requires the torch backend (native " + "input_to_node / node_to_node / regressor)") if self.requires_sequence: X, y, sequence_ranges = concatenate_sequences(X, y) self._input_to_node.fit(X) @@ -314,12 +372,182 @@ def fit(self, X: np.ndarray, y: np.ndarray, validate_data(self, X, y, multi_output=True) self._input_to_node.fit(X) self._node_to_node.fit(self._input_to_node.transform(X)) - # self._regressor = self._regressor.__class__() + if self._use_torch: + self._build_torch_backend(torch.float64) + if self.requires_sequence: + return self._torch_sequence_fit(X, y, sequence_ranges) + states, _ = self._torch_states(X) + states = states[self.washout:] + ys = torch.as_tensor(np.asarray(y), dtype=torch.float64) + cast(IncrementalRidge, self._torch_readout).fit( + states, ys[self.washout:]) + return self if self.requires_sequence: return self._sequence_fit(X, y, sequence_ranges, n_jobs) else: return self.partial_fit(X, y, postpone_inverse=False) + def _build_torch_backend(self, dtype: torch.dtype) -> None: + """Build the torch backend modules from the fitted blocks.""" + self._torch_input_map = build_input_map( + self._input_to_node, dtype=dtype) + self._torch_reservoir = build_reservoir( + self._node_to_node, dtype=dtype) + self._torch_readout = build_readout(self._regressor, dtype=dtype) + + def _torch_states(self, seq: np.ndarray, + initial_state: np.ndarray | None = None + ) -> tuple[torch.Tensor, torch.Tensor]: + """Return ``(states (L, hidden*dir), final_state (hidden*dir,))``. + + ``initial_state`` seeds the reservoir (zeros by default). + """ + Z = self._torch_input_map( + torch.as_tensor(np.asarray(seq), dtype=torch.float64)) + if initial_state is None: + states, final = self._torch_reservoir(Z.unsqueeze(0)) + else: + init = torch.as_tensor( + np.asarray(initial_state), dtype=torch.float64).reshape(1, -1) + states, final = self._torch_reservoir(Z.unsqueeze(0), init) + return states.squeeze(0), final.squeeze(0) + + def _torch_sequence_fit(self, X_cat: np.ndarray, y_cat: np.ndarray, + sequence_ranges: np.ndarray) -> ESNRegressor: + """Accumulate the readout over sequences with a fresh zero state. + + The first ``washout`` states (and matching targets) of each sequence + are dropped so the initial transient does not train the readout. + """ + n_seq = len(sequence_ranges) + for i, (start, stop) in enumerate(sequence_ranges): + states, _ = self._torch_states(X_cat[start:stop]) + ys = torch.as_tensor( + np.asarray(y_cat[start:stop]), dtype=torch.float64) + states, ys = states[self.washout:], ys[self.washout:] + cast(IncrementalRidge, self._torch_readout).partial_fit( + states, ys, reset=(i == 0), + postpone_inverse=(i < n_seq - 1)) + return self + + def _torch_gradient_fit(self, X: np.ndarray, y: np.ndarray, + sequence_ranges: (np.ndarray | None) + ) -> ESNRegressor: + """Train a fresh ``LinearReadout`` on the fixed reservoir states. + + States are computed once through the frozen reservoir, dropping the + first ``washout`` states (and matching targets) per sequence, then a + ``LinearReadout`` is trained with an optimizer loop. + """ + if sequence_ranges is not None: + states_list = [] + y_list = [] + for start, stop in sequence_ranges: + states, _ = self._torch_states(X[start:stop]) + ys = torch.as_tensor( + np.asarray(y[start:stop]), dtype=torch.float64) + states_list.append(states[self.washout:]) + y_list.append(ys[self.washout:]) + all_states = torch.cat(states_list, dim=0) + all_y = torch.cat(y_list, dim=0) + else: + states, _ = self._torch_states(X) + all_states = states[self.washout:] + all_y = torch.as_tensor( + np.asarray(y), dtype=torch.float64)[self.washout:] + y2 = all_y.reshape(all_states.shape[0], -1) + gen = torch_generator(self._input_to_node.random_state) + readout = LinearReadout( + all_states.shape[1], y2.shape[1], + fit_intercept=self._regressor.fit_intercept, generator=gen, + dtype=torch.float64) + train_readout( + readout, all_states, y2, optimizer=self.optimizer, + learning_rate=self.learning_rate, epochs=self.epochs, + batch_size=self.batch_size, loss=self.loss, + weight_decay=self._regressor.alpha, generator=gen) + self._torch_readout = readout + return self + + def _torch_trainable_fit(self, X: np.ndarray, y: np.ndarray, + sequence_ranges: (np.ndarray | None) + ) -> ESNRegressor: + """Jointly train input/reservoir weights and the readout (BPTT). + + Whichever of the input feature map and the reservoir is marked + trainable is optimized together with the readout by backpropagating + through the recurrence; each epoch recomputes the states. A fixed + input feature map is applied once and detached (cheaper); a trainable + one is recomputed each step so gradients reach its weights. + Mini-batching (``batch_size``) is over sequences; ``None`` is + full-batch. In non-sequence mode there is a single sequence, so it is + always full-BPTT over that sequence. + """ + dtype = torch.float64 + train_input = self.trainable_input + if train_input: + self._torch_input_map.set_input_trainable(True) + if self.trainable_reservoir: + self._torch_reservoir.set_recurrent_trainable(True) + if sequence_ranges is not None: + segments = [(X[a:b], y[a:b]) for a, b in sequence_ranges] + else: + segments = [(X, y)] + inputs = [torch.as_tensor(np.asarray(xs), dtype=dtype) + for xs, _ in segments] + targets = [torch.as_tensor(np.asarray(ys), dtype=dtype)[self.washout:] + for _, ys in segments] + fixed_feats = (None if train_input + else [self._torch_input_map(x).detach() + for x in inputs]) + n_targets = 1 if targets[0].ndim == 1 else targets[0].shape[1] + gen = torch_generator(self._input_to_node.random_state) + with torch.no_grad(): + probe_feats = (self._torch_input_map(inputs[0]) + if fixed_feats is None else fixed_feats[0]) + probe, _ = self._torch_reservoir(probe_feats.unsqueeze(0)) + readout = LinearReadout( + probe.shape[-1], n_targets, + fit_intercept=self._regressor.fit_intercept, generator=gen, + dtype=dtype) + params: list = [] + if train_input: + params += [p for p in self._torch_input_map.parameters() + if p.requires_grad] + if self.trainable_reservoir: + params += [p for p in self._torch_reservoir.parameters() + if p.requires_grad] + params += list(readout.parameters()) + optimizer = OPTIMIZERS[self.optimizer]( + params, lr=self.learning_rate, weight_decay=self._regressor.alpha) + loss_fn = LOSSES[self.loss]() + n_seq = len(inputs) + step = n_seq if self.batch_size is None else min( + int(self.batch_size), n_seq) + readout.train() + for _ in range(int(self.epochs)): + order = torch.randperm(n_seq, generator=gen).tolist() + for begin in range(0, n_seq, step): + batch = order[begin:begin + step] + optimizer.zero_grad() + states_parts = [] + target_parts = [] + for j in batch: + feats = (self._torch_input_map(inputs[j]) + if fixed_feats is None else fixed_feats[j]) + states, _ = self._torch_reservoir(feats.unsqueeze(0)) + states_parts.append(states.squeeze(0)[self.washout:]) + target_parts.append(targets[j]) + predicted = readout(torch.cat(states_parts, dim=0)) + expected = torch.cat(target_parts, dim=0).reshape( + predicted.shape[0], -1) + loss = loss_fn(predicted, expected) + loss.backward() + optimizer.step() + readout.eval() + self._torch_readout = readout + return self + def _sequence_fit(self, X: np.ndarray, y: np.ndarray, sequence_ranges: np.ndarray, n_jobs: (int | np.integer | @@ -337,26 +565,18 @@ def _sequence_fit(self, X: np.ndarray, y: np.ndarray, n_jobs : Union[int, np.integer, None], default=None The number of jobs to run in parallel. ```-1``` means using all processors. - See :term:`Glossary ` for more details. + See the scikit-learn glossary for n_jobs. Returns ------- self : Returns a trained ESNRegressor model. """ - if n_jobs is not None and n_jobs > 1: - reg = Parallel(n_jobs=n_jobs)(delayed(ESNRegressor.partial_fit) - (clone(self), X[idx[0]:idx[1], ...], - y[idx[0]:idx[1], ...], - postpone_inverse=True) - for idx in sequence_ranges[:-1]) - reg = sum(reg) - self._regressor = reg._regressor - else: - [ESNRegressor.partial_fit(self, - X[idx[0]:idx[1], ...], - y[idx[0]:idx[1], ...], - postpone_inverse=True) - for idx in sequence_ranges[:-1]] + # n_jobs is accepted for API compatibility but ignored: the torch + # fast path batches this, and the numpy fallback runs serially. + for idx in sequence_ranges[:-1]: + ESNRegressor.partial_fit(self, X[idx[0]:idx[1], ...], + y[idx[0]:idx[1], ...], + postpone_inverse=True) # last sequence, calculate inverse and bias ESNRegressor.partial_fit(self, X=X[sequence_ranges[-1][0]:, ...], @@ -364,22 +584,59 @@ def _sequence_fit(self, X: np.ndarray, y: np.ndarray, postpone_inverse=False) return self - def predict(self, X: np.ndarray) -> np.ndarray: + def predict(self, X: np.ndarray, initial_state: np.ndarray | None = None, + return_state: bool = False + ) -> np.ndarray | tuple[np.ndarray, np.ndarray]: """ Predict the targets using the trained ```ESNRegressor```. Parameters ---------- X : ndarray of shape (n_samples, n_features) + initial_state : ndarray of shape (hidden_layer_size,), default=None + Reservoir state to start from (zeros by default). Applied to each + sequence in sequence mode. Requires the torch backend. + return_state : bool, default=False + If True, also return the final reservoir state(s). Requires the + torch backend. Returns ------- y : ndarray of (n_samples,) or (n_samples, n_targets) - The predicted targets + The predicted targets. If ``return_state`` is True, a tuple + ``(y, final_state)`` is returned instead. """ if self._input_to_node is None or self._regressor is None: raise NotFittedError(self) + if getattr(self, "_use_torch", False): + squeeze = getattr(self, "_target_1d", False) + # No grad in inference: a trainable reservoir has requires_grad + # weights, so its forward would otherwise track gradients. + with torch.no_grad(): + if self.requires_sequence is False: + states, final = self._torch_states(X, initial_state) + pred = self._torch_readout.predict(states) + if squeeze: + pred = pred.squeeze(-1) + y = pred.numpy() + return (y, final.numpy()) if return_state else y + y = np.empty(shape=X.shape, dtype=object) + finals = np.empty(shape=X.shape, dtype=object) + for k, seq in enumerate(X): + states, final = self._torch_states(seq, initial_state) + pred = self._torch_readout.predict(states) + if squeeze: + pred = pred.squeeze(-1) + y[k] = pred.numpy() + finals[k] = final.numpy() + return (y, finals) if return_state else y + + if initial_state is not None or return_state: + raise NotImplementedError( + "initial_state / return_state require the torch backend " + "(native input_to_node / node_to_node / regressor)") + if self.requires_sequence is False: # input_to_node hidden_layer_state = self._input_to_node.transform(X) @@ -421,12 +678,47 @@ def _validate_hyperparameters(self) -> None: raise ValueError('Invalid value for requires_sequence, got {}' .format(self._requires_sequence)) + if not isinstance(self.washout, int) or self.washout < 0: + raise ValueError('Invalid value for washout, got {}' + .format(self.washout)) + if not is_regressor(self._regressor): raise TypeError("The last step should be a regressor and " "implement fit and predict '{}' (type {})" "doesn't".format(self._regressor, type(self._regressor))) + if self.solver not in ("closed_form", "gradient"): + raise ValueError('Invalid value for solver, got {}' + .format(self.solver)) + + if self.optimizer not in OPTIMIZERS: + raise ValueError('Invalid value for optimizer, got {}' + .format(self.optimizer)) + + if self.loss not in LOSSES: + raise ValueError('Invalid value for loss, got {}' + .format(self.loss)) + + if (not isinstance(self.epochs, int) + or isinstance(self.epochs, bool) + or self.epochs <= 0): + raise ValueError('Invalid value for epochs, got {}' + .format(self.epochs)) + + if (not isinstance(self.learning_rate, (int, float)) + or isinstance(self.learning_rate, bool) + or self.learning_rate <= 0): + raise ValueError('Invalid value for learning_rate, got {}' + .format(self.learning_rate)) + + if ((self.trainable_reservoir or self.trainable_input) + and self.solver != "gradient"): + raise ValueError( + "trainable_reservoir / trainable_input require " + "solver='gradient' (trainable weights cannot use the " + "closed-form solver)") + def __sizeof__(self) -> int: """ Return the size of the object in bytes. @@ -642,6 +934,32 @@ class ESNClassifier(ClassifierMixin, ESNRegressor): default='winner_takes_all' Decision strategy for sequence-to-label task. Ignored if the target output is a sequence + washout : int, default=0 + Number of initial reservoir states (and matching targets) to drop + per sequence when fitting the readout, discarding the start + transient. Training-only; requires the torch backend. + solver : {"closed_form", "gradient"}, default="closed_form" + Readout training method. ``"closed_form"`` solves the ridge normal + equations (requires a fixed reservoir); ``"gradient"`` trains the + readout with an optimizer loop and enables the ``trainable_*`` flags. + optimizer : {"adam", "adamw", "sgd", "rmsprop", "adagrad"}, default="adam" + Optimizer used when ``solver="gradient"``. + learning_rate : float, default=1e-3 + Learning rate used when ``solver="gradient"``. + epochs : int, default=100 + Number of training epochs when ``solver="gradient"``. + batch_size : Optional[int], default=None + Mini-batch size when ``solver="gradient"`` (``None`` means full + batch). With a trainable reservoir/input, batches are over sequences. + loss : {"mse", "mae", "huber"}, default="mse" + Loss used when ``solver="gradient"``. + trainable_reservoir : bool, default=False + If True (requires ``solver="gradient"``), train the recurrent + reservoir weights by backpropagation through the recurrence. + trainable_input : bool, default=False + If True (requires ``solver="gradient"``), train the input + feature-map weights. Combine with ``trainable_reservoir`` for a + fully trainable RNN. verbose : bool = False Verbosity output kwargs : Any, default = None @@ -656,13 +974,26 @@ def __init__(self, *, requires_sequence: Literal["auto"] | bool = "auto", decision_strategy: Literal["winner_takes_all", "median", "last_value"] = "winner_takes_all", + washout: int = 0, verbose: bool = False, + solver: str = "closed_form", + optimizer: str = "adam", + learning_rate: float = 1e-3, + epochs: int = 100, + batch_size: int | None = None, + trainable_reservoir: bool = False, + trainable_input: bool = False, + loss: str = "mse", **kwargs: Any) -> None: """Construct the ESNClassifier.""" super().__init__(input_to_node=input_to_node, node_to_node=node_to_node, regressor=regressor, - requires_sequence=requires_sequence, verbose=verbose, - **kwargs) + requires_sequence=requires_sequence, washout=washout, + verbose=verbose, solver=solver, optimizer=optimizer, + learning_rate=learning_rate, epochs=epochs, + batch_size=batch_size, + trainable_reservoir=trainable_reservoir, + trainable_input=trainable_input, loss=loss, **kwargs) self._decision_strategy = decision_strategy self._encoder = LabelBinarizer() self._sequence_to_value = False @@ -720,7 +1051,7 @@ def fit(self, X: np.ndarray, y: np.ndarray, n_jobs : int, default=None The number of jobs to run in parallel. ```-1``` means using all processors. - See :term:`Glossary ` for more details. + See the scikit-learn glossary for n_jobs. transformer_weights : ignored Returns @@ -730,6 +1061,22 @@ def fit(self, X: np.ndarray, y: np.ndarray, self._validate_hyperparameters() if self.requires_sequence == "auto": self._check_if_sequence(X, y) + backable = ( + input_is_backable(self._input_to_node) + and node_is_backable(self._node_to_node) + and regressor_is_backable(self._regressor)) + if self.solver == "gradient": + if not backable: + raise NotImplementedError( + "the gradient solver requires torch-backable " + "input_to_node, node_to_node and an " + "IncrementalRegression readout") + else: + self._use_torch = backable + if self.washout > 0 and not self._use_torch: + raise NotImplementedError( + "washout > 0 requires the torch backend (native " + "input_to_node / node_to_node / regressor)") if self.requires_sequence: self._check_if_sequence_to_value(X, y) X, y, sequence_ranges = concatenate_sequences( @@ -742,14 +1089,33 @@ def fit(self, X: np.ndarray, y: np.ndarray, self._node_to_node.fit(self._input_to_node.transform(X)) self._encoder = LabelBinarizer().fit(y) y = self._encoder.transform(y) - # self._regressor = self._regressor.__class__() + self._target_1d = (np.asarray(y).ndim == 1) + if self.solver == "gradient": + self._build_torch_backend(torch.float64) + self._use_torch = True + ranges = sequence_ranges if self.requires_sequence else None + if self.trainable_reservoir or self.trainable_input: + return self._torch_trainable_fit(X, y, ranges) + return self._torch_gradient_fit(X, y, ranges) + if self._use_torch: + self._build_torch_backend(torch.float64) + if self.requires_sequence: + return self._torch_sequence_fit(X, y, sequence_ranges) + states, _ = self._torch_states(X) + states = states[self.washout:] + ys = torch.as_tensor(np.asarray(y), dtype=torch.float64) + cast(IncrementalRidge, self._torch_readout).fit( + states, ys[self.washout:]) + return self if self.requires_sequence: return self._sequence_fit(X, y, sequence_ranges, n_jobs) else: super().partial_fit(X, y) return self - def predict(self, X: np.ndarray) -> np.ndarray: + def predict(self, X: np.ndarray, initial_state: np.ndarray | None = None, + return_state: bool = False + ) -> np.ndarray | tuple[np.ndarray, np.ndarray]: """ Predict the classes using the trained ```ESNClassifier```. @@ -757,13 +1123,21 @@ def predict(self, X: np.ndarray) -> np.ndarray: ---------- X : ndarray of shape (n_samples, n_features) The input data. + initial_state : ndarray of shape (hidden_layer_size,), default=None + Reservoir state to start from (zeros by default). Requires the + torch backend. + return_state : bool, default=False + Not supported for classifiers. Returns ------- y_pred : ndarray of shape (n_samples,) or (n_samples, n_classes) The predicted classes. """ - y = super().predict(X) + if return_state: + raise NotImplementedError( + "return_state is not supported for classifiers") + y = cast(np.ndarray, super().predict(X, initial_state=initial_state)) if self.requires_sequence and self._sequence_to_value: for k, _ in enumerate(y): y[k] = MatrixToValueProjection( @@ -775,8 +1149,7 @@ def predict(self, X: np.ndarray) -> np.ndarray: y[k] = self._encoder.inverse_transform(y[k], threshold=None) return y else: - return self._encoder.inverse_transform(super().predict(X), - threshold=None) + return self._encoder.inverse_transform(y, threshold=None) def predict_proba(self, X: np.ndarray) -> np.ndarray: """ @@ -792,7 +1165,7 @@ def predict_proba(self, X: np.ndarray) -> np.ndarray: y_pred : ndarray of shape (n_samples,) or (n_samples, n_classes) The predicted probability estimates. """ - y = super().predict(X) + y = cast(np.ndarray, super().predict(X)) if self.requires_sequence and self._sequence_to_value: for k, _ in enumerate(y): y[k] = MatrixToValueProjection( diff --git a/src/pyrcn/extreme_learning_machine/_elm.py b/src/pyrcn/extreme_learning_machine/_elm.py index 8c345db..05a3e12 100644 --- a/src/pyrcn/extreme_learning_machine/_elm.py +++ b/src/pyrcn/extreme_learning_machine/_elm.py @@ -10,14 +10,19 @@ import sys from typing import Any -from joblib import Parallel, delayed import numpy as np +import torch from sklearn.base import (BaseEstimator, ClassifierMixin, MultiOutputMixin, - RegressorMixin, clone, is_regressor) + RegressorMixin, is_regressor) from sklearn.exceptions import NotFittedError from sklearn.preprocessing import LabelBinarizer from sklearn.utils.validation import validate_data +from ..nn._bridge import ( + build_input_map, build_readout, input_is_backable, regressor_is_backable) +from ..nn._input import InputFeatureMap +from ..nn._readout import IncrementalRidge, LinearReadout +from ..nn._training import LOSSES, OPTIMIZERS, torch_generator, train_readout from ..base.blocks import InputToNode from ..linear_model import IncrementalRegression @@ -44,6 +49,19 @@ class ELMRegressor(RegressorMixin, MultiOutputMixin, BaseEstimator): chunk_size : Optional[int], default=None if X.shape[0] > chunk_size, calculate results incrementally with partial_fit + solver : {"closed_form", "gradient"}, default="closed_form" + Readout training method. ``"closed_form"`` solves the ridge normal + equations; ``"gradient"`` trains the readout with an optimizer loop. + optimizer : {"adam", "adamw", "sgd", "rmsprop", "adagrad"}, default="adam" + Optimizer used when ``solver="gradient"``. + learning_rate : float, default=1e-3 + Learning rate used when ``solver="gradient"``. + epochs : int, default=100 + Number of training epochs when ``solver="gradient"``. + batch_size : Optional[int], default=None + Mini-batch size when ``solver="gradient"`` (``None`` = full batch). + loss : {"mse", "mae", "huber"}, default="mse" + Loss used when ``solver="gradient"``. verbose : bool = False Verbosity output kwargs : Any, default = None @@ -57,6 +75,12 @@ def __init__(self, *, RegressorMixin | None) = None, chunk_size: int | None = None, verbose: bool = False, + solver: str = "closed_form", + optimizer: str = "adam", + learning_rate: float = 1e-3, + epochs: int = 100, + batch_size: int | None = None, + loss: str = "mse", **kwargs: Any) -> None: """Construct the ELMRegressor.""" if input_to_node is None: @@ -82,45 +106,16 @@ def __init__(self, *, self._regressor = self.regressor self._chunk_size = chunk_size self.verbose = verbose - - def __add__(self, other: ELMRegressor) -> ELMRegressor: - """ - Sum up two instances of an ```ELMRegressor```. - - We always need to update the correlation matrices of the regressor. - - Parameters - ---------- - other : ELMRegressor - ```ELMRegressor``` to be added to ```self``` - - Returns - ------- - self : returns the sum of two ```ELMRegressor``` instances. - """ - self.regressor._K = self.regressor._K + other.regressor._K - self.regressor._xTy = self.regressor._xTy + other.regressor._xTy - return self - - def __radd__(self, other: ELMRegressor) -> ELMRegressor: - """ - Sum up multiple instances of an ```ELMRegressor```. - - We always need to update the correlation matrices of the regressor. - - Parameters - ---------- - other : ELMRegressor - ```ELMRegressor``` to be added to ```self``` - - Returns - ------- - self : returns the sum of multiple ```ELMRegressor``` instances. - """ - if other == 0: - return self - else: - return self.__add__(other) + self.solver = solver + self.optimizer = optimizer + self.learning_rate = learning_rate + self.epochs = epochs + self.batch_size = batch_size + self.loss = loss + self._use_torch: bool = False + self._target_1d: bool = False + self._torch_input_map: InputFeatureMap + self._torch_readout: IncrementalRidge | LinearReadout def get_params(self, deep: bool = True) -> dict: """Get all parameters of the ESNRegressor.""" @@ -130,7 +125,13 @@ def get_params(self, deep: bool = True) -> dict: else: return {"input_to_node": self.input_to_node, "regressor": self.regressor, - "chunk_size": self.chunk_size} + "chunk_size": self.chunk_size, + "solver": self.solver, + "optimizer": self.optimizer, + "learning_rate": self.learning_rate, + "epochs": self.epochs, + "batch_size": self.batch_size, + "loss": self.loss} def set_params(self, **parameters: dict) -> ELMRegressor: """Set all possible parameters of the ELMRegressor.""" @@ -177,6 +178,7 @@ def partial_fit(self, X: np.ndarray, y: np.ndarray, f"got {self._regressor}") self._validate_hyperparameters() validate_data(self, X, y, multi_output=True) + self._use_torch = False # input_to_node try: @@ -207,7 +209,7 @@ def fit(self, X: np.ndarray, y: np.ndarray, n_jobs : int, default=None The number of jobs to run in parallel. ```-1``` means using all processors. - See :term:`Glossary ` for more details. + See the scikit-learn glossary for n_jobs. transformer_weights : Union[np.ndarray, None], default=None ignored @@ -218,7 +220,50 @@ def fit(self, X: np.ndarray, y: np.ndarray, self._validate_hyperparameters() validate_data(self, X, y, multi_output=True) + self._target_1d = (np.asarray(y).ndim == 1) + backable = (input_is_backable(self._input_to_node) + and regressor_is_backable(self._regressor)) + + if self.solver == "gradient": + if not backable: + raise NotImplementedError( + "the gradient solver requires a torch-backable " + "input_to_node and an IncrementalRegression readout") + self._input_to_node.fit(X) + dtype = torch.float64 + gen = torch_generator(self._input_to_node.random_state) + fm = build_input_map(self._input_to_node, dtype=dtype) + feats = fm(torch.as_tensor(np.asarray(X), dtype=dtype)) + y2 = torch.as_tensor( + np.asarray(y), dtype=dtype).reshape(feats.shape[0], -1) + lin_readout = LinearReadout( + feats.shape[1], y2.shape[1], + fit_intercept=self._regressor.fit_intercept, + generator=gen, dtype=dtype) + train_readout( + lin_readout, feats, y2, optimizer=self.optimizer, + learning_rate=self.learning_rate, epochs=self.epochs, + batch_size=self.batch_size, loss=self.loss, + weight_decay=self._regressor.alpha, generator=gen) + self._torch_input_map = fm + self._torch_readout = lin_readout + self._use_torch = True + return self + self._input_to_node.fit(X) + self._use_torch = False + + if (input_is_backable(self._input_to_node) + and regressor_is_backable(self._regressor)): + dtype = torch.float64 + fm = build_input_map(self._input_to_node, dtype=dtype) + Z = fm(torch.as_tensor(np.asarray(X), dtype=dtype)) + readout = build_readout(self._regressor, dtype=dtype) + readout.fit(Z, torch.as_tensor(np.asarray(y), dtype=dtype)) + self._torch_input_map = fm + self._torch_readout = readout + self._use_torch = True + return self if self._chunk_size is None or self._chunk_size >= X.shape[0]: # input_to_node @@ -230,23 +275,14 @@ def fit(self, X: np.ndarray, y: np.ndarray, elif self._chunk_size < X.shape[0]: # setup chunk list chunks = list(range(0, X.shape[0], self._chunk_size)) - # postpone inverse calculation for chunks n-1 - if n_jobs is None or n_jobs < 2: - [ELMRegressor.partial_fit( + # n_jobs is accepted for API compatibility but ignored; chunks + # are accumulated serially (the torch fast path batches instead). + for idx in chunks[:-1]: + ELMRegressor.partial_fit( self, X[idx:idx + self._chunk_size, ...], y[idx:idx + self._chunk_size, ...], transformer_weights=transformer_weights, postpone_inverse=True) - for idx in chunks[:-1]] - else: - reg = Parallel(n_jobs=n_jobs)( - delayed(ELMRegressor.partial_fit) - (clone(self), X[idx:idx + self._chunk_size, ...], - y[idx:idx + self._chunk_size, ...], - transformer_weights=transformer_weights, - postpone_inverse=True) for idx in chunks[:-1]) - reg = sum(reg) - self._regressor = reg._regressor # last chunk, calculate inverse and bias ELMRegressor.partial_fit(self, X=X[chunks[-1]:, ...], y=y[chunks[-1]:, ...], @@ -269,6 +305,14 @@ def predict(self, X: np.ndarray) -> np.ndarray: y : ndarray of (n_samples,) or (n_samples, n_targets) The predicted targets """ + if getattr(self, "_use_torch", False): + feats = self._torch_input_map( + torch.as_tensor(np.asarray(X), dtype=torch.float64)) + pred = self._torch_readout.predict(feats) + if getattr(self, "_target_1d", False): + pred = pred.squeeze(-1) + return pred.numpy() + hidden_layer_state = self._input_to_node.transform(X) return self._regressor.predict(hidden_layer_state) @@ -295,6 +339,30 @@ def _validate_hyperparameters(self) -> None: "doesn't".format(self._regressor, type(self._regressor))) + if self.solver not in ("closed_form", "gradient"): + raise ValueError('Invalid value for solver, got {}' + .format(self.solver)) + + if self.optimizer not in OPTIMIZERS: + raise ValueError('Invalid value for optimizer, got {}' + .format(self.optimizer)) + + if self.loss not in LOSSES: + raise ValueError('Invalid value for loss, got {}' + .format(self.loss)) + + if (not isinstance(self.epochs, int) + or isinstance(self.epochs, bool) + or self.epochs <= 0): + raise ValueError('Invalid value for epochs, got {}' + .format(self.epochs)) + + if (not isinstance(self.learning_rate, (int, float)) + or isinstance(self.learning_rate, bool) + or self.learning_rate <= 0): + raise ValueError('Invalid value for learning_rate, got {}' + .format(self.learning_rate)) + def __sizeof__(self) -> int: """ Return the size of the object in bytes. @@ -417,6 +485,19 @@ class ELMClassifier(ClassifierMixin, ELMRegressor): chunk_size : Optional[int], default=None if X.shape[0] > chunk_size, calculate results incrementally with partial_fit + solver : {"closed_form", "gradient"}, default="closed_form" + Readout training method. ``"closed_form"`` solves the ridge normal + equations; ``"gradient"`` trains the readout with an optimizer loop. + optimizer : {"adam", "adamw", "sgd", "rmsprop", "adagrad"}, default="adam" + Optimizer used when ``solver="gradient"``. + learning_rate : float, default=1e-3 + Learning rate used when ``solver="gradient"``. + epochs : int, default=100 + Number of training epochs when ``solver="gradient"``. + batch_size : Optional[int], default=None + Mini-batch size when ``solver="gradient"`` (``None`` = full batch). + loss : {"mse", "mae", "huber"}, default="mse" + Loss used when ``solver="gradient"``. verbose : bool = False Verbosity output kwargs : Any, default = None @@ -429,10 +510,19 @@ def __init__(self, *, regressor: (IncrementalRegression | RegressorMixin | None) = None, chunk_size: int | None = None, verbose: bool = False, + solver: str = "closed_form", + optimizer: str = "adam", + learning_rate: float = 1e-3, + epochs: int = 100, + batch_size: int | None = None, + loss: str = "mse", **kwargs: Any) -> None: """Construct the ELMClassifier.""" super().__init__(input_to_node=input_to_node, regressor=regressor, - chunk_size=chunk_size, verbose=verbose, **kwargs) + chunk_size=chunk_size, verbose=verbose, + solver=solver, optimizer=optimizer, + learning_rate=learning_rate, epochs=epochs, + batch_size=batch_size, loss=loss, **kwargs) self._encoder = LabelBinarizer() def partial_fit(self, X: np.ndarray, y: np.ndarray, @@ -489,7 +579,7 @@ def fit(self, X: np.ndarray, y: np.ndarray, n_jobs : Union[int, np.integer, None], default=None The number of jobs to run in parallel. ```-1``` means using all processors. - See :term:`Glossary ` for more details. + See the scikit-learn glossary for n_jobs. transformer_weights : Optional[np.ndarray], default=None ignored diff --git a/src/pyrcn/metrics/_classification.py b/src/pyrcn/metrics/_classification.py index 9f78222..3a6bbb3 100644 --- a/src/pyrcn/metrics/_classification.py +++ b/src/pyrcn/metrics/_classification.py @@ -67,13 +67,11 @@ def _check_targets(y_true: np.ndarray, y_pred: np.ndarray, def accuracy_score(y_true: np.ndarray, y_pred: np.ndarray, *, normalize: bool = True, sample_weight: np.ndarray | None = None) -> float: - """ - Accuracy classification score. + """Accuracy classification score. In multilabel classification, this function computes subset accuracy: - the set of labels predicted for a sample must *exactly* match the + the set of labels predicted for a sample must exactly match the corresponding set of labels in y_true. - Read more in the :ref:`User Guide `. Parameters ---------- @@ -121,16 +119,13 @@ def confusion_matrix(y_true: np.ndarray, y_pred: np.ndarray, *, sample_weight: np.ndarray | None = None, normalize: Literal["true", "predicted"] | None = None)\ -> np.ndarray: - """ - Compute confusion matrix to evaluate the accuracy of a classification. + """Compute confusion matrix to evaluate classification accuracy. - By definition a confusion matrix :math:`C` is such that :math:`C_{i, j}` - is equal to the number of observations known to be in group :math:`i` and - predicted to be in group :math:`j`. - Thus in binary classification, the count of true negatives is - :math:`C_{0,0}`, false negatives is :math:`C_{1,0}`, true positives is - :math:`C_{1,1}` and false positives is :math:`C_{0,1}`. - Read more in the :ref:`User Guide `. + By definition a confusion matrix C is such that C[i, j] is equal to the + number of observations known to be in group i and predicted to be in + group j. Thus in binary classification, the count of true negatives is + C[0, 0], false negatives is C[1, 0], true positives is C[1, 1] and false + positives is C[0, 1]. Parameters ---------- @@ -153,21 +148,9 @@ def confusion_matrix(y_true: np.ndarray, y_pred: np.ndarray, *, Returns ------- C : ndarray of shape (n_classes, n_classes) - Confusion matrix whose i-th row and j-th - column entry indicates the number of - samples with true label being i-th class - and predicted label being j-th class. - - See Also - -------- - plot_confusion_matrix : Plot Confusion Matrix. - ConfusionMatrixDisplay : Confusion Matrix visualization. - References - ---------- - .. [1] `Wikipedia entry for the Confusion matrix - `_ - (Wikipedia and other references may use a different - convention for axes). + Confusion matrix whose i-th row and j-th column entry indicates the + number of samples with true label being i-th class and predicted + label being j-th class. """ y_type, y_true, y_pred, sample_weight = _check_targets( y_true, y_pred, sample_weight) @@ -184,21 +167,16 @@ def multilabel_confusion_matrix(y_true: np.ndarray, y_pred: np.ndarray, *, sample_weight: np.ndarray | None = None, labels: np.ndarray | None = None, samplewise: bool = False) -> np.ndarray: - """ - Compute a confusion matrix for each class or sample. + """Compute a confusion matrix for each class or sample. - .. versionadded:: 0.21 Compute class-wise (default) or sample-wise (samplewise=True) multilabel - confusion matrix to evaluate the accuracy of a classification, and output - confusion matrices for each class or sample. - In multilabel confusion matrix :math:`MCM`, the count of true negatives - is :math:`MCM_{:,0,0}`, false negatives is :math:`MCM_{:,1,0}`, - true positives is :math:`MCM_{:,1,1}` and false positives is - :math:`MCM_{:,0,1}`. - Multiclass data will be treated as if binarized under a one-vs-rest - transformation. Returned confusion matrices will be in the order of - sorted unique labels in the union of (y_true, y_pred). - Read more in the :ref:`User Guide `. + confusion matrix to evaluate classification accuracy, and output + confusion matrices for each class or sample. In the multilabel confusion + matrix MCM, the count of true negatives is MCM[:, 0, 0], false negatives + is MCM[:, 1, 0], true positives is MCM[:, 1, 1] and false positives is + MCM[:, 0, 1]. Multiclass data are treated as if binarized under a + one-vs-rest transformation. Returned confusion matrices are in the order + of sorted unique labels in the union of (y_true, y_pred). Parameters ---------- @@ -223,12 +201,8 @@ def multilabel_confusion_matrix(y_true: np.ndarray, y_pred: np.ndarray, *, When calculating class-wise multi_confusion (default), then n_outputs = n_labels; when calculating sample-wise multi_confusion (samplewise=True), n_outputs = n_samples. If ``labels`` is defined, - the results will be returned in the order specified in ``labels``, - otherwise the results will be returned in sorted order by default - - See Also - -------- - confusion_matrix + the results are returned in the order specified in ``labels``, + otherwise the results are returned in sorted order by default. Notes ----- @@ -252,20 +226,15 @@ def cohen_kappa_score(y1: np.ndarray, y2: np.ndarray, *, labels: np.ndarray | None = None, weights: Literal["linear", "quadratic"] | None = None, sample_weight: np.ndarray | None = None) -> float: - r""" - Cohen' kappa: a statistic that measures inter-annotator agreement. + """Cohen's kappa: a statistic that measures inter-annotator agreement. - This function computes Cohen's kappa [1]_, a score that expresses the level + This function computes Cohen's kappa, a score that expresses the level of agreement between two annotators on a classification problem. It is - defined as - .. math:: - \kappa = (p_o - p_e) / (1 - p_e) - where :math:`p_o` is the empirical probability of agreement on the label - assigned to any sample (the observed agreement ratio), and :math:`p_e` is - the expected agreement when both annotators assign labels randomly. - :math:`p_e` is estimated using a per-annotator empirical prior over the - class labels [2]_. - Read more in the :ref:`User Guide `. + defined as kappa = (p_o - p_e) / (1 - p_e), where p_o is the empirical + probability of agreement on the label assigned to any sample (the + observed agreement ratio), and p_e is the expected agreement when both + annotators assign labels randomly. p_e is estimated using a + per-annotator empirical prior over the class labels. Parameters ---------- @@ -289,17 +258,6 @@ class labels [2]_. kappa : float The kappa statistic, which is a number between -1 and 1. The maximum value means complete agreement; zero or lower means chance agreement. - - References - ---------- - .. [1] J. Cohen (1960). "A coefficient of agreement for nominal scales". - Educational and Psychological Measurement 20(1):37-46. - doi:10.1177/001316446002000104. - .. [2] `R. Artstein and M. Poesio (2008). "Inter-coder agreement for - computational linguistics". Computational Linguistics 34(4):555-596 - `_. - .. [3] `Wikipedia entry for the Cohen's kappa - `_. """ y_type, y1, y2, sample_weight = _check_targets(y1, y2, sample_weight) if sample_weight is not None: @@ -319,14 +277,12 @@ def jaccard_score(y_true: np.ndarray, y_pred: np.ndarray, *, sample_weight: np.ndarray | None = None, zero_division: Literal["warn", 0, 1] = "warn")\ -> float | np.ndarray: - """ - Jaccard similarity coefficient score. + """Jaccard similarity coefficient score. - The Jaccard index [1], or Jaccard similarity coefficient, defined as - the size of the intersection divided by the size of the union of two label - sets, is used to compare set of predicted labels for a sample to the + The Jaccard index, or Jaccard similarity coefficient, defined as the size + of the intersection divided by the size of the union of two label sets, + is used to compare the set of predicted labels for a sample to the corresponding set of labels in ``y_true``. - Read more in the :ref:`User Guide `. Parameters ---------- @@ -347,53 +303,39 @@ def jaccard_score(y_true: np.ndarray, y_pred: np.ndarray, *, If the data are multiclass or multilabel, this will be ignored; setting ``labels=[pos_label]`` and ``average != 'binary'`` will report scores for that label only. - average : Optional[Literal['micro', 'macro', 'samples', 'weighted', - 'binary']], default='binary' - If ``None``, the scores for each class are returned. Otherwise, this - determines the type of averaging performed on the data: - ``'binary'``: - Only report results for the class specified by ``pos_label``. - This is applicable only if targets (``y_{true,pred}``) are binary. - ``'micro'``: - Calculate metrics globally by counting the total true positives, - false negatives and false positives. - ``'macro'``: - Calculate metrics for each label, and find their unweighted - mean. This does not take label imbalance into account. - ``'weighted'``: - Calculate metrics for each label, and find their average, weighted - by support (the number of true instances for each label). This - alters 'macro' to account for label imbalance. - ``'samples'``: - Calculate metrics for each instance, and find their average (only - meaningful for multilabel classification). + average : str or None, default='binary' + One of {'micro', 'macro', 'samples', 'weighted', 'binary'} or None. + If None, the scores for each class are returned. Otherwise, this + determines the type of averaging performed on the data. With + 'binary', only results for the class specified by ``pos_label`` are + reported (applicable only if targets are binary). With 'micro', + metrics are calculated globally by counting the total true positives, + false negatives and false positives. With 'macro', metrics are + calculated for each label and their unweighted mean is taken, which + does not take label imbalance into account. With 'weighted', metrics + are calculated for each label and averaged, weighted by support (the + number of true instances for each label), which accounts for label + imbalance. With 'samples', metrics are calculated for each instance + and averaged (only meaningful for multilabel classification). sample_weight : Optional[np.ndarray], default=None Sample weights. - zero_division : Literal["warn", 0.0, 1.0], default="warn" - Sets the value to return when there is a zero division, i.e. when there + zero_division : {"warn", 0, 1}, default="warn" + Sets the value to return when there is a zero division, i.e. when there are no negative values in predictions and labels. If set to "warn", this acts like 0, but a warning is also raised. Returns ------- - score : float (if average is not None) or array of floats, shape = - [n_unique_labels] - - See Also - -------- - accuracy_score, f_score, multilabel_confusion_matrix + score : float or ndarray of floats + A float if average is not None, otherwise an array of shape + (n_unique_labels,). Notes ----- - :func:`jaccard_score` may be a poor metric if there are no - positives for some samples or classes. Jaccard is undefined if there are - no true or predicted labels, and our implementation will return a score - of 0 with a warning. - - References - ---------- - .. [1] `Wikipedia entry for the Jaccard index - `_. + The Jaccard score may be a poor metric if there are no positives for + some samples or classes. Jaccard is undefined if there are no true or + predicted labels, and this implementation returns a score of 0 with a + warning. """ y_type, y_true, y_pred, sample_weight = _check_targets( y_true, y_pred, sample_weight) @@ -409,21 +351,18 @@ def jaccard_score(y_true: np.ndarray, y_pred: np.ndarray, *, def matthews_corrcoef(y_true: np.ndarray, y_pred: np.ndarray, *, sample_weight: np.ndarray | None = None,) -> float: - """ - Compute the Matthews correlation coefficient (MCC). - - The Matthews correlation coefficient is used in machine learning as a - measure of the quality of binary and multiclass classifications. It takes - into account true and false positives and negatives and is generally - regarded as a balanced measure which can be used even if the classes are of - very different sizes. The MCC is in essence a correlation coefficient value - between -1 and +1. A coefficient of +1 represents a perfect prediction, 0 - an average random prediction and -1 an inverse prediction. The statistic - is also known as the phi coefficient. [source: Wikipedia] - Binary and multiclass labels are supported. Only in the binary case does - this relate to information about true and false positives and negatives. - See references below. - Read more in the :ref:`User Guide `. + """Compute the Matthews correlation coefficient (MCC). + + The Matthews correlation coefficient is used as a measure of the quality + of binary and multiclass classifications. It takes into account true and + false positives and negatives and is generally regarded as a balanced + measure which can be used even if the classes are of very different + sizes. The MCC is in essence a correlation coefficient value between -1 + and +1. A coefficient of +1 represents a perfect prediction, 0 an + average random prediction and -1 an inverse prediction. The statistic is + also known as the phi coefficient. Binary and multiclass labels are + supported. Only in the binary case does this relate to information about + true and false positives and negatives. Parameters ---------- @@ -438,23 +377,8 @@ def matthews_corrcoef(y_true: np.ndarray, y_pred: np.ndarray, *, ------- mcc : float The Matthews correlation coefficient (+1 represents a perfect - prediction, 0 an average random prediction and -1 and inverse + prediction, 0 an average random prediction and -1 an inverse prediction). - - References - ---------- - .. [1] `Baldi, Brunak, Chauvin, Andersen and Nielsen, (2000). Assessing the - accuracy of prediction algorithms for classification: an overview - `_. - .. [2] `Wikipedia entry for the Matthews Correlation Coefficient - `_. - .. [3] `Gorodkin, (2004). Comparing two K-category assignments by a - K-category correlation coefficient - `_. - .. [4] `Jurman, Riccadonna, Furlanello, (2012). A Comparison of MCC and CEN - Error Measures in MultiClass Prediction - `_. """ y_type, y_true, y_pred, sample_weight = _check_targets( y_true, y_pred, sample_weight) @@ -469,13 +393,11 @@ def matthews_corrcoef(y_true: np.ndarray, y_pred: np.ndarray, *, def zero_one_loss(y_true: np.ndarray, y_pred: np.ndarray, *, normalize: bool = True, sample_weight: np.ndarray | None = None) -> float: - """ - Zero-one classification loss. + """Zero-one classification loss. If normalize is ``True``, return the fraction of misclassifications - (float), else it returns the number of misclassifications (int). The best + (float), else return the number of misclassifications (int). The best performance is 0. - Read more in the :ref:`User Guide `. Parameters ---------- @@ -491,19 +413,16 @@ def zero_one_loss(y_true: np.ndarray, y_pred: np.ndarray, *, Returns ------- - loss : float or int, + loss : float or int If ``normalize == True``, return the fraction of misclassifications - (float), else it returns the number of misclassifications (int). + (float), else return the number of misclassifications (int). Notes ----- In multilabel classification, the zero_one_loss function corresponds to - the subset zero-one loss: for each sample, the entire set of labels must be - correctly predicted, otherwise the loss for that sample is equal to one. - - See Also - -------- - accuracy_score, hamming_loss, jaccard_score + the subset zero-one loss: for each sample, the entire set of labels must + be correctly predicted, otherwise the loss for that sample is equal to + one. """ y_type, y_true, y_pred, sample_weight = _check_targets( y_true, y_pred, sample_weight) @@ -523,18 +442,15 @@ def f1_score(y_true: np.ndarray, y_pred: np.ndarray, *, sample_weight: np.ndarray | None = None, zero_division: Literal["warn", 0, 1] = "warn")\ -> float | np.ndarray: - """ - Compute the F1 score, also known as balanced F-score or F-measure. - - The F1 score can be interpreted as a weighted average of the precision and - recall, where an F1 score reaches its best value at 1 and worst score at 0. - The relative contribution of precision and recall to the F1 score are - equal. The formula for the F1 score is:: - F1 = 2 * (precision * recall) / (precision + recall) - In the multi-class and multi-label case, this is the average of - the F1 score of each class with weighting depending on the ``average`` - parameter. - Read more in the :ref:`User Guide `. + """Compute the F1 score, also known as balanced F-score or F-measure. + + The F1 score can be interpreted as a weighted average of precision and + recall, where an F1 score reaches its best value at 1 and worst score at + 0. The relative contribution of precision and recall to the F1 score are + equal. The formula for the F1 score is + F1 = 2 * (precision * recall) / (precision + recall). In the multi-class + and multi-label case, this is the average of the F1 score of each class + with weighting depending on the ``average`` parameter. Parameters ---------- @@ -550,36 +466,27 @@ def f1_score(y_true: np.ndarray, y_pred: np.ndarray, *, result in 0 components in a macro average. For multilabel targets, labels are column indices. By default, all labels in ``y_true`` and ``y_pred`` are used in sorted order. - .. versionchanged:: 0.17 - Parameter `labels` improved for multiclass problem. pos_label : Union[str, int], default=1 The class to report if ``average='binary'`` and the data is binary. If the data are multiclass or multilabel, this will be ignored; setting ``labels=[pos_label]`` and ``average != 'binary'`` will report scores for that label only. - average : Optional[Literal['micro', 'macro', 'samples','weighted', - 'binary']], default='binary' + average : str or None, default='binary' + One of {'micro', 'macro', 'samples', 'weighted', 'binary'} or None. This parameter is required for multiclass/multilabel targets. - If ``None``, the scores for each class are returned. Otherwise, this - determines the type of averaging performed on the data: - ``'binary'``: - Only report results for the class specified by ``pos_label``. - This is applicable only if targets (``y_{true,pred}``) are binary. - ``'micro'``: - Calculate metrics globally by counting the total true positives, - false negatives and false positives. - ``'macro'``: - Calculate metrics for each label, and find their unweighted - mean. This does not take label imbalance into account. - ``'weighted'``: - Calculate metrics for each label, and find their average weighted - by support (the number of true instances for each label). This - alters 'macro' to account for label imbalance; it can result in an - F-score that is not between precision and recall. - ``'samples'``: - Calculate metrics for each instance, and find their average (only - meaningful for multilabel classification where this differs from - :func:`accuracy_score`). + If None, the scores for each class are returned. Otherwise, this + determines the type of averaging performed on the data. + With 'binary', only results for the class specified by ``pos_label`` + are reported (applicable only if targets are binary). With 'micro', + metrics are calculated globally by counting the total true positives, + false negatives and false positives. With 'macro', metrics are + calculated for each label and their unweighted mean is taken, which + does not take label imbalance into account. With 'weighted', metrics + are calculated for each label and averaged, weighted by support (the + number of true instances for each label), which can result in an + F-score that is not between precision and recall. With 'samples', + metrics are calculated for each instance and averaged (only + meaningful for multilabel classification). sample_weight : Optional[np.ndarray], default=None Sample weights. zero_division : Literal["warn", 0, 1], default="warn" @@ -589,27 +496,18 @@ def f1_score(y_true: np.ndarray, y_pred: np.ndarray, *, Returns ------- - f1_score : float or array of float, shape = [n_unique_labels] + f1_score : float or ndarray of floats F1 score of the positive class in binary classification or weighted - average of the F1 scores of each class for the multiclass task. - - See Also - -------- - fbeta_score, precision_recall_fscore_support, jaccard_score, - multilabel_confusion_matrix - - References - ---------- - .. [1] `Wikipedia entry for the F1-score - `_. + average of the F1 scores of each class for the multiclass task. An + array of shape (n_unique_labels,) is returned if average is None. Notes ----- When ``true positive + false positive == 0``, precision is undefined. When ``true positive + false negative == 0``, recall is undefined. - In such cases, by default the metric will be set to 0, as will f-score, - and ``UndefinedMetricWarning`` will be raised. This behavior can be - modified with ``zero_division``. + In such cases, by default the metric is set to 0, as is f-score, and + ``UndefinedMetricWarning`` is raised. This behavior can be modified with + ``zero_division``. """ return fbeta_score( y_true, y_pred, beta=1, labels=labels, pos_label=pos_label, @@ -625,16 +523,14 @@ def fbeta_score(y_true: np.ndarray, y_pred: np.ndarray, beta: float, *, sample_weight: np.ndarray | None = None, zero_division: Literal["warn", 0, 1] = "warn")\ -> float | np.ndarray: - """ - Compute the F-beta score. + """Compute the F-beta score. The F-beta score is the weighted harmonic mean of precision and recall, - reaching its optimal value at 1 and its worst value at 0. - The `beta` parameter determines the weight of recall in the combined - score. ``beta < 1`` lends more weight to precision, while ``beta > 1`` - favors recall (``beta -> 0`` considers only precision, ``beta -> +inf`` - only recall). - Read more in the :ref:`User Guide `. + reaching its optimal value at 1 and its worst value at 0. The ``beta`` + parameter determines the weight of recall in the combined score. + ``beta < 1`` lends more weight to precision, while ``beta > 1`` favors + recall (``beta -> 0`` considers only precision, ``beta -> +inf`` only + recall). Parameters ---------- @@ -652,36 +548,27 @@ def fbeta_score(y_true: np.ndarray, y_pred: np.ndarray, beta: float, *, result in 0 components in a macro average. For multilabel targets, labels are column indices. By default, all labels in ``y_true`` and ``y_pred`` are used in sorted order. - .. versionchanged:: 0.17 - Parameter `labels` improved for multiclass problem. pos_label : Union[str, int], default=1 The class to report if ``average='binary'`` and the data is binary. If the data are multiclass or multilabel, this will be ignored; setting ``labels=[pos_label]`` and ``average != 'binary'`` will report scores for that label only. - average : Optional[Literal['micro', 'macro', 'samples','weighted', - 'binary']], default='binary' + average : str or None, default='binary' + One of {'micro', 'macro', 'samples', 'weighted', 'binary'} or None. This parameter is required for multiclass/multilabel targets. - If ``None``, the scores for each class are returned. Otherwise, this - determines the type of averaging performed on the data: - ``'binary'``: - Only report results for the class specified by ``pos_label``. - This is applicable only if targets (``y_{true,pred}``) are binary. - ``'micro'``: - Calculate metrics globally by counting the total true positives, - false negatives and false positives. - ``'macro'``: - Calculate metrics for each label, and find their unweighted - mean. This does not take label imbalance into account. - ``'weighted'``: - Calculate metrics for each label, and find their average weighted - by support (the number of true instances for each label). This - alters 'macro' to account for label imbalance; it can result in an - F-score that is not between precision and recall. - ``'samples'``: - Calculate metrics for each instance, and find their average (only - meaningful for multilabel classification where this differs from - :func:`accuracy_score`). + If None, the scores for each class are returned. Otherwise, this + determines the type of averaging performed on the data. + With 'binary', only results for the class specified by ``pos_label`` + are reported (applicable only if targets are binary). With 'micro', + metrics are calculated globally by counting the total true positives, + false negatives and false positives. With 'macro', metrics are + calculated for each label and their unweighted mean is taken, which + does not take label imbalance into account. With 'weighted', metrics + are calculated for each label and averaged, weighted by support (the + number of true instances for each label), which can result in an + F-score that is not between precision and recall. With 'samples', + metrics are calculated for each instance and averaged (only + meaningful for multilabel classification). sample_weight : Optional[np.ndarray], default=None Sample weights. zero_division : Literal["warn", 0, 1], default="warn" @@ -691,26 +578,18 @@ def fbeta_score(y_true: np.ndarray, y_pred: np.ndarray, beta: float, *, Returns ------- - fbeta_score : float (if average is not None) or array of float, shape = - [n_unique_labels] - F-beta score of the positive class in binary classification or weighted - average of the F-beta score of each class for the multiclass task. + fbeta_score : float or ndarray of floats + F-beta score of the positive class in binary classification or + weighted average of the F-beta score of each class for the + multiclass task. An array of shape (n_unique_labels,) is returned if + average is None. - See Also - -------- - precision_recall_fscore_support, multilabel_confusion_matrix Notes ----- When ``true positive + false positive == 0`` or ``true positive + false negative == 0``, f-score returns 0 and raises - ``UndefinedMetricWarning``. This behavior can be - modified with ``zero_division``. - References - ---------- - .. [1] R. Baeza-Yates and B. Ribeiro-Neto (2011). - Modern Information Retrieval. Addison Wesley, pp. 327-328. - .. [2] `Wikipedia entry for the F1-score - `_. + ``UndefinedMetricWarning``. This behavior can be modified with + ``zero_division``. """ _, _, f, _ = precision_recall_fscore_support( y_true, y_pred, beta=beta, labels=labels, pos_label=pos_label, @@ -733,26 +612,21 @@ def precision_recall_fscore_support(y_true: np.ndarray, y_pred: np.ndarray, *, 0, 1] = "warn")\ -> tuple[float | np.ndarray, float | np.ndarray, float | np.ndarray, np.ndarray | None]: - """ - Compute precision, recall, F-measure and support for each class. - - The precision is the ratio ``tp / (tp + fp)`` where ``tp`` is the number of - true positives and ``fp`` the number of false positives. The precision is - intuitively the ability of the classifier not to label as positive a sample - that is negative. - The recall is the ratio ``tp / (tp + fn)`` where ``tp`` is the number of - true positives and ``fn`` the number of false negatives. The recall is - intuitively the ability of the classifier to find all the positive samples. - The F-beta score can be interpreted as a weighted harmonic mean of - the precision and recall, where an F-beta score reaches its best - value at 1 and worst score at 0. - The F-beta score weights recall more than precision by a factor of - ``beta``. ``beta == 1.0`` means recall and precision are equally important. - The support is the number of occurrences of each class in ``y_true``. - If ``pos_label is None`` and in binary classification, this function - returns the average precision, recall and F-measure if ``average`` - is one of ``'micro'``, ``'macro'``, ``'weighted'`` or ``'samples'``. - Read more in the :ref:`User Guide `. + """Compute precision, recall, F-measure and support for each class. + + The precision is the ratio ``tp / (tp + fp)`` where ``tp`` is the number + of true positives and ``fp`` the number of false positives. The + precision is intuitively the ability of the classifier not to label as + positive a sample that is negative. The recall is the ratio + ``tp / (tp + fn)`` where ``tp`` is the number of true positives and + ``fn`` the number of false negatives. The recall is intuitively the + ability of the classifier to find all the positive samples. The F-beta + score can be interpreted as a weighted harmonic mean of the precision + and recall, where an F-beta score reaches its best value at 1 and worst + score at 0. The F-beta score weights recall more than precision by a + factor of ``beta``. ``beta == 1.0`` means recall and precision are + equally important. The support is the number of occurrences of each + class in ``y_true``. Parameters ---------- @@ -775,69 +649,54 @@ def precision_recall_fscore_support(y_true: np.ndarray, y_pred: np.ndarray, *, If the data are multiclass or multilabel, this will be ignored; setting ``labels=[pos_label]`` and ``average != 'binary'`` will report scores for that label only. - average : Optional[Literal['micro', 'macro', 'samples', 'weighted', - 'binary']], default=None - If ``None``, the scores for each class are returned. Otherwise, this - determines the type of averaging performed on the data: - ``'binary'``: - Only report results for the class specified by ``pos_label``. - This is applicable only if targets (``y_{true,pred}``) are binary. - ``'micro'``: - Calculate metrics globally by counting the total true positives, - false negatives and false positives. - ``'macro'``: - Calculate metrics for each label, and find their unweighted - mean. This does not take label imbalance into account. - ``'weighted'``: - Calculate metrics for each label, and find their average weighted - by support (the number of true instances for each label). This - alters 'macro' to account for label imbalance; it can result in an - F-score that is not between precision and recall. - ``'samples'``: - Calculate metrics for each instance, and find their average (only - meaningful for multilabel classification where this differs from - :func:`accuracy_score`). + average : str or None, default=None + One of {'micro', 'macro', 'samples', 'weighted', 'binary'} or None. + If None, the scores for each class are returned. Otherwise, this + determines the type of averaging performed on the data. + With 'binary', only results for the class specified by ``pos_label`` + are reported (applicable only if targets are binary). With 'micro', + metrics are calculated globally by counting the total true positives, + false negatives and false positives. With 'macro', metrics are + calculated for each label and their unweighted mean is taken, which + does not take label imbalance into account. With 'weighted', metrics + are calculated for each label and averaged, weighted by support (the + number of true instances for each label), which can result in an + F-score that is not between precision and recall. With 'samples', + metrics are calculated for each instance and averaged (only + meaningful for multilabel classification). warn_for : tuple or set, for internal use This determines which warnings will be made in the case that this function is being used to return only one of its metrics. sample_weight : Optional[np.ndarray], default=None Sample weights. - zero_division : "warn", 0 or 1, default="warn" - Sets the value to return when there is a zero division: - - recall: when there are no positive labels - - precision: when there are no positive predictions - - f-score: both - If set to "warn", this acts as 0, but warnings are also raised. + zero_division : {"warn", 0, 1}, default="warn" + Sets the value to return when there is a zero division: for recall + when there are no positive labels, for precision when there are no + positive predictions, and for f-score for both cases. If set to + "warn", this acts as 0, but warnings are also raised. Returns ------- - precision : float (if average is not None) or array of float, shape = - [n_unique_labels] - recall : float (if average is not None) or array of float, , shape = - [n_unique_labels] - fbeta_score : float (if average is not None) or array of float, shape = - [n_unique_labels] - support : None (if average is not None) or array of int, shape = - [n_unique_labels] - The number of occurrences of each label in ``y_true``. + precision : float or ndarray of floats + A float if average is not None, otherwise an array of shape + (n_unique_labels,). + recall : float or ndarray of floats + A float if average is not None, otherwise an array of shape + (n_unique_labels,). + fbeta_score : float or ndarray of floats + A float if average is not None, otherwise an array of shape + (n_unique_labels,). + support : None or ndarray of ints + The number of occurrences of each label in ``y_true``. None if + average is not None, otherwise an array of shape (n_unique_labels,). Notes ----- When ``true positive + false positive == 0``, precision is undefined. When ``true positive + false negative == 0``, recall is undefined. - In such cases, by default the metric will be set to 0, as will f-score, - and ``UndefinedMetricWarning`` will be raised. This behavior can be - modified with ``zero_division``. - References - ---------- - .. [1] `Wikipedia entry for the Precision and recall - `_. - .. [2] `Wikipedia entry for the F1-score - `_. - .. [3] `Discriminative Methods for Multi-labeled Classification Advances - in Knowledge Discovery and Data Mining (2004), pp. 22-30 by Shantanu - Godbole, Sunita Sarawagi - `_. + In such cases, by default the metric is set to 0, as is f-score, and + ``UndefinedMetricWarning`` is raised. This behavior can be modified with + ``zero_division``. """ y_type, y_true, y_pred, sample_weight = _check_targets( y_true, y_pred, sample_weight) @@ -860,15 +719,13 @@ def precision_score(y_true: np.ndarray, y_pred: np.ndarray, *, sample_weight: np.ndarray | None = None, zero_division: Literal["warn", 0, 1] = "warn")\ -> float | np.ndarray: - """ - Compute the precision. + """Compute the precision. - The precision is the ratio ``tp / (tp + fp)`` where ``tp`` is the number of - true positives and ``fp`` the number of false positives. The precision is - intuitively the ability of the classifier not to label as positive a sample - that is negative. - The best value is 1 and the worst value is 0. - Read more in the :ref:`User Guide `. + The precision is the ratio ``tp / (tp + fp)`` where ``tp`` is the number + of true positives and ``fp`` the number of false positives. The + precision is intuitively the ability of the classifier not to label as + positive a sample that is negative. The best value is 1 and the worst + value is 0. Parameters ---------- @@ -884,36 +741,27 @@ def precision_score(y_true: np.ndarray, y_pred: np.ndarray, *, result in 0 components in a macro average. For multilabel targets, labels are column indices. By default, all labels in ``y_true`` and ``y_pred`` are used in sorted order. - .. versionchanged:: 0.17 - Parameter `labels` improved for multiclass problem. pos_label : str or int, default=1 The class to report if ``average='binary'`` and the data is binary. If the data are multiclass or multilabel, this will be ignored; setting ``labels=[pos_label]`` and ``average != 'binary'`` will report scores for that label only. - average : {'micro', 'macro', 'samples', 'weighted', 'binary'} - default='binary' + average : str or None, default='binary' + One of {'micro', 'macro', 'samples', 'weighted', 'binary'} or None. This parameter is required for multiclass/multilabel targets. - If ``None``, the scores for each class are returned. Otherwise, this - determines the type of averaging performed on the data: - ``'binary'``: - Only report results for the class specified by ``pos_label``. - This is applicable only if targets (``y_{true,pred}``) are binary. - ``'micro'``: - Calculate metrics globally by counting the total true positives, - false negatives and false positives. - ``'macro'``: - Calculate metrics for each label, and find their unweighted - mean. This does not take label imbalance into account. - ``'weighted'``: - Calculate metrics for each label, and find their average weighted - by support (the number of true instances for each label). This - alters 'macro' to account for label imbalance; it can result in an - F-score that is not between precision and recall. - ``'samples'``: - Calculate metrics for each instance, and find their average (only - meaningful for multilabel classification where this differs from - :func:`accuracy_score`). + If None, the scores for each class are returned. Otherwise, this + determines the type of averaging performed on the data. + With 'binary', only results for the class specified by ``pos_label`` + are reported (applicable only if targets are binary). With 'micro', + metrics are calculated globally by counting the total true positives, + false negatives and false positives. With 'macro', metrics are + calculated for each label and their unweighted mean is taken, which + does not take label imbalance into account. With 'weighted', metrics + are calculated for each label and averaged, weighted by support (the + number of true instances for each label), which can result in an + F-score that is not between precision and recall. With 'samples', + metrics are calculated for each instance and averaged (only + meaningful for multilabel classification). sample_weight : array-like of shape (n_samples,), default=None Sample weights. zero_division : "warn", 0 or 1, default="warn" @@ -922,20 +770,16 @@ def precision_score(y_true: np.ndarray, y_pred: np.ndarray, *, Returns ------- - precision : float (if average is not None) or array of float of shape - (n_unique_labels,) + precision : float or ndarray of floats Precision of the positive class in binary classification or weighted - average of the precision of each class for the multiclass task. - - See Also - -------- - precision_recall_fscore_support, multilabel_confusion_matrix + average of the precision of each class for the multiclass task. An + array of shape (n_unique_labels,) is returned if average is None. Notes ----- When ``true positive + false positive == 0``, precision returns 0 and - raises ``UndefinedMetricWarning``. This behavior can be - modified with ``zero_division``. + raises ``UndefinedMetricWarning``. This behavior can be modified with + ``zero_division``. """ p, _, _, _ = precision_recall_fscore_support( y_true=y_true, y_pred=y_pred, labels=labels, pos_label=pos_label, @@ -952,14 +796,12 @@ def recall_score(y_true: np.ndarray, y_pred: np.ndarray, *, sample_weight: np.ndarray | None = None, zero_division: Literal["warn", 0, 1] = "warn")\ -> float | np.ndarray: - """ - Compute the recall. + """Compute the recall. The recall is the ratio ``tp / (tp + fn)`` where ``tp`` is the number of true positives and ``fn`` the number of false negatives. The recall is - intuitively the ability of the classifier to find all the positive samples. - The best value is 1 and the worst value is 0. - Read more in the :ref:`User Guide `. + intuitively the ability of the classifier to find all the positive + samples. The best value is 1 and the worst value is 0. Parameters ---------- @@ -975,36 +817,27 @@ def recall_score(y_true: np.ndarray, y_pred: np.ndarray, *, result in 0 components in a macro average. For multilabel targets, labels are column indices. By default, all labels in ``y_true`` and ``y_pred`` are used in sorted order. - .. versionchanged:: 0.17 - Parameter `labels` improved for multiclass problem. pos_label : str or int, default=1 The class to report if ``average='binary'`` and the data is binary. If the data are multiclass or multilabel, this will be ignored; setting ``labels=[pos_label]`` and ``average != 'binary'`` will report scores for that label only. - average : {'micro', 'macro', 'samples', 'weighted', 'binary'}, - default='binary' + average : str or None, default='binary' + One of {'micro', 'macro', 'samples', 'weighted', 'binary'} or None. This parameter is required for multiclass/multilabel targets. - If ``None``, the scores for each class are returned. Otherwise, this - determines the type of averaging performed on the data: - ``'binary'``: - Only report results for the class specified by ``pos_label``. - This is applicable only if targets (``y_{true,pred}``) are binary. - ``'micro'``: - Calculate metrics globally by counting the total true positives, - false negatives and false positives. - ``'macro'``: - Calculate metrics for each label, and find their unweighted - mean. This does not take label imbalance into account. - ``'weighted'``: - Calculate metrics for each label, and find their average weighted - by support (the number of true instances for each label). This - alters 'macro' to account for label imbalance; it can result in an - F-score that is not between precision and recall. - ``'samples'``: - Calculate metrics for each instance, and find their average (only - meaningful for multilabel classification where this differs from - :func:`accuracy_score`). + If None, the scores for each class are returned. Otherwise, this + determines the type of averaging performed on the data. + With 'binary', only results for the class specified by ``pos_label`` + are reported (applicable only if targets are binary). With 'micro', + metrics are calculated globally by counting the total true positives, + false negatives and false positives. With 'macro', metrics are + calculated for each label and their unweighted mean is taken, which + does not take label imbalance into account. With 'weighted', metrics + are calculated for each label and averaged, weighted by support (the + number of true instances for each label), which can result in an + F-score that is not between precision and recall. With 'samples', + metrics are calculated for each instance and averaged (only + meaningful for multilabel classification). sample_weight : array-like of shape (n_samples,), default=None Sample weights. zero_division : "warn", 0 or 1, default="warn" @@ -1013,19 +846,15 @@ def recall_score(y_true: np.ndarray, y_pred: np.ndarray, *, Returns ------- - recall : float (if average is not None) or array of float of shape - (n_unique_labels,) + recall : float or ndarray of floats Recall of the positive class in binary classification or weighted - average of the recall of each class for the multiclass task. - See Also - -------- - precision_recall_fscore_support, balanced_accuracy_score, - multilabel_confusion_matrix + average of the recall of each class for the multiclass task. An + array of shape (n_unique_labels,) is returned if average is None. Notes ----- - When ``true positive + false negative == 0``, recall returns 0 and raises - ``UndefinedMetricWarning``. This behavior can be modified with + When ``true positive + false negative == 0``, recall returns 0 and + raises ``UndefinedMetricWarning``. This behavior can be modified with ``zero_division``. """ _, r, _, _ = precision_recall_fscore_support( @@ -1038,15 +867,12 @@ def recall_score(y_true: np.ndarray, y_pred: np.ndarray, *, def balanced_accuracy_score(y_true: np.ndarray, y_pred: np.ndarray, *, sample_weight: np.ndarray | None = None, adjusted: bool = False) -> float: - """ - Compute the balanced accuracy. + """Compute the balanced accuracy. - The balanced accuracy in binary and multiclass classification problems to - deal with imbalanced datasets. It is defined as the average of recall - obtained on each class. - The best value is 1 and the worst value is 0 when ``adjusted=False``. - Read more in the :ref:`User Guide `. - .. versionadded:: 0.20 + The balanced accuracy in binary and multiclass classification problems + deals with imbalanced datasets. It is defined as the average of recall + obtained on each class. The best value is 1 and the worst value is 0 + when ``adjusted=False``. Parameters ---------- @@ -1063,29 +889,13 @@ def balanced_accuracy_score(y_true: np.ndarray, y_pred: np.ndarray, *, Returns ------- balanced_accuracy : float - - See Also - -------- - recall_score, roc_auc_score + The balanced accuracy score. Notes ----- - Some literature promotes alternative definitions of balanced accuracy. Our - definition is equivalent to :func:`accuracy_score` with class-balanced + Some literature promotes alternative definitions of balanced accuracy. + This definition is equivalent to accuracy_score with class-balanced sample weights, and shares desirable properties with the binary case. - See the :ref:`User Guide `. - - References - ---------- - .. [1] Brodersen, K.H.; Ong, C.S.; Stephan, K.E.; Buhmann, J.M. (2010). - The balanced accuracy and its posterior distribution. - Proceedings of the 20th International Conference on Pattern - Recognition, 3121-24. - .. [2] John. D. Kelleher, Brian Mac Namee, Aoife D'Arcy, (2015). - `Fundamentals of Machine Learning for Predictive Data Analytics: - Algorithms, Worked Examples, and Case Studies - `_. """ y_type, y_true, y_pred, sample_weight = _check_targets( y_true, y_pred, sample_weight) @@ -1105,10 +915,7 @@ def classification_report(y_true: np.ndarray, y_pred: np.ndarray, *, digits: int = 2, output_dict: bool = False, zero_division: Literal["warn", 0, 1] = "warn") \ -> str | dict: - """ - Build a text report showing the main classification metrics. - - Read more in the :ref:`User Guide `. + """Build a text report showing the main classification metrics. Parameters ---------- @@ -1128,41 +935,26 @@ def classification_report(y_true: np.ndarray, y_pred: np.ndarray, *, returned values will not be rounded. output_dict : bool, default=False If True, return output as dict. - .. versionadded:: 0.20 - zero_division : "warn", 0 or 1, default="warn" + zero_division : {"warn", 0, 1}, default="warn" Sets the value to return when there is a zero division. If set to "warn", this acts as 0, but warnings are also raised. Returns ------- - report : string / dict - Text summary of the precision, recall, F1 score for each class. - Dictionary returned if output_dict is True. Dictionary has the - following structure:: - {'label 1': {'precision':0.5, - 'recall':1.0, - 'f1-score':0.67, - 'support':1}, - 'label 2': { ... }, - ... - } - The reported averages include macro average (averaging the unweighted - mean per label), weighted average (averaging the support-weighted mean - per label), and sample average (only for multilabel classification). - Micro average (averaging the total true positives, false negatives and - false positives) is only shown for multi-label or multi-class - with a subset of classes, because it corresponds to accuracy - otherwise and would be the same for all metrics. - See also :func:`precision_recall_fscore_support` for more details - on averages. - Note that in binary classification, recall of the positive class - is also known as "sensitivity"; recall of the negative class is - "specificity". - - See Also - -------- - precision_recall_fscore_support, confusion_matrix, - multilabel_confusion_matrix + report : str or dict + Text summary of the precision, recall, F1 score for each class. A + dictionary is returned if output_dict is True, keyed by label with a + nested dict of 'precision', 'recall', 'f1-score' and 'support' for + each class. The reported averages include macro average (averaging + the unweighted mean per label), weighted average (averaging the + support-weighted mean per label), and sample average (only for + multilabel classification). Micro average (averaging the total true + positives, false negatives and false positives) is only shown for + multi-label or multi-class with a subset of classes, because it + corresponds to accuracy otherwise and would be the same for all + metrics. Note that in binary classification, recall of the positive + class is also known as "sensitivity"; recall of the negative class + is "specificity". """ y_type, y_true, y_pred, sample_weight = _check_targets( y_true, y_pred, sample_weight) @@ -1178,11 +970,10 @@ def classification_report(y_true: np.ndarray, y_pred: np.ndarray, *, def hamming_loss(y_true: np.ndarray, y_pred: np.ndarray, *, sample_weight: np.ndarray | None = None) -> float: - """ - Compute the average Hamming loss. + """Compute the average Hamming loss. - The Hamming loss is the fraction of labels that are incorrectly predicted. - Read more in the :ref:`User Guide `. + The Hamming loss is the fraction of labels that are incorrectly + predicted. Parameters ---------- @@ -1192,40 +983,25 @@ def hamming_loss(y_true: np.ndarray, y_pred: np.ndarray, *, Predicted labels, as returned by a classifier. sample_weight : array-like of shape (n_samples,), default=None Sample weights. - .. versionadded:: 0.18 Returns ------- loss : float or int - Return the average Hamming loss between element of ``y_true`` and + The average Hamming loss between elements of ``y_true`` and ``y_pred``. - See Also - -------- - accuracy_score, jaccard_score, zero_one_loss - Notes ----- - In multiclass classification, the Hamming loss corresponds to the Hamming - distance between ``y_true`` and ``y_pred`` which is equivalent to the - subset ``zero_one_loss`` function, when `normalize` parameter is set to - True. - In multilabel classification, the Hamming loss is different from the - subset zero-one loss. The zero-one loss considers the entire set of labels - for a given sample incorrect if it does not entirely match the true set of - labels. Hamming loss is more forgiving in that it penalizes only the - individual labels. - The Hamming loss is upperbounded by the subset zero-one loss, when - `normalize` parameter is set to True. It is always between 0 and 1, - lower being better. - - References - ---------- - .. [1] Grigorios Tsoumakas, Ioannis Katakis. Multi-Label Classification: - An Overview. International Journal of Data Warehousing & Mining, - 3(3), 1-13, July-September 2007. - .. [2] `Wikipedia entry on the Hamming distance - `_. + In multiclass classification, the Hamming loss corresponds to the + Hamming distance between ``y_true`` and ``y_pred`` which is equivalent + to the subset ``zero_one_loss`` function, when the normalize parameter + is set to True. In multilabel classification, the Hamming loss is + different from the subset zero-one loss. The zero-one loss considers the + entire set of labels for a given sample incorrect if it does not + entirely match the true set of labels. Hamming loss is more forgiving in + that it penalizes only the individual labels. The Hamming loss is + upper-bounded by the subset zero-one loss, when the normalize parameter + is set to True. It is always between 0 and 1, lower being better. """ y_type, y_true, y_pred, sample_weight = _check_targets( y_true, y_pred, sample_weight) @@ -1241,59 +1017,47 @@ def log_loss(y_true: np.ndarray, y_pred: np.ndarray, *, eps: float = 1e-15, normalize: bool = True, sample_weight: np.ndarray | None = None, labels: np.ndarray | None = None) -> float: - r""" - Log loss, aka logistic loss or cross-entropy loss. + """Log loss, aka logistic loss or cross-entropy loss. - This is the loss function used in (multinomial) logistic regression - and extensions of it such as neural networks, defined as the negative + This is the loss function used in (multinomial) logistic regression and + extensions of it such as neural networks, defined as the negative log-likelihood of a logistic model that returns ``y_pred`` probabilities - for its training data ``y_true``. - The log loss is only defined for two or more labels. - For a single sample with true label :math:`y \in \{0,1\}` and - and a probability estimate :math:`p = \operatorname{Pr}(y = 1)`, the log - loss is: - .. math:: - L_{\log}(y, p) = -(y \log (p) + (1 - y) \log (1 - p)) - Read more in the :ref:`User Guide `. + for its training data ``y_true``. The log loss is only defined for two + or more labels. For a single sample with true label y in {0, 1} and a + probability estimate p = Pr(y = 1), the log loss is + -(y * log(p) + (1 - y) * log(1 - p)). Parameters ---------- y_true : array-like or label indicator matrix Ground truth (correct) labels for n_samples samples. - y_pred : array-like of float, shape = (n_samples, n_classes) - or (n_samples,) - Predicted probabilities, as returned by a classifier's - predict_proba method. If ``y_pred.shape = (n_samples,)`` - the probabilities provided are assumed to be that of the - positive class. The labels in ``y_pred`` are assumed to be - ordered alphabetically, as done by - :class:`preprocessing.LabelBinarizer`. + y_pred : array-like of floats + Predicted probabilities of shape (n_samples, n_classes) or + (n_samples,), as returned by a classifier's predict_proba method. If + ``y_pred.shape = (n_samples,)`` the probabilities provided are + assumed to be that of the positive class. The labels in ``y_pred`` + are assumed to be ordered alphabetically. eps : float, default=1e-15 - Log loss is undefined for p=0 or p=1, so probabilities are - clipped to max(eps, min(1 - eps, p)). + Log loss is undefined for p=0 or p=1, so probabilities are clipped to + max(eps, min(1 - eps, p)). normalize : bool, default=True - If true, return the mean loss per sample. - Otherwise, return the sum of the per-sample losses. + If true, return the mean loss per sample. Otherwise, return the sum + of the per-sample losses. sample_weight : array-like of shape (n_samples,), default=None Sample weights. labels : array-like, default=None - If not provided, labels will be inferred from y_true. If ``labels`` - is ``None`` and ``y_pred`` has shape (n_samples,) the labels are - assumed to be binary and are inferred from ``y_true``. - .. versionadded:: 0.18 + If not provided, labels are inferred from y_true. If ``labels`` is + ``None`` and ``y_pred`` has shape (n_samples,) the labels are assumed + to be binary and are inferred from ``y_true``. Returns ------- loss : float + The log loss. Notes ----- The logarithm used is the natural logarithm (base-e). - - References - ---------- - C.M. Bishop (2006). Pattern Recognition and Machine Learning. Springer, - p. 209. """ y_type, y_true, y_pred, sample_weight = _check_targets( y_true, y_pred, sample_weight) @@ -1309,20 +1073,19 @@ def log_loss(y_true: np.ndarray, y_pred: np.ndarray, *, eps: float = 1e-15, def hinge_loss(y_true: np.ndarray, pred_decision: np.ndarray, *, labels: np.ndarray | None = None, sample_weight: np.ndarray | None = None) -> float: - """ - Average hinge loss (non-regularized). - - In binary class case, assuming labels in y_true are encoded with +1 and -1, - when a prediction mistake is made, ``margin = y_true * pred_decision`` is - always negative (since the signs disagree), implying ``1 - margin`` is - always greater than 1. The cumulated hinge loss is therefore an upper - bound of the number of mistakes made by the classifier. - In multiclass case, the function expects that either all the labels are - included in y_true or an optional labels argument is provided which - contains all the labels. The multilabel margin is calculated according - to Crammer-Singer's method. As in the binary case, the cumulated hinge loss - is an upper bound of the number of mistakes made by the classifier. - Read more in the :ref:`User Guide `. + """Average hinge loss (non-regularized). + + In the binary case, assuming labels in y_true are encoded with +1 and + -1, when a prediction mistake is made, ``margin = y_true * + pred_decision`` is always negative (since the signs disagree), implying + ``1 - margin`` is always greater than 1. The cumulated hinge loss is + therefore an upper bound of the number of mistakes made by the + classifier. In the multiclass case, the function expects that either all + the labels are included in y_true or an optional labels argument is + provided which contains all the labels. The multilabel margin is + calculated according to Crammer-Singer's method. As in the binary case, + the cumulated hinge loss is an upper bound of the number of mistakes made + by the classifier. Parameters ---------- @@ -1339,19 +1102,7 @@ def hinge_loss(y_true: np.ndarray, pred_decision: np.ndarray, *, Returns ------- loss : float - - References - ---------- - .. [1] `Wikipedia entry on the Hinge loss - `_. - .. [2] Koby Crammer, Yoram Singer. On the Algorithmic - Implementation of Multiclass Kernel-based Vector - Machines. Journal of Machine Learning Research 2, - (2001), 265-292. - .. [3] `L1 AND L2 Regularization for Multiclass Hinge Loss Models - by Robert C. Moore, John DeNero - `_. + The average hinge loss. """ y_type, y_true, pred_decision, sample_weight = _check_targets( y_true, pred_decision, sample_weight) @@ -1367,26 +1118,23 @@ def hinge_loss(y_true: np.ndarray, pred_decision: np.ndarray, *, def brier_score_loss(y_true: np.ndarray, y_prob: np.ndarray, *, sample_weight: np.ndarray | None = None, pos_label: int | None = None) -> float: - """ - Compute the Brier score loss. - - The smaller the Brier score loss, the better, hence the naming with "loss". - The Brier score measures the mean squared difference between the predicted - probability and the actual outcome. The Brier score always - takes on a value between zero and one, since this is the largest - possible difference between a predicted probability (which must be - between zero and one) and the actual outcome (which can take on values - of only 0 and 1). It can be decomposed is the sum of refinement loss and - calibration loss. - The Brier score is appropriate for binary and categorical outcomes that - can be structured as true or false, but is inappropriate for ordinal - variables which can take on three or more values (this is because the - Brier score assumes that all possible outcomes are equivalently + """Compute the Brier score loss. + + The smaller the Brier score loss, the better, hence the naming with + "loss". The Brier score measures the mean squared difference between the + predicted probability and the actual outcome. The Brier score always + takes on a value between zero and one, since this is the largest possible + difference between a predicted probability (which must be between zero + and one) and the actual outcome (which can take on values of only 0 and + 1). It can be decomposed as the sum of refinement loss and calibration + loss. The Brier score is appropriate for binary and categorical outcomes + that can be structured as true or false, but is inappropriate for + ordinal variables which can take on three or more values (this is because + the Brier score assumes that all possible outcomes are equivalently "distant" from one another). Which label is considered to be the positive - label is controlled via the parameter `pos_label`, which defaults to - the greater label unless `y_true` is all 0 or all -1, in which case - `pos_label` defaults to 1. - Read more in the :ref:`User Guide `. + label is controlled via the parameter ``pos_label``, which defaults to + the greater label unless ``y_true`` is all 0 or all -1, in which case + ``pos_label`` defaults to 1. Parameters ---------- @@ -1397,23 +1145,17 @@ def brier_score_loss(y_true: np.ndarray, y_prob: np.ndarray, *, sample_weight : array-like of shape (n_samples,), default=None Sample weights. pos_label : int or str, default=None - Label of the positive class. `pos_label` will be infered in the - following manner: - * if `y_true` in {-1, 1} or {0, 1}, `pos_label` defaults to 1; - * else if `y_true` contains string, an error will be raised and - `pos_label` should be explicitely specified; - * otherwise, `pos_label` defaults to the greater label, - i.e. `np.unique(y_true)[-1]`. + Label of the positive class. ``pos_label`` is inferred as follows: + if ``y_true`` is in {-1, 1} or {0, 1}, ``pos_label`` defaults to 1; + else if ``y_true`` contains a string, an error is raised and + ``pos_label`` should be explicitly specified; otherwise + ``pos_label`` defaults to the greater label, i.e. + ``np.unique(y_true)[-1]``. Returns ------- score : float Brier score loss. - - References - ---------- - .. [1] `Wikipedia entry for the Brier score - `_. """ y_type, y_true, y_prob, sample_weight = _check_targets( y_true, y_prob, sample_weight) diff --git a/src/pyrcn/metrics/_regression.py b/src/pyrcn/metrics/_regression.py index a4c4cff..5d18074 100644 --- a/src/pyrcn/metrics/_regression.py +++ b/src/pyrcn/metrics/_regression.py @@ -83,8 +83,6 @@ def mean_absolute_error(y_true: np.ndarray, y_pred: np.ndarray, *, -> float: """Mean absolute error regression loss. - Read more in the :ref:`User Guide `. - Parameters ---------- y_true : array-like of shape (n_samples,) or (n_samples, n_outputs) @@ -93,23 +91,22 @@ def mean_absolute_error(y_true: np.ndarray, y_pred: np.ndarray, *, Estimated target values. sample_weight : array-like of shape (n_samples,), default=None Sample weights. - multioutput : {'raw_values', 'uniform_average'} or array-like of shape - (n_outputs,), default='uniform_average' - Defines aggregating of multiple output values. - Array-like value defines weights used to average errors. - 'raw_values' : - Returns a full set of errors in case of multioutput input. - 'uniform_average' : - Errors of all outputs are averaged with uniform weight. + multioutput : str or array-like, default='uniform_average' + Defines aggregating of multiple output values, one of + {'raw_values', 'uniform_average'} or an array-like of shape + (n_outputs,). An array-like value defines weights used to average + errors. With 'raw_values', a full set of errors is returned in case + of multioutput input. With 'uniform_average', errors of all outputs + are averaged with uniform weight. Returns ------- loss : float or ndarray of floats If multioutput is 'raw_values', then mean absolute error is returned - for each output separately. - If multioutput is 'uniform_average' or an ndarray of weights, then the - weighted average of all output errors is returned. - MAE output is non-negative floating point. The best value is 0.0. + for each output separately. If multioutput is 'uniform_average' or + an ndarray of weights, then the weighted average of all output + errors is returned. MAE output is non-negative floating point. The + best value is 0.0. """ y_type, y_true, y_pred, sample_weight, multioutput = _check_reg_targets( y_true, y_pred, sample_weight, multioutput=multioutput) @@ -129,13 +126,10 @@ def mean_absolute_percentage_error(y_true: np.ndarray, y_pred: np.ndarray, "variance_weighted"] | None) = "uniform_average")\ -> float: - """ - Mean absolute percentage error regression loss. + """Mean absolute percentage error regression loss. - Note here that we do not represent the output as a percentage in range - [0, 100]. Instead, we represent it in range [0, 1/eps]. Read more in the - :ref:`User Guide `. - .. versionadded:: 0.24 + Note that the output is not represented as a percentage in range + [0, 100]. Instead, it is represented in range [0, 1/eps]. Parameters ---------- @@ -145,26 +139,25 @@ def mean_absolute_percentage_error(y_true: np.ndarray, y_pred: np.ndarray, Estimated target values. sample_weight : array-like of shape (n_samples,), default=None Sample weights. - multioutput : {'raw_values', 'uniform_average'} or array-like - Defines aggregating of multiple output values. - Array-like value defines weights used to average errors. - If input is list then the shape must be (n_outputs,). - 'raw_values' : - Returns a full set of errors in case of multioutput input. - 'uniform_average' : - Errors of all outputs are averaged with uniform weight. + multioutput : str or array-like, default='uniform_average' + Defines aggregating of multiple output values, one of + {'raw_values', 'uniform_average'} or an array-like of shape + (n_outputs,). An array-like value defines weights used to average + errors. With 'raw_values', a full set of errors is returned in case + of multioutput input. With 'uniform_average', errors of all outputs + are averaged with uniform weight. Returns ------- loss : float or ndarray of floats in the range [0, 1/eps] If multioutput is 'raw_values', then mean absolute percentage error - is returned for each output separately. - If multioutput is 'uniform_average' or an ndarray of weights, then the - weighted average of all output errors is returned. - MAPE output is non-negative floating point. The best value is 0.0. - But note the fact that bad predictions can lead to arbitarily large - MAPE values, especially if some y_true values are very close to zero. - Note that we return a large value instead of `inf` when y_true is zero. + is returned for each output separately. If multioutput is + 'uniform_average' or an ndarray of weights, then the weighted average + of all output errors is returned. MAPE output is non-negative + floating point. The best value is 0.0. Note that bad predictions can + lead to arbitrarily large MAPE values, especially if some y_true + values are very close to zero. A large value is returned instead of + ``inf`` when y_true is zero. """ y_type, y_true, y_pred, sample_weight, multioutput = _check_reg_targets( y_true, y_pred, sample_weight, multioutput=multioutput) @@ -185,8 +178,6 @@ def mean_squared_error(y_true: np.ndarray, y_pred: np.ndarray, *, squared: bool = True) -> float: """Mean squared error regression loss. - Read more in the :ref:`User Guide `. - Parameters ---------- y_true : array-like of shape (n_samples,) or (n_samples, n_outputs) @@ -195,14 +186,13 @@ def mean_squared_error(y_true: np.ndarray, y_pred: np.ndarray, *, Estimated target values. sample_weight : array-like of shape (n_samples,), default=None Sample weights. - multioutput : {'raw_values', 'uniform_average'} or array-like of shape - (n_outputs,), default='uniform_average' - Defines aggregating of multiple output values. - Array-like value defines weights used to average errors. - 'raw_values' : - Returns a full set of errors in case of multioutput input. - 'uniform_average' : - Errors of all outputs are averaged with uniform weight. + multioutput : str or array-like, default='uniform_average' + Defines aggregating of multiple output values, one of + {'raw_values', 'uniform_average'} or an array-like of shape + (n_outputs,). An array-like value defines weights used to average + errors. With 'raw_values', a full set of errors is returned in case + of multioutput input. With 'uniform_average', errors of all outputs + are averaged with uniform weight. squared : bool, default=True If True returns MSE value, if False returns RMSE value. @@ -235,8 +225,6 @@ def mean_squared_log_error(y_true: np.ndarray, y_pred: np.ndarray, *, "uniform_average") -> float: """Mean squared logarithmic error regression loss. - Read more in the :ref:`User Guide `. - Parameters ---------- y_true : array-like of shape (n_samples,) or (n_samples, n_outputs) @@ -245,15 +233,13 @@ def mean_squared_log_error(y_true: np.ndarray, y_pred: np.ndarray, *, Estimated target values. sample_weight : array-like of shape (n_samples,), default=None Sample weights. - multioutput : {'raw_values', 'uniform_average'} or array-like of shape - (n_outputs,), default='uniform_average' - Defines aggregating of multiple output values. - Array-like value defines weights used to average errors. - 'raw_values' : - Returns a full set of errors when the input is of multioutput - format. - 'uniform_average' : - Errors of all outputs are averaged with uniform weight. + multioutput : str or array-like, default='uniform_average' + Defines aggregating of multiple output values, one of + {'raw_values', 'uniform_average'} or an array-like of shape + (n_outputs,). An array-like value defines weights used to average + errors. With 'raw_values', a full set of errors is returned in case + of multioutput input. With 'uniform_average', errors of all outputs + are averaged with uniform weight. Returns ------- @@ -279,34 +265,32 @@ def median_absolute_error(y_true: np.ndarray, y_pred: np.ndarray, *, sample_weight: np.ndarray | None = None) -> float: """Median absolute error regression loss. - Median absolute error output is non-negative floating point. The best value - is 0.0. Read more in the :ref:`User Guide `. + Median absolute error output is non-negative floating point. The best + value is 0.0. Parameters ---------- - y_true : array-like of shape = (n_samples) or (n_samples, n_outputs) + y_true : array-like of shape (n_samples,) or (n_samples, n_outputs) Ground truth (correct) target values. - y_pred : array-like of shape = (n_samples) or (n_samples, n_outputs) + y_pred : array-like of shape (n_samples,) or (n_samples, n_outputs) Estimated target values. - multioutput : {'raw_values', 'uniform_average'} or array-like of shape - (n_outputs,), default='uniform_average' - Defines aggregating of multiple output values. Array-like value defines - weights used to average errors. - 'raw_values' : - Returns a full set of errors in case of multioutput input. - 'uniform_average' : - Errors of all outputs are averaged with uniform weight. + multioutput : str or array-like, default='uniform_average' + Defines aggregating of multiple output values, one of + {'raw_values', 'uniform_average'} or an array-like of shape + (n_outputs,). An array-like value defines weights used to average + errors. With 'raw_values', a full set of errors is returned in case + of multioutput input. With 'uniform_average', errors of all outputs + are averaged with uniform weight. sample_weight : array-like of shape (n_samples,), default=None Sample weights. - .. versionadded:: 0.24 Returns ------- loss : float or ndarray of floats If multioutput is 'raw_values', then mean absolute error is returned - for each output separately. - If multioutput is 'uniform_average' or an ndarray of weights, then the - weighted average of all output errors is returned. + for each output separately. If multioutput is 'uniform_average' or + an ndarray of weights, then the weighted average of all output + errors is returned. """ y_type, y_true, y_pred, sample_weight, multioutput = _check_reg_targets( y_true, y_pred, sample_weight, multioutput=multioutput) @@ -328,7 +312,6 @@ def explained_variance_score(y_true: np.ndarray, y_pred: np.ndarray, *, """Explained variance regression score function. Best possible score is 1.0, lower values are worse. - Read more in the :ref:`User Guide `. Parameters ---------- @@ -338,17 +321,15 @@ def explained_variance_score(y_true: np.ndarray, y_pred: np.ndarray, *, Estimated target values. sample_weight : array-like of shape (n_samples,), default=None Sample weights. - multioutput : {'raw_values', 'uniform_average', 'variance_weighted'} or - array-like of shape (n_outputs,), default='uniform_average' - Defines aggregating of multiple output scores. - Array-like value defines weights used to average scores. - 'raw_values' : - Returns a full set of scores in case of multioutput input. - 'uniform_average' : - Scores of all outputs are averaged with uniform weight. - 'variance_weighted' : - Scores of all outputs are averaged, weighted by the variances - of each individual output. + multioutput : str or array-like, default='uniform_average' + Defines aggregating of multiple output scores, one of + {'raw_values', 'uniform_average', 'variance_weighted'} or an + array-like of shape (n_outputs,). An array-like value defines + weights used to average scores. With 'raw_values', a full set of + scores is returned in case of multioutput input. With + 'uniform_average', scores of all outputs are averaged with uniform + weight. With 'variance_weighted', scores of all outputs are + averaged, weighted by the variances of each individual output. Returns ------- @@ -375,16 +356,13 @@ def r2_score(y_true: np.ndarray, y_pred: np.ndarray, *, multioutput: (np.ndarray | Literal[ "raw_values", "uniform_average", "variance_weighted"] | None) = "uniform_average") -> float: - """ - R^2 (coefficient of determination) regression score function. + """R^2 (coefficient of determination) regression score function. Best possible score is 1.0 and it can be negative (because the model can be arbitrarily worse). A constant model that always predicts the expected value of y, disregarding the input features, would get a R^2 score of 0.0. - Read more in the :ref:`User Guide `. - Parameters ---------- y_true : array-like of shape (n_samples,) or (n_samples, n_outputs) @@ -393,20 +371,15 @@ def r2_score(y_true: np.ndarray, y_pred: np.ndarray, *, Estimated target values. sample_weight : array-like of shape (n_samples,), default=None Sample weights. - multioutput : {'raw_values', 'uniform_average', 'variance_weighted'}, - array-like of shape (n_outputs,) or None, default='uniform_average' - Defines aggregating of multiple output scores. - Array-like value defines weights used to average scores. - Default is "uniform_average". - 'raw_values' : - Returns a full set of scores in case of multioutput input. - 'uniform_average' : - Scores of all outputs are averaged with uniform weight. - 'variance_weighted' : - Scores of all outputs are averaged, weighted by the variances - of each individual output. - .. versionchanged:: 0.19 - Default value of multioutput is 'uniform_average'. + multioutput : str, array-like or None, default='uniform_average' + Defines aggregating of multiple output scores, one of + {'raw_values', 'uniform_average', 'variance_weighted'}, an + array-like of shape (n_outputs,) or None. An array-like value + defines weights used to average scores. With 'raw_values', a full + set of scores is returned in case of multioutput input. With + 'uniform_average', scores of all outputs are averaged with uniform + weight. With 'variance_weighted', scores of all outputs are + averaged, weighted by the variances of each individual output. Returns ------- @@ -416,16 +389,10 @@ def r2_score(y_true: np.ndarray, y_pred: np.ndarray, *, Notes ----- - This is not a symmetric function. - Unlike most other scores, R^2 score may be negative (it need not actually - be the square of a quantity R). - This metric is not well-defined for single samples and will return a NaN - value if n_samples is less than two. - - References - ---------- - .. [1] `Wikipedia entry on the Coefficient of determination - `_ + This is not a symmetric function. Unlike most other scores, the R^2 + score may be negative (it need not actually be the square of a quantity + R). This metric is not well-defined for single samples and will return a + NaN value if n_samples is less than two. """ y_type, y_true, y_pred, sample_weight, multioutput = _check_reg_targets( y_true, y_pred, sample_weight, multioutput=multioutput) @@ -439,10 +406,7 @@ def r2_score(y_true: np.ndarray, y_pred: np.ndarray, *, def max_error(y_true: np.ndarray, y_pred: np.ndarray) -> float: - """ - max_error metric calculates the maximum residual error. - - Read more in the :ref:`User Guide `. + """Calculate the maximum residual error. Parameters ---------- @@ -470,8 +434,6 @@ def mean_tweedie_deviance(y_true: np.ndarray, y_pred: np.ndarray, *, power: float = 0) -> float: """Mean Tweedie deviance regression loss. - Read more in the :ref:`User Guide `. - Parameters ---------- y_true : array-like of shape (n_samples,) @@ -481,21 +443,19 @@ def mean_tweedie_deviance(y_true: np.ndarray, y_pred: np.ndarray, *, sample_weight : array-like of shape (n_samples,), default=None Sample weights. power : float, default=0 - Tweedie power parameter. Either power <= 0 or power >= 1. - The higher `p` the less weight is given to extreme - deviations between true and predicted targets. - - power < 0: Extreme stable distribution. Requires: y_pred > 0. - - power = 0 : Normal distribution, output corresponds to - mean_squared_error. y_true and y_pred can be any real numbers. - - power = 1 : Poisson distribution. Requires: y_true >= 0 and - y_pred > 0. - - 1 < p < 2 : Compound Poisson distribution. Requires: y_true >= 0 - and y_pred > 0. - - power = 2 : Gamma distribution. Requires: y_true > 0 and y_pred > 0. - - power = 3 : Inverse Gaussian distribution. Requires: y_true > 0 - and y_pred > 0. - - otherwise : Positive stable distribution. Requires: y_true > 0 - and y_pred > 0. + Tweedie power parameter. Either power <= 0 or power >= 1. The higher + the power, the less weight is given to extreme deviations between + true and predicted targets. The distribution depends on power: + power < 0 is the extreme stable distribution (requires y_pred > 0); + power = 0 is the normal distribution (output corresponds to + mean_squared_error, y_true and y_pred can be any real numbers); + power = 1 is the Poisson distribution (requires y_true >= 0 and + y_pred > 0); 1 < power < 2 is the compound Poisson distribution + (requires y_true >= 0 and y_pred > 0); power = 2 is the Gamma + distribution (requires y_true > 0 and y_pred > 0); power = 3 is the + inverse Gaussian distribution (requires y_true > 0 and y_pred > 0); + otherwise it is the positive stable distribution (requires + y_true > 0 and y_pred > 0). Returns ------- @@ -516,9 +476,8 @@ def mean_poisson_deviance(y_true: np.ndarray, y_pred: np.ndarray, *, sample_weight: np.ndarray | None = None) -> float: """Mean Poisson deviance regression loss. - Poisson deviance is equivalent to the Tweedie deviance with - the power parameter `power=1`. - Read more in the :ref:`User Guide `. + Poisson deviance is equivalent to the Tweedie deviance with the power + parameter power=1. Parameters ---------- @@ -540,13 +499,11 @@ def mean_poisson_deviance(y_true: np.ndarray, y_pred: np.ndarray, *, def mean_gamma_deviance(y_true: np.ndarray, y_pred: np.ndarray, *, sample_weight: np.ndarray | None = None) -> float: - """ - Mean Gamma deviance regression loss. + """Mean Gamma deviance regression loss. - Gamma deviance is equivalent to the Tweedie deviance with - the power parameter `power=2`. It is invariant to scaling of - the target variable, and measures relative errors. - Read more in the :ref:`User Guide `. + Gamma deviance is equivalent to the Tweedie deviance with the power + parameter power=2. It is invariant to scaling of the target variable, + and measures relative errors. Parameters ---------- diff --git a/src/pyrcn/model_selection/_search.py b/src/pyrcn/model_selection/_search.py index f1fccf2..ec45b67 100644 --- a/src/pyrcn/model_selection/_search.py +++ b/src/pyrcn/model_selection/_search.py @@ -197,40 +197,41 @@ def cv_results_(self) -> dict: A dict with keys as column headers and values as columns. It can be imported into a pandas DataFrame. For instance the below - given table will be represented by a cv_results_ dict of: - - { - 'param_kernel' : masked_array(data = ['rbf', 'rbf', 'rbf'], - mask = False), - 'param_gamma' : masked_array(data = [0.1 0.2 0.3], mask = False), - 'split0_test_score' : [0.80, 0.84, 0.70], - 'split1_test_score' : [0.82, 0.50, 0.70], - 'mean_test_score' : [0.81, 0.67, 0.70], - 'std_test_score' : [0.01, 0.24, 0.00], - 'rank_test_score' : [1, 3, 2], - 'split0_train_score' : [0.80, 0.92, 0.70], - 'split1_train_score' : [0.82, 0.55, 0.70], - 'mean_train_score' : [0.81, 0.74, 0.70], - 'std_train_score' : [0.01, 0.19, 0.00], - 'mean_fit_time' : [0.73, 0.63, 0.43], - 'std_fit_time' : [0.01, 0.02, 0.01], - 'mean_score_time' : [0.01, 0.06, 0.04], - 'std_score_time' : [0.00, 0.00, 0.00], - 'params' : [{'kernel' : 'rbf', 'gamma' : 0.1}, ...], - } + given table will be represented by a ``cv_results_`` dict of:: + + { + 'param_kernel' : masked_array(data = ['rbf', 'rbf', 'rbf'], + mask = False), + 'param_gamma' : masked_array(data = [0.1 0.2 0.3], + mask = False), + 'split0_test_score' : [0.80, 0.84, 0.70], + 'split1_test_score' : [0.82, 0.50, 0.70], + 'mean_test_score' : [0.81, 0.67, 0.70], + 'std_test_score' : [0.01, 0.24, 0.00], + 'rank_test_score' : [1, 3, 2], + 'split0_train_score' : [0.80, 0.92, 0.70], + 'split1_train_score' : [0.82, 0.55, 0.70], + 'mean_train_score' : [0.81, 0.74, 0.70], + 'std_train_score' : [0.01, 0.19, 0.00], + 'mean_fit_time' : [0.73, 0.63, 0.43], + 'std_fit_time' : [0.01, 0.02, 0.01], + 'mean_score_time' : [0.01, 0.06, 0.04], + 'std_score_time' : [0.00, 0.00, 0.00], + 'params' : [{'kernel' : 'rbf', 'gamma' : 0.1}, ...], + } Note ---- The key 'params' is used to store a list of parameter settings dicts for all the parameter candidates. - The mean_fit_time, std_fit_time, mean_score_time and std_score_time - are all in seconds. + The mean_fit_time, std_fit_time, mean_score_time and + std_score_time are all in seconds. For multi-metric evaluation, the scores for all the scorers are - available in the cv_results_ dict at the keys ending with that scorer’s - name ('_') instead of '_score' shown above. - (‘split0_test_precision’, ‘mean_train_precision’ etc.) + available in the ``cv_results_`` dict at the keys ending with that + scorer's name ('_') instead of '_score' shown above + ('split0_test_precision', 'mean_train_precision' etc.). Returns ------- @@ -294,12 +295,12 @@ def best_params_(self) -> dict: @property def best_index_(self) -> int | np.integer: """ - The index (of the cv_results_ arrays) which corresponds to the best - candidate. + The index of the ``cv_results_`` arrays for the best candidate. - The dict at search.cv_results_['params'][search.best_index_] gives the - parameter setting for the best model, that gives the highest mean score - (search.best_score_). + The dict at + ``search.cv_results_['params'][search.best_index_]`` gives the + parameter setting for the best model, that gives the highest mean + score (``search.best_score_``). For multi-metric evaluation, this is present only if refit is specified. @@ -396,15 +397,16 @@ class SHGOSearchCV(BaseSearchCV): objective function. constraints : Dict, List[Dict], None, default=None Constraints definitions, where each definition is a dictionary with - fields: - type : str - Constraint type: ‘eq’ for equality, ‘ineq’ for inequality. - fun : Callable - The function defining the constraint. - jac : Optional[Callable] - The Jacobian of ``fun`` (only for SLSQP). - args : List, Tuple - Extra arguments to be passed to the function and Jacobian. + the following fields: + + - type : str + Constraint type: 'eq' for equality, 'ineq' for inequality. + - fun : Callable + The function defining the constraint. + - jac : Optional[Callable] + The Jacobian of ``fun`` (only for SLSQP). + - args : List, Tuple + Extra arguments to be passed to the function and Jacobian. refit : bool, default=True Refit an estimator using the best found parameters on the whole dataset. @@ -415,16 +417,17 @@ class SHGOSearchCV(BaseSearchCV): cv : int, cross-validation generator or an iterable, default=None Determines the cross-validation splitting strategy. Possible inputs for cv are: + - None, to use the default 5-fold cross validation, - - integer, to specify the number of folds in a `(Stratified)KFold`, - - :term:`CV splitter`, - - An iterable yielding (train, test) splits as arrays of indices. - For integer/None inputs, if the estimator is a classifier and ``y`` is - either binary or multiclass, :class:`StratifiedKFold` is used. In all - other cases, :class:`KFold` is used. These splitters are instantiated - with `shuffle=False` so the splits will be the same across calls. - Refer :ref:`User Guide ` for the various - cross-validation strategies that can be used here. + - integer, to specify the number of folds in a + ``(Stratified)KFold``, + - a CV splitter, + - an iterable yielding (train, test) splits as arrays of indices. + + For integer/None inputs, if the estimator is a classifier and ``y`` + is either binary or multiclass, ``StratifiedKFold`` is used. In all + other cases, ``KFold`` is used. These splitters are instantiated + with ``shuffle=False`` so the splits will be the same across calls. return_train_score : bool, default=False If ``False``, the ``cv_results_`` attribute will not include training scores. @@ -455,11 +458,11 @@ class SHGOSearchCV(BaseSearchCV): classes_ : ndarray of shape (n_classes,) The classes labels. This is present only if ``refit`` is specified and the underlying estimator is a classifier. - feature_names_in_ : ndarray of shape (`n_features_in_`,) - Names of features seen during :term:`fit`. Only defined if - `best_estimator_` is defined (see the documentation for the `refit` - parameter for more details) and that `best_estimator_` exposes - `feature_names_in_` when fit. + feature_names_in_ : ndarray of shape (``n_features_in_``,) + Names of features seen during fit. Only defined if + ``best_estimator_`` is defined (see the documentation for the + ``refit`` parameter for more details) and that ``best_estimator_`` + exposes ``feature_names_in_`` when fit. See Also -------- @@ -494,9 +497,9 @@ def fit(self, X: np.ndarray, y: np.ndarray | None = None, *, Target relative to X for classification or regression; None for unsupervised learning. groups : np.ndarray of shape(n_samples, ), default = None - Group labels for the samples used while splitting the dataset into - train/test set. Only used in conjunction with a "Group" :term:`cv` - instance (e.g., :class:`~sklearn.model_selection.GroupKFold`). + Group labels for the samples used while splitting the dataset + into train/test set. Only used in conjunction with a "Group" cv + instance (e.g., ``sklearn.model_selection.GroupKFold``). **fit_params : dict of str -> object Parameters passed to the ``fit`` method of the estimator. diff --git a/src/pyrcn/nn/__init__.py b/src/pyrcn/nn/__init__.py new file mode 100644 index 0000000..6aef0df --- /dev/null +++ b/src/pyrcn/nn/__init__.py @@ -0,0 +1,40 @@ +"""PyTorch building blocks for reservoir computing (``pyrcn.nn``). + +The tensor-level, ``torch.nn``-style components that power PyRCN's +scikit-learn estimators, usable directly for pure-PyTorch workflows: + +* reservoir layers -- :class:`Reservoir`, :class:`EulerReservoir` -- and their + single-step cells :class:`LeakyESNCell`, :class:`EulerESNCell`; +* the input feature map :class:`InputFeatureMap`; +* the closed-form ridge readout :class:`IncrementalRidge`. + +Weight initializers (spectral-radius normalization, sparse fan-in, the +minimum-complexity topologies, ...) live in :mod:`pyrcn.nn.init`. + +Examples +-------- +>>> import torch +>>> from pyrcn.nn import InputFeatureMap, IncrementalRidge, Reservoir, init +>>> generator = torch.Generator().manual_seed(0) +>>> feature_map = InputFeatureMap(1, 100) +>>> reservoir = Reservoir(100, spectral_radius=0.9, leakage=0.8) +>>> reservoir.set_recurrent_weights( +... init.normal_recurrent_weights(100, generator=generator)) +>>> x = torch.randn(1, 50, 1) # (batch, length, features) +>>> states, final_state = reservoir(feature_map(x[0]).unsqueeze(0)) +>>> _ = IncrementalRidge(alpha=1e-3).fit(states[0], torch.randn(50, 1)) +""" + +# Authors: Peter Steiner +# License: BSD 3 clause + +from . import init +from ._input import InputFeatureMap +from ._readout import IncrementalRidge, LinearReadout +from ._reservoir import (EulerESNCell, EulerReservoir, LeakyESNCell, + Reservoir) +from ._training import torch_generator, train_readout + +__all__ = ["Reservoir", "EulerReservoir", "LeakyESNCell", "EulerESNCell", + "InputFeatureMap", "IncrementalRidge", "LinearReadout", + "train_readout", "torch_generator", "init"] diff --git a/src/pyrcn/nn/_activations.py b/src/pyrcn/nn/_activations.py new file mode 100644 index 0000000..16e465e --- /dev/null +++ b/src/pyrcn/nn/_activations.py @@ -0,0 +1,23 @@ +"""Activation functions shared by the torch backend modules.""" + +# Authors: Peter Steiner +# License: BSD 3 clause + +from __future__ import annotations + +from collections.abc import Callable + +import torch + +#: Activations that ``nn.RNNCell`` provides as a fused single-step op. +FUSED = ("tanh", "relu") + +ACTIVATIONS: dict[str, Callable[[torch.Tensor], torch.Tensor]] = { + "tanh": torch.tanh, + "relu": torch.relu, + "logistic": torch.sigmoid, + "identity": lambda t: t, + "bounded_relu": lambda t: t.clamp(0.0, 1.0), +} + +SUPPORTED = tuple(ACTIVATIONS) diff --git a/src/pyrcn/nn/_bridge.py b/src/pyrcn/nn/_bridge.py new file mode 100644 index 0000000..c24db21 --- /dev/null +++ b/src/pyrcn/nn/_bridge.py @@ -0,0 +1,103 @@ +"""Bridge from the scikit-learn config blocks to the torch backend modules. + +Given a *fitted* PyRCN block (``InputToNode`` / ``NodeToNode`` family) or a +readout config, build the equivalent torch backend module with the block's +weights injected, so the torch compute reproduces the block's NumPy +``transform`` within floating-point precision. This is what lets the estimators +run their standard configuration entirely on the torch backend (the fast path) +while still falling back to the legacy NumPy path for arbitrary sub-estimators +(e.g. a ``FeatureUnion`` input stage or an external ``Ridge`` readout). + +Fast-path membership is decided by *exact* type: only blocks whose +``transform`` is the standard affine-plus-activation (input) or the standard +leaky/Euler recurrence (reservoir) qualify, so subclasses with bespoke +transforms (e.g. ``BatchIntrinsicPlasticity``) correctly take the fallback. +""" + +# Authors: Peter Steiner +# License: BSD 3 clause + +from __future__ import annotations + +import numpy as np +import torch + +from ..base.blocks import (EulerNodeToNode, HebbianNodeToNode, InputToNode, + NodeToNode, PredefinedWeightsInputToNode, + PredefinedWeightsNodeToNode) +from ..linear_model import IncrementalRegression +from ._input import InputFeatureMap +from ._readout import IncrementalRidge +from ._reservoir import EulerReservoir, Reservoir + +_INPUT_TYPES = (InputToNode, PredefinedWeightsInputToNode) +_LEAKY_TYPES = (NodeToNode, PredefinedWeightsNodeToNode, HebbianNodeToNode) +_EULER_TYPES = (EulerNodeToNode,) + + +def _dense(weights: object) -> np.ndarray: + if hasattr(weights, "toarray"): + return weights.toarray() # type: ignore[union-attr] + return np.asarray(weights) + + +def input_is_backable(block: object) -> bool: + """True if ``block``'s transform is the standard affine + activation.""" + return type(block) in _INPUT_TYPES + + +def node_is_backable(block: object) -> bool: + """True if ``block``'s transform is the standard reservoir recurrence.""" + return type(block) in _LEAKY_TYPES + _EULER_TYPES + + +def regressor_is_backable(regressor: object) -> bool: + """True if ``regressor`` is the ridge readout with normalize off.""" + return (type(regressor) is IncrementalRegression + and not regressor.normalize) + + +def build_input_map(block: InputToNode, *, dtype: torch.dtype | None = None, + device: torch.device | str | int | None = None + ) -> InputFeatureMap: + """Torch ``InputFeatureMap`` reproducing a fitted ``InputToNode``.""" + weights = _dense(block.input_weights) + bias = _dense(block.bias_weights) + feature_map = InputFeatureMap( + weights.shape[0], weights.shape[1], + input_scaling=block.input_scaling, input_shift=block.input_shift, + bias_scaling=block.bias_scaling, bias_shift=block.bias_shift, + activation=block.input_activation, dtype=dtype, device=device) + feature_map.set_input_weights(weights, bias) + return feature_map + + +def build_reservoir(block: NodeToNode, *, dtype: torch.dtype | None = None, + device: torch.device | str | int | None = None + ) -> Reservoir | EulerReservoir: + """Torch reservoir module reproducing a fitted ``NodeToNode``.""" + weights = _dense(block.recurrent_weights) + hidden_size = weights.shape[0] + reservoir: Reservoir | EulerReservoir + if isinstance(block, EulerNodeToNode): + reservoir = EulerReservoir( + hidden_size, recurrent_scaling=block.recurrent_scaling, + gamma=block.gamma, epsilon=block.epsilon, + activation=block.reservoir_activation, dtype=dtype, device=device) + else: + reservoir = Reservoir( + hidden_size, spectral_radius=block.spectral_radius, + leakage=block.leakage, activation=block.reservoir_activation, + bidirectional=block.bidirectional, dtype=dtype, device=device) + reservoir.set_recurrent_weights(weights) + return reservoir + + +def build_readout(regressor: IncrementalRegression, *, + dtype: torch.dtype | None = None, + device: torch.device | str | int | None = None + ) -> IncrementalRidge: + """Torch closed-form ridge readout from an ``IncrementalRegression``.""" + return IncrementalRidge( + alpha=regressor.alpha, fit_intercept=regressor.fit_intercept, + dtype=dtype, device=device) diff --git a/src/pyrcn/nn/_input.py b/src/pyrcn/nn/_input.py new file mode 100644 index 0000000..c2d33ad --- /dev/null +++ b/src/pyrcn/nn/_input.py @@ -0,0 +1,75 @@ +"""PyTorch input feature map built on ``nn.Linear``. + +This is the torch companion of ``InputToNode``: an affine input projection +(input weights + bias) with PyRCN's scaling/shift and an activation. It is the +feature map for ELMs and the input stage for ESNs. ``input_scaling`` is folded +into ``weight`` and the shifts/``bias_scaling`` into ``bias``, so ``forward`` +reduces to ``activation(super().forward(x))``. +""" + +# Authors: Peter Steiner +# License: BSD 3 clause + +from __future__ import annotations + +import torch +import torch.nn as nn + +from ._activations import ACTIVATIONS, SUPPORTED + + +class InputFeatureMap(nn.Linear): + """Affine input projection + activation, matching ``InputToNode``:: + + s = f(input_scaling * (x @ W_in) + input_shift + + bias_scaling * bias + bias_shift) + + Weights are assigned via :meth:`set_input_weights`; all parameters are + frozen (``requires_grad=False``). + """ + + def __init__(self, in_features: int, out_features: int, *, + input_scaling: float = 1.0, input_shift: float = 0.0, + bias_scaling: float = 1.0, bias_shift: float = 0.0, + activation: str = "tanh", + device: torch.device | str | int | None = None, + dtype: torch.dtype | None = None) -> None: + if activation not in SUPPORTED: + raise ValueError( + f"unknown activation {activation!r}; supported: " + f"{sorted(SUPPORTED)}") + super().__init__(in_features, out_features, bias=True, device=device, + dtype=dtype) + self.input_scaling = float(input_scaling) + self.input_shift = float(input_shift) + self.bias_scaling = float(bias_scaling) + self.bias_shift = float(bias_shift) + self.activation = activation + for p in self.parameters(): + p.requires_grad_(False) + + def set_input_trainable(self, trainable: bool = True) -> None: + """Enable/disable gradient training of the input weights and bias.""" + for p in self.parameters(): + p.requires_grad_(trainable) + + def set_input_weights(self, input_weights: object, + bias_weights: object) -> None: + """Load input weights ``(in_features, out_features)`` and bias. + + ``input_scaling`` is folded into ``weight`` and the shifts / + ``bias_scaling`` into ``bias`` so a plain ``nn.Linear`` reproduces + PyRCN's affine. + """ + w = torch.as_tensor( + input_weights, dtype=self.weight.dtype, device=self.weight.device) + b = torch.as_tensor( + bias_weights, dtype=self.weight.dtype, + device=self.weight.device).reshape(-1) + with torch.no_grad(): + self.weight.copy_(self.input_scaling * w.T) + self.bias.copy_( + self.bias_scaling * b + (self.input_shift + self.bias_shift)) + + def forward(self, input: torch.Tensor) -> torch.Tensor: + return ACTIVATIONS[self.activation](super().forward(input)) diff --git a/src/pyrcn/nn/_readout.py b/src/pyrcn/nn/_readout.py new file mode 100644 index 0000000..d5f4f6e --- /dev/null +++ b/src/pyrcn/nn/_readout.py @@ -0,0 +1,163 @@ +"""PyTorch readouts for :mod:`pyrcn.nn`. + +:class:`IncrementalRidge` is the closed-form (ridge) readout, the torch +companion of ``IncrementalRegression``: it accumulates the normal-equation +statistics ``K = sum(Z^T Z)`` and ``xTy = sum(Z^T y)`` over one or more +batches and solves ``(K + alpha I) W = xTy`` once (mirroring the sequence-fit +flow, where each sequence contributes with ``postpone_inverse=True`` and the +final call triggers the single solve). The statistics are additive, so two +readouts fitted on disjoint data merge with ``+`` (map-reduce). +``fit_intercept`` folds a bias into the weights by appending a column of ones +to ``Z``; its weights carry no gradient. + +:class:`LinearReadout` is the trainable counterpart -- a plain ``nn.Linear`` +whose weights are optimized by a gradient loop (see +:func:`pyrcn.nn.train_readout`), used when ``solver='gradient'``. +""" + +# Authors: Peter Steiner +# License: BSD 3 clause + +from __future__ import annotations + +import math + +import torch +import torch.nn as nn + + +class IncrementalRidge(nn.Module): + """Accumulate ``Z^T Z`` / ``Z^T y`` and solve ``(K + alpha I) W = xTy``. + + Parameters + ---------- + alpha : float, default=1e-5 + L2 (ridge) regularization strength. + fit_intercept : bool, default=True + Append a column of ones to the features so the last weight row is the + intercept. + """ + + def __init__(self, *, alpha: float = 1e-5, fit_intercept: bool = True, + device: torch.device | str | int | None = None, + dtype: torch.dtype | None = None) -> None: + super().__init__() + self.alpha = float(alpha) + self.fit_intercept = bool(fit_intercept) + self._device = device + self._dtype = dtype + self.register_buffer("_K", None) + self.register_buffer("_xTy", None) + self.register_buffer("output_weights", None) + + def _preprocess(self, Z: torch.Tensor) -> torch.Tensor: + if not self.fit_intercept: + return Z + ones = torch.ones(Z.shape[0], 1, dtype=Z.dtype, device=Z.device) + return torch.cat([Z, ones], dim=1) + + def partial_fit(self, Z: torch.Tensor, y: torch.Tensor, *, + reset: bool = False, + postpone_inverse: bool = False) -> IncrementalRidge: + """Accumulate one batch; solve unless ``postpone_inverse``. + + With ``postpone_inverse=True`` only the statistics are updated (used + for all but the last sequence); the following non-postponed call + performs the single closed-form solve over the accumulated data. + """ + if reset: + self._K = None + self._xTy = None + self.output_weights = None + Zp = self._preprocess(Z) + gram = Zp.T @ Zp + rhs = Zp.T @ y + self._K = gram if self._K is None else self._K + gram + self._xTy = rhs if self._xTy is None else self._xTy + rhs + if postpone_inverse and self.output_weights is None: + return self + self.solve() + return self + + def fit(self, Z: torch.Tensor, y: torch.Tensor) -> IncrementalRidge: + """Fit on a single batch (reset, accumulate, solve).""" + return self.partial_fit(Z, y, reset=True, postpone_inverse=False) + + def solve(self) -> IncrementalRidge: + """Solve ``(K + alpha I) W = xTy`` from the accumulated statistics.""" + if self._K is None or self._xTy is None: + raise RuntimeError("no statistics accumulated; call partial_fit") + identity = torch.eye( + self._K.shape[0], dtype=self._K.dtype, device=self._K.device) + self.output_weights = torch.linalg.solve( + self._K + self.alpha * identity, self._xTy) + return self + + def predict(self, Z: torch.Tensor) -> torch.Tensor: + """Return ``Z @ output_weights`` (intercept folded in).""" + if self.output_weights is None: + raise RuntimeError("readout is not fitted; call fit/partial_fit") + return self._preprocess(Z) @ self.output_weights + + def forward(self, Z: torch.Tensor) -> torch.Tensor: + return self.predict(Z) + + def __add__(self, other: IncrementalRidge) -> IncrementalRidge: + """Merge accumulated statistics (map-reduce over disjoint data). + + Returns an unsolved readout; call :meth:`solve` before predicting. + """ + if (self._K is None or other._K is None + or self._xTy is None or other._xTy is None): + raise RuntimeError( + "both readouts must have accumulated statistics") + merged = IncrementalRidge( + alpha=self.alpha, fit_intercept=self.fit_intercept, + device=self._device, dtype=self._dtype) + merged._K = self._K + other._K + merged._xTy = self._xTy + other._xTy + return merged + + +class LinearReadout(nn.Module): + """Trainable linear readout (a thin wrapper over ``nn.Linear``). + + The gradient-mode counterpart of :class:`IncrementalRidge`: its weights + are trainable ``Parameter`` objects optimized by + :func:`pyrcn.nn.train_readout` + rather than solved in closed form. + + Parameters + ---------- + in_features, out_features : int + Readout input and output sizes. + fit_intercept : bool, default=True + Whether the underlying ``nn.Linear`` has a bias term. + """ + + def __init__(self, in_features: int, out_features: int, *, + fit_intercept: bool = True, + generator: torch.Generator | None = None, + device: torch.device | str | int | None = None, + dtype: torch.dtype | None = None) -> None: + super().__init__() + self.linear = nn.Linear( + in_features, out_features, bias=fit_intercept, device=device, + dtype=dtype) + if generator is not None: + # Reproducible init from a seeded generator (nn.Linear's default + # init draws from torch's global RNG). + bound = 1.0 / math.sqrt(in_features) + with torch.no_grad(): + self.linear.weight.uniform_(-bound, bound, generator=generator) + if self.linear.bias is not None: + self.linear.bias.uniform_( + -bound, bound, generator=generator) + + def forward(self, Z: torch.Tensor) -> torch.Tensor: + return self.linear(Z) + + def predict(self, Z: torch.Tensor) -> torch.Tensor: + """Return predictions ``(n_samples, out_features)`` without grad.""" + with torch.no_grad(): + return self.linear(Z) diff --git a/src/pyrcn/nn/_reservoir.py b/src/pyrcn/nn/_reservoir.py new file mode 100644 index 0000000..53ae10c --- /dev/null +++ b/src/pyrcn/nn/_reservoir.py @@ -0,0 +1,222 @@ +"""PyTorch reservoir cells and layers built on ``nn.RNNCell``. + +Reuses torch's fused single-step cell op for ``tanh``/``relu`` and adds only +what torch lacks: leaky integration (:class:`LeakyESNCell`), Euler / EuSN +integration (:class:`EulerESNCell`), and the extra PyRCN activations +(``logistic``/``identity``/``bounded_relu``). In every cell ``weight_ih`` is a +frozen identity (the reservoir input is added directly; input weights belong to +``InputToNode``); the effective recurrent matrix is folded into ``weight_hh``. +""" + +# Authors: Peter Steiner +# License: BSD 3 clause + +from __future__ import annotations + +import torch +import torch.nn as nn +import torch.nn.functional as F + +from ._activations import ACTIVATIONS, FUSED, SUPPORTED + + +class _ReservoirCell(nn.RNNCell): + """Shared machinery for the reservoir cells. + + Sets ``weight_ih`` to a frozen identity, freezes all parameters, and + provides :meth:`_activate` = ``f(x + h @ weight_hh)`` (reusing the fused op + for ``tanh``/``relu``). Subclasses fold their effective recurrent matrix + into ``weight_hh`` and combine ``_activate`` into their state update. + """ + + def __init__(self, hidden_size: int, activation: str, + device: torch.device | str | int | None, + dtype: torch.dtype | None) -> None: + if activation not in SUPPORTED: + raise ValueError( + f"unknown activation {activation!r}; supported: " + f"{sorted(SUPPORTED)}") + nonlinearity = activation if activation in FUSED else "tanh" + super().__init__(input_size=hidden_size, hidden_size=hidden_size, + bias=False, nonlinearity=nonlinearity, device=device, + dtype=dtype) + self.activation = activation + with torch.no_grad(): + self.weight_ih.copy_(torch.eye( + hidden_size, device=device, dtype=self.weight_ih.dtype)) + for p in self.parameters(): + p.requires_grad_(False) + + def _as_square(self, weights: object) -> torch.Tensor: + w = torch.as_tensor( + weights, dtype=self.weight_hh.dtype, device=self.weight_hh.device) + if w.shape != (self.hidden_size, self.hidden_size): + raise ValueError( + f"expected weights of shape " + f"{(self.hidden_size, self.hidden_size)}, got " + f"{tuple(w.shape)}") + return w + + def _zeros_like_state(self, x: torch.Tensor) -> torch.Tensor: + return torch.zeros( + x.shape[0], self.hidden_size, dtype=x.dtype, device=x.device) + + def _activate(self, x: torch.Tensor, h: torch.Tensor) -> torch.Tensor: + if self.activation in FUSED: + return super().forward(x, h) # fused cell op + pre = x + F.linear(h, self.weight_hh) # weight_ih is I + return ACTIVATIONS[self.activation](pre) + + +class LeakyESNCell(_ReservoirCell): + """Leaky reservoir step. + + ``h' = (1 - leakage) * h + leakage * f(x + spectral_radius * (h @ W))`` + """ + + def __init__(self, hidden_size: int, spectral_radius: float = 1.0, + leakage: float = 1.0, activation: str = "tanh", + device: torch.device | str | int | None = None, + dtype: torch.dtype | None = None) -> None: + super().__init__(hidden_size, activation, device, dtype) + self.spectral_radius = float(spectral_radius) + self.leakage = float(leakage) + + def set_recurrent_weights(self, weights: object) -> None: + """Load ``(hidden_size, hidden_size)`` weights (transposed, scaled).""" + w = self._as_square(weights) + with torch.no_grad(): + self.weight_hh.copy_(self.spectral_radius * w.T) + + def forward(self, input: torch.Tensor, + hx: torch.Tensor | None = None) -> torch.Tensor: + if hx is None: + hx = self._zeros_like_state(input) + return (1.0 - self.leakage) * hx + self.leakage * self._activate( + input, hx) + + +class EulerESNCell(_ReservoirCell): + """Euler (EuSN) reservoir step. + + ``h' = h + epsilon * f(x + h @ (recurrent_scaling * W + gamma * I))`` + """ + + def __init__(self, hidden_size: int, recurrent_scaling: float = 1.0, + gamma: float = 0.001, epsilon: float = 0.01, + activation: str = "tanh", + device: torch.device | str | int | None = None, + dtype: torch.dtype | None = None) -> None: + super().__init__(hidden_size, activation, device, dtype) + self.recurrent_scaling = float(recurrent_scaling) + self.gamma = float(gamma) + self.epsilon = float(epsilon) + + def set_recurrent_weights(self, weights: object) -> None: + """Fold ``recurrent_scaling * W + gamma * I`` into ``weight_hh``.""" + w = self._as_square(weights) + identity = torch.eye( + self.hidden_size, dtype=w.dtype, device=w.device) + effective = self.recurrent_scaling * w + self.gamma * identity + with torch.no_grad(): + self.weight_hh.copy_(effective.T) + + def forward(self, input: torch.Tensor, + hx: torch.Tensor | None = None) -> torch.Tensor: + if hx is None: + hx = self._zeros_like_state(input) + return hx + self.epsilon * self._activate(input, hx) + + +def _iterate(cell: _ReservoirCell, x: torch.Tensor, + h: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]: + outputs = [] + for t in range(x.shape[1]): + h = cell(x[:, t, :], h) + outputs.append(h) + return torch.stack(outputs, dim=1), h + + +class Reservoir(nn.Module): + """Run a :class:`LeakyESNCell` over batched (padded) sequences. + + ``forward`` takes ``x`` of shape ``(n_sequences, length, hidden_size)`` and + an optional ``initial_state`` ``(n_sequences, hidden_size)`` (zeros by + default), returning ``(states, final_state)``. With ``bidirectional=True`` + the same cell is run forward and over the time-reversed input (shared + weights, as ``NodeToNode`` does); the two state sequences are concatenated + on the feature axis, so ``states`` has ``2 * hidden_size`` features. + """ + + def __init__(self, hidden_size: int, spectral_radius: float = 1.0, + leakage: float = 1.0, activation: str = "tanh", + bidirectional: bool = False, + device: torch.device | str | int | None = None, + dtype: torch.dtype | None = None) -> None: + super().__init__() + self.hidden_size = hidden_size + self.bidirectional = bidirectional + self.cell = LeakyESNCell( + hidden_size, spectral_radius=spectral_radius, leakage=leakage, + activation=activation, device=device, dtype=dtype) + + def set_recurrent_weights(self, weights: object) -> None: + """Load the recurrent weight matrix into the cell.""" + self.cell.set_recurrent_weights(weights) + + def set_recurrent_trainable(self, trainable: bool = True) -> None: + """Enable/disable gradient training of the recurrent weights.""" + self.cell.weight_hh.requires_grad_(trainable) + + def forward(self, x: torch.Tensor, + initial_state: torch.Tensor | None = None + ) -> tuple[torch.Tensor, torch.Tensor]: + if not self.bidirectional: + if initial_state is None: + initial_state = self.cell._zeros_like_state(x) + return _iterate(self.cell, x, initial_state) + if initial_state is not None: + raise ValueError( + "initial_state is not supported with bidirectional=True") + states_fw, final_fw = _iterate( + self.cell, x, self.cell._zeros_like_state(x)) + reversed_states, final_bw = _iterate( + self.cell, torch.flip(x, dims=[1]), + self.cell._zeros_like_state(x)) + states = torch.cat( + [states_fw, torch.flip(reversed_states, dims=[1])], dim=-1) + return states, torch.cat([final_fw, final_bw], dim=-1) + + +class EulerReservoir(nn.Module): + """Run an :class:`EulerESNCell` over batched sequences (unidirectional). + + ``forward`` matches :class:`Reservoir` (``(states, final_state)``); + ``EulerNodeToNode`` is single-direction. + """ + + def __init__(self, hidden_size: int, recurrent_scaling: float = 1.0, + gamma: float = 0.001, epsilon: float = 0.01, + activation: str = "tanh", + device: torch.device | str | int | None = None, + dtype: torch.dtype | None = None) -> None: + super().__init__() + self.hidden_size = hidden_size + self.cell = EulerESNCell( + hidden_size, recurrent_scaling=recurrent_scaling, gamma=gamma, + epsilon=epsilon, activation=activation, device=device, dtype=dtype) + + def set_recurrent_weights(self, weights: object) -> None: + """Load the recurrent weight matrix into the cell.""" + self.cell.set_recurrent_weights(weights) + + def set_recurrent_trainable(self, trainable: bool = True) -> None: + """Enable/disable gradient training of the recurrent weights.""" + self.cell.weight_hh.requires_grad_(trainable) + + def forward(self, x: torch.Tensor, + initial_state: torch.Tensor | None = None + ) -> tuple[torch.Tensor, torch.Tensor]: + if initial_state is None: + initial_state = self.cell._zeros_like_state(x) + return _iterate(self.cell, x, initial_state) diff --git a/src/pyrcn/nn/_training.py b/src/pyrcn/nn/_training.py new file mode 100644 index 0000000..87a0d08 --- /dev/null +++ b/src/pyrcn/nn/_training.py @@ -0,0 +1,100 @@ +"""Gradient-based training utility for :mod:`pyrcn.nn` readouts. + +A minimal optimizer loop that trains a readout module (typically +:class:`~pyrcn.nn.LinearReadout`) on precomputed features/states. Optimizer +and loss are selected by name so the choice stays serializable and tunable +via scikit-learn model selection. ``weight_decay`` supplies the L2 (ridge) +regularization, so with a fixed reservoir this converges to the closed-form +:class:`~pyrcn.nn.IncrementalRidge` solution. +""" + +# Authors: Peter Steiner +# License: BSD 3 clause + +from __future__ import annotations + +import numpy as np +import torch +import torch.nn as nn +from sklearn.utils import check_random_state + +OPTIMIZERS = { + "adam": torch.optim.Adam, + "adamw": torch.optim.AdamW, + "sgd": torch.optim.SGD, + "rmsprop": torch.optim.RMSprop, + "adagrad": torch.optim.Adagrad, +} +LOSSES = { + "mse": nn.MSELoss, + "mae": nn.L1Loss, + "huber": nn.HuberLoss, +} + + +def torch_generator( + random_state: int | np.random.RandomState | None +) -> torch.Generator: + """Return a ``torch.Generator`` seeded deterministically from a + scikit-learn ``random_state`` (int / ``RandomState`` / ``None``).""" + rng = check_random_state(random_state) + seed = int(rng.randint(np.iinfo(np.int32).max)) + return torch.Generator().manual_seed(seed) + + +def train_readout(readout: nn.Module, Z: torch.Tensor, y: torch.Tensor, *, + optimizer: str = "adam", learning_rate: float = 1e-3, + epochs: int = 100, batch_size: int | None = None, + weight_decay: float = 0.0, loss: str = "mse", + generator: torch.Generator | None = None) -> nn.Module: + """Train ``readout`` on ``(Z, y)`` with a gradient optimizer loop. + + Parameters + ---------- + readout : nn.Module + The readout to train in place (e.g. a + :class:`~pyrcn.nn.LinearReadout`). + Z : torch.Tensor of shape (n_samples, n_features) + Readout inputs (reservoir states / input features). + y : torch.Tensor of shape (n_samples, n_targets) + Training targets. + optimizer : {"adam", "sgd"}, default="adam" + learning_rate : float, default=1e-3 + epochs : int, default=100 + batch_size : int or None, default=None + Mini-batch size; ``None`` uses full-batch updates. + weight_decay : float, default=0.0 + L2 penalty (ridge strength). + loss : {"mse"}, default="mse" + generator : torch.Generator or None, default=None + Seeds the mini-batch shuffling for reproducibility. + + Returns + ------- + readout : nn.Module + The trained readout (same object). + """ + if optimizer not in OPTIMIZERS: + raise ValueError( + f"unknown optimizer {optimizer!r}; supported: " + f"{sorted(OPTIMIZERS)}") + if loss not in LOSSES: + raise ValueError( + f"unknown loss {loss!r}; supported: {sorted(LOSSES)}") + opt = OPTIMIZERS[optimizer]( + readout.parameters(), lr=learning_rate, weight_decay=weight_decay) + loss_fn = LOSSES[loss]() + n_samples = Z.shape[0] + step = n_samples if batch_size is None else int(batch_size) + readout.train() + for _ in range(int(epochs)): + permutation = torch.randperm( + n_samples, generator=generator, device=Z.device) + for start in range(0, n_samples, step): + index = permutation[start:start + step] + opt.zero_grad() + batch_loss = loss_fn(readout(Z[index]), y[index]) + batch_loss.backward() + opt.step() + readout.eval() + return readout diff --git a/src/pyrcn/nn/init.py b/src/pyrcn/nn/init.py new file mode 100644 index 0000000..eebd0f7 --- /dev/null +++ b/src/pyrcn/nn/init.py @@ -0,0 +1,143 @@ +"""Weight initializers for the :mod:`pyrcn.nn` reservoir components. + +Random and sparse recurrent designs are normalized to unit spectral radius; +a reservoir's ``spectral_radius`` then scales them at runtime. The +minimum-complexity topologies (Rodan & Tino, 2011) are deterministic +structured matrices whose scale is the ``forward_weight`` (and, for the +delay-line-with-feedback design, the ``feedback_weight``). Input designs +cover dense/sparse uniform weights, uniform bias, and signed-constant +(Bernoulli) input weights. + +All initializers accept an optional :class:`torch.Generator` for +reproducibility and the usual ``dtype`` / ``device`` placement arguments. +""" + +# Authors: Peter Steiner +# License: BSD 3 clause + +from __future__ import annotations + +import torch + + +def spectral_normalize(weights: torch.Tensor, + radius: float = 1.0) -> torch.Tensor: + """Scale a square matrix to the given maximum absolute eigenvalue.""" + eigenvalues = torch.linalg.eigvals(weights) + return weights * (radius / eigenvalues.abs().max()) + + +def normal_recurrent_weights( + hidden_size: int, *, fan_in: int | None = None, + generator: torch.Generator | None = None, + dtype: torch.dtype | None = None, + device: torch.device | str | int | None = None) -> torch.Tensor: + """Normally distributed recurrent weights, unit spectral radius. + + If ``fan_in`` is given and smaller than ``hidden_size``, exactly ``fan_in`` + entries are kept per column (the rest zeroed) before normalization. + """ + weights = torch.randn(hidden_size, hidden_size, generator=generator, + dtype=dtype, device=device) + if fan_in is not None and fan_in < hidden_size: + for column in range(hidden_size): + order = torch.randperm( + hidden_size, generator=generator, device=device) + weights[order[fan_in:], column] = 0.0 + return spectral_normalize(weights) + + +def antisymmetric_recurrent_weights( + hidden_size: int, *, fan_in: int | None = None, + generator: torch.Generator | None = None, + dtype: torch.dtype | None = None, + device: torch.device | str | int | None = None) -> torch.Tensor: + """Antisymmetric uniform recurrent weights ``U - U.T`` (used by EuSN). + + ``U`` is uniform in ``[-1, 1)`` (optionally sparsified to ``fan_in`` + entries per column before antisymmetrization). + """ + u = torch.rand(hidden_size, hidden_size, generator=generator, dtype=dtype, + device=device) * 2.0 - 1.0 + if fan_in is not None and fan_in < hidden_size: + for column in range(hidden_size): + order = torch.randperm( + hidden_size, generator=generator, device=device) + u[order[fan_in:], column] = 0.0 + return u - u.T + + +def uniform_input_weights( + n_features: int, hidden_size: int, *, fan_in: int | None = None, + generator: torch.Generator | None = None, + dtype: torch.dtype | None = None, + device: torch.device | str | int | None = None) -> torch.Tensor: + """Uniform input weights in ``[-1, 1)`` of shape ``(n_features, hidden)``. + + If ``fan_in`` is given, exactly ``fan_in`` entries are kept per column + (hidden unit). + """ + weights = torch.rand(n_features, hidden_size, generator=generator, + dtype=dtype, device=device) * 2.0 - 1.0 + if fan_in is not None and fan_in < n_features: + for column in range(hidden_size): + order = torch.randperm( + n_features, generator=generator, device=device) + weights[order[fan_in:], column] = 0.0 + return weights + + +def uniform_bias_weights( + hidden_size: int, *, generator: torch.Generator | None = None, + dtype: torch.dtype | None = None, + device: torch.device | str | int | None = None) -> torch.Tensor: + """Uniform bias in ``[-1, 1)`` of shape ``(hidden_size,)``.""" + return torch.rand(hidden_size, generator=generator, dtype=dtype, + device=device) * 2.0 - 1.0 + + +def bernoulli_input_weights( + n_features: int, hidden_size: int, *, value: float = 1.0, + p: float = 0.5, generator: torch.Generator | None = None, + dtype: torch.dtype | None = None, + device: torch.device | str | int | None = None) -> torch.Tensor: + """Signed-constant input weights ``+/- value`` (min-complexity ESNs).""" + probabilities = torch.full((n_features, hidden_size), p, dtype=dtype, + device=device) + signs = torch.bernoulli(probabilities, generator=generator) + return (2.0 * signs - 1.0) * value + + +def simple_cycle_weights( + hidden_size: int, forward_weight: float = 0.9, *, + dtype: torch.dtype | None = None, + device: torch.device | str | int | None = None) -> torch.Tensor: + """Simple Cycle Reservoir: one cycle with weight ``forward_weight``.""" + weights = torch.zeros(hidden_size, hidden_size, dtype=dtype, device=device) + for i in range(hidden_size): + weights[i, i - 1] = forward_weight # subdiagonal + corner + return weights + + +def delay_line_weights( + hidden_size: int, forward_weight: float = 0.9, *, + dtype: torch.dtype | None = None, + device: torch.device | str | int | None = None) -> torch.Tensor: + """Delay Line Reservoir: subdiagonal only (nilpotent).""" + weights = torch.zeros(hidden_size, hidden_size, dtype=dtype, device=device) + for i in range(1, hidden_size): + weights[i, i - 1] = forward_weight + return weights + + +def delay_line_feedback_weights( + hidden_size: int, forward_weight: float = 0.9, + feedback_weight: float = 0.1, *, + dtype: torch.dtype | None = None, + device: torch.device | str | int | None = None) -> torch.Tensor: + """Delay Line Reservoir with feedback: subdiagonal + superdiagonal.""" + weights = torch.zeros(hidden_size, hidden_size, dtype=dtype, device=device) + for i in range(1, hidden_size): + weights[i, i - 1] = forward_weight # forward + weights[i - 1, i] = feedback_weight # feedback + return weights diff --git a/src/pyrcn/util/__init__.py b/src/pyrcn/util/__init__.py index 3909332..a622752 100644 --- a/src/pyrcn/util/__init__.py +++ b/src/pyrcn/util/__init__.py @@ -7,8 +7,10 @@ from __future__ import annotations from ._feature_extractor import FeatureExtractor +from ._sequences import SequenceBatch, check_sequences from ._util import (argument_parser, batched, concatenate_sequences, get_mnist, new_logger, value_to_tuple) __all__ = ('new_logger', 'get_mnist', 'argument_parser', 'FeatureExtractor', - 'concatenate_sequences', 'value_to_tuple', 'batched') + 'concatenate_sequences', 'value_to_tuple', 'batched', + 'check_sequences', 'SequenceBatch') diff --git a/src/pyrcn/util/_sequences.py b/src/pyrcn/util/_sequences.py new file mode 100644 index 0000000..1f4a424 --- /dev/null +++ b/src/pyrcn/util/_sequences.py @@ -0,0 +1,209 @@ +"""Normalization of sequence input into a canonical padded batch. + +This is the single place where the various accepted public input forms (a 2-D +single sequence, a list / object-array of 2-D sequences, or a 3-D equal-length +batch) are validated and converted to the canonical representation consumed by +the redesigned backend: a zero-padded ``(N, L_max, n_features)`` array plus a +per-sequence ``lengths`` vector. It replaces the ad-hoc +``concatenate_sequences`` plus the ``ndim``-based sequence heuristics. +""" + +# Authors: Peter Steiner +# License: BSD 3 clause + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Literal + +import numpy as np + +Task = Literal["sequence-to-sequence", "sequence-to-value"] + + +@dataclass +class SequenceBatch: + """Canonical, validated batch of sequences. + + Attributes + ---------- + X : np.ndarray of shape (n_sequences, max_length, n_features) + Zero-padded input sequences. + lengths : np.ndarray of shape (n_sequences,) + The true (unpadded) length of each sequence. + task : {"sequence-to-sequence", "sequence-to-value"} or None + Resolved task kind. ``None`` when no targets were supplied. + n_features : int + Number of input features. + y : np.ndarray or None + Normalized targets. For ``"sequence-to-sequence"`` this is + ``(n_sequences, max_length, n_targets)`` (zero-padded); for + ``"sequence-to-value"`` this is ``(n_sequences, n_targets)``. + n_targets : int or None + Number of target outputs, or ``None`` when no targets were supplied. + single_sequence : bool + ``True`` when the input was a single 2-D ``(T, n_features)`` array (so + the caller should return plain, non-object output). + """ + + X: np.ndarray + lengths: np.ndarray + task: Task | None + n_features: int + y: np.ndarray | None = None + n_targets: int | None = None + single_sequence: bool = False + + +def _as_2d(x: object) -> np.ndarray: + arr = np.asarray(x, dtype=float) + if arr.ndim != 2: + raise ValueError( + f"each sequence must be 2-D (n_samples, n_features), got ndim " + f"{arr.ndim}") + return arr + + +def _as_sequence_list(X: object) -> tuple[list[np.ndarray], bool]: + """Return X as a list of 2-D arrays plus a single-sequence flag.""" + if isinstance(X, np.ndarray): + if X.dtype == object: + if X.ndim != 1: + raise ValueError( + f"object array of sequences must be 1-D, got ndim " + f"{X.ndim}") + return [_as_2d(x) for x in X], False + if X.ndim == 2: + return [np.asarray(X, dtype=float)], True + if X.ndim == 3: + return [np.asarray(x, dtype=float) for x in X], False + raise ValueError( + f"X array must be 2-D, 3-D or a 1-D object array, got ndim " + f"{X.ndim}") + if isinstance(X, (list, tuple)): + if len(X) == 0: + raise ValueError("empty sequence input") + return [_as_2d(x) for x in X], False + raise ValueError(f"unsupported X type {type(X)!r}") + + +def _pad_inputs( + seqs: list[np.ndarray]) -> tuple[np.ndarray, np.ndarray, int]: + n_features = seqs[0].shape[1] + for s in seqs: + if s.shape[1] != n_features: + raise ValueError( + f"inconsistent n_features across sequences: {s.shape[1]} vs " + f"{n_features}") + lengths = np.array([s.shape[0] for s in seqs], dtype=int) + max_length = int(lengths.max()) + X = np.zeros((len(seqs), max_length, n_features), dtype=float) + for i, s in enumerate(seqs): + X[i, :s.shape[0]] = s + return X, lengths, n_features + + +def _as_target_list(y: object, single_sequence: bool) -> list[np.ndarray]: + if single_sequence: + return [np.asarray(y)] + if isinstance(y, (list, tuple)): + return [np.asarray(v) for v in y] + if isinstance(y, np.ndarray): + return [np.asarray(v) for v in y] + raise ValueError(f"unsupported y type {type(y)!r}") + + +def _resolve_task( + y_list: list[np.ndarray], lengths: np.ndarray, + task: Literal["auto"] | Task) -> Task: + per_timestep = [ + yi.ndim >= 1 and yi.shape[0] == length + for yi, length in zip(y_list, lengths)] + if task == "sequence-to-sequence": + if not all(per_timestep): + raise ValueError( + "task='sequence-to-sequence' but some targets are not aligned " + "per timestep with their sequence") + return task + if task == "sequence-to-value": + return task + if task != "auto": + raise ValueError(f"unknown task {task!r}") + if all(per_timestep): + return "sequence-to-sequence" + if not any(per_timestep): + return "sequence-to-value" + raise ValueError( + "targets mix sequence-to-sequence and sequence-to-value shapes; pass " + "task= explicitly") + + +def _pad_targets( + y_list: list[np.ndarray], + lengths: np.ndarray) -> tuple[np.ndarray, int]: + rows = [yi.reshape(yi.shape[0], -1) for yi in y_list] + n_targets = rows[0].shape[1] + for a in rows: + if a.shape[1] != n_targets: + raise ValueError("inconsistent n_targets across sequences") + max_length = int(lengths.max()) + Y = np.zeros((len(rows), max_length, n_targets), dtype=float) + for i, a in enumerate(rows): + Y[i, :a.shape[0]] = a + return Y, n_targets + + +def _stack_values(y_list: list[np.ndarray]) -> tuple[np.ndarray, int]: + rows = [np.atleast_1d(yi).reshape(-1) for yi in y_list] + n_targets = rows[0].shape[0] + for r in rows: + if r.shape[0] != n_targets: + raise ValueError("inconsistent n_targets across sequences") + return np.stack(rows, axis=0), n_targets + + +def check_sequences( + X: object, y: object = None, *, + task: Literal["auto"] | Task = "auto") -> SequenceBatch: + """Validate and normalize sequence input to a canonical padded batch. + + Parameters + ---------- + X : list/tuple of 2-D arrays, 1-D object array of 2-D arrays, a 2-D + ``(T, n_features)`` array (a single sequence), or a 3-D + ``(n_sequences, length, n_features)`` array. + y : optional targets, parallel to ``X``. ``None`` for prediction. + task : {"auto", "sequence-to-sequence", "sequence-to-value"} + Default "auto". + ``"auto"`` infers the task from the target shapes (per-timestep targets + whose leading dimension equals the sequence length are treated as + sequence-to-sequence); pass an explicit value to resolve the ambiguous + case where a per-sequence value has the same length as the sequence. + + Returns + ------- + batch : SequenceBatch + """ + seqs, single_sequence = _as_sequence_list(X) + X_batch, lengths, n_features = _pad_inputs(seqs) + + if y is None: + resolved: Task | None = None if task == "auto" else task + return SequenceBatch( + X=X_batch, lengths=lengths, task=resolved, n_features=n_features, + single_sequence=single_sequence) + + y_list = _as_target_list(y, single_sequence) + if len(y_list) != len(seqs): + raise ValueError( + f"X and y have different numbers of sequences: {len(seqs)} vs " + f"{len(y_list)}") + + resolved = _resolve_task(y_list, lengths, task) + if resolved == "sequence-to-sequence": + y_batch, n_targets = _pad_targets(y_list, lengths) + else: + y_batch, n_targets = _stack_values(y_list) + return SequenceBatch( + X=X_batch, lengths=lengths, task=resolved, n_features=n_features, + y=y_batch, n_targets=n_targets, single_sequence=single_sequence) diff --git a/src/pyrcn/util/_util.py b/src/pyrcn/util/_util.py index f6bb835..18f0278 100644 --- a/src/pyrcn/util/_util.py +++ b/src/pyrcn/util/_util.py @@ -50,10 +50,7 @@ def batched(iterable: Iterable, n: int) -> Iterator[tuple]: Notes ----- - Starting from Python 3.12, this is included in `itertools`_. - - .. _itertools: - https://docs.python.org/3/library/itertools.html#itertools.batched + Starting from Python 3.12, this is available as ``itertools.batched``. """ # batched('ABCDEFG', 3) --> ABC DEF G if n < 1: diff --git a/tests/test_activations.py b/tests/test_activations.py index 95e729f..fde341e 100644 --- a/tests/test_activations.py +++ b/tests/test_activations.py @@ -71,14 +71,6 @@ def test_logistic() -> None: np.testing.assert_array_equal(X, X_true) -def test_softplus() -> None: - print('\ttest_softplus():') - X = np.arange(-5, 5, dtype=float) - X_true = np.log(1 + np.exp(X)) - ACTIVATIONS["softplus"](X) - np.testing.assert_array_equal(X, X_true) - - def test_relu() -> None: print('\ttest_relu():') X = np.concatenate((np.full((1, ), -np.inf), np.arange(-5, 5), @@ -91,16 +83,6 @@ def test_relu() -> None: np.testing.assert_array_equal(X, X_true) -def test_softmax() -> None: - print('\ttest_softmax():') - X = np.arange(-5, 5, dtype=float) - X_true = np.exp(X) / np.exp(X).sum() - ACTIVATIONS["softmax"](X) - np.testing.assert_array_equal(X, X_true) - assert any([not X.sum() == 1.0, not X_true.sum() == 1.0, - not X.sum() == X_true.sum()]) is False - - def test_tanh() -> None: print('\ttest_l():') X = np.concatenate((np.full((1, ), -np.inf), np.arange(-5, 5), diff --git a/tests/test_backend_bridge.py b/tests/test_backend_bridge.py new file mode 100644 index 0000000..c5b349b --- /dev/null +++ b/tests/test_backend_bridge.py @@ -0,0 +1,125 @@ +"""The bridge builds torch modules from fitted blocks that reproduce the +block's NumPy ``transform``, and correctly classifies fast-path membership. +""" +from __future__ import annotations + +import numpy as np +import torch +from sklearn.linear_model import Ridge +from sklearn.pipeline import FeatureUnion + +from pyrcn.nn._bridge import ( + build_input_map, build_readout, build_reservoir, input_is_backable, + node_is_backable, regressor_is_backable) +from pyrcn.base.blocks import (BatchIntrinsicPlasticity, EulerNodeToNode, + HebbianNodeToNode, InputToNode, NodeToNode, + PredefinedWeightsInputToNode, + PredefinedWeightsNodeToNode) +from pyrcn.linear_model import IncrementalRegression + +RTOL, ATOL = 1e-8, 1e-11 + + +def _seq(length: int, features: int, seed: int) -> np.ndarray: + return np.random.RandomState(seed).normal(size=(length, features)) * 0.3 + + +def test_input_is_backable_classification() -> None: + assert input_is_backable(InputToNode()) + assert input_is_backable( + PredefinedWeightsInputToNode( + predefined_input_weights=np.zeros((3, 5)))) + # bespoke transform -> fallback + assert not input_is_backable(BatchIntrinsicPlasticity()) + assert not input_is_backable(FeatureUnion([("a", InputToNode())])) + + +def test_node_is_backable_classification() -> None: + assert node_is_backable(NodeToNode()) + assert node_is_backable(EulerNodeToNode()) + assert node_is_backable(HebbianNodeToNode()) + assert node_is_backable( + PredefinedWeightsNodeToNode( + predefined_recurrent_weights=np.zeros((5, 5)))) + assert not node_is_backable(InputToNode()) + + +def test_regressor_is_backable_classification() -> None: + assert regressor_is_backable(IncrementalRegression()) + assert not regressor_is_backable(IncrementalRegression(normalize=True)) + assert not regressor_is_backable(Ridge()) + + +def test_build_input_map_matches_transform() -> None: + i2n = InputToNode( + hidden_layer_size=20, input_scaling=0.5, input_shift=0.2, + bias_scaling=2.0, bias_shift=-0.1, input_activation="tanh", + random_state=42) + X = _seq(15, 6, 0) + i2n.fit(X) + expected = i2n.transform(X) + + fm = build_input_map(i2n, dtype=torch.float64) + got = fm(torch.as_tensor(X, dtype=torch.float64)).numpy() + np.testing.assert_allclose(got, expected, rtol=RTOL, atol=ATOL) + + +def test_build_reservoir_matches_transform_leaky() -> None: + hidden = 20 + n2n = NodeToNode( + hidden_layer_size=hidden, spectral_radius=0.9, leakage=0.6, + reservoir_activation="tanh", random_state=42) + n2n.fit(np.zeros((2 * hidden, hidden))) + X = _seq(30, hidden, 1) + expected = n2n.transform(X) + + res = build_reservoir(n2n, dtype=torch.float64) + states, _ = res(torch.as_tensor(X, dtype=torch.float64).unsqueeze(0)) + np.testing.assert_allclose( + states.squeeze(0).numpy(), expected, rtol=RTOL, atol=ATOL) + + +def test_build_reservoir_matches_transform_bidirectional() -> None: + hidden = 20 + n2n = NodeToNode( + hidden_layer_size=hidden, spectral_radius=0.9, leakage=0.6, + reservoir_activation="tanh", bidirectional=True, random_state=3) + n2n.fit(np.zeros((2 * hidden, hidden))) + X = _seq(30, hidden, 1) + expected = n2n.transform(X) # (30, 2*hidden) + + res = build_reservoir(n2n, dtype=torch.float64) + states, _ = res(torch.as_tensor(X, dtype=torch.float64).unsqueeze(0)) + assert states.shape == (1, 30, 2 * hidden) + np.testing.assert_allclose( + states.squeeze(0).numpy(), expected, rtol=RTOL, atol=ATOL) + + +def test_build_reservoir_matches_transform_euler() -> None: + hidden = 20 + euler = EulerNodeToNode( + hidden_layer_size=hidden, recurrent_scaling=0.5, gamma=0.01, + epsilon=0.1, reservoir_activation="tanh", random_state=42) + euler.fit(np.zeros((2 * hidden, hidden))) + X = _seq(30, hidden, 1) + expected = euler.transform(X) + + res = build_reservoir(euler, dtype=torch.float64) + states, _ = res(torch.as_tensor(X, dtype=torch.float64).unsqueeze(0)) + np.testing.assert_allclose( + states.squeeze(0).numpy(), expected, rtol=RTOL, atol=ATOL) + + +def test_build_readout_matches_incremental_regression() -> None: + rng = np.random.RandomState(0) + Z = rng.normal(size=(40, 8)) + y = rng.normal(size=(40, 2)) + reg = IncrementalRegression(alpha=1e-3, fit_intercept=True) + reg.fit(Z, y) + expected = reg.predict(Z) + + readout = build_readout(reg, dtype=torch.float64) + readout.fit(torch.as_tensor(Z, dtype=torch.float64), + torch.as_tensor(y, dtype=torch.float64)) + got = readout.predict(torch.as_tensor(Z, dtype=torch.float64)).numpy() + np.testing.assert_allclose(got, expected, rtol=1e-7, atol=1e-9) diff --git a/tests/test_backend_device.py b/tests/test_backend_device.py new file mode 100644 index 0000000..659ad7b --- /dev/null +++ b/tests/test_backend_device.py @@ -0,0 +1,52 @@ +"""The torch backend modules run on CPU and honour the ``device`` argument. + +The estimators are CPU/float64 today; device selection is a backend-module +capability (``InputFeatureMap`` / ``Reservoir`` / ``IncrementalRidge`` all take +``device=``). The CUDA test is skipped when no GPU is available. +""" +from __future__ import annotations + +import numpy as np +import pytest +import torch + +from pyrcn.nn import InputFeatureMap, IncrementalRidge, Reservoir + + +def _run_pipeline(device: str) -> tuple[torch.Tensor, torch.Tensor]: + """Feature map -> reservoir -> readout on ``device``; return outputs.""" + n_features, hidden, length = 3, 8, 12 + fm = InputFeatureMap(n_features, hidden, dtype=torch.float64, + device=device) + x = torch.randn(length, n_features, dtype=torch.float64, device=device) + feats = fm(x) + + res = Reservoir(hidden, spectral_radius=0.9, leakage=0.6, + dtype=torch.float64, device=device) + res.set_recurrent_weights(np.eye(hidden)) + states, final = res(feats.unsqueeze(0)) + states = states.squeeze(0) + + readout = IncrementalRidge(dtype=torch.float64, device=device) + y = torch.randn(length, 2, dtype=torch.float64, device=device) + readout.fit(states, y) + pred = readout.predict(states) + return states, pred + + +def test_backend_runs_on_cpu() -> None: + states, pred = _run_pipeline("cpu") + assert states.device.type == "cpu" + assert pred.device.type == "cpu" + assert states.shape == (12, 8) + assert pred.shape == (12, 2) + assert torch.isfinite(pred).all() + + +@pytest.mark.skipif(not torch.cuda.is_available(), + reason="CUDA not available") +def test_backend_runs_on_cuda() -> None: + states, pred = _run_pipeline("cuda") + assert states.device.type == "cuda" + assert pred.device.type == "cuda" + assert torch.isfinite(pred).all() diff --git a/tests/test_backend_euler.py b/tests/test_backend_euler.py new file mode 100644 index 0000000..5908b1e --- /dev/null +++ b/tests/test_backend_euler.py @@ -0,0 +1,71 @@ +"""Parity: torch Euler reservoir vs the legacy EulerNodeToNode. + +Recurrent weights come from a real ``EulerNodeToNode`` (antisymmetric-uniform, +possibly sparse) and are injected into the torch module, so initializations are +identical and this compares the Euler recurrence only. float64, tight tol. +""" +from __future__ import annotations + +import numpy as np +import pytest +import torch + +from pyrcn.nn import EulerReservoir +from pyrcn.nn.init import antisymmetric_recurrent_weights +from pyrcn.base.blocks import EulerNodeToNode + +RTOL, ATOL = 1e-8, 1e-11 + + +def _fit_euler(hidden_size: int, *, recurrent_scaling: float, gamma: float, + epsilon: float, activation: str, random_state: int, + k_rec: int | None = None) -> EulerNodeToNode: + euler = EulerNodeToNode( + hidden_layer_size=hidden_size, recurrent_scaling=recurrent_scaling, + gamma=gamma, epsilon=epsilon, reservoir_activation=activation, + k_rec=k_rec, random_state=random_state) + euler.fit(np.zeros((2 * hidden_size, hidden_size))) + return euler + + +def _dense(W: object) -> np.ndarray: + return W.toarray() if hasattr(W, "toarray") else np.asarray(W) + + +@pytest.mark.parametrize("activation", ["tanh", "identity", "relu"]) +@pytest.mark.parametrize("recurrent_scaling,gamma,epsilon", + [(1.0, 0.001, 0.01), (0.8, 0.01, 0.05)]) +def test_euler_parity(activation: str, recurrent_scaling: float, gamma: float, + epsilon: float) -> None: + hidden_size, length = 20, 40 + euler = _fit_euler( + hidden_size, recurrent_scaling=recurrent_scaling, gamma=gamma, + epsilon=epsilon, activation=activation, random_state=42) + X = np.random.RandomState(0).normal(size=(length, hidden_size)) * 0.3 + + expected = euler.transform(X) + + res = EulerReservoir( + hidden_size=hidden_size, recurrent_scaling=recurrent_scaling, + gamma=gamma, epsilon=epsilon, activation=activation, + dtype=torch.float64) + res.set_recurrent_weights(_dense(euler._recurrent_weights)) + states, _ = res(torch.as_tensor(X, dtype=torch.float64).unsqueeze(0)) + + np.testing.assert_allclose( + states.squeeze(0).numpy(), expected, rtol=RTOL, atol=ATOL) + + +def test_antisymmetric_init_is_antisymmetric() -> None: + W = antisymmetric_recurrent_weights( + 20, generator=torch.Generator().manual_seed(0), dtype=torch.float64) + assert W.shape == (20, 20) + assert torch.allclose(W, -W.T) + + +def test_antisymmetric_init_reproducible() -> None: + W1 = antisymmetric_recurrent_weights( + 16, generator=torch.Generator().manual_seed(1), dtype=torch.float64) + W2 = antisymmetric_recurrent_weights( + 16, generator=torch.Generator().manual_seed(1), dtype=torch.float64) + assert torch.equal(W1, W2) diff --git a/tests/test_backend_gradient.py b/tests/test_backend_gradient.py new file mode 100644 index 0000000..96e9c4c --- /dev/null +++ b/tests/test_backend_gradient.py @@ -0,0 +1,128 @@ +"""Gradient training core (B1): the trainable LinearReadout + train_readout. + +Consistency check (D9): with a fixed set of features, a gradient-trained +linear readout converges to the closed-form ridge (IncrementalRidge) solution. +The features are whitened (orthonormal columns) so the objective is +well-conditioned and gradient descent converges tightly in few epochs. +""" +from __future__ import annotations + +import numpy as np +import pytest +import torch + +from pyrcn.base.blocks import InputToNode, NodeToNode +from pyrcn.nn import IncrementalRidge, LinearReadout, train_readout +from pyrcn.nn._bridge import build_input_map, build_reservoir + + +def _whitened(n: int, p: int, t: int, seed: int + ) -> tuple[torch.Tensor, torch.Tensor]: + rng = np.random.RandomState(seed) + Q, _ = np.linalg.qr(rng.normal(size=(n, p))) # orthonormal columns + W_true = rng.normal(size=(p, t)) + y = Q @ W_true + 0.001 * rng.normal(size=(n, t)) + return (torch.as_tensor(Q, dtype=torch.float64), + torch.as_tensor(y, dtype=torch.float64)) + + +def test_gradient_readout_matches_closed_form() -> None: + Z, y = _whitened(300, 8, 2, seed=0) + alpha = 1e-8 + expected = IncrementalRidge( + alpha=alpha, fit_intercept=False, dtype=torch.float64 + ).fit(Z, y).predict(Z).numpy() + + torch.manual_seed(0) + readout = LinearReadout(8, 2, fit_intercept=False, dtype=torch.float64) + train_readout(readout, Z, y, optimizer="adam", learning_rate=0.05, + epochs=600, weight_decay=alpha) + got = readout.predict(Z).numpy() + + np.testing.assert_allclose(got, expected, rtol=1e-3, atol=1e-4) + + +def test_train_readout_reduces_loss() -> None: + Z, y = _whitened(200, 6, 1, seed=1) + torch.manual_seed(0) + readout = LinearReadout(6, 1, fit_intercept=True, dtype=torch.float64) + loss_fn = torch.nn.MSELoss() + with torch.no_grad(): + before = loss_fn(readout(Z), y).item() + train_readout(readout, Z, y, optimizer="adam", learning_rate=0.05, + epochs=200) + with torch.no_grad(): + after = loss_fn(readout(Z), y).item() + assert after < before * 1e-2 + + +def test_linear_readout_predict_shape_and_no_grad() -> None: + readout = LinearReadout(4, 3, dtype=torch.float64) + pred = readout.predict(torch.zeros(10, 4, dtype=torch.float64)) + assert pred.shape == (10, 3) + assert not pred.requires_grad + + +def test_train_readout_unknown_optimizer_raises() -> None: + readout = LinearReadout(4, 1, dtype=torch.float64) + Z = torch.zeros(5, 4, dtype=torch.float64) + y = torch.zeros(5, 1, dtype=torch.float64) + with pytest.raises(ValueError): + train_readout(readout, Z, y, optimizer="bogus") + + +def test_train_readout_unknown_loss_raises() -> None: + readout = LinearReadout(4, 1, dtype=torch.float64) + Z = torch.zeros(5, 4, dtype=torch.float64) + y = torch.zeros(5, 1, dtype=torch.float64) + with pytest.raises(ValueError): + train_readout(readout, Z, y, loss="bogus") + + +def test_gradient_reaches_closed_form_through_real_reservoir() -> None: + """Tight-consistency demo: gradient == closed-form through the reservoir. + + The reservoir states are highly correlated (large condition number), so + first-order optimizers converge slowly -- but the objective is still a + convex quadratic with a unique minimum equal to the closed-form ridge + solution. Minimizing the *same* ridge objective (SSE + alpha||params||^2) + with a second-order optimizer (L-BFGS) reaches that minimum, so the + gradient-trained readout matches IncrementalRidge to a tight tolerance + even through the raw, ill-conditioned reservoir map. + """ + X = np.sin(np.linspace(0, 8 * np.pi, 400)).reshape(-1, 1) + y = np.roll(X.ravel(), -1).reshape(-1, 1) # next-step target + i2n = InputToNode(hidden_layer_size=40, random_state=42) + i2n.fit(X) + n2n = NodeToNode(hidden_layer_size=40, spectral_radius=0.9, + leakage=0.7, random_state=42) + n2n.fit(i2n.transform(X)) + feats = build_input_map(i2n, dtype=torch.float64)( + torch.as_tensor(X, dtype=torch.float64)) + states, _ = build_reservoir(n2n, dtype=torch.float64)(feats.unsqueeze(0)) + states = states.squeeze(0) + targets = torch.as_tensor(y, dtype=torch.float64) + alpha = 1e-3 + + expected = IncrementalRidge( + alpha=alpha, fit_intercept=True, dtype=torch.float64 + ).fit(states, targets).predict(states).numpy() + + torch.manual_seed(0) + readout = LinearReadout(40, 1, fit_intercept=True, dtype=torch.float64) + optimizer = torch.optim.LBFGS( + readout.parameters(), lr=1.0, max_iter=500, history_size=50, + line_search_fn="strong_wolfe") + + def closure() -> torch.Tensor: + optimizer.zero_grad() + residual = readout(states) - targets + penalty = sum((p ** 2).sum() for p in readout.parameters()) + loss = (residual ** 2).sum() + alpha * penalty + loss.backward() + return loss + + optimizer.step(closure) + got = readout.predict(states).numpy() + + np.testing.assert_allclose(got, expected, rtol=1e-3, atol=1e-3) diff --git a/tests/test_backend_init.py b/tests/test_backend_init.py new file mode 100644 index 0000000..d2d3148 --- /dev/null +++ b/tests/test_backend_init.py @@ -0,0 +1,73 @@ +"""Numerical correctness of the torch-native reservoir weight init. + +These produce the recurrent matrix for real (non-parity) use, so they are +checked on their defining numerical properties: unit spectral radius for the +random/sparse designs, and the exact structure (and resulting spectral radius) +for the minimum-complexity topologies. Reproducibility uses a torch Generator. +""" +from __future__ import annotations + +import numpy as np +import torch + +from pyrcn.nn.init import (delay_line_feedback_weights, delay_line_weights, + normal_recurrent_weights, simple_cycle_weights) + + +def _spectral_radius(W: torch.Tensor) -> float: + return float(torch.linalg.eigvals(W).abs().max()) + + +def test_normal_recurrent_has_unit_spectral_radius() -> None: + g = torch.Generator().manual_seed(0) + W = normal_recurrent_weights(50, generator=g, dtype=torch.float64) + assert W.shape == (50, 50) + np.testing.assert_allclose(_spectral_radius(W), 1.0, rtol=1e-6) + + +def test_sparse_recurrent_fan_in_and_radius() -> None: + g = torch.Generator().manual_seed(0) + W = normal_recurrent_weights( + 40, fan_in=8, generator=g, dtype=torch.float64) + nonzeros_per_column = (W != 0).sum(dim=0) + assert int(nonzeros_per_column.max()) == 8 # exactly fan_in kept + assert int(nonzeros_per_column.min()) == 8 + np.testing.assert_allclose(_spectral_radius(W), 1.0, rtol=1e-6) + + +def test_simple_cycle_structure_and_radius() -> None: + W = simple_cycle_weights(6, forward_weight=0.9, dtype=torch.float64) + expected = torch.zeros(6, 6, dtype=torch.float64) + for i in range(6): + expected[i, i - 1] = 0.9 # subdiagonal + corner + assert torch.equal(W, expected) + np.testing.assert_allclose(_spectral_radius(W), 0.9, rtol=1e-6) + + +def test_delay_line_structure_is_nilpotent() -> None: + W = delay_line_weights(6, forward_weight=0.9, dtype=torch.float64) + expected = torch.zeros(6, 6, dtype=torch.float64) + for i in range(1, 6): + expected[i, i - 1] = 0.9 # subdiagonal only + assert torch.equal(W, expected) + assert _spectral_radius(W) < 1e-9 # nilpotent + + +def test_delay_line_feedback_structure() -> None: + W = delay_line_feedback_weights( + 6, forward_weight=0.9, feedback_weight=0.1, dtype=torch.float64) + expected = torch.zeros(6, 6, dtype=torch.float64) + for i in range(1, 6): + expected[i, i - 1] = 0.9 # forward + expected[i - 1, i] = 0.1 # feedback + assert torch.equal(W, expected) + + +def test_reproducible_with_generator() -> None: + W1 = normal_recurrent_weights( + 20, fan_in=5, generator=torch.Generator().manual_seed(1), + dtype=torch.float64) + W2 = normal_recurrent_weights( + 20, fan_in=5, generator=torch.Generator().manual_seed(1), + dtype=torch.float64) + assert torch.equal(W1, W2) diff --git a/tests/test_backend_input.py b/tests/test_backend_input.py new file mode 100644 index 0000000..4b102a7 --- /dev/null +++ b/tests/test_backend_input.py @@ -0,0 +1,92 @@ +"""Parity: torch InputFeatureMap vs the legacy InputToNode.transform. + +Input weights and bias come from a real ``InputToNode`` and are injected into +the torch module, so this compares the feature-map computation (scaling/shift + +activation) only. float64, tight tolerance. Plus torch-native init checks. +""" +from __future__ import annotations + +import numpy as np +import pytest +import torch + +from pyrcn.nn import InputFeatureMap +from pyrcn.nn.init import (bernoulli_input_weights, uniform_bias_weights, + uniform_input_weights) +from pyrcn.base.blocks import InputToNode + +RTOL, ATOL = 1e-8, 1e-11 + + +def _fit_i2n(in_features: int, hidden_size: int, *, input_scaling: float, + input_shift: float, bias_scaling: float, bias_shift: float, + activation: str, random_state: int) -> InputToNode: + i2n = InputToNode( + hidden_layer_size=hidden_size, input_scaling=input_scaling, + input_shift=input_shift, bias_scaling=bias_scaling, + bias_shift=bias_shift, input_activation=activation, + random_state=random_state) + i2n.fit(np.zeros((3, in_features))) + return i2n + + +def _dense(W: object) -> np.ndarray: + return W.toarray() if hasattr(W, "toarray") else np.asarray(W) + + +@pytest.mark.parametrize("activation", + ["tanh", "identity", "relu", "logistic", + "bounded_relu"]) +@pytest.mark.parametrize( + "input_scaling,input_shift,bias_scaling,bias_shift", + [(1.0, 0.0, 1.0, 0.0), (0.5, 0.2, 2.0, -0.1)]) +def test_input_parity(activation: str, input_scaling: float, + input_shift: float, bias_scaling: float, + bias_shift: float) -> None: + in_features, hidden_size, length = 6, 20, 15 + i2n = _fit_i2n( + in_features, hidden_size, input_scaling=input_scaling, + input_shift=input_shift, bias_scaling=bias_scaling, + bias_shift=bias_shift, activation=activation, random_state=42) + X = np.random.RandomState(0).normal(size=(length, in_features)) * 0.3 + + expected = i2n.transform(X) + + fm = InputFeatureMap( + in_features, hidden_size, input_scaling=input_scaling, + input_shift=input_shift, bias_scaling=bias_scaling, + bias_shift=bias_shift, activation=activation, dtype=torch.float64) + fm.set_input_weights(_dense(i2n._input_weights), i2n._bias_weights) + got = fm(torch.as_tensor(X, dtype=torch.float64)) + + np.testing.assert_allclose(got.numpy(), expected, rtol=RTOL, atol=ATOL) + + +def test_uniform_input_weights_shape_and_range() -> None: + W = uniform_input_weights( + 6, 20, generator=torch.Generator().manual_seed(0), dtype=torch.float64) + assert W.shape == (6, 20) + assert float(W.max()) < 1.0 and float(W.min()) >= -1.0 + + +def test_sparse_input_weights_fan_in() -> None: + W = uniform_input_weights( + 10, 20, fan_in=3, generator=torch.Generator().manual_seed(0), + dtype=torch.float64) + nonzeros_per_column = (W != 0).sum(dim=0) + assert int(nonzeros_per_column.max()) == 3 + assert int(nonzeros_per_column.min()) == 3 + + +def test_bernoulli_input_weights_are_signed_constants() -> None: + W = bernoulli_input_weights( + 8, 20, value=0.5, generator=torch.Generator().manual_seed(0), + dtype=torch.float64) + assert set(torch.unique(W).tolist()) <= {0.5, -0.5} + + +def test_uniform_bias_shape_and_range() -> None: + b = uniform_bias_weights( + 20, generator=torch.Generator().manual_seed(0), dtype=torch.float64) + assert b.shape == (20,) + assert float(b.max()) < 1.0 and float(b.min()) >= -1.0 diff --git a/tests/test_backend_parity_estimators.py b/tests/test_backend_parity_estimators.py new file mode 100644 index 0000000..e7ce236 --- /dev/null +++ b/tests/test_backend_parity_estimators.py @@ -0,0 +1,264 @@ +"""Backend parity harness for the pyrcn estimators. + +This module proves that the PyTorch "fast path" and the legacy numpy +"fallback" path compute the *same* thing for an identical estimator +configuration. + +Technique +--------- +For each configuration we build two twin estimators: + +* ``native`` -- every sub-estimator is a native pyrcn block + (``InputToNode`` / ``NodeToNode`` + ``IncrementalRegression``), so the + estimator selects the torch backend and ``_use_torch`` becomes ``True``. +* ``fallback`` -- identical, except the input stage is wrapped in a + single-transformer ``FeatureUnion([("x", InputToNode(...))])``. A + one-transformer ``FeatureUnion`` emits exactly the inner + ``InputToNode.transform`` output, and with the same ``random_state`` and + data the weights are identical. The estimator can no longer recognise a + native input block, so it drops to the numpy path + (``_use_torch`` is ``False``) while computing the same reservoir input, + states and readout. + +Because the two twins are mathematically identical and differ only in the +compute backend, their predictions must match. Any divergence larger than +the acceptance bar (rtol=1e-4, atol=1e-5) signals a real fast-path vs. +fallback discrepancy. +""" + +from __future__ import annotations + +import numpy as np +from sklearn.datasets import load_iris +from sklearn.pipeline import FeatureUnion + +from pyrcn.base.blocks import InputToNode, NodeToNode +from pyrcn.datasets import load_digits, mackey_glass +from pyrcn.echo_state_network import ESNClassifier, ESNRegressor +from pyrcn.extreme_learning_machine import ELMClassifier, ELMRegressor +from pyrcn.linear_model import IncrementalRegression + +RTOL = 1e-4 +ATOL = 1e-5 + + +def _wrap(i2n: InputToNode) -> FeatureUnion: + """Force the numpy fallback by hiding the native input block.""" + return FeatureUnion([("x", i2n)]) + + +def _build_esn_regressor( + forced_fallback: bool, *, hidden_layer_size: int, + spectral_radius: float, leakage: float, alpha: float, + bidirectional: bool = False, +) -> ESNRegressor: + """Build an ESN regressor twin (native or forced-fallback).""" + i2n = InputToNode(hidden_layer_size=hidden_layer_size, random_state=42) + n2n = NodeToNode( + hidden_layer_size=hidden_layer_size, + spectral_radius=spectral_radius, leakage=leakage, + bidirectional=bidirectional, random_state=42) + return ESNRegressor( + input_to_node=_wrap(i2n) if forced_fallback else i2n, + node_to_node=n2n, + regressor=IncrementalRegression(alpha=alpha), + verbose=False) + + +def _build_esn_classifier( + forced_fallback: bool, *, hidden_layer_size: int, + spectral_radius: float, leakage: float, alpha: float, +) -> ESNClassifier: + """Build an ESN classifier twin (native or forced-fallback).""" + i2n = InputToNode(hidden_layer_size=hidden_layer_size, random_state=42) + n2n = NodeToNode( + hidden_layer_size=hidden_layer_size, + spectral_radius=spectral_radius, leakage=leakage, + random_state=42) + return ESNClassifier( + input_to_node=_wrap(i2n) if forced_fallback else i2n, + node_to_node=n2n, + regressor=IncrementalRegression(alpha=alpha), + verbose=False) + + +def _build_elm_regressor( + forced_fallback: bool, *, hidden_layer_size: int, alpha: float, +) -> ELMRegressor: + """Build an ELM regressor twin (native or forced-fallback).""" + i2n = InputToNode(hidden_layer_size=hidden_layer_size, random_state=42) + return ELMRegressor( + input_to_node=_wrap(i2n) if forced_fallback else i2n, + regressor=IncrementalRegression(alpha=alpha), + verbose=False) + + +def _build_elm_classifier( + forced_fallback: bool, *, hidden_layer_size: int, alpha: float, +) -> ELMClassifier: + """Build an ELM classifier twin (native or forced-fallback).""" + i2n = InputToNode(hidden_layer_size=hidden_layer_size, random_state=42) + return ELMClassifier( + input_to_node=_wrap(i2n) if forced_fallback else i2n, + regressor=IncrementalRegression(alpha=alpha), + verbose=False) + + +def _assert_backends(native: object, fallback: object) -> None: + """Assert the twins really used different compute backends.""" + assert native._use_torch is True + assert getattr(fallback, "_use_torch", False) is False + + +def _max_diff(a: np.ndarray, b: np.ndarray) -> tuple[float, float]: + """Return (max abs diff, max rel diff) for reporting.""" + a = np.asarray(a, dtype=float) + b = np.asarray(b, dtype=float) + abs_diff = np.abs(a - b) + rel_diff = abs_diff / (np.abs(b) + 1e-12) + return float(abs_diff.max()), float(rel_diff.max()) + + +def _mackey_glass_2d() -> tuple[np.ndarray, np.ndarray]: + """Return a small non-sequence mackey-glass dataset.""" + X, y = mackey_glass(n_timesteps=600) + return X.reshape(-1, 1), y + + +def test_esn_regressor_nonsequence_parity() -> None: + """ESN regressor parity on 2-D (non-sequence) input.""" + X, y = _mackey_glass_2d() + X_train, X_test = X[:-100], X[-100:] + y_train = y[:-100] + + cfg = dict(hidden_layer_size=50, spectral_radius=0.9, + leakage=0.7, alpha=1e-3) + native = _build_esn_regressor(False, **cfg) + fallback = _build_esn_regressor(True, **cfg) + native.fit(X_train, y_train) + fallback.fit(X_train, y_train) + _assert_backends(native, fallback) + + got = native.predict(X_test) + ref = fallback.predict(X_test) + print("esn_nonsequence max abs/rel:", _max_diff(got, ref)) + np.testing.assert_allclose(got, ref, rtol=RTOL, atol=ATOL) + + +def test_esn_regressor_bidirectional_parity() -> None: + """ESN regressor parity with a bidirectional reservoir.""" + X, y = _mackey_glass_2d() + X_train, X_test = X[:-100], X[-100:] + y_train = y[:-100] + + cfg = dict(hidden_layer_size=50, spectral_radius=0.9, + leakage=0.7, alpha=1e-3, bidirectional=True) + native = _build_esn_regressor(False, **cfg) + fallback = _build_esn_regressor(True, **cfg) + native.fit(X_train, y_train) + fallback.fit(X_train, y_train) + _assert_backends(native, fallback) + + got = native.predict(X_test) + ref = fallback.predict(X_test) + print("esn_bidirectional max abs/rel:", _max_diff(got, ref)) + np.testing.assert_allclose(got, ref, rtol=RTOL, atol=ATOL) + + +def test_esn_regressor_sequence_parity() -> None: + """ESN regressor parity on an object-array of sequences.""" + X_flat, y_flat = mackey_glass(n_timesteps=600) + n_seq = 4 + length = 120 + X = np.empty(shape=(n_seq,), dtype=object) + y = np.empty(shape=(n_seq,), dtype=object) + for k in range(n_seq): + start = k * length + stop = start + length + X[k] = X_flat[start:stop].reshape(-1, 1) + y[k] = y_flat[start:stop] + + cfg = dict(hidden_layer_size=50, spectral_radius=0.9, + leakage=0.7, alpha=1e-3) + native = _build_esn_regressor(False, **cfg) + fallback = _build_esn_regressor(True, **cfg) + native.fit(X, y) + fallback.fit(X, y) + _assert_backends(native, fallback) + + got = native.predict(X) + ref = fallback.predict(X) + assert len(got) == len(ref) + worst = (0.0, 0.0) + for k in range(len(ref)): + worst = tuple(np.maximum(worst, _max_diff(got[k], ref[k]))) + np.testing.assert_allclose(got[k], ref[k], rtol=RTOL, atol=ATOL) + print("esn_sequence max abs/rel:", worst) + + +def test_esn_classifier_sequence_to_value_parity() -> None: + """ESN classifier parity for the sequence-to-value task.""" + X, y = load_digits(return_X_y=True, as_sequence=True) + X, y = X[:40], y[:40] + + cfg = dict(hidden_layer_size=50, spectral_radius=0.9, + leakage=0.7, alpha=1e-3) + native = _build_esn_classifier(False, **cfg) + fallback = _build_esn_classifier(True, **cfg) + native.fit(X, y) + fallback.fit(X, y) + _assert_backends(native, fallback) + + got = native.predict(X) + ref = fallback.predict(X) + assert len(got) == len(ref) + for k in range(len(ref)): + assert np.array_equal(got[k], ref[k]) + + got_p = native.predict_proba(X) + ref_p = fallback.predict_proba(X) + worst = (0.0, 0.0) + for k in range(len(ref_p)): + worst = tuple(np.maximum(worst, _max_diff(got_p[k], ref_p[k]))) + np.testing.assert_allclose(got_p[k], ref_p[k], rtol=RTOL, + atol=ATOL) + print("esn_classifier_seq proba max abs/rel:", worst) + + +def test_elm_regressor_parity() -> None: + """ELM regressor parity on a 2-output regression target.""" + X = np.linspace(0, 10, 400).reshape(-1, 1) + y = np.hstack((np.sin(X), np.cos(X))) + + cfg = dict(hidden_layer_size=50, alpha=1e-3) + native = _build_elm_regressor(False, **cfg) + fallback = _build_elm_regressor(True, **cfg) + native.fit(X, y) + fallback.fit(X, y) + _assert_backends(native, fallback) + + got = native.predict(X) + ref = fallback.predict(X) + print("elm_regressor max abs/rel:", _max_diff(got, ref)) + np.testing.assert_allclose(got, ref, rtol=RTOL, atol=ATOL) + + +def test_elm_classifier_parity() -> None: + """ELM classifier parity on the iris dataset.""" + X, y = load_iris(return_X_y=True) + + cfg = dict(hidden_layer_size=100, alpha=1e-3) + native = _build_elm_classifier(False, **cfg) + fallback = _build_elm_classifier(True, **cfg) + native.fit(X, y) + fallback.fit(X, y) + _assert_backends(native, fallback) + + got = native.predict(X) + ref = fallback.predict(X) + assert np.array_equal(got, ref) + + got_p = native.predict_proba(X) + ref_p = fallback.predict_proba(X) + print("elm_classifier proba max abs/rel:", _max_diff(got_p, ref_p)) + np.testing.assert_allclose(got_p, ref_p, rtol=RTOL, atol=ATOL) diff --git a/tests/test_backend_readout.py b/tests/test_backend_readout.py new file mode 100644 index 0000000..3d31240 --- /dev/null +++ b/tests/test_backend_readout.py @@ -0,0 +1,124 @@ +"""Parity: torch IncrementalRidge vs the legacy IncrementalRegression. + +The legacy readout and the torch readout are fed the *identical* feature +matrix Z and targets y; both compute the closed-form ridge solution +``W = (K + alpha I)^-1 xTy`` over accumulated statistics. float64, tight +tolerance. Plus torch-specific property tests (accumulation, merge, postpone). +""" +from __future__ import annotations + +import numpy as np +import pytest +import torch + +from pyrcn.nn import IncrementalRidge +from pyrcn.linear_model import IncrementalRegression + +RTOL, ATOL = 1e-7, 1e-9 + + +def _legacy(alpha: float, fit_intercept: bool) -> IncrementalRegression: + return IncrementalRegression(alpha=alpha, fit_intercept=fit_intercept) + + +def _rng(seed: int) -> np.random.RandomState: + return np.random.RandomState(seed) + + +@pytest.mark.parametrize("fit_intercept", [True, False]) +@pytest.mark.parametrize("n_targets", [1, 3]) +def test_fit_parity_single_batch(fit_intercept: bool, n_targets: int) -> None: + n_samples, n_features, alpha = 40, 8, 1e-3 + Z = _rng(0).normal(size=(n_samples, n_features)) + y = _rng(1).normal(size=(n_samples, n_targets)) + + expected = _legacy(alpha, fit_intercept).fit(Z, y).predict(Z) + + ridge = IncrementalRidge( + alpha=alpha, fit_intercept=fit_intercept, dtype=torch.float64) + ridge.fit(torch.as_tensor(Z, dtype=torch.float64), + torch.as_tensor(y, dtype=torch.float64)) + got = ridge.predict(torch.as_tensor(Z, dtype=torch.float64)) + + np.testing.assert_allclose(got.numpy(), expected, rtol=RTOL, atol=ATOL) + + +def test_fit_parity_1d_target() -> None: + n_samples, n_features, alpha = 30, 6, 1e-4 + Z = _rng(2).normal(size=(n_samples, n_features)) + y = _rng(3).normal(size=(n_samples,)) + + expected = _legacy(alpha, True).fit(Z, y).predict(Z) + + ridge = IncrementalRidge(alpha=alpha, fit_intercept=True, + dtype=torch.float64) + ridge.fit(torch.as_tensor(Z, dtype=torch.float64), + torch.as_tensor(y, dtype=torch.float64)) + got = ridge.predict(torch.as_tensor(Z, dtype=torch.float64)) + + assert got.ndim == 1 + np.testing.assert_allclose(got.numpy(), expected, rtol=RTOL, atol=ATOL) + + +def test_sequence_postpone_parity() -> None: + """Accumulate several batches with postpone_inverse, solve once at end. + + Mirrors the ESN/ELM sequence-fit flow: partial_fit per sequence with + postpone_inverse=True, final partial_fit solves. Equivalent to a single + closed-form solve over the concatenated data. + """ + alpha, n_features, n_targets = 1e-3, 5, 2 + batches = [(_rng(10 + i).normal(size=(12, n_features)), + _rng(50 + i).normal(size=(12, n_targets))) for i in range(4)] + + legacy = _legacy(alpha, True) + for i, (Z, y) in enumerate(batches): + legacy.partial_fit(Z, y, reset=(i == 0), + postpone_inverse=(i < len(batches) - 1)) + + ridge = IncrementalRidge(alpha=alpha, fit_intercept=True, + dtype=torch.float64) + for i, (Z, y) in enumerate(batches): + ridge.partial_fit( + torch.as_tensor(Z, dtype=torch.float64), + torch.as_tensor(y, dtype=torch.float64), reset=(i == 0), + postpone_inverse=(i < len(batches) - 1)) + + Z_test = _rng(99).normal(size=(15, n_features)) + expected = legacy.predict(Z_test) + got = ridge.predict(torch.as_tensor(Z_test, dtype=torch.float64)) + np.testing.assert_allclose(got.numpy(), expected, rtol=RTOL, atol=ATOL) + + +def test_merge_equals_concatenated_fit() -> None: + """Merged readouts (summed stats) == one readout on concatenated data.""" + alpha, n_features, n_targets = 1e-3, 5, 2 + Za = _rng(20).normal(size=(18, n_features)) + ya = _rng(21).normal(size=(18, n_targets)) + Zb = _rng(22).normal(size=(14, n_features)) + yb = _rng(23).normal(size=(14, n_targets)) + + ra = IncrementalRidge(alpha=alpha, dtype=torch.float64) + ra.partial_fit(torch.as_tensor(Za, dtype=torch.float64), + torch.as_tensor(ya, dtype=torch.float64), + reset=True, postpone_inverse=True) + rb = IncrementalRidge(alpha=alpha, dtype=torch.float64) + rb.partial_fit(torch.as_tensor(Zb, dtype=torch.float64), + torch.as_tensor(yb, dtype=torch.float64), + reset=True, postpone_inverse=True) + merged = ra + rb + merged.solve() + + whole = IncrementalRidge(alpha=alpha, dtype=torch.float64) + whole.fit(torch.as_tensor(np.vstack([Za, Zb]), dtype=torch.float64), + torch.as_tensor(np.vstack([ya, yb]), dtype=torch.float64)) + + np.testing.assert_allclose( + merged.output_weights.numpy(), whole.output_weights.numpy(), + rtol=RTOL, atol=ATOL) + + +def test_predict_before_fit_raises() -> None: + ridge = IncrementalRidge(dtype=torch.float64) + with pytest.raises(RuntimeError): + ridge.predict(torch.zeros(3, 4, dtype=torch.float64)) diff --git a/tests/test_backend_reservoir.py b/tests/test_backend_reservoir.py new file mode 100644 index 0000000..2052585 --- /dev/null +++ b/tests/test_backend_reservoir.py @@ -0,0 +1,125 @@ +"""Parity: the torch Reservoir must equal the legacy NumPy NodeToNode. + +The recurrent weights are produced by PyRCN's own ``NodeToNode`` initialization +(dense and sparse ``k_rec`` variants, already spectral-radius-normalized by +PyRCN) and injected verbatim into the torch module. The initializations are +therefore identical by construction, so this compares the *compute* (the +recurrence) only. Run in float64 for a tight tolerance. +""" +from __future__ import annotations + +import numpy as np +import pytest +import torch + +from pyrcn.nn import Reservoir +from pyrcn.base.blocks import HebbianNodeToNode, NodeToNode + +RTOL, ATOL = 1e-8, 1e-11 + + +def _fit_nodetonode(hidden_size: int, *, spectral_radius: float, + leakage: float, activation: str, random_state: int, + k_rec: int | None = None, + bidirectional: bool = False) -> NodeToNode: + """A real NodeToNode with PyRCN-initialized recurrent weights.""" + n2n = NodeToNode( + hidden_layer_size=hidden_size, spectral_radius=spectral_radius, + leakage=leakage, reservoir_activation=activation, k_rec=k_rec, + bidirectional=bidirectional, random_state=random_state) + # NodeToNode's input dimension equals hidden_size (the InputToNode output). + n2n.fit(np.zeros((2 * hidden_size, hidden_size))) + return n2n + + +def _dense(W: object) -> np.ndarray: + return W.toarray() if hasattr(W, "toarray") else np.asarray(W) + + +@pytest.mark.parametrize("activation", + ["tanh", "identity", "relu", "logistic", + "bounded_relu"]) +@pytest.mark.parametrize("k_rec", [None, 5]) # dense vs sparse init +@pytest.mark.parametrize("spectral_radius,leakage", [(1.0, 1.0), (0.9, 0.5)]) +def test_reservoir_parity(activation: str, k_rec: int | None, + spectral_radius: float, leakage: float) -> None: + hidden_size, length = 20, 40 + n2n = _fit_nodetonode( + hidden_size, spectral_radius=spectral_radius, leakage=leakage, + activation=activation, random_state=42, k_rec=k_rec) + X = np.random.RandomState(0).normal(size=(length, hidden_size)) * 0.3 + + expected = n2n._pass_through_recurrent_weights(X) + + res = Reservoir( + hidden_size=hidden_size, spectral_radius=spectral_radius, + leakage=leakage, activation=activation, dtype=torch.float64) + res.set_recurrent_weights(_dense(n2n._recurrent_weights)) + states, final = res(torch.as_tensor(X, dtype=torch.float64).unsqueeze(0)) + + np.testing.assert_allclose( + states.squeeze(0).numpy(), expected, rtol=RTOL, atol=ATOL) + np.testing.assert_allclose( + final.squeeze(0).numpy(), expected[-1], rtol=RTOL, atol=ATOL) + + +@pytest.mark.parametrize("training_method", + ["hebbian", "anti_hebbian", "oja", "anti_oja"]) +def test_hebbian_learned_weights_parity(training_method: str) -> None: + # Hebbian only *learns* the weights (in fit); transform is the standard + # recurrence, so the torch form is the standard Reservoir with the + # NumPy-learned weights injected. + hidden_size = 20 + hebbian = HebbianNodeToNode( + hidden_layer_size=hidden_size, spectral_radius=0.9, leakage=0.8, + reservoir_activation="tanh", random_state=42, learning_rate=1e-3, + epochs=2, training_method=training_method) + X = np.random.RandomState(0).normal(size=(30, hidden_size)) * 0.3 + hebbian.fit(X) + expected = hebbian.transform(X) + + res = Reservoir(hidden_size=hidden_size, spectral_radius=0.9, leakage=0.8, + activation="tanh", dtype=torch.float64) + res.set_recurrent_weights(_dense(hebbian._recurrent_weights)) + states, _ = res(torch.as_tensor(X, dtype=torch.float64).unsqueeze(0)) + np.testing.assert_allclose( + states.squeeze(0).numpy(), expected, rtol=RTOL, atol=ATOL) + + +def test_reservoir_bidirectional_parity() -> None: + hidden_size = 20 + n2n = _fit_nodetonode( + hidden_size, spectral_radius=0.9, leakage=0.6, activation="tanh", + random_state=3, bidirectional=True) + X = np.random.RandomState(0).normal(size=(30, hidden_size)) * 0.3 + expected = n2n.transform(X) # (30, 2*hidden_size) + + res = Reservoir(hidden_size=hidden_size, spectral_radius=0.9, leakage=0.6, + activation="tanh", bidirectional=True, dtype=torch.float64) + res.set_recurrent_weights(_dense(n2n._recurrent_weights)) + states, _ = res(torch.as_tensor(X, dtype=torch.float64).unsqueeze(0)) + + assert states.shape == (1, 30, 2 * hidden_size) + np.testing.assert_allclose( + states.squeeze(0).numpy(), expected, rtol=RTOL, atol=ATOL) + + +def test_reservoir_batched_matches_per_sequence() -> None: + hidden_size = 20 + n2n = _fit_nodetonode( + hidden_size, spectral_radius=0.9, leakage=0.6, activation="tanh", + random_state=7) + rng = np.random.RandomState(1) + X1 = rng.normal(size=(30, hidden_size)) * 0.3 + X2 = rng.normal(size=(30, hidden_size)) * 0.3 + e1 = n2n._pass_through_recurrent_weights(X1) + e2 = n2n._pass_through_recurrent_weights(X2) + + res = Reservoir(hidden_size=hidden_size, spectral_radius=0.9, leakage=0.6, + activation="tanh", dtype=torch.float64) + res.set_recurrent_weights(_dense(n2n._recurrent_weights)) + Xb = torch.as_tensor(np.stack([X1, X2]), dtype=torch.float64) + states, _ = res(Xb) + + np.testing.assert_allclose(states[0].numpy(), e1, rtol=RTOL, atol=ATOL) + np.testing.assert_allclose(states[1].numpy(), e2, rtol=RTOL, atol=ATOL) diff --git a/tests/test_elm_gradient.py b/tests/test_elm_gradient.py new file mode 100644 index 0000000..5f832b3 --- /dev/null +++ b/tests/test_elm_gradient.py @@ -0,0 +1,91 @@ +"""Phase B1 gradient solver for the ELM estimators (plumbing + contracts). + +The gradient solver trains a ``LinearReadout`` with an optimizer loop instead +of the closed-form ridge solve. These tests check that the gradient path is +wired correctly (runs, learns, right shapes, right guards). The tight +gradient-vs-closed-form numerical equivalence is proven separately on +well-conditioned features in ``test_backend_gradient.py``; through the raw ELM +feature map gradient descent converges only slowly, so here we assert that +learning happens rather than exact agreement. + +A torch seed is set per test because the readout's initial weights come from +torch's global RNG (gradient reproducibility w.r.t. ``random_state`` is a +later refinement). +""" +from __future__ import annotations + +import numpy as np +import pytest +import torch +from sklearn.datasets import load_digits +from sklearn.pipeline import FeatureUnion + +from pyrcn.base.blocks import InputToNode +from pyrcn.extreme_learning_machine import ELMClassifier, ELMRegressor + + +def test_elm_gradient_learns_and_matches_shapes() -> None: + torch.manual_seed(0) + X = np.linspace(0, 10, 200).reshape(-1, 1) + y = np.sin(X).ravel() + grad = ELMRegressor( + hidden_layer_size=50, alpha=1e-8, solver="gradient", + optimizer="adam", learning_rate=0.2, epochs=400, + random_state=42).fit(X, y) + + assert grad._use_torch is True + pred = grad.predict(X) + assert pred.shape == y.shape # 1-D target -> 1-D prediction + assert grad.score(X, y) > 0.4 # gradient training learns + + +def test_elm_gradient_multitarget_shape() -> None: + torch.manual_seed(0) + X = np.linspace(0, 10, 200).reshape(-1, 1) + y = np.column_stack([np.sin(X).ravel(), np.cos(X).ravel()]) + grad = ELMRegressor( + hidden_layer_size=50, solver="gradient", learning_rate=0.1, + epochs=200, random_state=42).fit(X, y) + assert grad.predict(X).shape == (200, 2) + + +def test_elm_gradient_requires_backable() -> None: + X = np.linspace(0, 10, 100).reshape(-1, 1) + y = np.sin(X).ravel() + elm = ELMRegressor( + solver="gradient", + input_to_node=FeatureUnion([("x", InputToNode())])) + with pytest.raises(NotImplementedError): + elm.fit(X, y) + + +def test_elm_invalid_solver() -> None: + X = np.linspace(0, 10, 100).reshape(-1, 1) + y = np.sin(X).ravel() + with pytest.raises(ValueError): + ELMRegressor(solver="bogus").fit(X, y) + + +def test_elm_gradient_reproducible() -> None: + # No torch.manual_seed: the estimator seeds its readout from random_state. + X = np.linspace(0, 10, 150).reshape(-1, 1) + y = np.sin(X).ravel() + + def _pred() -> np.ndarray: + elm = ELMRegressor( + hidden_layer_size=40, solver="gradient", optimizer="adam", + learning_rate=0.1, epochs=150, random_state=42) + return elm.fit(X, y).predict(X) + + np.testing.assert_array_equal(_pred(), _pred()) + + +def test_elm_gradient_classifier() -> None: + torch.manual_seed(0) + X, y = load_digits(return_X_y=True) + X, y = X[:400], y[:400] + clf = ELMClassifier( + hidden_layer_size=100, solver="gradient", optimizer="adam", + learning_rate=0.1, epochs=200, random_state=42).fit(X, y) + assert clf._use_torch is True + assert clf.score(X, y) > 0.8 diff --git a/tests/test_elm_torch_backend.py b/tests/test_elm_torch_backend.py new file mode 100644 index 0000000..b17cfb9 --- /dev/null +++ b/tests/test_elm_torch_backend.py @@ -0,0 +1,81 @@ +"""Torch-backend fast-path parity tests for the ELM estimators.""" + +from __future__ import annotations + +import numpy as np +from sklearn.datasets import load_digits +from sklearn.linear_model import Ridge +from sklearn.pipeline import FeatureUnion + +from pyrcn.base.blocks import InputToNode +from pyrcn.extreme_learning_machine import ELMClassifier, ELMRegressor +from pyrcn.linear_model import IncrementalRegression + + +def _regression_data() -> tuple: + rng = np.random.RandomState(0) + X = np.linspace(0, 10, 200).reshape(-1, 1) + y = np.sin(X).ravel() + 0.01 * rng.randn(X.shape[0]) + return X, y + + +def test_elm_uses_torch_when_backable() -> None: + X, y = _regression_data() + elm = ELMRegressor(hidden_layer_size=25, random_state=42) + elm.fit(X, y) + assert elm._use_torch is True + + +def test_elm_falls_back_for_featureunion() -> None: + X, y = _regression_data() + elm = ELMRegressor( + input_to_node=FeatureUnion([ + ("x", InputToNode(hidden_layer_size=20, random_state=42))])) + elm.fit(X, y) + assert getattr(elm, "_use_torch", False) is False + y_pred = elm.predict(X) + assert y_pred.shape[0] == X.shape[0] + + +def test_elm_falls_back_for_ridge() -> None: + X, y = _regression_data() + elm = ELMRegressor( + input_to_node=InputToNode(hidden_layer_size=20, random_state=42), + regressor=Ridge()) + elm.fit(X, y) + assert getattr(elm, "_use_torch", False) is False + + +def test_elm_fastpath_matches_numpy_reference() -> None: + X = np.linspace(0, 10, 200).reshape(-1, 1) + X_test = np.linspace(0, 10, 40).reshape(-1, 1) + targets = { + "1d": np.sin(X).ravel(), + "2d": np.hstack((np.sin(X), np.cos(X))), + } + for y in targets.values(): + i2n = InputToNode(hidden_layer_size=30, bias_scaling=1.0, + random_state=42) + i2n.fit(X) + H = i2n.transform(X) + reg = IncrementalRegression(alpha=1e-3) + reg.fit(H, y) + expected = reg.predict(i2n.transform(X_test)) + + elm = ELMRegressor( + input_to_node=InputToNode(hidden_layer_size=30, + bias_scaling=1.0, random_state=42), + regressor=IncrementalRegression(alpha=1e-3)) + elm.fit(X, y) + assert elm._use_torch is True + got = elm.predict(X_test) + np.testing.assert_allclose( + got, expected, rtol=1e-6, atol=1e-7) + + +def test_elm_classifier_fastpath() -> None: + X, y = load_digits(return_X_y=True) + elm = ELMClassifier(hidden_layer_size=500, random_state=42) + elm.fit(X, y) + assert elm._use_torch is True + assert elm.score(X, y) > 0.9 diff --git a/tests/test_esn_gradient.py b/tests/test_esn_gradient.py new file mode 100644 index 0000000..a2d4a89 --- /dev/null +++ b/tests/test_esn_gradient.py @@ -0,0 +1,109 @@ +"""Phase B1 gradient solver for the ESN estimators (plumbing + contracts). + +The gradient solver keeps the reservoir fixed, computes its states once +(dropping ``washout`` per sequence) and trains a ``LinearReadout`` with an +optimizer loop instead of the closed-form ridge solve. These tests check that +the gradient path is wired correctly (runs, learns, right shapes, right +guards) rather than asserting tight closed-form equivalence. + +Gradient fits are reproducible through the estimator's ``random_state`` (the +readout init and shuffle are seeded from it), checked by +``test_esn_gradient_reproducible`` without any global torch seeding. +""" +from __future__ import annotations + +import numpy as np +import pytest +import torch +from sklearn.pipeline import FeatureUnion + +from pyrcn.base.blocks import InputToNode +from pyrcn.datasets import load_digits, mackey_glass +from pyrcn.echo_state_network import ESNClassifier, ESNRegressor + + +def test_esn_gradient_nonsequence_learns() -> None: + torch.manual_seed(0) + X, y = mackey_glass(n_timesteps=500) + X = np.asarray(X).reshape(-1, 1) + y = np.asarray(y) + esn = ESNRegressor( + hidden_layer_size=50, spectral_radius=0.9, leakage=0.7, + alpha=1e-8, solver="gradient", optimizer="adam", + learning_rate=0.1, epochs=400, random_state=42).fit(X, y) + + assert esn._use_torch is True + pred = esn.predict(X) + assert pred.shape == y.shape # 1-D target -> 1-D prediction + assert esn.score(X, y) > 0.3 # gradient training learns + + +def test_esn_gradient_sequence_runs() -> None: + torch.manual_seed(0) + Xs, ys = mackey_glass(n_timesteps=600) + Xs = np.asarray(Xs).reshape(-1, 1) + ys = np.asarray(ys) + X = np.empty(shape=(3,), dtype=object) + y = np.empty(shape=(3,), dtype=object) + for k in range(3): + X[k] = Xs[k * 200:(k + 1) * 200] + y[k] = ys[k * 200:(k + 1) * 200] + + esn = ESNRegressor( + hidden_layer_size=50, spectral_radius=0.9, leakage=0.7, + alpha=1e-8, solver="gradient", optimizer="adam", + learning_rate=0.1, epochs=200, random_state=42).fit(X, y) + + assert esn._use_torch is True + pred = esn.predict(X) + assert pred.dtype == object + assert len(pred) == 3 + for k in range(3): + assert np.asarray(pred[k]).ndim == 1 + + +def test_esn_gradient_requires_backable() -> None: + X, y = mackey_glass(n_timesteps=200) + X = np.asarray(X).reshape(-1, 1) + y = np.asarray(y) + esn = ESNRegressor( + solver="gradient", + input_to_node=FeatureUnion([("x", InputToNode())])) + with pytest.raises(NotImplementedError): + esn.fit(X, y) + + +def test_esn_invalid_solver() -> None: + X, y = mackey_glass(n_timesteps=200) + X = np.asarray(X).reshape(-1, 1) + y = np.asarray(y) + with pytest.raises(ValueError): + ESNRegressor(solver="bogus").fit(X, y) + + +def test_esn_gradient_reproducible() -> None: + # No torch.manual_seed: the estimator seeds its readout from random_state. + X, y = mackey_glass(n_timesteps=300) + X = np.asarray(X).reshape(-1, 1) + y = np.asarray(y) + + def _pred() -> np.ndarray: + return ESNRegressor( + hidden_layer_size=40, spectral_radius=0.9, leakage=0.7, + solver="gradient", optimizer="adam", learning_rate=0.1, + epochs=150, random_state=42).fit(X, y).predict(X) + + np.testing.assert_array_equal(_pred(), _pred()) + + +def test_esn_gradient_classifier() -> None: + torch.manual_seed(0) + X, y = load_digits(return_X_y=True, as_sequence=True) + X, y = X[:40], y[:40] + clf = ESNClassifier( + hidden_layer_size=50, solver="gradient", optimizer="adam", + learning_rate=0.05, epochs=200, random_state=42).fit(X, y) + + assert clf._use_torch is True + pred = clf.predict(X) + assert len(pred) == len(X) diff --git a/tests/test_esn_torch_backend.py b/tests/test_esn_torch_backend.py new file mode 100644 index 0000000..23f0f65 --- /dev/null +++ b/tests/test_esn_torch_backend.py @@ -0,0 +1,133 @@ +"""Parity tests for the ESN torch backend fast path.""" +from __future__ import annotations + +import numpy as np +from numpy.testing import assert_allclose +from sklearn.linear_model import Ridge +from sklearn.pipeline import FeatureUnion + +from pyrcn.base.blocks import InputToNode, NodeToNode +from pyrcn.echo_state_network import ESNRegressor +from pyrcn.linear_model import IncrementalRegression +from pyrcn.util import concatenate_sequences + + +def _make_i2n(**kwargs: object) -> InputToNode: + params = dict(hidden_layer_size=50, input_activation="identity", + bias_scaling=0.1, input_scaling=1.0, random_state=42) + params.update(kwargs) + return InputToNode(**params) + + +def _make_n2n(**kwargs: object) -> NodeToNode: + params = dict(hidden_layer_size=50, spectral_radius=0.9, + reservoir_activation="tanh", random_state=42) + params.update(kwargs) + return NodeToNode(**params) + + +def _nonsequence_data() -> tuple[np.ndarray, ...]: + rng = np.random.default_rng(0) + X = rng.standard_normal((200, 1)) + y = np.sin(X[:, 0]) + 0.1 * rng.standard_normal(200) + return X[:150], X[150:], y[:150], y[150:] + + +def _sequence_data() -> tuple[np.ndarray, np.ndarray]: + rng = np.random.default_rng(1) + X = np.empty(shape=(4,), dtype=object) + y = np.empty(shape=(4,), dtype=object) + for k in range(4): + length = 30 + k + X[k] = rng.standard_normal((length, 1)) + y[k] = np.cos(X[k][:, 0]) + return X, y + + +def test_esn_uses_torch_when_backable() -> None: + X_train, X_test, y_train, y_test = _nonsequence_data() + esn = ESNRegressor(input_to_node=_make_i2n(), node_to_node=_make_n2n()) + esn.fit(X_train, y_train) + assert esn._use_torch is True + + +def test_esn_falls_back_for_featureunion() -> None: + X_train, X_test, y_train, y_test = _nonsequence_data() + union = FeatureUnion([ + ("a", _make_i2n(hidden_layer_size=20)), + ("b", _make_i2n(hidden_layer_size=20, random_state=7))]) + esn = ESNRegressor(input_to_node=union, + node_to_node=_make_n2n(hidden_layer_size=40)) + esn.fit(X_train, y_train) + assert esn._use_torch is False + y_pred = esn.predict(X_test) + assert y_pred.shape[0] == X_test.shape[0] + + +def test_esn_falls_back_for_ridge() -> None: + X_train, X_test, y_train, y_test = _nonsequence_data() + esn = ESNRegressor(input_to_node=_make_i2n(), node_to_node=_make_n2n(), + regressor=Ridge(alpha=1e-3)) + esn.fit(X_train, y_train) + assert esn._use_torch is False + y_pred = esn.predict(X_test) + assert y_pred.shape[0] == X_test.shape[0] + + +def _fit_numpy_reference(i2n: InputToNode, n2n: NodeToNode, + reg: IncrementalRegression, X: np.ndarray, + y: np.ndarray) -> np.ndarray: + i2n.fit(X) + n2n.fit(i2n.transform(X)) + states = n2n.transform(i2n.transform(X)) + reg.fit(states, y) + return states + + +def test_esn_fastpath_matches_numpy_reference_nonsequence() -> None: + X_train, X_test, y_train, y_test = _nonsequence_data() + for bidirectional in (False, True): + i2n = _make_i2n() + n2n = _make_n2n(bidirectional=bidirectional) + reg = IncrementalRegression(alpha=1e-3) + _fit_numpy_reference(i2n, n2n, reg, X_train, y_train) + s_test = n2n.transform(i2n.transform(X_test)) + expected = reg.predict(s_test) + + esn = ESNRegressor( + input_to_node=_make_i2n(), + node_to_node=_make_n2n(bidirectional=bidirectional), + regressor=IncrementalRegression(alpha=1e-3)) + esn.fit(X_train, y_train) + assert esn._use_torch is True + got = esn.predict(X_test) + assert_allclose(got, expected, rtol=1e-6, atol=1e-6) + + +def test_esn_fastpath_matches_numpy_reference_sequence() -> None: + X_train, y_train = _sequence_data() + X_test, _ = _sequence_data() + + i2n = _make_i2n() + n2n = _make_n2n() + reg = IncrementalRegression(alpha=1e-3) + X_cat, y_cat, ranges = concatenate_sequences(X_train, y_train) + i2n.fit(X_cat) + n2n.fit(i2n.transform(X_cat)) + n_seq = len(ranges) + for i, (start, stop) in enumerate(ranges): + states = n2n.transform(i2n.transform(X_cat[start:stop])) + reg.partial_fit(states, y_cat[start:stop], reset=(i == 0), + postpone_inverse=(i < n_seq - 1)) + expected = [] + for seq in X_test: + expected.append(reg.predict(n2n.transform(i2n.transform(seq)))) + + esn = ESNRegressor(input_to_node=_make_i2n(), node_to_node=_make_n2n(), + regressor=IncrementalRegression(alpha=1e-3)) + esn.fit(X_train, y_train) + assert esn._use_torch is True + got = esn.predict(X_test) + assert len(got) == len(expected) + for k in range(len(expected)): + assert_allclose(got[k], expected[k], rtol=1e-5) diff --git a/tests/test_esn_trainable.py b/tests/test_esn_trainable.py new file mode 100644 index 0000000..b1d565e --- /dev/null +++ b/tests/test_esn_trainable.py @@ -0,0 +1,131 @@ +"""Phase B2: trainable reservoir (end-to-end gradient training via BPTT). + +With ``trainable_reservoir=True`` (and ``solver='gradient'``) the reservoir's +recurrent weights are optimized jointly with the readout by backpropagating +through the recurrence. The RC initialization is the starting point. These +tests check the plumbing: the weights become trainable and actually move, the +fit learns, the invalid ``trainable + closed_form`` combo is rejected, and +fits are reproducible via ``random_state``. +""" +from __future__ import annotations + +import numpy as np +import pytest + +from pyrcn.datasets import load_digits +from pyrcn.echo_state_network import ESNClassifier, ESNRegressor + + +def _sin_next_step(n: int = 160) -> tuple[np.ndarray, np.ndarray]: + x = np.sin(np.linspace(0, 8 * np.pi, n)).reshape(-1, 1) + return x, np.roll(x.ravel(), -1) + + +def _reg(**kw: object) -> ESNRegressor: + params: dict = dict( + hidden_layer_size=20, spectral_radius=0.9, leakage=0.7, + solver="gradient", optimizer="adam", learning_rate=0.05, + epochs=120, random_state=42) + params.update(kw) + return ESNRegressor(**params) + + +def test_trainable_reservoir_fits_and_flags() -> None: + X, y = _sin_next_step() + esn = _reg(trainable_reservoir=True).fit(X, y) + assert esn._use_torch is True + assert esn._torch_reservoir.cell.weight_hh.requires_grad is True + assert esn.score(X, y) > 0.6 # end-to-end training learns + + +def test_trainable_reservoir_updates_recurrent_weights() -> None: + X, y = _sin_next_step() + fixed = _reg(trainable_reservoir=False, epochs=60).fit(X, y) + trained = _reg(trainable_reservoir=True, epochs=60).fit(X, y) + w_fixed = fixed._torch_reservoir.cell.weight_hh.detach().numpy() + w_trained = trained._torch_reservoir.cell.weight_hh.detach().numpy() + # the fixed path keeps the RC init; the trainable path moved away from it + assert not np.allclose(w_fixed, w_trained) + + +def test_trainable_requires_gradient_solver() -> None: + with pytest.raises(ValueError): + ESNRegressor(trainable_reservoir=True, solver="closed_form").fit( + np.zeros((10, 1)), np.zeros(10)) + + +def test_trainable_reproducible() -> None: + X, y = _sin_next_step(120) + + def _pred() -> np.ndarray: + return _reg(trainable_reservoir=True, epochs=60).fit(X, y).predict(X) + + np.testing.assert_array_equal(_pred(), _pred()) + + +def test_trainable_classifier_runs() -> None: + X, y = load_digits(return_X_y=True, as_sequence=True) + X, y = X[:30], y[:30] + clf = ESNClassifier( + hidden_layer_size=25, solver="gradient", trainable_reservoir=True, + optimizer="adam", learning_rate=0.05, epochs=60, + random_state=42).fit(X, y) + assert clf._use_torch is True + assert clf._torch_reservoir.cell.weight_hh.requires_grad is True + assert len(clf.predict(X)) == len(X) + + +def test_trainable_input_learns_and_updates_weights() -> None: + X, y = _sin_next_step() + fixed = _reg(trainable_input=False, epochs=80).fit(X, y) + trained = _reg(trainable_input=True, epochs=80).fit(X, y) + assert all(p.requires_grad + for p in trained._torch_input_map.parameters()) + w_fixed = fixed._torch_input_map.weight.detach().numpy() + w_trained = trained._torch_input_map.weight.detach().numpy() + assert not np.allclose(w_fixed, w_trained) # input weights moved + assert trained.score(X, y) > 0.6 + + +def test_trainable_input_and_reservoir_together() -> None: + X, y = _sin_next_step() + esn = _reg(trainable_input=True, trainable_reservoir=True, + epochs=80).fit(X, y) + assert all(p.requires_grad for p in esn._torch_input_map.parameters()) + assert esn._torch_reservoir.cell.weight_hh.requires_grad is True + assert esn.score(X, y) > 0.6 + + +def test_trainable_input_requires_gradient_solver() -> None: + with pytest.raises(ValueError): + ESNRegressor(trainable_input=True, solver="closed_form").fit( + np.zeros((10, 1)), np.zeros(10)) + + +def _sequences(n_seq: int = 6, length: int = 100 + ) -> tuple[np.ndarray, np.ndarray]: + flat = np.sin(np.linspace(0, 12 * np.pi, n_seq * length)) + X = np.empty(n_seq, dtype=object) + y = np.empty(n_seq, dtype=object) + for k in range(n_seq): + seg = flat[k * length:(k + 1) * length].reshape(-1, 1) + X[k] = seg + y[k] = np.roll(seg.ravel(), -1) + return X, y + + +def test_bptt_minibatching_runs_and_is_reproducible() -> None: + X, y = _sequences() + + def _pred(batch_size: int) -> np.ndarray: + esn = ESNRegressor( + hidden_layer_size=20, spectral_radius=0.9, leakage=0.7, + solver="gradient", trainable_reservoir=True, optimizer="adam", + learning_rate=0.05, epochs=40, batch_size=batch_size, + random_state=42) + return esn.fit(X, y).predict(X) + + out = _pred(2) # 2 sequences / step + assert len(out) == 6 + a, b = _pred(3), _pred(3) + assert all(np.array_equal(a[k], b[k]) for k in range(6)) diff --git a/tests/test_esn_washout_state.py b/tests/test_esn_washout_state.py new file mode 100644 index 0000000..d2c1418 --- /dev/null +++ b/tests/test_esn_washout_state.py @@ -0,0 +1,124 @@ +"""A4b: ESN washout (training-only) and predict initial_state / return_state. + +Defaults (washout=0, initial_state=None, return_state=False) are behavior- +preserving; the new capabilities require the torch backend and raise on the +numpy fallback. +""" +from __future__ import annotations + +import numpy as np +import pytest +from sklearn.linear_model import Ridge + +from pyrcn.base.blocks import InputToNode, NodeToNode +from pyrcn.echo_state_network import ESNRegressor +from pyrcn.linear_model import IncrementalRegression + +HID = 30 + + +def _data(n: int = 200, seed: int = 0) -> tuple[np.ndarray, np.ndarray]: + rng = np.random.RandomState(seed) + X = rng.normal(size=(n, 1)) * 0.3 + y = np.sin(np.linspace(0, 8, n)) + return X, y + + +def _numpy_reference(X: np.ndarray, y: np.ndarray, X_test: np.ndarray, + washout: int) -> np.ndarray: + """Legacy numpy pipeline with the first ``washout`` states dropped.""" + i2n = InputToNode(hidden_layer_size=HID, random_state=42) + i2n.fit(X) + n2n = NodeToNode(hidden_layer_size=HID, spectral_radius=0.9, leakage=0.6, + random_state=42) + n2n.fit(i2n.transform(X)) + states = n2n.transform(i2n.transform(X)) + reg = IncrementalRegression(alpha=1e-3) + reg.fit(states[washout:], y[washout:]) + test_states = n2n.transform(i2n.transform(X_test)) + return reg.predict(test_states) + + +def _esn(washout: int = 0) -> ESNRegressor: + return ESNRegressor( + hidden_layer_size=HID, spectral_radius=0.9, leakage=0.6, alpha=1e-3, + washout=washout, random_state=42) + + +def test_washout_zero_is_behavior_preserving() -> None: + X, y = _data() + expected = _numpy_reference(X, y, X, washout=0) + got = _esn(washout=0).fit(X, y).predict(X) + np.testing.assert_allclose(got, expected, rtol=1e-5, atol=1e-6) + + +def test_washout_matches_numpy_reference() -> None: + X, y = _data() + expected = _numpy_reference(X, y, X, washout=10) + got = _esn(washout=10).fit(X, y).predict(X) + np.testing.assert_allclose(got, expected, rtol=1e-5, atol=1e-6) + + +def test_washout_changes_output() -> None: + X, y = _data() + y0 = _esn(washout=0).fit(X, y).predict(X) + y1 = _esn(washout=25).fit(X, y).predict(X) + assert not np.allclose(y0, y1) + + +def test_washout_negative_raises() -> None: + X, y = _data() + with pytest.raises(ValueError): + _esn(washout=-1).fit(X, y) + + +def test_washout_fallback_raises() -> None: + X, y = _data() + esn = ESNRegressor(regressor=Ridge(), washout=3, hidden_layer_size=HID) + with pytest.raises(NotImplementedError): + esn.fit(X, y) + + +def test_initial_state_zero_matches_default() -> None: + X, y = _data() + esn = _esn().fit(X, y) + y_none = esn.predict(X) + y_zero = esn.predict(X, initial_state=np.zeros(HID)) + np.testing.assert_allclose(y_zero, y_none, rtol=1e-10, atol=1e-12) + + +def test_initial_state_changes_output() -> None: + X, y = _data() + esn = _esn().fit(X, y) + y_none = esn.predict(X) + y_seed = esn.predict(X, initial_state=np.full(HID, 0.5)) + assert not np.allclose(y_none, y_seed) + + +def test_return_state_nonsequence() -> None: + X, y = _data() + esn = _esn().fit(X, y) + y_only = esn.predict(X) + y_out, final = esn.predict(X, return_state=True) + assert final.shape == (HID,) + np.testing.assert_allclose(y_out, y_only, rtol=1e-12, atol=1e-12) + + +def test_return_state_carry_continues_trajectory() -> None: + # Splitting a sequence and carrying the final state reproduces running + # the whole sequence in one pass. + X, y = _data(n=120) + esn = _esn().fit(X, y) + whole = esn.predict(X) + first, second = X[:60], X[60:] + y1, state = esn.predict(first, return_state=True) + y2 = esn.predict(second, initial_state=state) + np.testing.assert_allclose( + np.concatenate([y1, y2]), whole, rtol=1e-8, atol=1e-9) + + +def test_initial_state_fallback_raises() -> None: + X, y = _data() + esn = ESNRegressor(regressor=Ridge(), hidden_layer_size=HID).fit(X, y) + with pytest.raises(NotImplementedError): + esn.predict(X, initial_state=np.zeros(HID)) diff --git a/tests/test_input_to_node.py b/tests/test_input_to_node.py index 0ecae37..9e0221e 100644 --- a/tests/test_input_to_node.py +++ b/tests/test_input_to_node.py @@ -184,6 +184,10 @@ def test_input_to_node_sparse() -> None: i2n.fit(X) assert i2n._input_weights.shape == (3, 5) assert safe_sparse_dot(X, i2n._input_weights).shape == (10, 5) + # k_in is the number of inputs connected to each hidden node. + W = i2n._input_weights + W = W.toarray() if hasattr(W, 'toarray') else np.asarray(W) + assert np.all((W != 0).sum(axis=0) == 2) assert i2n.__sizeof__() != 0 assert i2n.input_weights is not None assert i2n.bias_weights is not None diff --git a/tests/test_model_selection_esn.py b/tests/test_model_selection_esn.py new file mode 100644 index 0000000..3664c5c --- /dev/null +++ b/tests/test_model_selection_esn.py @@ -0,0 +1,109 @@ +"""Model-selection tests for torch-backed ESN estimators. + +These prove that GridSearchCV, RandomizedSearchCV and the library's +SequentialSearchCV work with the torch-backed ESN estimators and preserve +the torch fast path through ``clone`` (``best_estimator_._use_torch`` stays +True). Bare-name grids (``spectral_radius``, ``alpha``, ...) are routed to the +sub-blocks by the estimator's ``set_params``. Reservoirs are kept tiny so the +per-timestep recurrence stays fast under repeated CV refits. +""" + +from __future__ import annotations + +import numpy as np +from sklearn.model_selection import (GridSearchCV, RandomizedSearchCV, + TimeSeriesSplit) + +from pyrcn.base.blocks import NodeToNode +from pyrcn.datasets import mackey_glass +from pyrcn.echo_state_network import ESNRegressor +from pyrcn.linear_model import IncrementalRegression +from pyrcn.model_selection import SequentialSearchCV + +HIDDEN = 20 + + +def _data() -> tuple[np.ndarray, np.ndarray]: + """Return small non-sequence mackey-glass data.""" + X, y = mackey_glass(n_timesteps=300) + return X.reshape(-1, 1), y + + +def _estimator() -> ESNRegressor: + """A small torch-backable ESN regressor (hidden size routed to blocks).""" + return ESNRegressor(hidden_layer_size=HIDDEN, verbose=False) + + +def test_gridsearch_esn_fastpath_survives_clone() -> None: + """GridSearchCV fits and keeps the torch fast path via clone.""" + X, y = _data() + gs = GridSearchCV( + _estimator(), + param_grid={'spectral_radius': [0.0, 0.9], 'alpha': [1e-2, 1e-5]}, + cv=TimeSeriesSplit(n_splits=2)) + gs.fit(X, y) + assert gs.best_estimator_._use_torch is True + y_pred = gs.predict(X) + assert y_pred.shape == y.shape + assert np.all(np.isfinite(y_pred)) + + +def test_gridsearch_varies_model() -> None: + """Bare-name grid params really take effect through set_params.""" + X, y = _data() + gs = GridSearchCV( + _estimator(), + param_grid={'spectral_radius': [0.0, 0.9], 'alpha': [1e-2, 1e-5]}, + cv=TimeSeriesSplit(n_splits=2)) + gs.fit(X, y) + assert len(gs.cv_results_['params']) == 4 + scores = np.asarray(gs.cv_results_['mean_test_score']) + assert np.unique(np.round(scores, 10)).size >= 2 + + +def test_randomizedsearch_esn_fastpath() -> None: + """RandomizedSearchCV keeps the torch fast path.""" + X, y = _data() + rs = RandomizedSearchCV( + _estimator(), + param_distributions={'spectral_radius': [0.1, 0.5, 0.9], + 'alpha': [1e-2, 1e-4]}, + n_iter=3, random_state=42, cv=TimeSeriesSplit(n_splits=2)) + rs.fit(X, y) + assert rs.best_estimator_._use_torch is True + + +def test_sequentialsearch_esn_fastpath() -> None: + """SequentialSearchCV keeps the torch fast path across searches.""" + X, y = _data() + cv = TimeSeriesSplit(n_splits=2) + ss = SequentialSearchCV( + _estimator(), + searches=[ + ('sr', GridSearchCV, {'spectral_radius': [0.0, 0.9]}, {'cv': cv}), + ('alpha', GridSearchCV, {'alpha': [1e-2, 1e-5]}, {'cv': cv}), + ]).fit(X, y) + assert ss.best_estimator_ is not None + assert ss.best_estimator_._use_torch is True + assert 'sr' in ss.all_best_params_ + assert 'alpha' in ss.all_best_params_ + assert isinstance(ss.best_score_, float) + + +def test_component_swap_grid_fastpath() -> None: + """Object-valued component swaps still fast-path through clone.""" + X, y = _data() + gs = GridSearchCV( + _estimator(), + param_grid={ + 'node_to_node': [ + NodeToNode(hidden_layer_size=HIDDEN, spectral_radius=0.0, + random_state=42), + NodeToNode(hidden_layer_size=HIDDEN, spectral_radius=0.9, + random_state=42)], + 'regressor': [IncrementalRegression(alpha=1e-2), + IncrementalRegression(alpha=1e-5)]}, + cv=TimeSeriesSplit(n_splits=2)) + gs.fit(X, y) + assert gs.best_estimator_._use_torch is True + assert np.all(np.isfinite(gs.predict(X))) diff --git a/tests/test_sequences.py b/tests/test_sequences.py new file mode 100644 index 0000000..6124b30 --- /dev/null +++ b/tests/test_sequences.py @@ -0,0 +1,82 @@ +"""Tests for the sequence normalization used by the redesigned backend.""" +from __future__ import annotations + +import numpy as np +import pytest + +from pyrcn.util import check_sequences + + +def test_2d_single_sequence() -> None: + X = np.arange(30, dtype=float).reshape(10, 3) + y = np.arange(10, dtype=float) + batch = check_sequences(X, y) + assert batch.X.shape == (1, 10, 3) + assert batch.lengths.tolist() == [10] + assert batch.single_sequence is True + assert batch.task == "sequence-to-sequence" + assert batch.n_features == 3 + np.testing.assert_array_equal(batch.X[0], X) + + +def test_list_ragged_seq2seq() -> None: + X = [np.ones((5, 2)), np.ones((3, 2)) * 2] + y = [np.ones((5, 1)), np.ones((3, 1))] + batch = check_sequences(X, y) + assert batch.X.shape == (2, 5, 2) + assert batch.lengths.tolist() == [5, 3] + assert batch.single_sequence is False + assert batch.task == "sequence-to-sequence" + assert np.all(batch.X[1, 3:] == 0) # zero-padded tail + np.testing.assert_array_equal(batch.X[1, :3], np.ones((3, 2)) * 2) + assert batch.y.shape == (2, 5, 1) + + +def test_object_array_seq2value() -> None: + rng = np.random.RandomState(0) + X = np.empty(4, dtype=object) + y = np.empty(4, dtype=object) + for k in range(4): + X[k] = rng.rand(8, 8) + y[k] = np.atleast_1d(k) # (1,) per sequence + batch = check_sequences(X, y) + assert batch.X.shape == (4, 8, 8) + assert batch.lengths.tolist() == [8, 8, 8, 8] + assert batch.task == "sequence-to-value" + assert batch.y.shape == (4, 1) + + +def test_3d_equal_length_seq2seq() -> None: + X = np.ones((4, 6, 2)) + y = [np.ones((6, 1)) for _ in range(4)] + batch = check_sequences(X, y) + assert batch.X.shape == (4, 6, 2) + assert batch.lengths.tolist() == [6, 6, 6, 6] + assert batch.task == "sequence-to-sequence" + + +def test_task_override_value() -> None: + # per-sequence target length equals L (ambiguous) -> force value + X = [np.ones((3, 2)), np.ones((3, 2))] + y = [np.arange(3), np.arange(3)] + batch = check_sequences(X, y, task="sequence-to-value") + assert batch.task == "sequence-to-value" + assert batch.y.shape == (2, 3) + + +def test_predict_no_targets() -> None: + X = [np.ones((5, 2)), np.ones((3, 2))] + batch = check_sequences(X) + assert batch.X.shape == (2, 5, 2) + assert batch.y is None + assert batch.task is None + + +def test_inconsistent_features_raises() -> None: + with pytest.raises(ValueError): + check_sequences([np.ones((5, 2)), np.ones((3, 3))]) + + +def test_bad_ndim_raises() -> None: + with pytest.raises(ValueError): + check_sequences(np.ones((2, 3, 4, 5)))