epiphanies: DeepNSM-facet Morton-comma — blind-convergence adjudication + implicit-256:256-tile probe

.claude/board/EPIPHANIES.md

@@ -1,3 +1,47 @@
+## 2026-07-19 — E-IMPLICIT-MORTON-TILE-1 — the 256:256=64k tile distance table is IMPLICIT, but the address does TWO jobs at TWO fidelities: pure shift/mask (address-only) is ROUTING (ρ≈0.51 hierarchical vs 0.41 flat — coarse is correct); the METRIC needs the stored 256-centroid codebook (ADC, ρ=1.0) — the 64k pairwise table is NEVER materialized in EITHER case (F11-adjacent named test, RUN)
+
+**Status:** FINDING (built + measured on committed data, hierarchical-vs-flat contrast). **Confidence:** High for the tier distinction + the 64k-never-materialized ruling; Medium-magnitude for the exact address-only ρ (8-genre committed data + rough deterministic 4-means, degenerate empty-branch leaves add noise — the *direction* hier>flat is robust, the absolute 0.51 is a floor). Runs the OGAR canon's F11-adjacent "hierarchical-4⁴ vs flat-256" named-but-unrun test. Operator: *"what about Morton-comma where the table is implicit 256:256=64k."* Deliverable: `implicit_morton_tile_probe.py` (scratchpad; std-only, deterministic, no RNG; buildable into `crates/deepnsm/examples/` on request).
+
+**The probe.** Build a 256-centroid codebook on the committed 8-genre semantic vectors as a **4-level 4-ary hierarchy** (byte = the four 2-bit ancestry groups). Compare two distances over all 32,640 leaf pairs: `d_mat` = materialized L2(centroid_a, centroid_b) (the 256×256 table, ground truth) vs `d_imp` = **address-only** distance from the two bytes (first-divergence-level Morton offset, scaled 4^−L — pure shift/mask, no centroid load).
+
+**Measured:**
+- `ρ(d_imp, d_mat)` **HIERARCHICAL 4⁴ = 0.5083**; **FLAT-relabel = 0.4118** → the hierarchy→address→distance correspondence is REAL (+0.09, ~22% relative; the canon's "flat k-means-256 breaks this, hierarchical 4⁴ preserves it" holds in DIRECTION).
+- All-levels weighted variant = 0.4168 (**worse** than divergence-level-only) → the address's real signal is the PREFIX, not the tail; post-divergence nibbles are arbitrary noise.
+- **ADC-implicit = 1.0 by construction:** store the 256 centroids (2 KB texture), compute L2 on demand — the 64k pairwise table is NEVER materialized, ρ trivially perfect.
+
+**The ruling — the address does TWO jobs at TWO fidelities, neither needs the 64k table:**
+| job | mechanism | fidelity | storage |
+|---|---|---|---|
+| **Routing / prefix-scoping** (`is_ancestor_of`, which codebook/basin, longest-prefix) | address-only Morton shift/mask | ρ≈0.51 — coarse is EXACTLY right | nothing, O(1) |
+| **Metric** (fine cell↔cell distance ordering) | ADC over stored 256 centroids | ρ=1.0 | 2 KB codebook, never 64k |
+
+So "implicit 256:256=64k" is SOUND — the 64k pairwise table is a materialization *choice* you always decline (store the 256-centroid **texture**, regenerate the distance **shader** on demand). But the RADICAL address-only-*metric* reading (distance from bytes alone, no codebook load) is a coarse ρ≈0.51 proxy: it captures ROUTING, not the fine metric. The **comma** supplies fine structure for a *continuous perturbation field placed over the tile* (the shader — cf. `E-PERTURBATION-CONVERGENCE-1`), NOT for discrete cell↔cell metric (no sub-cell to interpolate between two quantized leaves).
+
+**Why this composes with the header/payload split (`E-DEEPNSM-FACET-BLIND-CONVERGENCE-1`):** the HEADER (classid/prefix) uses address-only ancestry → ROUTING, where coarse is correct — which is exactly why **frequency-as-header works** (a routing coordinate). The PAYLOAD (6×256:256) carries the METRIC → needs the stored centroids (ADC/Fisher-z, ρ=1.0) — which is exactly why **frequency-in-payload corrupts** (it perturbs a metric coordinate). Same address, two fidelities, two jobs — the two findings are one architecture. **Honest boundary:** 8-genre committed data with a rough k-means; the real 96D/6-subspace codebook (gitignored) may shift the address-only ρ, but the tier distinction (routing-coarse vs metric-needs-codebook; 64k never materialized) is structural. Additive: probe + board only, no core change. **Cross-ref:** E-DEEPNSM-FACET-BLIND-CONVERGENCE-1, E-FREQ-IS-COSINE-REPLACEMENT-1 (distance immaterializes because the centroid IS the cosine — the ADC ρ=1.0 leg), E-PERTURBATION-CONVERGENCE-1 (the comma as the continuous-field shader), OGAR `256×256 CENTROID TILE` canon + the F11 hierarchical-4⁴ condition, `soa_layout/le-contract.md` L4 `6×palette256:palette256`.
+
+## 2026-07-19 — E-DEEPNSM-FACET-BLIND-CONVERGENCE-1 — two INDEPENDENT blind implementations of PROBE-DEEPNSM-FACET (spec §3.2–3.5, `lemmas_5k.csv`) converged BYTE-IDENTICALLY (ρ_all=0.9992, NN-flip 12/256, anchor we/will −1915 ranks) → header-invariance CONFIRMED, PF=Header equilibrium CONFIRMED; and the pre-registered §3b double-decile falsifier is STRUCTURALLY MIS-SCOPED (empty even with full frequency range) — the SECOND criterion-vs-data miscalibration this arc after the Codex P1
+
+**Status:** FINDING (diverse-redundancy replication, operator-adjudicated) + SPEC-CORRECTION (§3b). **Confidence:** High for header-invariance + convergence; High for the §3b mis-scope (proven on committed data, two independent runs + a rank-stratified widening). Provenance: the blind protocol the operator requested — *"you both test the outcome without telling the other session the results before I confirm that it was done"* — applied to the merged #758 design/spec handoff (`deepnsm-morton-comma-facet-v1.md`). Deliverables: the spec (merged #758); two independent Python impls (session A + operator-run session B, both from spec alone, neither importing the other); the decisive widened cell run this session.
+
+**The two blind runs (top-256 by rank, 32,640 pairs, deterministic — no RNG):**
+| measurement | session A | session B | verdict |
+|---|---|---|---|
+| ρ_all (Spearman d_payload vs d_truth) | 0.9992 | 0.9992 | identical |
+| HEADER freq-shuffle invariance | invariant (d_header=d_truth) | invariant (mean \|Δd_payload\|=0.2417) | identical |
+| nearest-neighbor flip rate | 12/256 (4.7%) | 12/256 (4.7%) | identical |
+| §3b corrupted set (near-decile ∩ freq-far-decile) | EMPTY (0 pairs) | EMPTY (0 pairs) | identical |
+| named anchor (near decile, max Δf) | we/will dt=0.950→dp=1.175 (+0.224), −1915 ranks | we/will dt=0.950→dp=1.175 (+0.224), −1915 ranks | identical |
+
+**Two independent implementations from the spec text alone produced BYTE-IDENTICAL numbers.** That is the diverse-redundancy result working as designed: spec §3.2–3.5 is deterministic and unambiguous, so the empty-3b-vs-strong-anchor tension is a REAL property of the data, not either session's artifact.
+
+**The rulings (operator-adjudicated):**
+1. **Header-invariance (orthogonality): CONFIRMED — the load-bearing half.** Frequency as a routing/attention HEADER does not leak into the semantic distance (`d_header=d_truth` by construction, freq-shuffle-invariant in both runs). This is the claim the whole facet design rests on.
+2. **Collapse mechanism: PRESENT and consistent.** Folding frequency into the payload demotes near-synonym / frequency-divergent pairs by +1256..1915 rank positions (we/will, and county/am in the widened cell) and flips 4.7–6.2% of nearest neighbors. PF=Payload measurably corrupts near-neighbor structure.
+3. **PF=Header is the game-matrix equilibrium: CONFIRMED.** (Address, Dist, Header) is the stable cell; (Address, Dist, Payload) is the defection trap the operator's game-matrix predicted. Frequency belongs in the `classid` header, the 6×(8:8) payload carries semantic distribution ONLY.
+4. **§3b (the pre-registered double-decile AND falsifier) is STRUCTURALLY MIS-SCOPED — retire it.** The decisive widened cell this session (256 words RANK-STRATIFIED across the full 5,050 → frequency z-span −1.14…6.23, full dynamic range restored) STILL returns an empty near∩freq-far double-decile set. So the empty-3b is NOT a top-256 window artifact: semantic-nearness and frequency-nearness are POSITIVELY correlated in this data (near-decile semantic pairs are also frequency-near; the extreme intersection is ~disjoint), so the strict AND can never fire regardless of window. The correct falsifier is the **anchor + NN-flip signal** (which both runs and the widened cell agree fires cleanly), NOT the double-decile set. This is the SECOND criterion-vs-data miscalibration on this probe — cf. the Codex P1 (impossible `d_payload<d_truth` under Pythagoras); both are the same lesson: **a pre-registered set-membership criterion must be checked against the data's actual joint distribution before it becomes the gate.**
+
+**Honest boundary.** ρ_all=0.9992 means the GLOBAL ranking is barely perturbed — the corruption is a TAIL effect concentrated on the high-frequency function-word core (we/will, county/am), exactly as Codex's Pythagoras math predicts (append can only ADD distance, `d_payload≥d_truth`, so damage lands where Δf is large relative to a small d_truth). "Collapse" here means localized near-neighbor demotion, not a global scramble. The richer distributional-payload (96-subgenre / palette256²) leg is gated on the gitignored `codebook_pq.bin` and would only refine the magnitude, not the ruling. Additive: probe + spec only, no core change. **Cross-ref:** E-FREQ-IS-COSINE-REPLACEMENT-1 (the ground the facet stands on), E-SURFACE-FORM-COLLAPSE-1 (the homograph front door), `.claude/plans/deepnsm-morton-comma-facet-v1.md` (the merged spec — §3b now superseded by this entry's anchor/NN-flip falsifier), the OGAR 256×256 CENTROID TILE canon, `soa_layout/le-contract.md` L4 `6×palette256:palette256`.
+
 ## 2026-07-19 — E-FREQ-IS-COSINE-REPLACEMENT-1 — the per-subgenre word FREQUENCY DISTRIBUTION reproduces semantic-distance ordering with NO embedding and NO cosine-on-vectors (Spearman ρ=0.762 on the 8 in-repo genres alone) — "the centroid IS the cosine", the empirical ground under the palette256 / Morton / perturbation-shader distance
 
 **Status:** FINDING (built + measured in both Python and Rust, KILL-gated ρ≥0.5). **Confidence:** High. Deliverable: `crates/deepnsm/examples/freq_is_cosine.rs` (std-only, zero-dep). Operator thread: *"the frequency translates to cosine replacement centroid"* + *"there's no way this isn't relevant for distribution"* — offered as a testable claim, tested.

.claude/plans/deepnsm-morton-comma-facet-v1.md

@@ -105,6 +105,18 @@ collapse semantic distance vs frequency-in-header — runs on **committed data**
    `d_truth`, `d_payload`, inflation, and how many rank positions it FALLS in the
    nearness ordering (a true near-pair demoted purely by frequency).
 
+> **⊘ SUPERSEDED (2026-07-19, `E-DEEPNSM-FACET-BLIND-CONVERGENCE-1`).** Two
+> independent blind runs + a rank-stratified widening (freq z-span −1.14…6.23,
+> full dynamic range) all return an **empty** near-decile∩freq-far-decile set
+> (3b) — semantic-nearness and frequency-nearness are POSITIVELY correlated in
+> this data, so the double-decile AND can never fire regardless of window. The
+> 3b **set-membership** clause of §3.5 is therefore RETIRED as mis-scoped; the
+> live falsifier is measurements **3a (NN-flip) + 4 (named anchor)**, which fire
+> cleanly and identically across both blind runs. Verdict this arc:
+> header-invariance CONFIRMED, PF=Header equilibrium CONFIRMED (localized tail
+> collapse via anchor we/will −1915 ranks). See the EPIPHANY for the full
+> adjudication.
+
 ### 3.5 PASS / COLLAPSE criteria (pre-registered — fill with YOUR numbers only)
 - **HEADER lane** must be **frequency-shuffle-invariant** (measurement 2) — the
   lane-orthogonality proof (frequency didn't leak into semantics).