Skip to content

Region::fit does not discriminate at embedding dimensionality #3

Description

@ops-ping

A fitted Region either admits every probe or rejects every probe, at every rank and coverage tried. b_member and s_depth are shipped as first-class operators but do not appear usable at 768 dimensions with the exemplar counts any caller supplies.

Evidence

Proclamation sentences — 10-exemplar cloud of royal proclamations, probed with 4 held-out proclamations and 6 plain messages, through the real fit_region / b_member in vrules-wasm:

rank / coverage proclamations inside plain inside
3 / 0.90 4/4 6/6
3 / 0.95 4/4 6/6
5 / 0.95 4/4 6/6
8 / 0.95 0/4 0/6

Royalty terms — 24-exemplar cloud of royal terms, probed with 6 royal and 10 non-royal terms:

rank / coverage royal inside non-royal inside
3 / 0.90 6/6 10/10
5 / 0.95 5/6 10/10
8 / 0.95 5/6 10/10
12 / 0.90 5/6 10/10

A 9-exemplar version of the same cloud admitted tractor, coffee and bicycle while at rank 5 excluding royal woman and the queen of Spain — the wrong points on both sides.

Likely cause

depth_of_unit sums squared components along rank principal directions plus a residual term scaled by residual_scale. With a handful of exemplars in 768 dimensions the residual dominates and is near-identical for every vector (concentration of measure), so depth is nearly constant across probes and tau — a quantile of the cloud's own depths — lands inside that constant band. Raising rank toward the exemplar count inverts it: in-cloud residuals collapse, the per-direction scales shrink, and depths blow up for everything.

Impact

  • Blocks using b_member for membership in the browser examples; a linear axis with c_project had to be used instead.
  • crates/vrules-core/tests/fraud_triage.rs fits bec_phrasing_v1 from 6 exemplars at rank 3 and the Fraud triage example gates on it, so that region is likely non-discriminating too. Relevant to Re-enable Address, Fraud triage, Streaming and Proof once their runs assert behaviour #2.
  • Existing tests pass because they only assert a region behaves consistently with the cloud it was fitted from, never that it excludes known negatives.

Suggested work

Establish empirically what exemplar count and dimensionality Region::fit needs to separate known negatives, and either document that floor or change the fit (e.g. shrinkage on the covariance, or dimensionality reduction before fitting). Add conformance coverage that probes a fitted region with negatives, not only with its own cloud.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions