Skip to content

feat(cli): L0 scoring, basis pinning, selection rule, and L1 projection with carry-forward (META-297) - #20

Merged
qmarcelle merged 5 commits into
mainfrom
feature/meta-297-commit-graph-mining-core
Aug 11, 2026
Merged

feat(cli): L0 scoring, basis pinning, selection rule, and L1 projection with carry-forward (META-297)#20
qmarcelle merged 5 commits into
mainfrom
feature/meta-297-commit-graph-mining-core

Conversation

@qmarcelle

@qmarcelle qmarcelle commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

What this is

META-297 Phase 3 (L0) and L1. The branch grew past its original scope; this description now matches the code rather than the state it was opened at.

  • L0 — scoring, basis pinning, and the producer-profile selection rule.
  • L1 — projecting a selection into generated.coChange, and deciding what an ordinary generation run does with it afterwards.

The pipeline is mine → score → select → project, each stage pure with respect to the one before it, so the extracted events and the uncapped scored set stay auditable behind whatever the selection capped.

Emission is opt-in. mineHistory is off unless a caller asks for it, so an ordinary run reads the working tree and never the commit graph.

Scoring (L0)

META-289 v2.2.1, implemented verbatim.

Parameter Value
size_weight min(1, 10/fileCount)
position_decay 2^(-Δpos/250)
Window 500 first-parent transitions
Scoring exclusion events with fileCount > 50
Threshold support >= 3

Selection rule (L0)

Threshold first, ranking second, cap third — in that order, because capping first would silently choose which pairs get ranked.

support DESC → occurrences ASC → files[0] ASC by UTF-8 bytes → files[1] ASC by UTF-8 bytes, cap 50.

Every key is an integer or a string. weightedSupport is a double whose precision ECMAScript leaves implementation-defined, so it ranks nothing and is never serialized. The receipt records minSupport, pairsBeforeCap, pairsEmitted, cap, rankingRule and capBound, so a 50-entry list that is everything is distinguishable from the top of 1,848.

L1 projection

Emits exactly files, support, occurrences. No derived value — no rate, probability, lift, confidence or ranking. No generated classification flag: this producer implements no deterministic tooling-coupling classifier, and ADR-003 A-010 defines absence as unclassified rather than false. The pre-A-010 producer emitted a constant false, which asserted that a lockfile and its manifest are a real source coupling.

Canonical endpoint order is established here, not upstream. score.ts sorts pair endpoints with <, which is UTF-16 code unit order, while the rule names UTF-8 byte order. The two disagree: U+1F600 is a surrogate pair beginning 0xD83D and sorts before U+E000 in UTF-16, while its UTF-8 encoding sorts after. Upstream order is fine for keying a map; it is not fine for bytes a second producer is compared against. Endpoint reversal now yields identical output.

Refusal rather than degradation. A non-mined completeness state projects nothing — never an empty array, which under A-009 is the positive finding "the analysis ran and found no qualifying pairs."

Carry-forward — the load-bearing decision

The standard replaces generated wholesale each run. The ruling says mining is explicit and opt-in and that ordinary generation must not recompute history or fabricate freshness. Taken together, the first erases what the second forbids recomputing.

So coChange and basisRevision are the one part of the producer-owned section that is preserved rather than rebuilt — not because they are manual, but because they derive from an input ordinary generation does not read. The working tree is scanned every run; the commit graph is not.

Three failure modes, each producing a plausible-looking artifact:

Failure Why it is dangerous
Drop Regenerating without carry-forward destroys mined evidence, and the result is indistinguishable from a repository never mined
Advance the pin Carrying observations forward while moving basisRevision to HEAD re-attributes old counts to a commit they were never counted at — the numbers stay plausible and become false
Recompute Every run pays seconds to tens of seconds, and the artifact churns on every commit

Preservation is by object identity, not field-by-field rebuild: a rebuilt entry re-orders keys and changes bytes while staying structurally identical, so identity is what makes byte-for-byte real. A pinned empty array is preserved — it is a positive finding, and dropping it would convert a real result into "never analyzed". Nothing is invented when no prior block exists.

generatedAt records the generation run. basisRevision is the authoritative freshness and provenance pin for the observations, and the only field saying which commit they were counted at.

Refresh outcome — Greptile P1, fixed

When a caller passed mineHistory: true and mining could not complete, generation fell back to the recorded block and returned a successful result. The caller received the previous revision's counts with no way to tell that from a completed refresh.

Falling back is correct and stays. Falling back quietly was the defect. GenerateResult.historyRefresh now reports requested / mined / preserved / refusal, and the refusal reason — which mineHistoryBlock was already computing and discarding, because the diagnostics object it accepts was never passed — is now surfaced. Absence of the field means no refresh was requested; reporting mined: false on an ordinary run would claim an attempt that never happened.

One accommodation, named

@workspacejson/spec@0.4.4 is published and its CoChangeEntry predates A-009 — it requires rate and has no support — so an observation entry is not assignable to it. Confined to a locally declared two-field HistoryBlock and one cast at the assembly boundary, documented in place and deleted when the amended spec publishes. No any, no duplicated standard type, and WorkspaceJsonValidator is untouched and still runs on every artifact.

Test boundaries

candidate-tests/ is excluded from this workspace's vitest run and executes against packed candidates in a disposable environment, because the observation form is rejected by the published dependencies this workspace legitimately pins. Repo-native coverage is validator-free by construction. Cases are placed by what they need, not by what is convenient — refusal cases run repo-native because a refusal emits no coChange; fallback-with-preservation cases need an observation-form artifact on disk and run as candidate-contract tests rather than being weakened to fit.

Verification

Suite Result
mining-core 97/97
cli 76/76
agents-audit-compat 44/44
Candidate-contract (packed env) 22/22
typecheck / build pass
check:architecture 93 files, 4 manifests, no boundary or clean-room violations
check:package-docs 3 manifests agree

Watched-red — each guarantee deliberately broken, rebuilt, repacked and clean-installed, then restored:

Mutation Result
Remove carry-forward (destroy evidence) 7 failed / 12 passed
Advance basisRevision without recomputing 6 failed / 13 passed
Recompute during ordinary generation 1 failed / 18 passed
Drop the refresh-outcome field (silent fallback) 2 failed / 20 passed
Report mined: true unconditionally 1 failed / 21 passed
Restored 22/22

The recompute case is worth noting: an earlier version passed its own mutation, because the fixture had no commit graph so mining refused and fell back — a producer that wrongly mined every run was indistinguishable from one that never did. The replacement mines a repository that does have history, moves the graph, asserts the pin did not advance and the new pairing is absent, then re-mines explicitly and asserts it is present, so the assertions cannot pass by the miner being broken.

Boundaries

Runtime dependency set unchanged — standard-owned packages remain exact published versions; mining-core is a bundled devDependency. No publication authorized. Emission being possible is not the A-009 step-3 gate being open.

META-297 Phases 1 and 2. Adds packages/mining-core, a private, unpublished
L0 that reads git and returns an in-memory observation set. Three consumers
are planned — the producer's L1 projection, the META-289 harness, and the
report — and this is the single implementation all three take, so the
META-140 defect class does not land in the numbers an independent producer
gets compared against.

REQ-001  First-parent extraction using META-289 v2.2.1's frozen commands
         verbatim. Rename detection records both paths. No filtering: events
         Phase 3 will exclude are returned so the exclusion stays countable.
REQ-002  Empty-tree object computed via `hash-object -t tree /dev/null`. The
         SHA-1 literal is absent from the package and a test asserts it,
         assembling the forbidden string from parts so the check does not
         match its own source.
REQ-003  One exported `normalizePath`. ADR-006 does not exist, so this is the
         de facto rule ahead of META-278's ratification: it answers 2 of the 6
         questions from schema prose and declares the other 4 as assumptions
         carried in every observation set rather than defaulting silently.
REQ-004  Deterministic serialization with sorted keys and no wall-clock value.
         Scope stated: these bytes are a function of the repository at the
         resolved basis commit and nothing else.
REQ-005  Four completeness states with reasons, and no code path collapsing
         two into one. The state 2/3 boundary is a recorded parameter
         defaulting to 1 rather than v2.2.1's rawSupport >= 3, because that
         threshold is a scoring decision and Phase 2 runs before scoring.
REQ-006  Shallow-clone guard, watched red. Without it, a --depth 1 clone of a
         coupled fixture returned QUALIFYING_RELATIONSHIP_OBSERVED with the
         same three pairs as the full clone and the thesis pair's support at
         1 instead of 3 — a confident wrong answer, not the graceful empty
         return AP-1 describes.

Cross-checked against workspace-json/billfold, the one repository whose answer
is known in advance. Extraction reproduces the Phase 0 census exactly, and
empty-tree handling is worth exactly one unit of support on the thesis pair
(6 with, 5 without), matching the audit.

Not implemented, deliberately: weighting, decay, the fileCount > 50 exclusion,
support thresholds, lift, ranking, pair caps (REQ-007, REQ-009), and any
projection into the artifact. L1 is blocked on a schema admission — the
published coChange item requires `rate` and sets additionalProperties: false,
so the counts-only shape is rejected rather than divergent.

tsconfig deliberately excludes types/ambient.d.ts, whose hand-written node:fs
and node:child_process declarations shadow @types/node; OWNERSHIP.md already
records that shadowing as an unfixed defect of the class META-244 fixed once
for @workspacejson/spec.

Refs: META-297, META-278, META-289, META-140
The state-4 assertion previously checked an unresolvable revision, which is
state 1 — so EVIDENCE_UNAVAILABLE was structurally reachable but never
exercised, and REQ-005's "one fixture per state" was met for three states,
not four.

Adds makeCorruptedRepo: deletes the root commit's loose object so HEAD still
resolves but the first-parent walk fails. Observed
EVIDENCE_UNAVAILABLE / GIT_FAILED. Returns undefined when objects are packed
so the test skips rather than passing against a repository that was never
corrupted.

Keeps the unresolvable-revision case as its own assertion, since "absent
history" and "broken history" landing in different states is the distinction
REQ-005 exists to carry.

Refs: META-297
…META-297 Phase 3)

Completes META-297 Phase 3 at L0. Nothing here writes to the artifact; L1
projection remains closed.

Scoring implements META-289 v2.2.1 verbatim: size_weight = min(1, 10/fileCount),
position_decay = 2^(-dPos/250), a 500-transition window, and exclusion of events
with fileCount > 50. dPos is measured from the newest EXTRACTED event, never the
newest scored one, so excluding a large event cannot shift the decay of
everything older than it. Excluded events stay in the observation set and are
named by commit — an exclusion nobody can point at is not auditable. The
recorded file-role and path exclusion set is empty, so the size rule is the only
exclusion applied.

Counts follow the standard's ratified A-009 amendment: support is the distinct
scored commits in which both files changed, occurrences the distinct scored
commits in which at least one changed — the symmetric union, not a per-file
marginal. Both integers, both over one boundary, support <= occurrences by
construction.

Basis pinning records the resolved basis as a full-length lowercase object name
per A-009's pattern, alongside both window edges and the frozen weighting
identifiers. A basis that cannot be pinned throws rather than being emitted, and
where there is no window to pin the field is absent rather than a placeholder.

The selection rule applies the frozen threshold, ranks by support DESC then
occurrences ASC then files ASC by UTF-8 bytes, and caps at 50 after ranking.
UTF-8 byte order is not a bare string comparison: U+1F600 sorts before U+E000
under UTF-16 code units and after it under UTF-8, so compareUtf8 is explicit.
Every ranking key is an integer or a byte sequence, so no float can decide an
order. Capping is pure — the scored set and the extracted events behind it are
untouched — and the execution receipt records threshold, pairsBeforeCap,
pairsEmitted, cap, the complete ranking rule and whether the cap bound, so a
truncated list is visibly truncated rather than silently short.

serializeSelection throws on any non-integer number. weightedSupport is a double
from 2 ** x, whose precision ECMAScript leaves implementation-defined; a float
in a committed artifact is the churn class A-009 exists to prevent. It stays in
memory on the scored set for diagnostics and never reaches artifact-bound output.

Honest degradation is the first thing both new stages do. A --depth 1 clone can
hand over real events that would otherwise produce a structurally identical
answer at reduced magnitude, so score and select gate on completeness state and
never on whether an events array happens to be non-empty.

Requirement labels corrected throughout. REQ-001..006 are the only identifiers
written down for this package and remain cited; the weighting, scoring
exclusion, basis pinning and selection rule carry no issue number and are now
NAMED rather than numbered. Earlier drafts inferred REQ-007/009/010/011 from
code comments; REQ-011 in particular is the generated-classifier investigation,
tracked separately as META-316, not basis pinning. An invented identifier reads
as a citation and cites nothing.

History mining is an explicit refresh operation, not part of default generation:
a bound 500-transition window costs 7.3-8.2s with a short PATH and 27.2-29.9s
with a 36-entry one on an Apple M4 Pro, because the frozen parameters spend two
git subprocesses per commit and Node re-resolves the binary through PATH on each
of the 1000 spawns. Extraction is spawn-bound, not git-bound; scoring the
resulting events costs 1-24ms. The public command name is deliberately left
unchosen.

Verified on three fixtures — billfold (44 transitions), motdotla/dotenv pinned
at 2fc7eac8 (634 available, window binds) and sindresorhus/execa pinned at
8017b279 (846 available, window binds). Both external fixtures were selected on
metadata observable before mining. Three dotenv pair supports were recounted by
hand through an independent git path and agree exactly (80, 24, 8). Selection
output is byte-identical across separate OS processes on all three.

Gates: 194/194 tests across 21 files, monorepo build, monorepo typecheck,
architecture check and package-docs check all pass.
@sourcery-ai

sourcery-ai Bot commented Aug 11, 2026

Copy link
Copy Markdown

Reviewer's Guide

Implements the L0 mining core scoring and selection pipeline (mine → score → select) for META-297 Phase 3, adding v2.2.1-based scoring, basis pinning, a deterministic selection rule with a cap and execution receipts, plus comprehensive tests, documentation, and packaging for the new private @workspacejson/mining-core package, without yet emitting artifacts.

Sequence diagram for the L0 mining pipeline (mine → score → select with completeness gating)

sequenceDiagram
  actor Caller
  participant mine
  participant score
  participant select

  Caller->>mine: mine(repoRoot, { basisRevision, windowTransitions, qualifyingMinCooccurrence })
  alt not a git repo or shallow / no commits / git failed
    mine-->>Caller: ObservationSet (completeness NOT_MINED or EVIDENCE_UNAVAILABLE)
  else real history mined
    mine-->>Caller: ObservationSet (completeness MINED_* + basisWindow + events)
    Caller->>score: score(observations, { minSupport })
    alt completeness NOT_MINED or EVIDENCE_UNAVAILABLE in observations
      score-->>Caller: ScoredSet via unscored() (no pairs, no scoringBasis)
    else mined and basisWindow present
      score->>score: apply sizeWeight, positionDecay, SCORING_MAX_FILE_COUNT
      score->>score: build ScoredPair[], ScoringBasis, ScoringExclusions
      score-->>Caller: ScoredSet (pairs in path order)
      Caller->>select: select(scoredSet, { minSupport, cap })
      alt scored.completeness NOT_MINED or EVIDENCE_UNAVAILABLE
        select-->>Caller: SelectionResult (empty pairs, receipt.capBound=false)
      else mined
        select->>select: filter by DEFAULT_MIN_SUPPORT
        select->>select: rank using compareUtf8
        select->>select: apply SELECTION_CAP, build SelectionReceipt
        select-->>Caller: SelectionResult (ranked & capped pairs)
      end
    end
  end
Loading

File-Level Changes

Change Details Files
Introduce L0 scoring logic implementing META-289 v2.2.1 weighting, scoring exclusions, A-009 count semantics, and basis pinning over mined observation sets.
  • Add score() implementation that applies size_weight and position_decay over mined events, counts support/occurrences per A-009, and records scoring metadata and exclusions.
  • Implement basis pinning that validates full-length object IDs, records window edges and weighting parameters, and throws on unpinnable bases.
  • Ensure honest degradation by short-circuiting scoring when completeness indicates NOT_MINED/EVIDENCE_UNAVAILABLE and by treating excluded large events as named exclusions, not deletions.
packages/mining-core/src/score.ts
Implement the provisional selection rule (threshold, ranking, cap) for producer profiles, including UTF-8-based ordering and float-free serialized output.
  • Add select() to filter scored pairs by support threshold, rank by support/occurrences/UTF-8 path order, apply a configurable cap after ranking, and preserve the original scored set.
  • Implement compareUtf8 and RANKING_RULE/SELECTION_CAP constants and expose a SelectionResult+receipt structure capturing thresholds, counts, cap, and ranking rule.
  • Add deterministic, integer-only JSON serialization for observation, scored, and selection outputs, enforcing no floats in artifact-bound selection via serializeSelection().
packages/mining-core/src/select.ts
packages/mining-core/src/serialize.ts
Define and harden the L0 mining pipeline for observation sets, git extraction, path normalization, and completeness semantics, with explicit handling of shallow clones and corrupted histories.
  • Implement mine() to walk first-parent history with a configurable window, count co-occurrences, compute BasisWindow, and produce ObservationSet with recorded thresholds and path-normalization assumptions.
  • Add git.ts with frozen META-289 extraction commands (rev-list, diff-tree -z), empty-tree computation, shallow-repo detection, and robust parsers for name-status output with strong error typing.
  • Introduce a single normalizePath() with an explicit PATH_NORMALIZATION_ASSUMPTIONS record and enforce its exclusive use; define CompletenessState/Reason pairs and helper completeness() with REASONS_BY_STATE validation.
packages/mining-core/src/mine.ts
packages/mining-core/src/git.ts
packages/mining-core/src/paths.ts
packages/mining-core/src/completeness.ts
Add comprehensive test coverage and fixtures validating determinism, completeness states, shallow clone behavior, scoring and selection semantics, and billfold cross-checks.
  • Create synthetic git fixtures for each completeness state, including shallow clones and corrupted repos, with fixed author/commit metadata for deterministic object IDs.
  • Add mine.test.ts, git.test.ts, score.test.ts, select.test.ts, paths.test.ts, and billfold.test.ts to validate extraction invariants, windowing, scoring math, exclusion behavior, basis pinning, selection ranking and cap, UTF-8 ordering, honest degradation, and absence of floats in selection output.
  • Configure Vitest with extended timeouts suited to real git-based tests and ensure tests can be skipped when external billfold fixture is unavailable.
packages/mining-core/src/testing/fixtures.ts
packages/mining-core/src/mine.test.ts
packages/mining-core/src/git.test.ts
packages/mining-core/src/score.test.ts
packages/mining-core/src/select.test.ts
packages/mining-core/src/paths.test.ts
packages/mining-core/src/billfold.test.ts
packages/mining-core/vitest.config.ts
Introduce the private @workspacejson/mining-core package with public API surface, documentation, TypeScript config, and repository README integration.
  • Add packages/mining-core/package.json, tsconfig, LICENSE, and build/test/typecheck scripts using tsup and Vitest, ensuring Node 20+ and real @types/node typings.
  • Create mining-core/README.md documenting scope (META-297 phases 1–3), scoring, basis pinning, selection rule, completeness semantics, path identity assumptions, refresh model, and runtime characteristics.
  • Expose the L0 API via src/index.ts, re-exporting mining, scoring, selection, git, paths, completeness, and serialization utilities; update root README to list the new private package and clarify its role relative to published packages.
packages/mining-core/package.json
packages/mining-core/tsconfig.json
packages/mining-core/LICENSE
packages/mining-core/src/index.ts
packages/mining-core/README.md
README.md

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

… (META-297 L1)

L1 turns a selection into `generated.coChange` and decides what an ordinary
generation run does with it afterwards. The second half is the load-bearing
part.

## Carry-forward, and why it exists

The standard replaces `generated` wholesale each run. The ruling says history
mining is explicit and opt-in, and that ordinary generation must not recompute
history or fabricate freshness. Taken together the first erases what the second
forbids recomputing, so `coChange` and `basisRevision` are the one part of the
producer-owned section that is PRESERVED rather than rebuilt — not because they
are manual, but because they derive from an input ordinary generation does not
read. The working tree is scanned every run; the commit graph is not.

Three failure modes, each producing a plausible-looking artifact:

- drop: regenerating without carry-forward destroys mined evidence, and the
  result is indistinguishable from a repository that was never mined;
- advance the pin: carrying observations forward while moving `basisRevision`
  to current HEAD re-attributes old counts to a commit they were never counted
  at — the numbers stay plausible and become false;
- recompute: every run pays seconds to tens of seconds, and the artifact churns
  on every commit.

Preservation is by object identity, not field-by-field rebuild: a rebuilt entry
re-orders keys and changes bytes while staying structurally identical, so
identity is what makes byte-for-byte real. A pinned empty array is preserved —
under A-009 it is the positive finding "analyzed, found nothing", and dropping
it would convert a real result into "never analyzed". Nothing is invented when
no prior block exists.

`generatedAt` records the generation run. `basisRevision` is the authoritative
freshness and provenance pin for the observations, and the only field that says
which commit they were counted at.

## Canonical order is established at projection, not upstream

`score.ts` sorts pair endpoints with `<`, which is UTF-16 code unit order, while
the ruling names UTF-8 byte order. The two disagree: U+1F600 is a surrogate pair
beginning 0xD83D and sorts before U+E000 in UTF-16, while its UTF-8 encoding
sorts after. Upstream order is fine for keying a map; it is not fine for bytes a
second producer is compared against. `project` re-orders endpoints under the
same `compareUtf8` the ranking uses, and endpoint reversal now yields identical
bytes.

## Refusal rather than degradation

A non-mined completeness state projects nothing — never an empty array, which
under A-009 is a positive claim that the analysis ran. A refused mining pass
falls back to carry-forward so a shallow clone cannot destroy earlier evidence.
No derived value is stored, and the A-010 classification flag is omitted because
this producer has no deterministic classifier.

## One accommodation, named

`@workspacejson/spec@0.4.4` is the published package and its `CoChangeEntry`
predates A-009 — it requires `rate` and has no `support` — so an observation
entry is not assignable to it. Confined to a locally declared two-field
`HistoryBlock` and ONE cast at the assembly boundary, both documented in place
and deleted when the amended spec publishes. No `any`, no duplicated standard
type, and `WorkspaceJsonValidator` is untouched and still runs on every
artifact.

## Test boundaries

`candidate-tests/` is excluded from this workspace's vitest run and executes
against packed candidates in a disposable environment, because the observation
form is rejected by the published dependencies this workspace legitimately
pins. Excluding it keeps the two boundaries from contaminating each other; it is
not a way of skipping it. Repo-native coverage is validator-free by
construction.

Gates: mining-core 97/97, cli 72/72, agents-audit-compat 44/44 (213 total);
typecheck, build, architecture (91 files, 4 manifests, no boundary or clean-room
violations) and package-docs all green. Runtime dependency set unchanged —
standard-owned packages remain exact published versions; mining-core is a
bundled devDependency.
@qmarcelle

Copy link
Copy Markdown
Contributor Author

@greptileai

@greptile-apps

greptile-apps Bot commented Aug 11, 2026

Copy link
Copy Markdown

Greptile Summary

The PR introduces the private mining-core pipeline and integrates explicit, basis-pinned history mining with ordinary-generation carry-forward behavior. The latest changes also expose whether a requested history refresh completed, was refused, or preserved prior observations.

  • Adds deterministic mining, scoring, selection, projection, and serialization primitives.
  • Preserves prior history during ordinary generation and after refused refreshes.
  • Returns a typed refresh outcome to programmatic callers.
  • Adds package, integration, fixture, and completeness-state coverage.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename Overview
packages/cli/src/producer/generate.ts Integrates explicit mining and history carry-forward while returning a consistent refresh outcome for completed and refused requests.
packages/cli/src/producer/history-mine.ts Adapts the mining-core pipeline into an artifact-ready history block and preserves concrete refusal diagnostics.
packages/cli/src/producer/history-carry-forward.ts Preserves conforming basis and co-change evidence without recomputation during ordinary generation or refused refreshes.
packages/mining-core/src/mine.ts Implements bounded first-parent extraction with explicit completeness and exclusion metadata.
packages/mining-core/src/score.ts Implements support, occurrence, weighting, decay, and exclusion semantics over extracted events.
packages/mining-core/src/select.ts Applies deterministic thresholding, UTF-8 ranking, and post-ranking capping with an execution receipt.
packages/mining-core/src/serialize.ts Serializes artifact-bound selection data while rejecting non-integer values.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Explicit mineHistory request] --> B[Mine commit history]
  B -->|Complete| C[Score observations]
  C --> D[Select ranked pairs]
  D --> E[Return newly mined history]
  B -->|Refused| F[Carry forward prior history if available]
  E --> G[Generate workspace result]
  F --> G
  G --> H[Return historyRefresh outcome]
  I[Ordinary generation] --> F
Loading

Reviews (2): Last reviewed commit: "fix(cli): report whether an explicit his..." | Re-trigger Greptile

Comment thread packages/cli/src/producer/generate.ts Outdated
Greptile P1 on #20. When a caller passed `mineHistory: true` and mining could
not complete, generation fell back to the previously recorded block and returned
a successful result. The caller received the PREVIOUS revision's counts with no
way to distinguish that from a refresh that ran — the artifact looks identical
either way, and `basisRevision` only helps a reader who already suspects
something is wrong.

Falling back is correct and stays: destroying evidence over a shallow clone or a
transient Git failure would be worse than keeping it. Falling back QUIETLY was
the defect.

`GenerateResult.historyRefresh` now reports the outcome:

  requested  always true — the field is absent unless a refresh was asked for
  mined      whether the commit graph was read and a new block produced
  preserved  whether a prior block was carried instead
  refusal    why nothing was mined; present iff `mined` is false

Absence of the field means no refresh was requested. Reporting `mined: false` on
an ordinary run would claim a refresh was attempted and failed, which is a
different and untrue statement.

The refusal reason was already being computed and thrown away: `mineHistoryBlock`
accepts a diagnostics object for exactly this and generate.ts never passed one.
It is passed now, so a caller can distinguish a shallow clone from a Git
invocation failure rather than receiving a bare boolean.

Test boundary, kept honest: the refusal cases run repo-native, because a refusal
emits no `coChange` and therefore touches nothing the published 0.4.4 validator
rejects. The fallback-with-preservation and completed-refresh cases need an
observation-form artifact on disk, so they run in `candidate-tests/` against the
amended schema rather than being weakened to fit the workspace.

Watched-red, packed and clean-installed per mutation:
  drop the outcome field (restore the silent fallback)  20 passed / 2 failed
  report `mined: true` unconditionally                  21 passed / 1 failed
  restored                                              22 passed / 0 failed

Gates: mining-core 97/97, cli 76/76, agents-audit-compat 44/44 (217 total);
typecheck, build, architecture (93 files, no boundary or clean-room violations)
and package-docs green. Candidate-contract suite 22/22.
@qmarcelle qmarcelle changed the title feat(mining-core): L0 scoring, basis pinning and the selection rule (META-297 Phase 3) feat(cli): L0 scoring, basis pinning, selection rule, and L1 projection with carry-forward (META-297) Aug 11, 2026
@qmarcelle

Copy link
Copy Markdown
Contributor Author

@sourcery-ai review

@sourcery-ai

sourcery-ai Bot commented Aug 11, 2026

Copy link
Copy Markdown

Sorry @qmarcelle, your pull request is larger than the review limit of 150000 diff characters

@qmarcelle

Copy link
Copy Markdown
Contributor Author

Greptile P1 fixed — head 12695a3

"Refresh refusal becomes stale success" — valid, and worse than the summary suggested: mineHistoryBlock already accepted a diagnostics object for exactly this, and generate.ts never passed one. The refusal reason was being computed and discarded on every refused refresh.

The thread auto-resolved as outdated when the fix touched those lines, so the disposition is recorded here rather than inline.

The fallback stays. Destroying evidence over a shallow clone or a transient Git failure would be worse than keeping it. What is fixed is that it is no longer silent.

GenerateResult.historyRefresh:

Field Meaning
requested always true — the field is absent unless a refresh was asked for
mined whether the commit graph was read and a new block produced
preserved whether a prior block was carried instead
refusal why nothing was mined; present iff mined is false

Absence means no refresh was requested. Reporting mined: false on an ordinary run would claim an attempt that never happened — a different and untrue statement, asserted by test.

Watched-red, packed and clean-installed per mutation:

Mutation Result
Drop the outcome field (restore the silent fallback) 20 passed / 2 failed
Hardcode mined: true so a refusal reads as success 21 passed / 1 failed
Restored 22/22

Cases are placed by what they need, not by convenience: refusal cases run repo-native because a refusal emits no coChange and touches nothing the published 0.4.4 validator rejects; fallback-with-preservation needs an observation-form artifact on disk and runs as a candidate-contract test rather than being weakened to fit.

The PR description was also stale — it still described this branch as L0-only, which stopped being true when L1 landed. Rewritten to match the code, including the carry-forward decision, the canonical-ordering fix, the single named type accommodation, and the full watched-red table.

Gates at this head: mining-core 97/97, cli 76/76, agents-audit-compat 44/44 (217); typecheck, build, architecture (93 files, no violations), package-docs green; candidate-contract 22/22.

@qmarcelle
qmarcelle marked this pull request as ready for review August 11, 2026 17:15
Copilot AI lite review requested due to automatic review settings August 11, 2026 17:15

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry @qmarcelle, your pull request is larger than the review limit of 150000 diff characters

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@qmarcelle
qmarcelle merged commit 031c350 into main Aug 11, 2026
5 checks passed
@qmarcelle
qmarcelle deleted the feature/meta-297-commit-graph-mining-core branch August 11, 2026 17:15
qmarcelle added a commit that referenced this pull request Aug 12, 2026
…er (META-321) (#22)

* tooling(review): repo-owned Greptile policy written from CLI producer failure classes (META-321)

The standard's rollout pattern is proven, but its rules are about a schema
surface this repository does not own. Copying them would have installed a gate
that fires on things the CLI cannot do and stays silent on the things it can.

So the rules here are derived from defects this producer has actually shipped or
nearly shipped. The load-bearing one is the P1 Greptile found on PR #20: an
explicit `mineHistory: true` request could fall back to stale preserved history
and return a successful-looking result, with the refusal reason computed and
then discarded. From the artifact alone that is indistinguishable from a refresh
that completed.

Every CLI-specific rule is a specialization of that one failure class — a
producer emitting a plausible artifact for evidence it did not gather:
refusal reported as success, an empty coChange block for a repository that could
not be analyzed, a basis pin advanced without recounting, a preserved block
rebuilt rather than passed through, UTF-16 ordering substituted for the UTF-8
byte order two producers are compared on.

Five ecosystem rules are carried because they are genuinely
repository-independent. Seven standard rules are deliberately not carried, and
`.greptile/rules.md` records which and why, so the omissions read as decisions
rather than oversights — four of them survive in producer form, stated from the
emitting side rather than the schema-authoring side.

`REVIEW.md` states what the checks mean. Check completion is not semantic
approval: the status check answers "did review complete on this head", and
conversation resolution answers "was every actionable finding dealt with".
Greptile Review is deliberately NOT made a required status here. A rule that has
not been shown to fail for the defect it names is not eligible to be a hard
gate, and installing one before calibration would be the same defect these rules
exist to catch in the producer. That decision follows the canary, on evidence.

Sourcery stays defense-in-depth. A check run existing on recent heads is not
calibration.

No release impact: every changed path is outside `packages/`, each publishable
manifest's `files` field lists only directory-local entries, and no workflow
copies root files into a package before pack. No published bytes, exports,
schema, or package metadata change, so no changeset accompanies this.

* fix(review): let the clean-room policy name the scopes it forbids (META-321)

The policy files tripped the guard they describe. `check:architecture` reported
four clean-room violations on `.greptile/config.json` and `.greptile/rules.md`,
both for `@marcelle-labs/` and `workspace.vreko.json` — the vocabulary the rule
has to name to be enforceable at all.

Greptile flagged this itself on PR #22 as a P1, citing the very rule the files
had just introduced, which is the first evidence that branch-local
configuration is read.

There were two ways out and only one of them is honest. Paraphrasing the scopes
out of the policy would turn the build green while leaving the reviewer without
the strings it matches on — adjusting the measurement to fit the behavior, which
is the move `.greptile/rules.md` explicitly prohibits. So the two files join
`SELF_REFERENTIAL`, which already holds OWNERSHIP.md, CONTRIBUTING.md, README.md
and the guard's own source for exactly this reason, and whose comment already
anticipated "the ownership documentation that explains the rules".

Membership is by exact path, not directory prefix. That distinction is invisible
from a passing run — "two files are exempt" and "the whole directory is exempt"
produce identical green — so it is asserted rather than commented: a new red
case writes `@marcelle-labs/` into `.greptile/notes.md` and requires the guard
to still reject it. Injecting the directory-wide mistake takes the suite to
20 passed / 1 failed; restoring it returns 21 / 0.

REVIEW.md is deliberately NOT exempted. It does not name the forbidden
vocabulary, and exempting a file against a hypothetical future edit would give
up real coverage for nothing.

Waived coverage, stated rather than glossed: `SELF_REFERENTIAL` gates four
checks. For `rules.md` the other three cannot apply — `copied-schema` is
JSON-only, `shadowed-standard-types` is `.d.ts`-only, `neutral-producer-purity`
is scoped to `packages/cli/`. For `config.json` the one additional waiver is
`copied-schema`, which needs `$schema` + `$id` + `type: "object"` + `properties`;
the Greptile config has none of them.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants