Skip to content

CANARY (do not merge): META-322 Greptile calibration positive control - #11

Closed
qmarcelle wants to merge 3 commits into
mainfrom
meta-322/greptile-calibration-canary
Closed

CANARY (do not merge): META-322 Greptile calibration positive control#11
qmarcelle wants to merge 3 commits into
mainfrom
meta-322/greptile-calibration-canary

Conversation

@qmarcelle

@qmarcelle qmarcelle commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Disposable canary for META-322. This PR will be closed without merging.

It carries the repo-owned policy from #10 plus one deliberate, reversible semantic defect, to measure whether Greptile Review actually catches the Integrations failure class it was configured to catch — before anyone considers making it a required merge gate.

The positive control

extension/src/pathMatch.ts reintroduces a symmetric suffix fallback into pathsMatch, dressed up as a plausible bug fix.

This is the one defect class this repository has already suffered and fixed:

  • META-291 / PR fix(path-identity): resolve absolute queries against a proven root, drop suffix fallback #8 removed exactly this fallback from the server matcher, where a stored src/a.ts matched /elsewhere/unrelated-repo/src/a.ts — a fragility assertion landing on a file in a different repository.
  • src/path-match.ts records that a second matcher is how a deny silently became a warn: the enforcement layer drifted to a symmetric fuzzy suffix match while the read layer was tightened.

It violates two branch-local custom rules at once:

Rule id Why
proven-root-path-identity ADR-006 §4/§8 — no suffix matching, no endsWith comparison, no symmetric fuzzy matching; no-match is a refusal, not a nearest guess
ported-copies-must-not-diverge extension/src/pathMatch.ts is a hand-synced port of src/path-match.ts; changing one copy's semantics without the other is the divergence defect itself

A finding that names either rule is evidence that branch-local .greptile/ configuration was read, rather than that a generic reviewer noticed loose code.

It is invisible to every deterministic gate — on purpose

CI is expected to stay green while the defect is present:

  • tsconfig.json sets "include": ["src/**/*"]typecheck and build never see extension/src/.
  • npm run test is vitest over tests/**; .check.ts files do not match its patterns.
  • CI never runs npm --prefix extension run test. The only job touching the extension is standard-candidate-consumption via npm run build:extension, which compiles and packages — a suffix fallback is type-valid.
  • No test covers extension/src/pathMatch.ts anywhere.

If CI stays green and the reviewer catches it, semantic review is adding detection CI cannot provide. If CI stays green and the reviewer misses it, that is the reportable outcome and it argues against making the check required.

Sequence

  1. This head (e22d6f5) carries the defect — record caught / not-caught and the rule id cited.
  2. A follow-up commit reverts it — record whether the finding repeats on the clean head and whether the push retriggers review.
  3. Findings reconciled individually, thread receipts preserved.
  4. Closed without merging.

Full protocol and results: docs/review/calibration-2026-08.md.

Summary by Sourcery

Introduce semantic review configuration and documentation for Greptile and Sourcery, along with a temporary path-matching change used as a calibration canary.

New Features:

  • Add repo-owned Greptile configuration, rules, and orientation files to govern semantic review behavior.
  • Document review and merge policy, including reviewer roles, required checks, and calibration protocol for Greptile on this repository.
  • Record a calibration evidence log for Greptile review behavior over time.

Enhancements:

  • Configure Sourcery to ignore build output and vendored directories while explicitly constraining its role to non-authoritative defense in depth.

Documentation:

  • Add detailed reviewer and merge-policy documentation describing current branch protection, CI behavior, and how automated reviewers interact with human review.
  • Add a calibration document capturing planned and observed behavior of Greptile Review as a potential required merge gate.

Tests:

  • Introduce a disposable semantic defect in extension path matching as a positive-control canary for reviewer calibration, without impacting existing deterministic CI gates.

Chores:

  • Establish repository-level governance for automated reviewers through new configuration and policy files under .greptile/ and docs/review/.

…ntation

Integrations has had no repository-owned review policy, so reviewer behavior
here has never been governed by this repository's own failure classes.

Rules are derived from verified Integrations surfaces and defects, not copied
from workspacejson/standard:

- proven-root-path-identity, ported-copies-must-not-diverge: META-291 / PR #8
  removed a suffix fallback that matched a stored key against a file in an
  unrelated repository; src/path-match.ts records that a second, drifted matcher
  is how a deny silently became a warn. extension/src/pathMatch.ts is still a
  hand-synced port.
- unavailable-is-not-approval: hooks/pre-edit-check.mjs routes failures through
  emitUnavailable and never emits an approval; src/reviewer.ts keeps UNAVAILABLE
  distinct from PASS. Exercised by the smoke suite.
- reader-tolerance-is-not-repair: normalizeWorkspace degrades to empty and never
  fabricates; no read path holds a write to the artifact.
- packed-artifact-must-be-measured: dist/ is gitignored but shipped, and the hook
  resolves ../dist/* at runtime, so source-only tests do not prove what a
  consumer receives.
- probes-must-not-be-destructive: META-285 records --help running a destructive
  install into the source repo; the surrounding installer surface is genuinely
  destructive, so probe versus action has to be explicit.
- host-contract-claims-need-a-watched-version: the Codex output contract is
  recorded as watched on 0.144.1 (2026-07-13) with a single adapter point.
- clean-room-public-boundary, consume-standard-do-not-vendor, and the four
  ecosystem evidence rules.

Standard's four-read-paths, negative-fixture, and derived-probability rules are
deliberately not carried over; CLI producer/history rules have no equivalent
verified failure mode here. .greptile/rules.md records each omission.

.sourcery.yaml is path scoping only. Sourcery keeps no semantic authority: a
success check from it is not review approval.

config/repository-structure.json gains .greptile and .sourcery.yaml via its
documented allowlist path; the checker script is untouched. The check was
watched failing with 2 violations before the amendment.

No product behavior change. No package bytes, exports, or metadata change.

Refs META-322
…op dropping

Decorations were not rendering in workspaces opened at a parent folder: the
editor hands us a path with extra leading segments and exact equality misses
the stored key. Fall back to a suffix comparison when exact equality fails.
Copilot AI lite review requested due to automatic review settings August 12, 2026 13:33
@sourcery-ai

sourcery-ai Bot commented Aug 12, 2026

Copy link
Copy Markdown

Reviewer's Guide

This PR introduces a deliberate semantic defect in the extension path matcher as a calibration canary for Greptile Review, and adds repository-owned reviewer policy and configuration files (.greptile/, Sourcery scope, and review docs) needed to measure whether Greptile can detect the defect and respect the repo’s rules.

File-Level Changes

Change Details Files
Reintroduce a symmetric suffix-based path matching fallback in the VS Code extension matcher to create a controlled, reversible semantic defect.
  • Refactor pathsMatch to normalize both query and storedKey into local variables before comparison.
  • Add a guard returning true on exact equality of normalized paths.
  • Add symmetric suffix matching using endsWith on both query and storedKey so that longer-leading-segment paths can still match.
extension/src/pathMatch.ts
Add detailed Greptile review rules and reasoning, defining repository-specific failure classes and constraints for path identity, normalization, duplication, host behavior, and packaging.
  • Define ecosystem-wide rules around evidence, absence, measurements, vacuous checks, clean-room boundaries, and daemon assumptions.
  • Define integrations-specific rules including proven-root path identity, non-divergent ported copies, reader tolerance vs repair, host adaptation posture, packaged artifact measurement, non-destructive probes, and watched host contracts.
  • Document which rules from other repositories are deliberately not carried over and why.
  • Specify scopes for each rule so Greptile can target the correct files.
.greptile/rules.md
Document the repository’s review and merge policy, including measured GitHub branch protection state and the intended role and authority of Greptile Review vs Sourcery and CI.
  • Record current required status checks, branch protection flags, and conversation resolution requirements as measured from GitHub.
  • Clarify that Sourcery is defense in depth without semantic authority and that Greptile’s semantic rules are repo-owned via .greptile/.
  • Define the calibration requirements and protocol before Greptile can be made a required merge gate, including behavior on positive-control defects and per-head SHA association.
  • Clarify what different check states (absent, completed, on older heads) do and do not imply for semantic approval.
docs/review/merge-policy.md
Add a calibration evidence record for August 2026 to track how Greptile Review behaves on this repository and whether it is eligible to be a required gate.
  • Capture pre-policy baseline state for reviewer apps, check runs, required contexts, and local gates.
  • Describe the rationale for the chosen canary defect (symmetric suffix path fallback) and its invisibility to deterministic CI gates.
  • Lay out the stepwise protocol for running the canary, measuring Greptile’s behavior, and deciding branch protection changes.
  • Reserve a results section to be filled from actual observations, explicitly distinguishing not-measured from pass.
docs/review/calibration-2026-08.md
Configure Sourcery as a scoped, non-authoritative reviewer by defining ignore paths for build output and vendored trees without granting semantic rule power.
  • Add a .sourcery.yaml file specifying versioned configuration.
  • Ignore node_modules, dist, vsix, and .parity-ci to keep Sourcery focused on source surfaces.
  • Document that Sourcery is defense in depth and that semantic review rules live in .greptile/ rather than in Sourcery config.
.sourcery.yaml
Introduce Greptile configuration stubs for rules and orientation files to wire the new policy docs into the reviewer pipeline.
  • Add .greptile/config.json to hold structured rule definitions, scopes, severities, and trigger behavior (content not shown in diff).
  • Add .greptile/files.json to point Greptile at orientation and policy documents (content not shown in diff).
.greptile/config.json
.greptile/files.json

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've left some high level feedback:

  • The new symmetric suffix matching in extension/src/pathMatch.ts reintroduces the previously-removed loose path identity behavior; pathsMatch should stay strict and align with src/path-match.ts to avoid cross-repo false matches.
  • To prevent future drift between the extension and server, consider sharing a single path-matching implementation (or a generated copy) rather than hand-porting pathsMatch in two places.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- The new symmetric suffix matching in `extension/src/pathMatch.ts` reintroduces the previously-removed loose path identity behavior; `pathsMatch` should stay strict and align with `src/path-match.ts` to avoid cross-repo false matches.
- To prevent future drift between the extension and server, consider sharing a single path-matching implementation (or a generated copy) rather than hand-porting `pathsMatch` in two places.

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

@greptile-apps

greptile-apps Bot commented Aug 12, 2026

Copy link
Copy Markdown

Greptile Summary

This PR establishes repository-owned semantic-review policy and records the disposable META-322 calibration protocol.

  • Adds scoped Greptile rules and orientation metadata for repository-specific failure classes.
  • Documents measured merge eligibility, reviewer authority, and the positive-control calibration procedure.
  • Adds Sourcery scope configuration and updates the repository-structure allowlist.
  • The previously reported suffix-matching canary has been reverted on the current head.

Confidence Score: 5/5

The current head appears safe to merge because the previously reported path-identity defect has been removed and no blocking failure remains.

No blocking failure remains.

Important Files Changed

Filename Overview
.greptile/config.json Adds trigger behavior, review presentation settings, repository context, and structured semantic rules.
.greptile/rules.md Documents the repository-specific failure classes, scopes, and rationale represented by the structured rules.
.greptile/files.json Adds orientation metadata directing semantic review toward key contracts and runtime boundaries.
docs/review/calibration-2026-08.md Records the positive-control calibration protocol and preserves unmeasured results as unmeasured.
docs/review/merge-policy.md Documents measured branch protection, reviewer roles, and the evidence required before making semantic review mandatory.
.sourcery.yaml Restricts Sourcery exclusions to generated and vendored surfaces without granting it semantic authority.
config/repository-structure.json Allows the newly introduced review configuration files and directory in the repository structure check.

Reviews (2): Last reviewed commit: "Revert "fix(extension): match nested-fol..." | Re-trigger Greptile

Comment thread extension/src/pathMatch.ts
qmarcelle added a commit that referenced this pull request Aug 12, 2026
Calibration ran on PR #11 (canary, closed unmerged) against this policy.

All seven criteria observed. Greptile caught the positive control (P1 on
extension/src/pathMatch.ts at e22d6f5) citing the branch-local rule by name,
stayed silent on the reverted clean head (447a318, 0 comments), and retriggered
on push. Merge eligibility behaved as claimed: PR #10 sat at BLOCKED on one
unresolved thread.

Two results qualify how a required check should be read, and are recorded rather
than smoothed over:

- the Greptile check concluded success on the head carrying the P1, so the check
  encodes review completion, not absence of findings;
- GitHub auto-resolved the canary thread when the revert removed its lines, so
  required_conversation_resolution can be satisfied without anyone answering a
  finding.

Decision: Greptile Review is eligible as a required current-head gate. The
required-contexts mutation is deliberately not applied here — PR #6 is BEHIND and
predates the policy, so the change costs a PR this issue does not own.

Reconciles review findings on #10 individually:

- Greptile P2: corrected a false coverage claim. extension/test/intelligence.check.ts
  does exercise pathsMatch; the real limitation is that no gate runs that suite.
  Chasing it down found the suite is already red on main against pre-META-291
  assertions — filed as META-329, not repaired here.
- Sourcery 1: audited rules.md/config.json scope parity, all nine scoped rules
  match; config.json declared authoritative for scope so drift has a resolution
  direction.
- Sourcery 2: added a refresh procedure and measurement stamp to the branch
  protection section, which transcribes settings that can change externally.

Refs META-322, META-329
@qmarcelle

Copy link
Copy Markdown
Contributor Author

Canary complete — closing without merging

Calibration receipt for META-322. Nothing here merges; the branch is deleted with the PR.

# Criterion Result
1 Branch-local .greptile/ read pass — finding cited Rule Used: …ADR-006 §4/§8…, source: .greptile
2 Current-head association pass — distinct check runs on aa456b4, e22d6f5, 447a318; comment commit_id=e22d6f5
3 Positive control caught pass — 1 of 1 (P1, 8 files reviewed, 1 comments added)
4 Clean head does not repeat pass — 0 of 0 (7 files reviewed, 0 comments added)
5 Push retriggers pass — new run on 447a318
6 Individual reconciliation pass, with caveat — see below
7 Merge eligibility as claimed pass#10 BLOCKED on one unresolved thread; this PR CLEAN once resolved

Canary SHAs: defect e22d6f5620ab6fce2bb06d5cefc20dbcfdc966b1, revert 447a318ff5dcd40e12c561b40d4602377e9171fe (diff vs policy head aa456b4: empty).

Two caveats recorded in docs/review/calibration-2026-08.md rather than smoothed over:

  1. The Greptile check concluded success on the head carrying the P1. Requiring the context enforces review completed on this head, not no findings.
  2. This PR's thread auto-resolved when the revert removed its lines (isOutdated=true, isResolved=true), with nobody answering it. required_conversation_resolution is real — it held chore(review): repo-owned Greptile policy and merge-policy documentation for integrations #10 at BLOCKED — but "resolved" alone is not proof of engagement.

Sourcery independently named the same defect in its review body while its check reported success. It stays defense-in-depth; the prior conclusion is not falsified, and its check state is confirmed non-authoritative.

Decision: Greptile Review is eligible as a required current-head gate. The required-contexts mutation is deliberately not applied by META-322 — PR #6 is BEHIND and predates the policy, so the change would cost a PR this issue does not own.

Incidental find, tracked as META-329, not repaired here: extension/test/intelligence.check.ts still asserts pre-META-291 path semantics, so the extension suite is red on main (81 pass / 1 fail) and no gate runs it.

@qmarcelle

Copy link
Copy Markdown
Contributor Author

Closing unmerged as designed — disposable calibration canary for META-322. Receipt in the comment above.

@qmarcelle qmarcelle closed this Aug 12, 2026
@qmarcelle
qmarcelle deleted the meta-322/greptile-calibration-canary branch August 12, 2026 13:43
qmarcelle added a commit that referenced this pull request Aug 12, 2026
…ion (#10)

Adds the first repository-owned review policy for workspacejson/integrations, derived from verified Integrations failure classes rather than copied from workspacejson/standard.

- .greptile/{config.json,rules.md,files.json}: 14 rules, each traced to a surface or defect in this repository
- .sourcery.yaml: path scoping only; Sourcery holds no semantic authority
- docs/review/merge-policy.md, docs/review/calibration-2026-08.md: measured merge eligibility and the calibration evidence record
- config/repository-structure.json: allowlist entries for the two new root paths (watched red before, green after)

Calibrated on canary PR #11 (closed unmerged): positive control caught 1 of 1 by
the branch-local rule, clean head 0 of 0, three retriggers observed, merge
eligibility behaved as claimed.

Two recorded caveats: the Greptile check concludes success even on a head
carrying a P1 finding, so it encodes review completion rather than semantic
approval; and GitHub can auto-resolve threads whose lines disappear, so
required_conversation_resolution needs written per-finding receipts to be
meaningful.

No product behavior change. No published package bytes, exports, or metadata
change. No changeset (repository is not Changesets-managed).

Refs META-322
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants