Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/better-harness-doc-links.mmd
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ flowchart LR
n_skills_better_harness_references_findings_review_md["findings-review.md"]
n_skills_better_harness_references_manual_direct_fix_md["manual-direct-fix.md"]
n_skills_better_harness_references_project_harness_md["project-harness.md"]
n_skills_better_harness_references_report_source_review_md["report-source-review.md"]
n_skills_better_harness_references_session_evidence_md["session-evidence.md"]
n_skills_better_harness_references_session_repeated_workflows_md["session-repeated-workflows.md"]
n_skills_better_harness_references_support_bootstrap_md["support-bootstrap.md"]
Expand All @@ -79,6 +80,7 @@ flowchart LR
n_skills_better_harness_SKILL_md --> n_skills_better_harness_references_findings_review_md
n_skills_better_harness_SKILL_md --> n_skills_better_harness_references_manual_direct_fix_md
n_skills_better_harness_SKILL_md --> n_skills_better_harness_references_project_harness_md
n_skills_better_harness_SKILL_md --> n_skills_better_harness_references_report_source_review_md
n_skills_better_harness_SKILL_md --> n_skills_better_harness_references_session_evidence_md
n_skills_better_harness_SKILL_md --> n_skills_better_harness_references_session_repeated_workflows_md
n_skills_better_harness_SKILL_md --> n_skills_better_harness_references_support_bootstrap_md
Expand Down
135 changes: 135 additions & 0 deletions docs/specs/2026-08-02-native-learning-report-integration.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,135 @@
# Native Learning Review in Report Evidence

## Traceability

- Spec ID: `native-learning-report-integration`
- Roadmap: `LC-05`
- AI involvement: Codex (GPT 5.6 Sol)
- Status: Implemented

## Intent

Make the already validated native recurring-correction method usable through
the standard Harness review packet, lead-decision, and source-apply path. A
normal source must surface a bounded nested packet; the existing lead decision
may carry only its native decisions; source apply projects only validated
matches into canonical Learning Capture diagnostics and reader evidence.

## Acceptance Scenarios

- **AC-1: Reviewable source packet.** When ordinary report Task Episodes form
native candidate groups, generated diagnostics retain a privacy-safe packet
and declare that a decision is required. The packet is deterministic and
contains no raw session, prompt, command, transcript, credential, or path.
- **AC-2: Validated report projection.** Given exact native decisions in the
normal lead decision, source apply validates them and projects accepted signals,
candidates, match/abstain counts, and binding digests into canonical Learning
Capture diagnostics.
- **AC-3: Fail closed across the review lifecycle.** A pending source validates
its generated packet against current Episodes. Review compile/apply rejects
stale packet/source digests, invented aliases, and incomplete decisions.
A persisted reviewed source independently revalidates its stored packet,
decision allowlists, and result aggregates, so tampering fails validation
even after legitimate outer review fields have changed the current Episodes.
- **AC-4: No-group compatibility.** A source with no native groups has no new
native review diagnostic and preserves the existing Learning Loop output.
- **AC-5: Reader visibility.** The neutral evidence brief reports whether a
native review is pending or validated, and reports bounded group/match/
abstain counts without exposing packet aliases or private source data.
- **AC-6: Standard review binding.** The outer Harness packet carries the
optional native subpacket but does not merge its opaque aliases into
`allowedEvidenceRefs`, including when another packet is built from a persisted
reviewed source.
- **AC-7: Public local review route.** A registered, parser-safe
`harness source-review` command exposes explicit `create`, `decision`, and
`apply` phases. `create` writes the bounded packet and a packet-bound decision
template, `decision` compiles only the caller-edited template, and `apply`
requires explicit non-interactive confirmation before atomically replacing
the selected source. The route does not call a model or author a decision.
- **AC-8: Strict persisted contract.** Stored Episode facts, groups, coverage,
decisions, matches, abstentions, and result metadata reject unknown fields,
unsafe values, cross-reference drift, and decision/result disagreement even
when packet digests are recomputed. The complete stored Learning Loop result
must equal deterministic reconstruction from the packet, review, final
Episodes, signals, interventions, and asset coverage. A packet with groups
requires complete native decisions before outer review apply; omission cannot
leave a stale pending packet after Episode review mutation.

## Non-goals

- Do not call an AI model or choose a provider/model.
- Do not create, alter, or apply a Skill, Rule, Memory, Hook, ledger entry, or
other Harness asset.
- Do not infer native `repeated-rediscovery` or another pattern.
- Do not claim intervention effectiveness, transfer, causality, or savings.
- Do not make unreviewed native candidates into canonical candidates.

## Plan and Tasks

1. Extend generated Learning Capture diagnostics with a tightly validated,
pending privacy-safe native review packet.
2. Add that packet as an independent nested contract in the Harness review
packet, without mixing its opaque aliases with source evidence references.
3. Require and compile native decisions when a nested packet is present, then
apply them only through `applyReportSourceReview` after both packet contracts
validate.
4. Surface aggregate native-review state in the neutral evidence brief.
5. Add the registered `harness source-review create|decision|apply` owner under
`report-source/`, with a packet-bound decision template, local JSON files,
parser-safe output, refusal to overwrite intermediate files, and explicit
confirmation for source apply.
6. Cover the standard outer packet -> lead decision -> apply path for positive,
abstain, stale/invented, dedupe, no-group, privacy, and stored-result
tampering behavior.

## Test and Review Evidence

- AC-1 to AC-8: native candidate, report-source review, public CLI, task-loop
source, analyzer, Learning Loop, Episode, and report-source contract tests.
- Public-path evidence invokes the root `scripts/better-harness.mjs` facade for
create, decision, refused unconfirmed apply, and confirmed atomic apply.
- Documentation and frozen-contract evidence covers Skill routing, generated
doc links, human help fixtures, command inventory, schema, and packaging.
- Regression and package evidence: `npm test` and `npm run pack:verify`.

## Implementation Evidence

- AC-1 and AC-4: `task-loop-source.mjs` emits the pending packet only when
bounded native groups exist; native and cross-module fixtures retain the
no-group behavior.
- AC-2, AC-3, AC-6, and AC-8: the outer packet -> lead decision -> apply test
rejects omitted decisions, stale digests, recomputed-digest nested packet
injection, invented or duplicate aliases, unknown nested result fields, and
decision/result drift. Synchronized edits to stored canonical candidates,
costs, or scores fail deterministic full-result reconstruction. Rebuilt outer
packets exclude every `native-learning-evidence` reference.
- AC-5: `evidence-brief.mjs` emits only pending/reviewed aggregate counts.
- AC-7: the registered root CLI E2E passed create, caller-authored decision,
unconfirmed-apply refusal, and confirmed atomic apply by editing the generated
template. Machine output and read errors omit absolute paths and raw validator
details, intermediate files use create-only writes, bare decision documents
are rejected without writing a review, and malformed help combinations fail
instead of hiding invalid arguments.
- Focused native/report-source run: 63/63. Cross-module source/analyzer/
Learning Loop/Episode run: 81/81.
- CLI, Skill, frozen CLI, and doc-link contract run: 66 passed with one Windows
package-bin symlink test skipped.
- Final full regression: 1089 passed, 4 failed, and 6 skipped. All four failures
are Windows `EPERM` symlink-creation restrictions in analysis-scope, render,
and workspace-topology tests outside the changed modules.
- Package verification passed with an isolated npm cache: 385 npm entries and
408 runtime ZIP entries.
- `git diff --check` passed.

## Privacy and Risk

- A review can become stale between runs. Rebuilding the packet and requiring
both digests fail closes stale input.
- Packets are source-visible data. Only the existing bounded, privacy-safe
packet projection is retained; the brief emits aggregate counts only.
- Existing unreviewed provider candidates remain unchanged. Native candidates
are added only after an exact validated match.
- The outer review packet schema advances from v2 to v3 because it gains an
optional nested field. Sources without native groups omit the field; v2
packet fixtures remain valid only against their historical contract and are
not silently reinterpreted as v3 packets.
9 changes: 8 additions & 1 deletion scripts/better-harness-cli/cli.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,7 @@ const GROUP_EXAMPLES = {
{ audience: "workflow", text: "better-harness harness checkup --phase scan --provider qoder --workspace . --json" },
{ audience: "advanced", text: "better-harness harness workspace-topology --workspace . --json" },
{ audience: "maintainer", text: "better-harness harness source --workspace . --source <scratch>/report.source.json --language en" },
{ audience: "maintainer", text: "better-harness harness source-review create --source <scratch>/report.source.json --packet <scratch>/review.packet.json --decision <scratch>/lead.decision.json --json" },
{ audience: "advanced", text: "better-harness harness render --findings <input>/findings.json --mode qoder-canvas --out .qoder/better-harness --target . --validate --json" },
{ audience: "advanced", text: "better-harness harness preview-canvas <run>/report.canvas.tsx --open" },
{ audience: "advanced", text: "better-harness harness report-quality --report <run>/report.md" },
Expand Down Expand Up @@ -261,6 +262,10 @@ function hasHelpFlag(argv) {
return argv.some((value) => value === "--help" || value === "-h");
}

function requiresOwnerHelpValidation(command, subcommand) {
return command === "harness" && subcommand === "source-review";
}

function hasJsonFlag(argv) {
for (const value of argv) {
if (value === "--") return false;
Expand Down Expand Up @@ -426,7 +431,9 @@ export function resolveDispatch(argv = []) {
return commandSchema(argv.slice(1));
}

if (hasHelpFlag(argv)) {
const ownerValidatedHelp = requiresOwnerHelpValidation(command, subcommand);
const canonicalOwnerHelp = ownerValidatedHelp && rest.length === 1 && hasHelpFlag(rest);
if (hasHelpFlag(argv) && (!ownerValidatedHelp || canonicalOwnerHelp)) {
const direct = directDispatchFor(command, subcommand);
if (direct) {
const metadata = commandMetadata(command);
Expand Down
7 changes: 7 additions & 0 deletions scripts/better-harness-cli/registry.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -227,6 +227,13 @@ const COMMANDS = [
summary: "Build the deterministic Agent Work Loop source envelope.",
description: "Collect repository, practice, and session candidates into report.source.json while preserving evidence boundaries.",
},
{
name: "source-review",
audience: "maintainer",
script: "harness-analysis/report-source/cli.mjs",
summary: "Create, compile, and apply a bounded report-source review.",
description: "Expose an explicit local create, caller-authored decision, and confirmed apply lifecycle without calling a model or merging native evidence aliases into the outer evidence namespace.",
},
{
name: "task-loop-report",
audience: "maintainer",
Expand Down
6 changes: 6 additions & 0 deletions scripts/harness-analysis/evidence-brief.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -170,6 +170,12 @@ function diagnosticsLines(repository = {}) {
const friction = rows(signals.frictionSignals);
if (Object.keys(learning).length > 0) {
output.push(`- Learning observation coverage: ${rows(learning.episodeRecords).length} episode record(s), ${rows(learning.recurringIssueCandidates).length} recurring candidate(s), ${observedSkills.length} scoped Skill observation(s), and ${unscopedSkills.length} unscoped Skill observation(s).`);
const native = learning.nativeLearningReview ?? {};
if (native.status === "review-required") {
output.push(` - Native recurring-correction review: ${rows(native.packet?.groups).length} bounded group(s) await a packet-bound decision.`);
} else if (native.status === "reviewed") {
output.push(` - Native recurring-correction review: validated ${rows(native.result?.matches).length} match(es) and ${rows(native.result?.abstentions).length} abstention(s).`);
}
const frictionList = list(friction.map((row) => `${row?.name}: ${count(row?.count)}`), 6);
if (frictionList.text) output.push(` - Aggregated friction signals: ${withOmitted(frictionList.text, frictionList.omitted)}.`);
const coverage = list(Object.entries(learning.coverage ?? {}).map(([key, value]) => `${humanLabel(key)}=${cleanText(value, 100)}`), 9);
Expand Down
Loading