feat(ic-skills-feedback): opt-in failure/coverage-gap feedback skill

[marc0olo]

What

Adds ic-skills-feedback, an opt-in skill that lets agents send anonymous reports
when ICP skill work fails or when no skill covered the task (coverage gap) —
so maintainers can turn recurring failures into evals and skill fixes. There are no
success reports; this is a lead generator for what breaks, not a satisfaction metric.

Design highlights

• Opt-in only. Sends only with project consent (feedback: on in the managed
  AGENTS.md block, wired through llms.txt onboarding) or an explicit one-time
  session yes. No consent → do nothing, silently.
• Privacy by construction. Two payload tiers gated on "did a human approve this
  exact text this turn?" — a structured tier (enums, skill name/hash, allowlisted
  error codes; safe for autonomous runs) that always sends, and a prose tier
  (task_summary / what_went_wrong) that rides along only after human approval.
  Server enforces a blocklist + PII scan as backstop.
• Stateless skill, server owns state (dedup/triage). Model id sent verbatim;
  server canonicalizes via an alias table, so no skill release per new model.
• Coverage gap is a first-class report (suspected_cause: missing_skill,
  skill_name: "(none)") — the highest-value lead, since it names the next skill to write.
• Endpoint: POST https://feedback-api.skills.internetcomputer.org/reports. No
  spam protection in v1 (public, anonymous, curl-able; impact neutralized by strict
  schema + dedup + the untrusted-lead stance).

⚠️ Draft — merge blocker

The ingest endpoint is not live yet. Do not merge until the backend canister
(feedback-api.skills.internetcomputer.org) is deployed, or agents will POST to a
dead URL. The backend + triage dashboard are built separately (tracked outside this repo).

Evals

evaluations/ic-skills-feedback.json — 5 output + 5 trigger cases. All pass with
the skill; large deltas vs baseline on the behaviors that matter.

Eval results (with-skill vs baseline)

Output evals
  no consent means no report ................. WITH 4/4 | WITHOUT 4/4
  autonomous run sends structured tier only .. WITH 6/6 | WITHOUT 1/6
  prose only w/ human approval, PII redacted . WITH 5/5 | WITHOUT 0/5
  failure-only refuses a success report ...... WITH 3/3 | WITHOUT 2/3
  coverage gap uses the (none) sentinel ...... WITH 5/5 | WITHOUT 0/5

Trigger evals
  should-trigger ....... 2/2 correct
  should-not-trigger ... 3/3 correct

Baseline (no skill) leaks the private repo name + path in the prose case, invents
non-schema fields in the autonomous and coverage-gap cases, and never limits itself to
the structured tier when running autonomously — the skill fixes all of it.

npm run validate passes (28/28 skills).

🤖 Generated with Claude Code

[github-actions]

Skill Validation Report

Validating skill: /home/runner/work/icskills/icskills/skills/ic-skills-feedback

Structure

• Pass: SKILL.md found
• Pass: internal link: references/error-signals.md (exists)
• Pass: all files in references/ are referenced

Frontmatter

• Pass: name: "ic-skills-feedback" (valid)
• Pass: description: (720 chars)
• Pass: license: "Apache-2.0"
• Pass: compatibility: (158 chars)
• Pass: metadata: (2 entries)

Markdown

• Pass: no unclosed code fences found

Tokens

File	Tokens
SKILL.md body	3,040
references/endpoint.md	1,081
references/error-signals.md	760
Total	4,881

Content Analysis

Metric	Value
Word count	1,958
Code block ratio	0.11
Imperative ratio	0.13
Information density	0.12
Instruction specificity	0.86
Sections	14
List items	30
Code blocks	7

References Content Analysis

Metric	Value
Word count	1,140
Code block ratio	0.00
Imperative ratio	0.08
Information density	0.04
Instruction specificity	0.89
Sections	10
List items	38
Code blocks	1

Contamination Analysis

Metric	Value
Contamination level	low
Contamination score	0.01
Primary language category	config
Scope breadth	2

• Warning: Language mismatch: shell (1 category differ from primary)

References Contamination Analysis

Metric	Value
Contamination level	low
Contamination score	0.00
Scope breadth	0

Result: passed

Project Checks

✓ Project checks passed for 1 skills (0 warnings)

[Copilot]

Pull request overview

Adds a new opt-in ic-skills-feedback skill that enables anonymous reporting of ICP skill failures / partial failures and coverage gaps (no success reporting), plus supporting docs and eval cases. This fits the repo’s goal of improving skill quality by turning real-world failures into actionable leads and future evals.

Changes:

• Adds the new ic-skills-feedback skill defining consent rules, payload schema, privacy constraints, and POST behavior.
• Adds reference docs for the server contract and an error_signals allowlist.
• Updates llms.txt onboarding guidance and adds an evaluation suite for the skill.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
src/pages/llms.txt.ts	Adds onboarding text to optionally enable anonymous failure reporting via feedback: on in AGENTS.md.
skills/ic-skills-feedback/SKILL.md	Introduces the new opt-in failure/coverage-gap feedback skill, including schema and sending rules.
skills/ic-skills-feedback/references/error-signals.md	Defines an allowlist for safe error_signals tokens.
skills/ic-skills-feedback/references/endpoint.md	Documents the intended server-side ingest/validation contract.
evaluations/ic-skills-feedback.json	Adds output and trigger eval cases for the new skill.

Suppressed comments (1)

evaluations/ic-skills-feedback.json:21

• The "autonomous run sends structured tier only" eval is about constructing the exact report payload, but the expected behaviors don't currently assert several required schema fields (e.g., schema_version, agent, model) or that unknown top-level keys are absent. This weakens the regression signal for the core contract of this skill.

        "Produces one JSON object with report_type set to failure or partial",
        "suspected_cause and outcome_after are values from the allowed enums (outcome_after is worked_around)",
        "error_signals contains \"M0220\" and only allowlisted tokens",
        "consent_basis is \"project\"",
        "Does NOT include task_summary or what_went_wrong, and does not set free_text_reviewed true, because no human approved prose this turn",

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

Suppressed comments (4)

skills/ic-skills-feedback/SKILL.md:257

• In the decision-flow code block, “feedback:on” is missing the space used everywhere else (feedback: on). Keeping the exact token consistent reduces the chance an agent copies the wrong form into AGENTS.md.

        project feedback:on ......... consent_basis=project

src/pages/llms.txt.ts:71

• The line break splits the sentence at “If the”, which will render awkwardly in the generated llms.txt. Rewrap so the break occurs at a sentence boundary (and consider marking AGENTS.md as code for consistency).

\`ic-skills-feedback\` skill, which sends nothing without opt-in and never includes free-text details without explicit confirmation. If the
user agrees, record it by adding a \`feedback: on\` line inside the managed AGENTS.md
block (see below). If they decline, or you cannot ask, leave it off and continue —
never block.

src/pages/llms.txt.ts:129

• The text describing the managed block markers shows the end marker as “:end -->”, but the actual marker used elsewhere in this file is <!-- ic-skills:managed:end -->. Keeping this accurate helps prevent agents from writing malformed blocks.

\`feedback: on\` inside the managed block you just wrote (between the
\`<!-- ic-skills:managed:start -->\` / \`:end -->\` markers). Its absence means reports
stay off. The \`ic-skills-feedback\` skill reads this line as standing consent.

skills/ic-skills-feedback/SKILL.md:89

• “Create the block if absent” is underspecified here: this repo’s ic-skills managed block has a specific required structure (see llms.txt onboarding), and creating a minimal block just to hold feedback: on could leave a malformed/partial managed section. Suggest pointing explicitly to the existing ic-skills managed block markers, and only creating the block using the standard onboarding snippet.

This issue also appears on line 257 of the same file.

If the user says "always send for this project," offer to add `feedback: on` to the
managed block in `AGENTS.md` (create the block if absent; never touch other content).