Add flag-and-release-change skill (apply counterpart to should-flag-change) - #102
Closed
Vadman97 wants to merge 2 commits into
Closed
Add flag-and-release-change skill (apply counterpart to should-flag-change)#102Vadman97 wants to merge 2 commits into
Vadman97 wants to merge 2 commits into
Conversation
The "apply" counterpart to the advisory should-flag-change skill: once a PR's change is judged flag-worthy, this skill creates the guarding flag, wires the new path behind it on the PR branch, and registers an automated rollout (create-automated-rollout-config) so the change releases safely on merge. Two-phase plan->implement, portable (public LD MCP + git). Includes references/auto-release.md (release policies, simple vs policy, precedence) and references/pr-wiring.md (three-dot diff, guarding, push). Wires into README + regenerated skills.json. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Skill eval results
Only suites whose source actually changed since their last recorded score were re-run. Soft-failing while we stabilise the baseline. |
Reframe flag-and-release-change as a PR-scoped orchestrator that composes should-flag-change (decide) + launchdarkly-flag-create (create flag & wire code) and adds the unique auto-release step. Delegate flag-creation and per-SDK guarding mechanics to flag-create instead of re-teaching them; slim pr-wiring.md to PR-specific mechanics (three-dot diff, commit/push) and point at flag-create's sdk-evaluation-patterns for guarding. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
bradsimantel
added a commit
that referenced
this pull request
Jul 15, 2026
Bring the flag-and-release-change skill (originally PR #102) onto the should-flag-change branch (#101) so the advisory "decide" step and the "apply" step ship together. Only the content of #102's two flag-and-release commits is included -- not that branch's unrelated observability skills (which sit on an unmerged #99 commit). README skill list updated and skills.json regenerated from the catalog script.
Contributor
|
Superseded by #101. The flag-and-release-change skill has been consolidated onto that branch, where it's been refactored into a thinner PR orchestrator that composes should-flag-change, launchdarkly-flag-create, and a new extracted flag-release skill (the automated-rollout recording step). #101 also adds eval suites for flag-release and flag-and-release-change. Closing this in favor of the consolidated PR. |
nieblara
added a commit
that referenced
this pull request
Jul 21, 2026
* Add should-flag-change advisory skill + eval suite
New read-only skill that decides whether a code change should ship behind a
LaunchDarkly feature flag. Invoked ad hoc or in CI on a PR: it reads the diff
and surrounding code, weighs a decision framework (favoring flags for
user-facing and risky changes, weighting false negatives over false positives),
and ends with a structured recommend-flag verdict. It never creates or
modifies flags.
- skills/feature-flags/should-flag-change: SKILL.md (read-only scope boundary,
confusion-matrix decision framework, recommend-flag verdict contract) plus
README and marketplace.json
- evals: recommend-flag tool definition + mock, git_diff provider var injected
as a <git_diff> tag, and a two-tier suite (judgment vs agentic providers,
targeted per fixture) with 5 labeled fixtures
- evals/scripts/build-pr-fixture.js: build a fixture from a real PR via
gh pr diff + refs/pull/N/head, bootstrapping the label from LD SDK call sites
- wiring: package.json scripts, _manifest.js, README skill list, skills.json
* Add already-gated fixtures, PR-fixture tooling, and templating fix
Extends the should-flag-change eval suite and its fixture tooling based on
testing against real pull requests.
- promptfooconfig.yaml: add three synthetic fixtures for the "already gated"
case the suite didn't cover — a change that ships behind a flag should get
recommend: false ("already handled"), and its ungated mirror should get
recommend: true. Uses public SDK idioms; no proprietary source.
- provider: fix a Nunjucks var-render crash on diffs containing `{{ ... }}`
(JSX props, Go templates). Fixtures may wrap such content in a `{% raw %}`
block; stripRawWrapper removes it before the agent sees the diff.
- build-pr-fixture.js: read changed files via the GitHub contents API instead
of fetching refs/pull/N/head (avoids cloning a large external repo); add
--flag-pattern / --flag-token / --flag-file-pattern to teach it a codebase's
flag conventions at call time, and --counterfactual to build positive-recall
fixtures by stripping the flag gate. Kept repo-agnostic — no baked-in
conventions.
- tests: unit-test stripRawWrapper and the builder's pure functions
(addedLines, detectLdUsage, splitDiffByFile, dropFilesFromDiff,
extractFlagTokens, stripFlagGate).
* Combine should-flag-change and flag-and-release-change onto one branch
Bring the flag-and-release-change skill (originally PR #102) onto the
should-flag-change branch (#101) so the advisory "decide" step and the
"apply" step ship together.
Only the content of #102's two flag-and-release commits is included --
not that branch's unrelated observability skills (which sit on an
unmerged #99 commit). README skill list updated and skills.json
regenerated from the catalog script.
* Improve both flag skills with ideas from auto-factory and gonfalon
should-flag-change:
- Ancestor-gate analysis: detect an enclosing flag the change already
sits behind, and judge on its rollout state.
- Prerequisite/dependency signal as a reason to flag.
- Optional risk/blast-radius field on the verdict, orthogonal to
confidence (added to the eval tool schema, additive/non-breaking).
- Refactor-that-changes-a-contract case; net-new-vs-incremental
measurability nuance; unverified-claim confidence discipline.
flag-and-release-change:
- Capture and honor human release intent (release / hold / notBefore /
segment / prerequisite), fail-closed; three-layer precedence
(intent -> override -> policy -> default).
- "The deploy is not the release" framing.
- Fail-closed on non-409 flag-creation errors.
- Paired flag-on/flag-off tests run to green before push.
- Metric-adequacy check for guarded policies; prerequisite coupling.
- Off-path-invokes-no-new-code invariant; edge cases for duplicate
config, registering before the PR exists, and double-toggling.
Sources: launchdarkly-labs/launchdarkly-auto-factory (AI-config prompts,
ADRs) and launchdarkly/gonfalon .agents/skills. Portable-safe only --
repo-specific specifics (env matrix, release tags, context kinds) left
out as the per-repo customization surface.
* Extract flag-release skill; slim flag-and-release-change to an orchestrator
Decompose the "apply" half into composable, harness-callable units,
matching gonfalon's planning/add-flag/rollout shape while staying generic:
- New skill feature-flags/flag-release: records a flag's automated rollout
for a PR (match-release-policies preview, release intent + fail-closed
hold, precedence, prerequisites, metric-adequacy). Operates on an
existing flag; creates no flags and edits no code. auto-release.md moves
here as its core reference.
- flag-and-release-change becomes a thin portable orchestrator that
composes should-flag-change -> launchdarkly-flag-create -> flag-release,
owning only the PR workflow (clone/diff/push) and plan->implement
sequencing. Notes that an automation harness can bypass it and invoke
the three atoms directly.
Repo-specific values (env matrix, release tags, targeting context) remain
out of these generic skills -- that is the AgentControl customization seam.
* Add eval coverage for the new flag decision/release logic
should-flag-change:
- New fixtures: ancestor-gate (agentic; explores to an enclosing flag not
in the diff) and prerequisite-dependency (judgment).
- risk-calibration asserts folded into the auth (expect high) and
docs-only (expect low/absent) fixtures. All pass.
New flag-release suite (adds match-release-policies / list-release-policies
/ create-automated-rollout-config tool defs + mocks; environmentKey mock
replacement):
- Release-on-merge happy path: previews the policy, records both envs. PASS.
- Hold intent: KNOWN-RED tracked signal. sonnet-4-6 records the held env as
`policy` (wrongly believing policy holds); opus omits it correctly. The
assertion is correct (a held env must be omitted); do not weaken it.
New flag-and-release-change suite (both PASS):
- Plan phase is side-effect-free (no mutating MCP calls during planning).
- Fails closed: a create-flag 403 (via the restricted-project mock hook)
stops the run before any release is recorded.
Wiring: manifest + package.json scripts for both new suites; READMEs for
flag-release and flag-and-release-change. Sharpened flag-release's
hold-honoring wording (helps stronger models; sonnet gap tracked above).
90/90 unit tests pass.
* Address review feedback on the should-flag-change PR
1. Catalog leaked third-party skills from evals/node_modules (dotenv,
dotenvx, playwright) into the public skills.json, and CI's
generate_catalog.py --check failed on the mismatch. Add node_modules to
the validator's EXCLUDED_DIRS (generate_catalog reuses it) and regenerate
skills.json — back to 44 skills, --check clean.
2. Enforce the recommend-flag contract. A new suite-wide verdict_contract
assertion requires the tool to be called EXACTLY once and as the final
tool call; a run that calls it early, twice, or keeps working afterward
now fails instead of passing on the first call's boolean.
3. Guard the read-only safety contract. A new read_only_guard assertion bans
Edit/Write and mutating shell commands (redirects, rm/mv/cp, git
commit/push, package installs) across the suite, so the agentic tier can't
modify code or state and still pass. Previously only flag-mutating MCP
tools were checked.
Both new assertions pass on all 10 should-flag-change fixtures.
* Declare js-yaml as a direct dependency of the eval tooling
build-pr-fixture.js required js-yaml via an explicit ../node_modules path,
but js-yaml was only present transitively (via promptfoo). A promptfoo bump
that dropped or relocated it would break the fixture builder and its unit
test. Add js-yaml (^4.1.1, already the resolved version) to evals
devDependencies, update the lockfile, and use a plain require. Also drop the
now-unused node:path import.
* Enrich should-flag-change + add targeting context-availability reference (#108)
* feat(feature-flags): enrich should-flag-change and add context-availability reference
Layer portable, de-LaunchDarkly-internal lessons from our flag-planning
skills onto the public feature-flag skills.
should-flag-change:
- Add an explicit, named user-observability test as a gate before any
`recommend: false` verdict.
- Add a `verdict` field (suggested | already-flagged | not-suited) to the
recommend-flag output, keeping already-flagged (protected by an existing or
ancestor gate) distinct from not-suited (nothing to flag). `recommend` stays
the boolean a CI check keys on.
- Generalize a decision-posture tie-breaker (conservative vs. low-overhead)
for genuinely balanced calls, without internal "dogfood" wording.
Add a new SDK-agnostic targeting context-availability reference: match the
context kind to the surface where the flag is read (server/client/anonymous),
key vs attribute, and rollout bucketing. Wire it into flag-targeting and
flag-create, and reference it from should-flag-change.
Bump versions and regenerate skills.json.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(feature-flags): cover verdict taxonomy and targeting-context reasoning
- should-flag-change suite: assert the verdict field across the already-flagged,
not-suited, and suggested fixtures, and validate the enum plus recommend/verdict
agreement in the ambiguous fixture.
- flag-create suite: add a fixture where a client-side flag is asked to target a
server-only signal, asserting the agent flags that browser context can't carry
it and suggests an available approach.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* Let the eval gate tolerate documented known-red fixtures
The flag-release suite carries an intentionally-red fixture (Sonnet mishandles
a held production environment) that must stay red until the skill or model
closes the gap. Previously any suite below 75% failed the CI job, so this
tracked signal blocked merge.
Declare such fixtures via a knownRed allowlist in _manifest.js (matched by the
fixture description, which promptfoo stores under testCase.description).
aggregate.js now keeps the honest score/badge red while gating on a gateScore
that excludes known-red fixtures, so only unexpected failures fail CI. A
known-red fixture that starts passing is surfaced so its entry can be removed.
* Revert "Let the eval gate tolerate documented known-red fixtures"
This reverts commit 5e87093.
* Restore known-red eval gate tolerance for flag-release hold fixture
Sonnet still records held production as `policy` (auto-releases on merge),
so the intentionally-red hold-intent fixture keeps failing the 75% suite
gate. Re-apply the gateScore allowlist so the tracked signal stays visible
without blocking CI.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(flag-release): make held-environment handling a mechanical step
The hold-intent fixture failed because the model maps "hold production" to
`releaseType: policy` — the tool describes `policy` as "defer to the
environment's release policy on merge", and "defer" reads like "hold". The
skill fought this with prose caveats, which the authors noted hadn't closed
the gap.
Restructure the Implement phase around a forced RELEASE-vs-HOLD bucket sort:
the environments array is built only from the RELEASE bucket, HOLD means
absence from the array, and the "policy defers to the policy, not to you"
misreading is called out at the point of the call. Remove the known-red gate
allowlist so the fixture counts again and the skill fix carries the suite.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(flag-release): de-trap the rollout tool description; name the hold-field hallucination
Root-cause dig on the hold-intent fixture: the model doesn't fail from a bare
reasoning gap — the eval's mock tool description diverged from the real hosted
tool and fed it a trap. The real create-automated-rollout-config says policy
"waits until merge, then performs the release"; the mock said policy "defers to
the release policy", and "defer" reads as "hold", steering the model to keep a
held env as policy. With louder omit-prose the model instead fabricated a
holdUntil field — which the mock silently accepts (.passthrough), so it never
learns the field is invalid.
- Make the mock tool description faithful to the real tool: drop "defer", say
"waits then performs", and state explicitly there is NO hold/notBefore/date
field (omit the env to hold it; unknown fields are rejected).
- Skill: name the exact tell — wanting holdUntil/notBefore/hold (or keeping an
env as policy "so it waits") means the env is HOLD; drop the entry, don't
invent a field. Date/reason go in the report.
- Keep the known-red gate as a backstop since the 2-test suite is a binary gate
that can flake even when the fix lands; the honest score still shows red if it
regresses.
Co-authored-by: Cursor <cursoragent@cursor.com>
* chore(evals): drop known-red backstop now flag-release hold fixture passes
The de-trapped tool description + skill hallucination-naming flips Sonnet to the
correct behavior: it omits the held environment (recorded [{staging,simple}],
production absent) and scores 0.83 > 0.75. With a real fix in place the known-red
gate tolerance would only mask a future regression, so remove it and let the
fixture gate on its own.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(flag-release): broaden hold coverage and lock in the no-fabricated-field fix
Expand the flag-release suite from 2 to 5 fixtures, all exercising the
release-vs-hold sort the earlier fix addressed:
- add a no_fabricated_fields assertion (rejects holdUntil/notBefore/hold on any
recorded env) to the hold fixtures, locking in the regression fix
- multi-env mixed intent (dev+staging release, production hold) — the sort must
scale past two envs
- single-target date hold (the only env is held, so nothing releases)
- fail closed on ambiguous intent (release staging, hold unconfirmed production)
Retire the stale KNOWN-RED comment on the original hold fixture (now a passing
regression guard). More fixtures also stabilize the 75% gate: a single stochastic
miss on a 5-test suite stays green, where on 2 tests it went red.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(flag-and-release-change): cover the orchestrator's decide gate and create step
The orchestrator suite only had plan-side-effect-free and fail-closed fixtures.
Add two more targeting its own distinct, sandbox-reachable decisions (the record
step needs a git push the MCP-only provider can't do, and is covered directly by
the flag-release suite):
- decide gate: a docs-only change is judged not flag-worthy and creates/records
nothing, even when told to "flag and release if warranted"
- create step: an approved change produces exactly one boolean kill-switch,
created OFF, never toggled on by hand (over-flag + created-OFF guards)
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Ramon Niebla <rniebla@launchdarkly.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
feature-flags/flag-and-release-change— the "apply" half of the software-factory flag pipeline, complementing Brad's advisoryshould-flag-change(#101).Where
should-flag-changedecides whether a PR's change should ship behind a flag (read-only), this skill acts on a "yes":create-automated-rollout-config— a per-environment plan (simple= serve true now;policy= on merge, resolve that env's release policy and run the matching immediate/progressive/guarded rollout).It runs plan → implement with a user check-in between, and creates nothing during planning.
Design choices (per @vkorolik)
gitCLI. Nofactory_*tools; no committed.release-flags/manifest.match-release-policies(preview) →create-automated-rollout-config, mirroring the factory'sstartAutomatedRelease/ release-policy semantics (ADR 0002 in launchdarkly-auto-factory).Contents
skills/feature-flags/flag-and-release-change/SKILL.mdreferences/auto-release.md— release policies,simplevspolicy, precedence (overrides → policy → defaults), preview flow.references/pr-wiring.md— clone, three-dot diff, guarding patterns, commit/push to the PR branch.marketplace.json; README skill list + regeneratedskills.json.Validation
python3 scripts/generate_catalog.py— regeneratedskills.json.python3 scripts/validate_skills.py— 46 SKILL.md files validated.Notes
should-flag-changeas the upstream decision step (that path resolves once Add should-flag-change advisory skill + eval suite #101 merges).auto-factory.ymlGitHub Action.🤖 Generated with Claude Code