Skip to content

Test Codex plugin skill contracts#171

Merged
ScriptedAlchemy merged 4 commits into
masterfrom
codex/skill-validation-tests
Jul 1, 2026
Merged

Test Codex plugin skill contracts#171
ScriptedAlchemy merged 4 commits into
masterfrom
codex/skill-validation-tests

Conversation

@ScriptedAlchemy

@ScriptedAlchemy ScriptedAlchemy commented Jul 1, 2026

Copy link
Copy Markdown
Owner

Summary

  • add Rust skill contract tests (tests/plugin_skill_contract_test.rs) that mirror Codex skill-creator quick_validate rules for bundled Codex skills
  • add Cursor-specific skill frontmatter linting so Cursor-only fields like paths and disable-model-invocation stay valid without being forced into the Codex schema
  • assert the generated/installed Codex and Cursor plugin skill trees are byte-identical copies of the source skill folders (hermetic against ambient TRACEDECAY_DATA_DIR)
  • add a canonical, CRLF-tolerant SKILL.md frontmatter parser (src/automation/skill_frontmatter.rs) shared by the contract tests and the Hermes skill inventory, replacing bespoke per-site parsing
  • remove Codex-only paths frontmatter from running-impacted-tests and narrow the documented divergence in src/agents/codex.rs to frontmatter only, so the skill bodies must still mirror byte-for-byte
  • rewrite 14 cursor-plugin slash-skill descriptions to lead with "Use to …" trigger language so their metadata follows skill-creator trigger guidance, enforced by a documented trigger-language rule

Tests

  • python3 /home/zack/.codex/skills/.system/skill-creator/scripts/quick_validate.py <each codex-plugin/skills/*>
  • cargo test --test plugin_skill_contract_test
  • cargo test codex_skills_match_the_cursor_source_for_parity
  • cargo test session_context_skill_index_matches_bundle_skills
  • cargo test --test update_plugin_test
  • cargo test -p tracedecay --lib -- automation::skill_frontmatter automation::hermes_skill_inventory
  • cargo fmt --all --check && cargo clippy -p tracedecay --lib --test plugin_skill_contract_test --test update_plugin_test -- -D warnings

@changeset-bot

changeset-bot Bot commented Jul 1, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: a77b5c6

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d0a96625b7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread codex-plugin/skills/running-impacted-tests/SKILL.md
@ScriptedAlchemy ScriptedAlchemy force-pushed the codex/skill-validation-tests branch from d0a9662 to b0375b2 Compare July 1, 2026 20:42
@ScriptedAlchemy ScriptedAlchemy force-pushed the codex/skill-validation-tests branch from b0375b2 to 00c2503 Compare July 1, 2026 20:44
ScriptedAlchemy and others added 3 commits July 1, 2026 22:05
Give the crate one line-ending-tolerant SKILL.md frontmatter parser with
typed Scalar/Block values and proper YAML single-quote unescaping, and
rewire the Hermes skill inventory onto it instead of its bespoke copy.
Reuse the canonical frontmatter parser so the contract tests survive CRLF
checkouts (this PR's Windows CI failure), replace the generated-bundle
contract re-runs with byte-parity against the source tree, pin
TRACEDECAY_DATA_DIR for hermetic installs, narrow the running-impacted-tests
parity exemption to frontmatter only, loosen the trigger-language heuristic
to a documented rule, document the size budgets, and rename the test to
plugin_skill_contract_test.rs to match its Codex+Cursor scope.
@ScriptedAlchemy

Copy link
Copy Markdown
Owner Author

Addressed in bc2129f: running-impacted-tests is now explicitly listed in CODEX_SKILL_FRONTMATTER_DIVERGENCES with a justification comment, and the parity test still compares its body line-for-line after stripping frontmatter, so body drift remains caught. codex_skills_match_the_cursor_source_for_parity passes on the branch.

@ScriptedAlchemy ScriptedAlchemy merged commit 66c33dd into master Jul 1, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant