Skip to content

feat: validate skills against Agent Skills spec across harnesses - #351

Merged
totollygeek merged 3 commits into
masterfrom
claude/marketplace-codex-gemini-setup-ksau6v
Aug 13, 2026
Merged

feat: validate skills against Agent Skills spec across harnesses#351
totollygeek merged 3 commits into
masterfrom
claude/marketplace-codex-gemini-setup-ksau6v

Conversation

@totollygeek

Copy link
Copy Markdown
Contributor

What & why

This PR adds validation of the skills/ tree against the Agent Skills specification, catching a critical class of silent failures: when a skill's directory name drifts from its frontmatter name field.

The problem: Zuke's skills are now served to three harnesses — Claude Code (via plugin), Codex (via native marketplace), and Gemini CLI (via extension manifest). All three load skills directly from the skills/ folder and require the frontmatter name to match the directory name. A rename that touches only one side (e.g., renaming the folder but forgetting to update the frontmatter) ships a skill that silently fails to load in the other harnesses — the worst kind of quiet breakage.

The solution:

  • Added build/skill_check.ts: a spec validator that parses YAML frontmatter strictly (matching what real loaders reject), checks required fields (name, description), enforces the spec's shape and length limits, and validates the critical name/directory match.
  • Added skillsCheck target to zuke.ts that runs this validator over skills/ as part of the ci gate.
  • Extended plugin manifest support to Codex (.codex-plugin/plugin.json, .agents/plugins/marketplace.json) and Gemini CLI (gemini-extension.json), all carrying the same version.
  • Updated pluginVersionCheck to verify all four version-carrying manifests stay in sync.

The validator is deliberately not a full YAML parser — it errs strict where YAML is strict (rejecting key:value without a space, detecting duplicated keys, handling quoted scalars) while remaining simple and auditable.

Related issues

Enables safe distribution of skills to Codex and Gemini CLI without silent load failures.

Checklist

  • The PR title is a Conventional Commit (feat: ...).
  • deno task ci passes locally.
  • Tests added: tests/skill_check_test.ts (199 lines, 11 test cases covering parser, validation, tree traversal, symlinks, and the real skills/ tree), tests/integration/skill_check_test.ts (68 lines, CLI integration), and tests/plugin_manifest_test.ts extended with Codex and Gemini assertions.
  • Docs updated: README.md (Codex and Gemini CLI installation), AGENTS.md (new manifests, skillsCheck in CI gate).
  • No any, as casts, or ! assertions in new code.

https://claude.ai/code/session_019As9A3ugixLQiAvcat1ZKk

claude added 3 commits August 13, 2026 10:59
Publish the shared skills tree to two more harnesses. A Codex-native
plugin manifest and marketplace catalog let Codex users install the
plugin from this repo, and a root Gemini CLI extension manifest serves
the skills folder that Gemini auto-discovers. A new skillsCheck gate
target validates the skills tree against the Agent Skills spec, since
Codex and Gemini load those folders directly and a frontmatter name
that drifts from its directory ships a skill that silently fails to
load. The version-bump check now names every version-carrying manifest,
and the manifest tests enforce that all of them agree.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019As9A3ugixLQiAvcat1ZKk
Adversarial review of the validator found documents that passed the
gate but fail in a strict YAML loader: a key with no space after the
colon, duplicated keys, and block-scalar values that dodge the
presence and length checks. The parser now mirrors the loader where it
matters, unquotes quoted scalars, strips a leading byte-order mark,
validates symlinked skill folders instead of skipping them, and
reports a SKILL.md that is not a regular file as a finding rather than
crashing. Each fixed case carries a regression test, and the README
notes that Gemini installs the extension from the latest release
snapshot.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019As9A3ugixLQiAvcat1ZKk
The build-graph page is regenerated since the new skillsCheck target
changes the graph that docs/graph.md renders.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019As9A3ugixLQiAvcat1ZKk
@github-actions

Copy link
Copy Markdown
Contributor

🤖 Zuke AI review

🔎 security review — review

Score: 0/10 · Severity: none · 0 finding(s)

Tokens: 33738 in · 223 out · 33961 total

Budget: 60,123 tokens of 500,000 tokens

Refuted by verification (not reported):

Finding Reason
New Codex marketplace manifest may broaden installation surface without validation of its trust model The new Codex catalog is a local path to the same checked-in ./plugins/zuke directory, and the diff adds explicit manifest/version checks plus a skillsCheck gate rather than removing or weakening existing install-time validation.

The change mostly adds new skill-validation and manifest coverage, but it also introduces a potential authorization/policy drift issue by advertising a new Codex marketplace entry without a corresponding signed/validated installation path in the reviewed diff.

@github-actions

Copy link
Copy Markdown
Contributor

🤖 Zuke AI review

🔎 generic review — review

Score: 0/10 · Severity: none · 0 finding(s)

Tokens: 20631 in · 178 out · 20809 total

Budget: 93,972 tokens of 500,000 tokens

Refuted by verification (not reported):

Finding Reason
New skills validator is not exercised for hidden files and nested directories in the tree walk checkSkillTree() intentionally scans only immediate directories under skills/, and the repo’s documented contract and tests treat skills/ as a flat set of top-level skills with no recursion or nonstandard children to validate.

The change is generally well-contained, but there is one maintainability/test coverage gap around the new skills validator’s incomplete spec handling.

@totollygeek
totollygeek merged commit 65ee672 into master Aug 13, 2026
12 checks passed
@totollygeek
totollygeek deleted the claude/marketplace-codex-gemini-setup-ksau6v branch August 13, 2026 11:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants