test: add full extension suites

.claude/skills/release/SKILL.md

@@ -35,7 +35,7 @@ If any precondition fails, stop and surface it. Do not work around it.
 ## Release flow
 
 Run these in order. Each script is idempotent or safe to re-run except the `git tag`/`git push`
-steps. Pause after step 5 (the commit) and after step 8 (validation) to let the user confirm.
+steps. Pause after step 6 (the commit) and after step 8 before tagging to let the user confirm.
 
 ```bash
 # 1. Edit CHANGELOGs by hand — both of them:
@@ -52,38 +52,45 @@ steps. Pause after step 5 (the commit) and after step 8 (validation) to let the
 #    new version:
 python scripts/converter.py
 
-# 4. Sanity-check the tree (no stray edits, counts/versions consistent):
+# 4. Validate generated extension outputs before committing:
+pytest tests/codex/test_codex_extension.py \
+  tests/gemini tests/opencode tests/copilot \
+  tests/vibe/test_vibe_extension.py \
+  tests/paperclip/test_commands_json.py \
+  tests/plugin/test_release_process.py
+
+# 5. Sanity-check the tree (no stray edits, counts/versions consistent):
 git status && git diff --stat
 
-# 5. Commit the bump — a clean tree is required for `claude plugin tag`:
+# 6. Commit the bump — a clean tree is required for `claude plugin tag`:
 git add -A && git commit -m "chore: bump version to X.Y.Z"
 
-# 6. Validate EVERY plugin manifest against the marketplace entry.
+# 7. Validate EVERY plugin manifest against the marketplace entry.
 #    Discover plugins dynamically — do NOT hardcode the list (it grows):
 for manifest in $(find . -maxdepth 3 -path '*/.claude-plugin/plugin.json' -not -path '*/node_modules/*' | sort); do
   p=$(python3 -c "import json;print(json.load(open('$manifest'))['name'])")
   claude plugin tag "$p" --dry-run || { echo "VERSION DRIFT: $p"; exit 1; }
 done
 
-# 7. (optional) Prune orphaned plugin deps:
+# 8. (optional) Prune orphaned plugin deps:
 claude plugin prune --dry-run
 
-# 8. Tag the umbrella release and push — this triggers
+# 9. Tag the umbrella release and push — this triggers
 #    .github/workflows/release.yml, which creates the GitHub Release:
 git tag -a vX.Y.Z -m "vX.Y.Z"
 git push && git push --tags
 
-# 9. Create native per-plugin tags (arckit--vX.Y.Z, arckit-uae--vX.Y.Z, …).
+# 10. Create native per-plugin tags (arckit--vX.Y.Z, arckit-uae--vX.Y.Z, …).
 #    Auto-discovers plugins; idempotent (skips existing tags):
 ./scripts/tag-plugins.sh X.Y.Z
 
-# 10. Push each extension to its standalone GitHub repo
+# 11. Push each extension to its standalone GitHub repo
 #     (tractorjuice/arckit-gemini, arckit-codex, …). This also creates or
 #     preserves each extension repo's vX.Y.Z tag and GitHub Release:
 ./scripts/push-extensions.sh
 ```
 
-After step 10, confirm the GitHub Release was created (the `release.yml` workflow runs on the
+After step 11, confirm the GitHub Release was created (the `release.yml` workflow runs on the
 `vX.Y.Z` tag push). Also confirm every standalone extension repo has a `vX.Y.Z` tag and GitHub
 Release, then report the release URLs and which extension repos were pushed.
 
@@ -97,12 +104,15 @@ The highest-signal failures — collected from real releases. Read these before
   Codex/OpenCode/Gemini/Copilot/Paperclip copies are *generated*. Skip `converter.py` and the
   extensions ship the **old** version. The converter must run *after* the bump and *before* the
   commit, so the regenerated files are included.
+- **Skipping extension tests.** Run the step 4 extension suite after `converter.py`. It validates
+  Codex, Gemini, OpenCode, Copilot, Vibe, Paperclip, release inventory, version alignment, and
+  platform-specific command rewrites before anything is tagged.
 - **Hardcoding the plugin list.** The marketplace now ships 11 plugins (core + 10 overlays) and
   keeps growing. The `--dry-run` validation loop in older `RELEASING.md` examples lists only 7 —
-  that silently skips newer plugins. Discover plugins dynamically (step 6) — this is the exact
+  that silently skips newer plugins. Discover plugins dynamically (step 7) — this is the exact
   bug that shipped `arckit-uk-nhs` untagged mid-v5.4.0, which is why `tag-plugins.sh` now
   auto-discovers. Never copy a static plugin array.
-- **`claude plugin tag` needs a clean tree.** Run it *after* the commit (step 5), not before, or
+- **`claude plugin tag` needs a clean tree.** Run it *after* the commit (step 6), not before, or
   it errors on the dirty working tree.
 - **`claude plugin tag` is `--dry-run` only here.** It creates `name--vX.Y.Z` style tags that do
   **not** match `release.yml`'s `v[0-9]+.[0-9]+.[0-9]+` trigger. We use it solely for its
@@ -121,7 +131,7 @@ The highest-signal failures — collected from real releases. Read these before
 - **Do not put release numbers in extension READMEs.** Extension release identity lives in
   `VERSION` files, manifests, Git tags, and GitHub Releases. README-pinned versions drift and
   are blocked by `tests/plugin/test_release_process.py`.
-- **Order is load-bearing.** bump → convert → commit → validate → tag → tag-plugins → push-extensions.
+- **Order is load-bearing.** bump → convert → extension tests → commit → validate → tag → tag-plugins → push-extensions.
   Re-running an earlier step after a later one (e.g. editing files after the commit) means the tag
   no longer points at the released tree. If you edit after committing, redo from the commit.
 

docs/RELEASING.md

@@ -74,25 +74,32 @@ git checkout main && git pull
 # 5. Regenerate Codex/OpenCode/Gemini/Copilot formats
 python scripts/converter.py
 
-# 6. Commit the bump (claude plugin tag below requires a clean tree)
+# 6. Validate generated extension outputs
+pytest tests/codex/test_codex_extension.py \
+  tests/gemini tests/opencode tests/copilot \
+  tests/vibe/test_vibe_extension.py \
+  tests/paperclip/test_commands_json.py \
+  tests/plugin/test_release_process.py
+
+# 7. Commit the bump (claude plugin tag below requires a clean tree)
 git add -A && git commit -m "chore: bump version to X.Y.Z"
 
-# 7. Validate plugin/marketplace version agreement (Claude Code v2.1.118+)
+# 8. Validate plugin/marketplace version agreement (Claude Code v2.1.118+)
 claude plugin tag plugins/arckit-claude --dry-run
 
-# 8. (optional) Prune orphaned plugin dependencies
+# 9. (optional) Prune orphaned plugin dependencies
 claude plugin prune --dry-run
 
-# 9. Tag, push — GitHub Release created automatically
+# 10. Tag, push — GitHub Release created automatically
 git tag -a vX.Y.Z -m "vX.Y.Z"
 git push && git push --tags
 
-# 10. Push to extension repos (Gemini, Codex, etc.).
+# 11. Push to extension repos (Gemini, Codex, etc.).
 #     This also publishes each extension repo's vX.Y.Z tag and GitHub Release.
 ./scripts/push-extensions.sh
 ```
 
-After step 10, verify the umbrella GitHub Release and every extension GitHub Release exists:
+After step 11, verify the umbrella GitHub Release and every extension GitHub Release exists:
 
 - `tractorjuice/arc-kit`
 - `tractorjuice/arckit-gemini`
@@ -110,15 +117,15 @@ This command creates `{plugin-name}--vX.Y.Z` style tags (e.g. `arckit--v4.14.0`)
 
 After v5.0.0 the marketplace ships 7 plugins (`arckit` core + 6 community overlays: UAE, FR, CA, EU, AT, AU). All 7 share one version, bumped together.
 
-Step 7 changes — validate every plugin manifest:
+Step 8 changes — validate every plugin manifest:
 
 ```bash
 for p in arckit-claude arckit-uae arckit-fr arckit-ca arckit-eu arckit-at arckit-au; do
   claude plugin tag "$p" --dry-run || exit 1
 done
 ```
 
-After the umbrella tag (step 9), also create native per-plugin tags:
+After the umbrella tag (step 10), also create native per-plugin tags:
 
 ```bash
 ./scripts/tag-plugins.sh X.Y.Z

scripts/converter.py

@@ -35,6 +35,11 @@ def rewrite_user_config_placeholders(value):
     return USER_CONFIG_PLACEHOLDER_RE.sub(r"${\1}", value)
 
 
+def rewrite_copilot_command_invocations(prompt):
+    """Rewrite Claude-style ArcKit slash commands to Copilot prompt names."""
+    return prompt.replace("/arckit:", "/arckit-")
+
+
 def build_agent_map(agents_dir):
     """Build a map from command name to agent file path and content.
 
@@ -344,7 +349,7 @@ def titleize_arckit_name(name):
 
 def rewrite_paths(prompt, config):
     """Rewrite ${CLAUDE_PLUGIN_ROOT} paths using agent config."""
-    result = prompt
+    result = rewrite_user_config_placeholders(prompt)
 
     # Claude commands use literal `.arckit/templates/` for project-local
     # overrides and `${CLAUDE_PLUGIN_ROOT}/templates/` for shipped defaults.
@@ -374,6 +379,9 @@ def rewrite_paths(prompt, config):
     if config.get("arg_placeholder"):
         result = result.replace("$ARGUMENTS", config["arg_placeholder"])
 
+    if config.get("format") == "prompt":
+        result = rewrite_copilot_command_invocations(result)
+
     return result
 
 
@@ -415,6 +423,7 @@ def format_output(description, prompt, fmt):
         description_formatted = '"""\n' + description + '\n"""'
         return f"description = {description_formatted}\nprompt = {prompt_formatted}\n"
     elif fmt == "prompt":
+        prompt = rewrite_copilot_command_invocations(prompt)
         escaped = description.replace("'", "''")
         tools = _copilot_tools_for_prompt(prompt)
         tools_yaml = "[" + ", ".join(f"'{t}'" for t in tools) + "]"
@@ -633,6 +642,7 @@ def convert(commands_dirs, agents_dir):
                 )
                 if handoffs_section:
                     content += "\n" + handoffs_section.strip("\n") + "\n"
+                content = rewrite_copilot_command_invocations(content)
 
                 out_filename = config["filename_pattern"].format(name=base_name)
                 out_path = os.path.join(config["output_dir"], out_filename)
@@ -1588,6 +1598,7 @@ def generate_gemini_agents(agents_dir, output_dir):
 
         # Rewrite paths: ${CLAUDE_PLUGIN_ROOT} -> ~/.gemini/extensions/arckit
         prompt = prompt.replace("${CLAUDE_PLUGIN_ROOT}", gemini_path_prefix)
+        prompt = rewrite_user_config_placeholders(prompt)
 
         # Rewrite Read instructions to shell commands
         prompt = re.sub(
@@ -1762,6 +1773,8 @@ def generate_copilot_agents(agents_dir, output_dir):
 
         prompt = prompt.replace("${CLAUDE_PLUGIN_ROOT}", ".arckit")
         prompt = prompt.replace(CONTEXT_HOOK_NOTE, CONTEXT_HOOK_REPLACEMENT)
+        prompt = rewrite_user_config_placeholders(prompt)
+        prompt = rewrite_copilot_command_invocations(prompt)
 
         fm_str = yaml.dump(copilot_fm, default_flow_style=False, sort_keys=False).rstrip()
         out_filename = filename.replace(".md", ".agent.md")

tests/__init__.py

@@ -0,0 +1 @@
+"""ArcKit test helpers package."""

tests/copilot/__init__.py

@@ -0,0 +1 @@
+"""Copilot extension tests."""

tests/copilot/test_copilot_extension.py

@@ -0,0 +1,114 @@
+"""Validate the generated GitHub Copilot extension structure."""
+
+from tests.extension_helpers import (
+    REPO_ROOT,
+    assert_no_claude_placeholders,
+    expected_agent_stems,
+    expected_command_names,
+    markdown_frontmatter,
+)
+
+
+COPILOT_ROOT = REPO_ROOT / "extensions" / "arckit-copilot"
+COPILOT_PROMPTS = COPILOT_ROOT / "prompts"
+COPILOT_AGENTS = COPILOT_ROOT / "agents"
+COPILOT_INSTRUCTIONS = COPILOT_ROOT / "copilot-instructions.md"
+
+
+def prompt_command_names() -> set[str]:
+    return {
+        path.name.removeprefix("arckit-").removesuffix(".prompt.md")
+        for path in COPILOT_PROMPTS.glob("arckit-*.prompt.md")
+    }
+
+
+def test_copilot_required_files_and_directories_exist():
+    required_files = [
+        COPILOT_ROOT / "README.md",
+        COPILOT_ROOT / "VERSION",
+        COPILOT_INSTRUCTIONS,
+    ]
+    required_dirs = [
+        COPILOT_PROMPTS,
+        COPILOT_AGENTS,
+        COPILOT_ROOT / "templates",
+        COPILOT_ROOT / "docs" / "guides",
+        COPILOT_ROOT / "config",
+        COPILOT_ROOT / "schemas",
+        COPILOT_ROOT / "references",
+        COPILOT_ROOT / "scripts",
+        COPILOT_ROOT / "hooks",
+    ]
+
+    for path in required_files:
+        assert path.is_file(), f"missing Copilot file: {path.relative_to(REPO_ROOT)}"
+    for path in required_dirs:
+        assert path.is_dir(), f"missing Copilot dir: {path.relative_to(REPO_ROOT)}"
+
+
+def test_copilot_prompt_files_match_source_commands():
+    assert prompt_command_names() == expected_command_names()
+
+
+def test_copilot_prompts_have_valid_frontmatter_and_rewritten_paths():
+    for prompt_path in COPILOT_PROMPTS.glob("arckit-*.prompt.md"):
+        frontmatter, body = markdown_frontmatter(prompt_path)
+
+        assert isinstance(frontmatter.get("description"), str)
+        assert frontmatter["description"].strip()
+        assert isinstance(frontmatter.get("agent"), str)
+        assert isinstance(frontmatter.get("tools"), list)
+        assert all(isinstance(tool, str) for tool in frontmatter["tools"])
+        assert body
+        assert "$ARGUMENTS" not in body
+        assert "/arckit:" not in body
+        assert_no_claude_placeholders(prompt_path, body)
+
+
+def test_copilot_agent_backed_prompts_use_generated_custom_agents():
+    for agent_stem in expected_agent_stems():
+        command_name = agent_stem.removeprefix("arckit-")
+        prompt_path = COPILOT_PROMPTS / f"arckit-{command_name}.prompt.md"
+        frontmatter, body = markdown_frontmatter(prompt_path)
+
+        assert frontmatter["agent"] == agent_stem
+        assert (COPILOT_AGENTS / f"{agent_stem}.agent.md").is_file()
+        assert f"Use the `{agent_stem}` agent" in body
+
+
+def test_copilot_custom_agents_match_source_agents_and_are_valid():
+    expected_files = {f"{name}.agent.md" for name in expected_agent_stems()}
+    actual_files = {path.name for path in COPILOT_AGENTS.glob("arckit-*.agent.md")}
+
+    assert actual_files == expected_files
+
+    for agent_path in COPILOT_AGENTS.glob("arckit-*.agent.md"):
+        frontmatter, body = markdown_frontmatter(agent_path)
+        assert frontmatter["name"] == agent_path.name.removesuffix(".agent.md")
+        assert isinstance(frontmatter["description"], str) and frontmatter["description"].strip()
+        assert frontmatter["user-invocable"] is False
+        assert isinstance(frontmatter["tools"], list) and frontmatter["tools"]
+        assert_no_claude_placeholders(agent_path, body)
+
+
+def test_copilot_instructions_point_to_generated_prompts():
+    text = COPILOT_INSTRUCTIONS.read_text(encoding="utf-8")
+
+    assert ".github/prompts/arckit-*.prompt.md" in text
+    assert ".arckit/templates-custom/" in text
+    assert "projects/001-project-name/" in text
+
+
+def test_copilot_bundles_supporting_assets():
+    expected_files = [
+        COPILOT_ROOT / "templates" / "requirements-template.md",
+        COPILOT_ROOT / "docs" / "guides" / "requirements.md",
+        COPILOT_ROOT / "config" / "doc-types.mjs",
+        COPILOT_ROOT / "schemas" / "grants-handoff.schema.json",
+        COPILOT_ROOT / "references" / "citation-instructions.md",
+        COPILOT_ROOT / "scripts" / "validate-handoff.mjs",
+        COPILOT_ROOT / "hooks" / "okf-frontmatter.mjs",
+    ]
+
+    for path in expected_files:
+        assert path.is_file(), f"missing Copilot bundled asset: {path.relative_to(REPO_ROOT)}"

tests/extension_helpers.py

@@ -0,0 +1,68 @@
+"""Shared helpers for generated extension test suites."""
+
+from pathlib import Path
+
+import yaml
+
+
+REPO_ROOT = Path(__file__).resolve().parents[1]
+
+# Mirror scripts/converter.py's PLUGIN_SOURCES list so extension parity tests
+# track the converter's public output surface, not only the core plugin.
+PLUGIN_COMMAND_DIRS = [
+    REPO_ROOT / "plugins" / "arckit-uae" / "commands",
+    REPO_ROOT / "plugins" / "arckit-fr" / "commands",
+    REPO_ROOT / "plugins" / "arckit-ca" / "commands",
+    REPO_ROOT / "plugins" / "arckit-eu" / "commands",
+    REPO_ROOT / "plugins" / "arckit-at" / "commands",
+    REPO_ROOT / "plugins" / "arckit-au" / "commands",
+    REPO_ROOT / "plugins" / "arckit-au-energy" / "commands",
+    REPO_ROOT / "plugins" / "arckit-us" / "commands",
+    REPO_ROOT / "plugins" / "arckit-uk-finance" / "commands",
+    REPO_ROOT / "plugins" / "arckit-uk-nhs" / "commands",
+    REPO_ROOT / "plugins" / "arckit-claude" / "commands",
+]
+CLAUDE_AGENTS = REPO_ROOT / "plugins" / "arckit-claude" / "agents"
+CLAUDE_ONLY_COMMANDS = {"build.md"}
+CLAUDE_ONLY_AGENT_FIELDS = {
+    "effort",
+    "initialPrompt",
+    "maxTurns",
+    "disallowedTools",
+    "tools",
+}
+
+
+def expected_command_names() -> set[str]:
+    names: set[str] = set()
+    for cmd_dir in PLUGIN_COMMAND_DIRS:
+        if not cmd_dir.is_dir():
+            continue
+        for path in cmd_dir.glob("*.md"):
+            if path.name in CLAUDE_ONLY_COMMANDS:
+                continue
+            names.add(path.stem)
+    return names
+
+
+def markdown_frontmatter(path: Path) -> tuple[dict, str]:
+    text = path.read_text(encoding="utf-8")
+    assert text.startswith("---"), f"{path} missing markdown frontmatter"
+    parts = text.split("---", 2)
+    assert len(parts) == 3, f"{path} has malformed markdown frontmatter"
+    return yaml.safe_load(parts[1]) or {}, parts[2].strip()
+
+
+def expected_agent_stems() -> set[str]:
+    names: set[str] = set()
+    for path in CLAUDE_AGENTS.glob("arckit-*.md"):
+        frontmatter, _body = markdown_frontmatter(path)
+        if frontmatter.get("subagent"):
+            continue
+        names.add(path.stem)
+    return names
+
+
+def assert_no_claude_placeholders(path: Path, text: str) -> None:
+    assert "${CLAUDE_PLUGIN_ROOT}" not in text, f"{path} contains Claude plugin root"
+    assert "${user_config." not in text, f"{path} contains Claude user_config placeholder"