From d1b32dc05a2835b431c2d88f1ff529983852b043 Mon Sep 17 00:00:00 2001 From: Cocoblood9527 Date: Thu, 19 Mar 2026 23:18:25 +0800 Subject: [PATCH] feat(ci): add locomo full-gate workflow for main --- .github/workflows/locomo-full-gate.yml | 74 ++++++++++++++++ README.md | 1 + ...locomo-ci-full-gate-implementation-plan.md | 49 +++++++++++ ...3-19-v2-r11-locomo-ci-full-gate-results.md | 49 +++++++++++ ...03-19-v2-r11-locomo-ci-full-gate-design.md | 84 +++++++++++++++++++ tools/locomo/README.md | 16 ++++ 6 files changed, 273 insertions(+) create mode 100644 .github/workflows/locomo-full-gate.yml create mode 100644 docs/superpowers/plans/2026-03-19-v2-r11-locomo-ci-full-gate-implementation-plan.md create mode 100644 docs/superpowers/releases/2026-03-19-v2-r11-locomo-ci-full-gate-results.md create mode 100644 docs/superpowers/specs/2026-03-19-v2-r11-locomo-ci-full-gate-design.md diff --git a/.github/workflows/locomo-full-gate.yml b/.github/workflows/locomo-full-gate.yml new file mode 100644 index 0000000..4a82b31 --- /dev/null +++ b/.github/workflows/locomo-full-gate.yml @@ -0,0 +1,74 @@ +name: locomo-full-gate + +on: + pull_request: + branches: + - main + push: + branches: + - main + workflow_dispatch: + +jobs: + full-gate: + runs-on: ubuntu-latest + timeout-minutes: 120 + env: + CARGO_TERM_COLOR: always + + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Setup Rust + uses: dtolnay/rust-toolchain@stable + + - name: Setup Python + uses: actions/setup-python@v5 + with: + python-version: "3.11" + + - name: Setup Node + uses: actions/setup-node@v4 + with: + node-version: "20" + + - name: Enable Corepack + run: corepack enable + + - name: Install Python dependencies + run: | + python -m venv .venv + ./.venv/bin/python -m pip install -U pip pytest maturin + ./.venv/bin/python -m pip install -e python + + - name: Install Node dependencies + run: | + corepack pnpm --dir packages/node install --frozen-lockfile + corepack pnpm --dir packages/mcp install --frozen-lockfile + + - name: Run LoCoMo full gate + shell: bash + run: | + set -euo pipefail + mkdir -p artifacts + REBUILD_PYTHON=0 tools/locomo/run-full-gate.sh 5 | tee artifacts/locomo-full-gate.log + + base_dir="$(grep -Eo '\[full-gate\] BASE_DIR=.*' artifacts/locomo-full-gate.log | tail -n1 | sed 's/.*BASE_DIR=//')" + result_json="$(grep -Eo '\[full-gate\] RESULT_JSON=.*' artifacts/locomo-full-gate.log | tail -n1 | sed 's/.*RESULT_JSON=//')" + + if [[ -n "${base_dir}" ]]; then + printf '%s\n' "${base_dir}" > artifacts/base_dir.txt + fi + + if [[ -n "${result_json}" && -f "${result_json}" ]]; then + cp "${result_json}" artifacts/ + fi + + - name: Upload gate artifacts + if: always() + uses: actions/upload-artifact@v4 + with: + name: locomo-full-gate + path: artifacts/ + if-no-files-found: warn diff --git a/README.md b/README.md index adf34c9..e1dca46 100644 --- a/README.md +++ b/README.md @@ -201,6 +201,7 @@ tools/locomo/refresh-failure-samples.sh 5 80 - `run-regression-gate.sh` default threshold is `hit@5 >= 0.55` (second arg can override). - `run-category-gate.sh` also checks default category floors for `multi-hop` and `open-domain`. - `run-full-gate.sh` runs baseline threshold checks, drift checks, and the required 4-command matrix in one command. +- CI gate workflow `.github/workflows/locomo-full-gate.yml` runs `REBUILD_PYTHON=0 tools/locomo/run-full-gate.sh 5` on PRs and pushes to `main` (plus manual dispatch), and uploads gate artifacts. - If LoCoMo scores unexpectedly remain old, rebuild local Python binding: `./.venv/bin/python -m pip install -e python` diff --git a/docs/superpowers/plans/2026-03-19-v2-r11-locomo-ci-full-gate-implementation-plan.md b/docs/superpowers/plans/2026-03-19-v2-r11-locomo-ci-full-gate-implementation-plan.md new file mode 100644 index 0000000..d186201 --- /dev/null +++ b/docs/superpowers/plans/2026-03-19-v2-r11-locomo-ci-full-gate-implementation-plan.md @@ -0,0 +1,49 @@ +# V2-R11 LoCoMo CI Full Gate Implementation Plan + +> **For agentic workers:** REQUIRED: Use superpowers:subagent-driven-development (if subagents available) or superpowers:executing-plans to implement this plan. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Add CI automation so LoCoMo full-gate is enforced for PR/push events on `main`. + +**Architecture:** Keep retrieval logic unchanged; add a GitHub Actions workflow that bootstraps Rust/Python/Node dependencies and executes existing `tools/locomo/run-full-gate.sh` as the single gate entrypoint, then uploads diagnostics artifacts. + +**Tech Stack:** GitHub Actions YAML + existing Bash/Python tooling + README docs. + +--- + +### Task 1: Add CI Workflow + +**Files:** +- Create: `.github/workflows/locomo-full-gate.yml` + +- [ ] Add workflow triggers for `pull_request` to `main`, `push` to `main`, and manual dispatch. +- [ ] Add environment bootstrap steps: Rust, Python `.venv`, Node/Corepack, and package installs. +- [ ] Run `REBUILD_PYTHON=0 tools/locomo/run-full-gate.sh 5` as canonical gate command. +- [ ] Capture and upload gate logs/result artifacts. + +### Task 2: Update Documentation + +**Files:** +- Modify: `tools/locomo/README.md` +- Modify: `README.md` + +- [ ] Document CI workflow behavior and trigger scope. +- [ ] Keep local usage commands unchanged; position CI gate as complement to local validation. + +### Task 3: Verification + +**Files:** +- No source edits + +- [ ] `cargo test --workspace` +- [ ] `./.venv/bin/pytest python/tests -q` +- [ ] `corepack pnpm --dir packages/node test` +- [ ] `corepack pnpm --dir packages/mcp test` +- [ ] `REBUILD_PYTHON=0 tools/locomo/run-full-gate.sh 5` + +### Task 4: Finish Round + +**Files:** +- Create: `docs/superpowers/releases/2026-03-19-v2-r11-locomo-ci-full-gate-results.md` + +- [ ] Record verification evidence and CI workflow summary. +- [ ] Commit changes on feature branch. diff --git a/docs/superpowers/releases/2026-03-19-v2-r11-locomo-ci-full-gate-results.md b/docs/superpowers/releases/2026-03-19-v2-r11-locomo-ci-full-gate-results.md new file mode 100644 index 0000000..a932e40 --- /dev/null +++ b/docs/superpowers/releases/2026-03-19-v2-r11-locomo-ci-full-gate-results.md @@ -0,0 +1,49 @@ +# V2-R11 LoCoMo CI Full Gate Results + +## Summary + +This round adds CI automation for the existing LoCoMo full gate. Retrieval logic is unchanged. + +Added: + +- GitHub Actions workflow to run full LoCoMo gate on PR/push to `main` +- workflow artifact upload for gate logs and result JSON +- README updates describing CI gate behavior +- spec + implementation plan docs for V2-R11 + +## New Files + +- `.github/workflows/locomo-full-gate.yml` +- `docs/superpowers/specs/2026-03-19-v2-r11-locomo-ci-full-gate-design.md` +- `docs/superpowers/plans/2026-03-19-v2-r11-locomo-ci-full-gate-implementation-plan.md` + +## LoCoMo Full Gate Evidence (K=5) + +Command: + +```bash +REBUILD_PYTHON=0 tools/locomo/run-full-gate.sh 5 +``` + +Result: + +- overall: `0.5822` +- multi-hop: `0.4255` +- open-domain: `0.3370` +- threshold checks: PASS +- drift checks: PASS +- required 4-command matrix: PASS + +## Required Verification Matrix + +All required commands passed in this round: + +1. `cargo test --workspace` +2. `./.venv/bin/pytest python/tests -q` +3. `corepack pnpm --dir packages/node test` +4. `corepack pnpm --dir packages/mcp test` + +## Outcome + +- LoCoMo retrieval guardrail is now enforceable in CI. +- PRs to `main` can be blocked automatically when retrieval metrics or drift budgets regress. diff --git a/docs/superpowers/specs/2026-03-19-v2-r11-locomo-ci-full-gate-design.md b/docs/superpowers/specs/2026-03-19-v2-r11-locomo-ci-full-gate-design.md new file mode 100644 index 0000000..892aefe --- /dev/null +++ b/docs/superpowers/specs/2026-03-19-v2-r11-locomo-ci-full-gate-design.md @@ -0,0 +1,84 @@ +# V2-R11 LoCoMo CI Full Gate Design + +## Goal + +Turn the current local LoCoMo full gate into an automatic CI guard so every PR can be blocked on retrieval regression risk, while preserving the current validated baseline: + +- overall `hit@5 = 0.5822` +- multi-hop `hit@5 = 0.4255` +- open-domain `hit@5 = 0.3370` + +## Problem + +`tools/locomo/run-full-gate.sh` already gives a strong local quality gate, but it is still manually triggered. This leaves room for accidental merge of retrieval/tooling regressions when contributors skip or partially run checks. + +## Options Considered + +1. Keep local-only gate and rely on contributor discipline +- Lowest implementation cost, highest process risk. + +2. Add a lightweight CI gate with only the 4-command matrix +- Faster CI, but does not protect LoCoMo retrieval baseline/drift. + +3. Add GitHub Actions PR gate that runs full LoCoMo gate script (chosen) +- Slightly higher CI time, but directly enforces the retrieval baseline and required matrix in one canonical command. + +## Chosen Design + +### 1) New GitHub Actions workflow + +Create `.github/workflows/locomo-full-gate.yml` with triggers: + +- `pull_request` to `main` +- `push` to `main` +- `workflow_dispatch` + +### 2) Standardized CI environment bootstrap + +In workflow job: + +- checkout repo +- setup Rust stable toolchain +- setup Python 3.11 + `.venv` +- install Python deps (`pip`, `pytest`, `maturin`, `pip install -e python`) +- setup Node 20 + Corepack +- install `packages/node` and `packages/mcp` deps with `pnpm` + +### 3) Canonical gate execution in CI + +Run: + +```bash +REBUILD_PYTHON=0 tools/locomo/run-full-gate.sh 5 +``` + +This preserves one source of truth for gate logic and includes: + +- threshold checks +- category checks +- drift checks +- required 4-command matrix + +### 4) Artifact retention for troubleshooting + +Persist gate log and generated result JSON as workflow artifacts to make CI failures diagnosable without rerunning locally. + +## Test Strategy + +1. Local command-level verification: +- `cargo test --workspace` +- `./.venv/bin/pytest python/tests -q` +- `corepack pnpm --dir packages/node test` +- `corepack pnpm --dir packages/mcp test` + +2. Local LoCoMo full gate dry run: +- `REBUILD_PYTHON=0 tools/locomo/run-full-gate.sh 5` + +3. CI syntax and path validation via workflow file review and local repo checks. + +## Success Criteria + +1. Workflow exists and is valid at `.github/workflows/locomo-full-gate.yml`. +2. Workflow runs one-command LoCoMo full gate on PRs to `main`. +3. Gate artifacts (log + result JSON when present) are uploaded. +4. Required local 4-command matrix passes in this round. diff --git a/tools/locomo/README.md b/tools/locomo/README.md index 1549953..dc3d02f 100644 --- a/tools/locomo/README.md +++ b/tools/locomo/README.md @@ -51,3 +51,19 @@ Failure samples output path defaults to: ```bash REBUILD_PYTHON=0 tools/locomo/run-full-gate.sh 5 ``` + +## CI Gate + +GitHub Actions workflow `.github/workflows/locomo-full-gate.yml` runs: + +```bash +REBUILD_PYTHON=0 tools/locomo/run-full-gate.sh 5 +``` + +Triggers: + +- pull requests to `main` +- pushes to `main` +- manual `workflow_dispatch` + +The workflow uploads gate logs (and result JSON when present) as artifacts.