Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 74 additions & 0 deletions .github/workflows/locomo-full-gate.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
name: locomo-full-gate

on:
pull_request:
branches:
- main
push:
branches:
- main
workflow_dispatch:

jobs:
full-gate:
runs-on: ubuntu-latest
timeout-minutes: 120
env:
CARGO_TERM_COLOR: always

steps:
- name: Checkout
uses: actions/checkout@v4

- name: Setup Rust
uses: dtolnay/rust-toolchain@stable

- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: "3.11"

- name: Setup Node
uses: actions/setup-node@v4
with:
node-version: "20"

- name: Enable Corepack
run: corepack enable

- name: Install Python dependencies
run: |
python -m venv .venv
./.venv/bin/python -m pip install -U pip pytest maturin
./.venv/bin/python -m pip install -e python

- name: Install Node dependencies
run: |
corepack pnpm --dir packages/node install --frozen-lockfile
corepack pnpm --dir packages/mcp install --frozen-lockfile

- name: Run LoCoMo full gate
shell: bash
run: |
set -euo pipefail
mkdir -p artifacts
REBUILD_PYTHON=0 tools/locomo/run-full-gate.sh 5 | tee artifacts/locomo-full-gate.log

base_dir="$(grep -Eo '\[full-gate\] BASE_DIR=.*' artifacts/locomo-full-gate.log | tail -n1 | sed 's/.*BASE_DIR=//')"
result_json="$(grep -Eo '\[full-gate\] RESULT_JSON=.*' artifacts/locomo-full-gate.log | tail -n1 | sed 's/.*RESULT_JSON=//')"

if [[ -n "${base_dir}" ]]; then
printf '%s\n' "${base_dir}" > artifacts/base_dir.txt
fi

if [[ -n "${result_json}" && -f "${result_json}" ]]; then
cp "${result_json}" artifacts/
fi

- name: Upload gate artifacts
if: always()
uses: actions/upload-artifact@v4
with:
name: locomo-full-gate
path: artifacts/
if-no-files-found: warn
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -201,6 +201,7 @@ tools/locomo/refresh-failure-samples.sh 5 80
- `run-regression-gate.sh` default threshold is `hit@5 >= 0.55` (second arg can override).
- `run-category-gate.sh` also checks default category floors for `multi-hop` and `open-domain`.
- `run-full-gate.sh` runs baseline threshold checks, drift checks, and the required 4-command matrix in one command.
- CI gate workflow `.github/workflows/locomo-full-gate.yml` runs `REBUILD_PYTHON=0 tools/locomo/run-full-gate.sh 5` on PRs and pushes to `main` (plus manual dispatch), and uploads gate artifacts.
- If LoCoMo scores unexpectedly remain old, rebuild local Python binding:
`./.venv/bin/python -m pip install -e python`

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# V2-R11 LoCoMo CI Full Gate Implementation Plan

> **For agentic workers:** REQUIRED: Use superpowers:subagent-driven-development (if subagents available) or superpowers:executing-plans to implement this plan. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Add CI automation so LoCoMo full-gate is enforced for PR/push events on `main`.

**Architecture:** Keep retrieval logic unchanged; add a GitHub Actions workflow that bootstraps Rust/Python/Node dependencies and executes existing `tools/locomo/run-full-gate.sh` as the single gate entrypoint, then uploads diagnostics artifacts.

**Tech Stack:** GitHub Actions YAML + existing Bash/Python tooling + README docs.

---

### Task 1: Add CI Workflow

**Files:**
- Create: `.github/workflows/locomo-full-gate.yml`

- [ ] Add workflow triggers for `pull_request` to `main`, `push` to `main`, and manual dispatch.
- [ ] Add environment bootstrap steps: Rust, Python `.venv`, Node/Corepack, and package installs.
- [ ] Run `REBUILD_PYTHON=0 tools/locomo/run-full-gate.sh 5` as canonical gate command.
- [ ] Capture and upload gate logs/result artifacts.

### Task 2: Update Documentation

**Files:**
- Modify: `tools/locomo/README.md`
- Modify: `README.md`

- [ ] Document CI workflow behavior and trigger scope.
- [ ] Keep local usage commands unchanged; position CI gate as complement to local validation.

### Task 3: Verification

**Files:**
- No source edits

- [ ] `cargo test --workspace`
- [ ] `./.venv/bin/pytest python/tests -q`
- [ ] `corepack pnpm --dir packages/node test`
- [ ] `corepack pnpm --dir packages/mcp test`
- [ ] `REBUILD_PYTHON=0 tools/locomo/run-full-gate.sh 5`

### Task 4: Finish Round

**Files:**
- Create: `docs/superpowers/releases/2026-03-19-v2-r11-locomo-ci-full-gate-results.md`

- [ ] Record verification evidence and CI workflow summary.
- [ ] Commit changes on feature branch.
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# V2-R11 LoCoMo CI Full Gate Results

## Summary

This round adds CI automation for the existing LoCoMo full gate. Retrieval logic is unchanged.

Added:

- GitHub Actions workflow to run full LoCoMo gate on PR/push to `main`
- workflow artifact upload for gate logs and result JSON
- README updates describing CI gate behavior
- spec + implementation plan docs for V2-R11

## New Files

- `.github/workflows/locomo-full-gate.yml`
- `docs/superpowers/specs/2026-03-19-v2-r11-locomo-ci-full-gate-design.md`
- `docs/superpowers/plans/2026-03-19-v2-r11-locomo-ci-full-gate-implementation-plan.md`

## LoCoMo Full Gate Evidence (K=5)

Command:

```bash
REBUILD_PYTHON=0 tools/locomo/run-full-gate.sh 5
```

Result:

- overall: `0.5822`
- multi-hop: `0.4255`
- open-domain: `0.3370`
- threshold checks: PASS
- drift checks: PASS
- required 4-command matrix: PASS

## Required Verification Matrix

All required commands passed in this round:

1. `cargo test --workspace`
2. `./.venv/bin/pytest python/tests -q`
3. `corepack pnpm --dir packages/node test`
4. `corepack pnpm --dir packages/mcp test`

## Outcome

- LoCoMo retrieval guardrail is now enforceable in CI.
- PRs to `main` can be blocked automatically when retrieval metrics or drift budgets regress.
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
# V2-R11 LoCoMo CI Full Gate Design

## Goal

Turn the current local LoCoMo full gate into an automatic CI guard so every PR can be blocked on retrieval regression risk, while preserving the current validated baseline:

- overall `hit@5 = 0.5822`
- multi-hop `hit@5 = 0.4255`
- open-domain `hit@5 = 0.3370`

## Problem

`tools/locomo/run-full-gate.sh` already gives a strong local quality gate, but it is still manually triggered. This leaves room for accidental merge of retrieval/tooling regressions when contributors skip or partially run checks.

## Options Considered

1. Keep local-only gate and rely on contributor discipline
- Lowest implementation cost, highest process risk.

2. Add a lightweight CI gate with only the 4-command matrix
- Faster CI, but does not protect LoCoMo retrieval baseline/drift.

3. Add GitHub Actions PR gate that runs full LoCoMo gate script (chosen)
- Slightly higher CI time, but directly enforces the retrieval baseline and required matrix in one canonical command.

## Chosen Design

### 1) New GitHub Actions workflow

Create `.github/workflows/locomo-full-gate.yml` with triggers:

- `pull_request` to `main`
- `push` to `main`
- `workflow_dispatch`

### 2) Standardized CI environment bootstrap

In workflow job:

- checkout repo
- setup Rust stable toolchain
- setup Python 3.11 + `.venv`
- install Python deps (`pip`, `pytest`, `maturin`, `pip install -e python`)
- setup Node 20 + Corepack
- install `packages/node` and `packages/mcp` deps with `pnpm`

### 3) Canonical gate execution in CI

Run:

```bash
REBUILD_PYTHON=0 tools/locomo/run-full-gate.sh 5
```

This preserves one source of truth for gate logic and includes:

- threshold checks
- category checks
- drift checks
- required 4-command matrix

### 4) Artifact retention for troubleshooting

Persist gate log and generated result JSON as workflow artifacts to make CI failures diagnosable without rerunning locally.

## Test Strategy

1. Local command-level verification:
- `cargo test --workspace`
- `./.venv/bin/pytest python/tests -q`
- `corepack pnpm --dir packages/node test`
- `corepack pnpm --dir packages/mcp test`

2. Local LoCoMo full gate dry run:
- `REBUILD_PYTHON=0 tools/locomo/run-full-gate.sh 5`

3. CI syntax and path validation via workflow file review and local repo checks.

## Success Criteria

1. Workflow exists and is valid at `.github/workflows/locomo-full-gate.yml`.
2. Workflow runs one-command LoCoMo full gate on PRs to `main`.
3. Gate artifacts (log + result JSON when present) are uploaded.
4. Required local 4-command matrix passes in this round.
16 changes: 16 additions & 0 deletions tools/locomo/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,3 +51,19 @@ Failure samples output path defaults to:
```bash
REBUILD_PYTHON=0 tools/locomo/run-full-gate.sh 5
```

## CI Gate

GitHub Actions workflow `.github/workflows/locomo-full-gate.yml` runs:

```bash
REBUILD_PYTHON=0 tools/locomo/run-full-gate.sh 5
```

Triggers:

- pull requests to `main`
- pushes to `main`
- manual `workflow_dispatch`

The workflow uploads gate logs (and result JSON when present) as artifacts.
Loading