From acaeb386f3ffd7355fe034fa2eb28b2f40b5b70c Mon Sep 17 00:00:00 2001 From: Eli Nimad Date: Wed, 15 Jul 2026 21:19:53 -0700 Subject: [PATCH 1/7] feat: add fail-closed test selection --- .no-mistakes.yaml | 13 +- CONTRIBUTING.md | 6 +- bin/fm-test-select.sh | 788 +++++++++++++++++++++++++++++++++++ tests/fm-test-select.test.sh | 660 +++++++++++++++++++++++++++++ 4 files changed, 1458 insertions(+), 9 deletions(-) create mode 100755 bin/fm-test-select.sh create mode 100755 tests/fm-test-select.test.sh diff --git a/.no-mistakes.yaml b/.no-mistakes.yaml index b95e69b41..df2980dfd 100644 --- a/.no-mistakes.yaml +++ b/.no-mistakes.yaml @@ -9,8 +9,8 @@ # HEAD-continuity guard; see docs/architecture.md "No-mistakes gate authority boundary." disable_project_settings: true -# Pin lint and the portable behavior suite to the same deterministic commands -# the Linux CI jobs run, instead of leaving them to no-mistakes' default handling. +# Pin lint to the deterministic command the Linux CI job runs, and load test +# policy only from target main instead of the untrusted branch under review. # CI separately owns platform-specific compatibility lanes, including the stock # macOS Bash snapshot checks. Without a configured # commands.lint, the gate's lint step never ran the deterministic @@ -19,13 +19,12 @@ disable_project_settings: true # them. commands.lint delegates to bin/fm-lint.sh, the single owner of the lint # definition that .github/workflows/ci.yml also invokes, so local can never # diverge from CI again (parity asserted by tests/fm-lint.test.sh). -# The test command mirrors the Linux behavior job in .github/workflows/ci.yml: -# iterate every tests/*.test.sh, run each, and fail the step if any one exits -# non-zero (an agent-driven test step has crashed the daemon). The e2e tests need -# tmux on PATH, which the firstmate environment provides. +# The target-main selector may shadow an exact focused mapping, but gate-shadow +# always runs the complete suite once and propagates every failure. The e2e tests +# need tmux on PATH, which the firstmate environment provides. commands: lint: 'bin/fm-lint.sh' - test: 'command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"' + test: 'umask 077; tmp_dir=$(mktemp -d "${TMPDIR:-/tmp}/fm-test-select-gate.XXXXXX") || exit 70; trap "rm -rf \"$tmp_dir\"" EXIT; trap "exit 70" HUP INT TERM; git clone --quiet --filter=blob:none --no-checkout --depth=1 --branch main https://github.com/xLabs-OS/firstmate "$tmp_dir/target" || exit 70; git -C "$tmp_dir/target" show HEAD:bin/fm-test-select.sh > "$tmp_dir/selector" || exit 70; chmod 500 "$tmp_dir/selector" || exit 70; bash "$tmp_dir/selector" gate-shadow' # Keep test evidence out of this repo; it stays in a temp dir instead. test: diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 4e7bd8154..f4b5d77b3 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -70,14 +70,16 @@ Check and test the toolbelt before pushing: ```sh for script in bin/*.sh bin/backends/*.sh; do bash -n "$script"; done # syntax-check the toolbelt bin/fm-lint.sh # lint the toolbelt and behavior tests; the single owner CI and the no-mistakes gate both run -for test_script in tests/*.test.sh; do bash "$test_script"; done # behavior tests, matching CI and no-mistakes commands.test +bin/fm-test-select.sh local # path-aware local feedback with fail-closed complete fallback [ "$(readlink CLAUDE.md)" = "AGENTS.md" ] [ "$(readlink .claude/skills)" = "../.agents/skills" ] tmp=$(mktemp -d) && printf 'done: smoke\n' > "$tmp/smoke.status" && FM_STATE_OVERRIDE="$tmp" FM_SIGNAL_GRACE=1 FM_POLL=1 FM_HEARTBEAT=999999 bin/fm-watch-arm.sh # watcher re-arm smoke test (prints arm status, then an actionable signal) ``` +Focused selection is local feedback only; no-mistakes still runs the complete suite once before delivery. +Running a specific test directly remains useful as supplementary feedback while developing its subject. Discover tests by listing `tests/*.test.sh`: each is a self-contained bash script named `.test.sh`, and its header comment describes what it covers, so run one directly to focus on a subject. -Tests that need a real optional backend or an explicit opt-in (real herdr/zellij/cmux smoke tests, the live Pi regression) skip themselves and print the tool or environment gate needed to enable them, so the run-all loop above is always safe. +Tests that need a real optional backend or an explicit opt-in (real herdr/zellij/cmux smoke tests, the live Pi regression) skip themselves and print the tool or environment gate needed to enable them, so the selector's complete fallback remains safe. ## Questions diff --git a/bin/fm-test-select.sh b/bin/fm-test-select.sh new file mode 100755 index 000000000..6a4746204 --- /dev/null +++ b/bin/fm-test-select.sh @@ -0,0 +1,788 @@ +#!/usr/bin/env bash +# fm-test-select.sh - fail-closed path-aware Firstmate test selection. +# +# This script is the single owner of the Phase 1 selection policy, execution +# order, snapshot checks, and terminal receipt. Focused execution is feedback, +# never final-gate authority: gate-shadow always runs the complete suite once, +# whether or not it first shadows an eligible focused plan. +# +# Public commands: +# fm-test-select.sh local focus an exact mapped contract change, else full +# fm-test-select.sh gate-shadow optionally shadow focus, then always run full +# fm-test-select.sh full run the complete suite without selecting focus +# fm-test-select.sh --help print this help +# +# The caller cannot supply a base, risk, path, test, skip, mapping, or policy. +# The selector fetches origin/main into a restrictive temporary object store so +# it does not update the checkout's refs, index, worktree, or configuration. +# +# Exit status: +# 0 requested tests passed, or local found no change +# 1 one or more requested tests failed, or shadow/full results mismatched +# 64 invalid usage +# 70 repository, trust, tool, base, or inventory failure blocked safe coverage +# 75 the repository or target snapshot changed during execution +set -u +set -o pipefail + +export LC_ALL=C +umask 077 + +POLICY_VERSION=instruction-scout-contract-v1 +TARGET_BASE_REF=origin/main +CONTRACT_BASE=54bb9c9d94ac3cc89d7e7d379a9a6eac95b01131 +# The backticks are literal Markdown from the one allowed trigger line. +# shellcheck disable=SC2016 +TRIGGER_LINE='- `scout-implementation-contract` - load before briefing a scout whose output may scope implementation, and before dispatching or promoting implementation from such a report.' + +FOCUS_TESTS=( + tests/fm-instruction-owners.test.sh + tests/fm-scout-implementation-contract.test.sh +) + +CONTRACT_DEPENDENCIES=( + .agents/skills/scout-implementation-contract/SKILL.md + AGENTS.md + tests/fm-instruction-owners.test.sh + tests/fm-scout-implementation-contract.test.sh + tests/lib.sh +) + +usage() { + cat <<'EOF' +usage: fm-test-select.sh {local|gate-shadow|full|--help} + +Run fail-closed Firstmate test selection. Focus is available only for the +exact instruction-scout-contract-v1 mapping; every unknown or unsafe change +falls back to the complete suite. gate-shadow always runs the complete suite. +EOF +} + +if [ "$#" -ne 1 ]; then + usage >&2 + exit 64 +fi + +case "$1" in + local|gate-shadow|full) CONTEXT=$1 ;; + --help) + usage + exit 0 + ;; + *) + usage >&2 + exit 64 + ;; +esac + +# Receipt fields default to documented unavailable/skipped values so failures +# never imply that an observation or a test run occurred. +TARGET_BASE_TIP=unavailable +MERGE_BASE=unavailable +HEAD_SHA=unavailable +WORKTREE_STATE=unavailable +DIFF_DIGEST=unavailable +DIFF_COUNT=0 +CLASSIFICATION=complete +REASON=unavailable +LOCAL_PLAN=full +GATE_PLAN=full +ORDERED_TESTS=none +FOCUS_EXECUTION=skipped +FOCUS_RESULTS=unavailable +FULL_EXECUTION=skipped +FULL_RESULTS=unavailable +COMPARISON=not-applicable +SNAPSHOT_STABILITY=unavailable +RESULT=error + +emit_receipt() { + printf '%s\n' \ + "firstmate.test-selection.v1 policy_version=$POLICY_VERSION context=$CONTEXT target_base_ref=$TARGET_BASE_REF target_base_tip=$TARGET_BASE_TIP merge_base=$MERGE_BASE head=$HEAD_SHA worktree_state=$WORKTREE_STATE diff_digest=$DIFF_DIGEST diff_count=$DIFF_COUNT classification=$CLASSIFICATION reason=$REASON local_plan=$LOCAL_PLAN gate_plan=$GATE_PLAN ordered_tests=$ORDERED_TESTS focus_execution=$FOCUS_EXECUTION focus_results=$FOCUS_RESULTS full_execution=$FULL_EXECUTION full_results=$FULL_RESULTS comparison=$COMPARISON snapshot_stability=$SNAPSHOT_STABILITY result=$RESULT" +} + +coverage_error() { + REASON=$1 + printf 'fm-test-select: %s\n' "$2" >&2 + RESULT=error + emit_receipt + exit 70 +} + +ROOT=$(git rev-parse --show-toplevel 2>/dev/null) || { + REASON=repository-unavailable + printf 'fm-test-select: a Git worktree is required\n' >&2 + emit_receipt + exit 70 +} +cd "$ROOT" 2>/dev/null || { + REASON=repository-unavailable + printf 'fm-test-select: the Git worktree is unavailable\n' >&2 + emit_receipt + exit 70 +} + +HEAD_SHA=$(git rev-parse --verify 'HEAD^{commit}' 2>/dev/null) \ + || coverage_error repository-head-unavailable "HEAD does not resolve to a commit" + +STATUS_OUTPUT=$(git status --porcelain=v1 --untracked-files=all 2>/dev/null) \ + || coverage_error repository-status-unavailable "cannot inspect repository status" +if [ -n "$STATUS_OUTPUT" ]; then + WORKTREE_STATE=dirty +else + WORKTREE_STATE=clean +fi + +if [ "$CONTEXT" = gate-shadow ] && [ "$WORKTREE_STATE" != clean ]; then + coverage_error gate-checkout-dirty "gate-shadow requires a clean committed checkout" +fi + +TMP_ROOT=$(mktemp -d "${TMPDIR:-/tmp}/fm-test-select.XXXXXX") \ + || coverage_error temporary-storage-unavailable "cannot create restrictive temporary storage" +chmod 700 "$TMP_ROOT" 2>/dev/null || { + rm -rf "$TMP_ROOT" + coverage_error temporary-storage-unavailable "cannot restrict temporary storage" +} +trap 'rm -rf "$TMP_ROOT"' EXIT +trap 'exit 70' HUP INT TERM + +hash_file() { + git hash-object --stdin < "$1" 2>/dev/null +} + +safe_path() { + local candidate=$1 + [ -n "$candidate" ] || return 1 + case "$candidate" in + /*|../*|*/../*|*/..|..|*$'\n'*|*$'\r'*|*$'\t'*) + return 1 + ;; + esac + if printf '%s' "$candidate" | grep -q '[[:cntrl:]]'; then + return 1 + fi + return 0 +} + +encode_path() { + local value=$1 + value=${value//'%'/'%25'} + value=${value//' '/'%20'} + value=${value//','/'%2C'} + value=${value//';'/'%3B'} + value=${value//'='/'%3D'} + value=${value//'['/'%5B'} + value=${value//']'/'%5D'} + printf '%s' "$value" +} + +join_encoded_tests() { + local joined='' test encoded + for test in "$@"; do + encoded=$(encode_path "$test") + if [ -n "$joined" ]; then + joined="$joined,$encoded" + else + joined=$encoded + fi + done + printf '%s' "$joined" +} + +# Fetch target main into a temporary bare repository. An alternate points at +# the checkout's existing object store, while the checkout later points at the +# temporary store for any newly fetched objects. No checkout ref is updated. +ORIGIN_URL= +TARGET_GIT="$TMP_ROOT/target.git" +TARGET_FETCHED=false +fetch_target() { + local object_dir + ORIGIN_URL=$(git remote get-url origin 2>/dev/null) || return 1 + [ -n "$ORIGIN_URL" ] || return 1 + git init --bare -q "$TARGET_GIT" >/dev/null 2>&1 || return 1 + object_dir=$(git rev-parse --path-format=absolute --git-path objects 2>/dev/null) \ + || return 1 + mkdir -p "$TARGET_GIT/objects/info" || return 1 + printf '%s\n' "$object_dir" > "$TARGET_GIT/objects/info/alternates" || return 1 + git -C "$TARGET_GIT" fetch --quiet --no-tags "$ORIGIN_URL" \ + '+refs/heads/main:refs/heads/main' >/dev/null 2>&1 || return 1 + TARGET_BASE_TIP=$(git -C "$TARGET_GIT" rev-parse --verify \ + 'refs/heads/main^{commit}' 2>/dev/null) || return 1 + case "$TARGET_BASE_TIP" in + *[!0-9a-f]*|'') return 1 ;; + esac + if [ -n "${GIT_ALTERNATE_OBJECT_DIRECTORIES:-}" ]; then + export GIT_ALTERNATE_OBJECT_DIRECTORIES="$TARGET_GIT/objects:$GIT_ALTERNATE_OBJECT_DIRECTORIES" + else + export GIT_ALTERNATE_OBJECT_DIRECTORIES="$TARGET_GIT/objects" + fi + TARGET_FETCHED=true + return 0 +} + +if ! fetch_target; then + TARGET_BASE_TIP=unavailable + REASON='target-fetch-unavailable' +fi + +# Inventory is the lexical filesystem set, not a caller-provided list. Every +# member must be a safe regular non-symlink before complete execution is safe. +TESTS=() +TEST_INVENTORY_DIGEST=unavailable +load_test_inventory() { + local test file_digest manifest="$TMP_ROOT/test-inventory" + TESTS=() + : > "$manifest" + shopt -s nullglob + TESTS=(tests/*.test.sh) + shopt -u nullglob + [ "${#TESTS[@]}" -gt 0 ] || return 1 + for test in "${TESTS[@]}"; do + safe_path "$test" || return 1 + [ -f "$test" ] && [ ! -L "$test" ] || return 1 + file_digest=$(git hash-object --stdin < "$test" 2>/dev/null) || return 1 + printf '%s\0%s\n' "$test" "$file_digest" >> "$manifest" || return 1 + done + TEST_INVENTORY_DIGEST=$(hash_file "$manifest") || return 1 + case "$TEST_INVENTORY_DIGEST" in + *[!0-9a-f]*|'') return 1 ;; + esac + return 0 +} + +load_test_inventory \ + || coverage_error unsafe-full-inventory "tests/*.test.sh must be a nonempty safe regular non-symlink inventory" + +FULL_TEST_LIST=$(join_encoded_tests "${TESTS[@]}") + +# Snapshot untracked names, types, and content without ever logging a path. +untracked_manifest() { + local output=$1 list="$TMP_ROOT/untracked-list" path digest target + git ls-files --others --exclude-standard -z -- > "$list" 2>/dev/null || return 1 + : > "$output" + while IFS= read -r -d '' path; do + printf '%s\0' "$path" >> "$output" || return 1 + if [ -L "$path" ]; then + target=$(readlink "$path" 2>/dev/null) || return 1 + digest=$(printf '%s' "$target" | git hash-object --stdin 2>/dev/null) || return 1 + printf 'symlink\0%s\n' "$digest" >> "$output" || return 1 + elif [ -f "$path" ]; then + digest=$(git hash-object --stdin < "$path" 2>/dev/null) || return 1 + printf 'regular\0%s\n' "$digest" >> "$output" || return 1 + else + printf 'other\0unavailable\n' >> "$output" || return 1 + fi + done < "$list" + return 0 +} + +snapshot_signature() { + local output=$1 index_file worktree_file untracked_file tests_file + local index_digest worktree_digest untracked_digest tests_digest snapshot_head + index_file="$TMP_ROOT/snapshot-index" + worktree_file="$TMP_ROOT/snapshot-worktree" + untracked_file="$TMP_ROOT/snapshot-untracked" + tests_file="$TMP_ROOT/snapshot-tests" + + snapshot_head=$(git rev-parse --verify 'HEAD^{commit}' 2>/dev/null) || return 1 + { + git ls-files --stage -z -- + git diff --cached --binary --full-index --no-ext-diff HEAD -- + } > "$index_file" 2>/dev/null || return 1 + git diff --binary --full-index --no-ext-diff -- > "$worktree_file" 2>/dev/null \ + || return 1 + untracked_manifest "$untracked_file" || return 1 + + : > "$tests_file" + local test file_digest + local -a current_tests + shopt -s nullglob + current_tests=(tests/*.test.sh) + shopt -u nullglob + [ "${#current_tests[@]}" -gt 0 ] || return 1 + for test in "${current_tests[@]}"; do + safe_path "$test" || return 1 + [ -f "$test" ] && [ ! -L "$test" ] || return 1 + file_digest=$(git hash-object --stdin < "$test" 2>/dev/null) || return 1 + printf '%s\0%s\n' "$test" "$file_digest" >> "$tests_file" || return 1 + done + + index_digest=$(hash_file "$index_file") || return 1 + worktree_digest=$(hash_file "$worktree_file") || return 1 + untracked_digest=$(hash_file "$untracked_file") || return 1 + tests_digest=$(hash_file "$tests_file") || return 1 + printf '%s\n' \ + "head=$snapshot_head" \ + "index=$index_digest" \ + "worktree=$worktree_digest" \ + "untracked=$untracked_digest" \ + "tests=$tests_digest" > "$output" || return 1 + return 0 +} + +INITIAL_SNAPSHOT="$TMP_ROOT/initial-snapshot" +snapshot_signature "$INITIAL_SNAPSHOT" \ + || coverage_error snapshot-unavailable "cannot capture a safe repository snapshot" +INITIAL_TEST_DIGEST=$(hash_file "$TMP_ROOT/snapshot-tests") \ + || coverage_error snapshot-unavailable "cannot verify the test inventory snapshot" +SNAPSHOT_HEAD=$(awk -F= '/^head=/ { print $2; exit }' "$INITIAL_SNAPSHOT") +SNAPSHOT_STATUS=$(git status --porcelain=v1 --untracked-files=all 2>/dev/null) \ + || coverage_error repository-status-unavailable "cannot recheck repository status" +if [ -n "$SNAPSHOT_STATUS" ]; then + SNAPSHOT_WORKTREE_STATE=dirty +else + SNAPSHOT_WORKTREE_STATE=clean +fi +if [ "$INITIAL_TEST_DIGEST" != "$TEST_INVENTORY_DIGEST" ] \ + || [ "$SNAPSHOT_HEAD" != "$HEAD_SHA" ] \ + || { [ "$CONTEXT" = gate-shadow ] && [ "$SNAPSHOT_WORKTREE_STATE" != clean ]; }; then + SNAPSHOT_STABILITY=changed + REASON=concurrent-snapshot-change + RESULT=error + emit_receipt + exit 75 +fi +WORKTREE_STATE=$SNAPSHOT_WORKTREE_STATE + +# Focus trust requires the fixed contract commit (or a successor with the exact +# dependency tree), the target base as an ancestor of HEAD, and no shallow or +# parser uncertainty. Any failure simply selects complete coverage. +FOCUS_TRUSTED=false +if "$TARGET_FETCHED"; then + if git -C "$TARGET_GIT" cat-file -e "$CONTRACT_BASE^{commit}" 2>/dev/null \ + && git -C "$TARGET_GIT" merge-base --is-ancestor \ + "$CONTRACT_BASE" "$TARGET_BASE_TIP" >/dev/null 2>&1 \ + && git -C "$TARGET_GIT" diff --quiet "$CONTRACT_BASE" "$TARGET_BASE_TIP" \ + -- "${CONTRACT_DEPENDENCIES[@]}" >/dev/null 2>&1 \ + && git -C "$TARGET_GIT" merge-base --is-ancestor \ + "$TARGET_BASE_TIP" "$HEAD_SHA" >/dev/null 2>&1; then + MERGE_BASE=$(git -C "$TARGET_GIT" merge-base "$TARGET_BASE_TIP" "$HEAD_SHA" 2>/dev/null) \ + || MERGE_BASE=unavailable + if [ "$MERGE_BASE" = "$TARGET_BASE_TIP" ]; then + FOCUS_TRUSTED=true + else + REASON='target-ancestry-unproven' + fi + else + REASON='target-contract-unproven' + fi +fi + +RAW_COMMITTED="$TMP_ROOT/diff-committed" +RAW_INDEX="$TMP_ROOT/diff-index" +RAW_WORKTREE="$TMP_ROOT/diff-worktree" +UNTRACKED_LIST="$TMP_ROOT/diff-untracked" +DIFF_BYTES="$TMP_ROOT/diff-bytes" +DIFF_AVAILABLE=false +if "$TARGET_FETCHED" \ + && git diff --raw -z --find-renames --no-ext-diff \ + "$TARGET_BASE_TIP" "$HEAD_SHA" -- > "$RAW_COMMITTED" 2>/dev/null \ + && git diff --cached --raw -z --find-renames --no-ext-diff \ + "$HEAD_SHA" -- > "$RAW_INDEX" 2>/dev/null \ + && git diff --raw -z --find-renames --no-ext-diff -- \ + > "$RAW_WORKTREE" 2>/dev/null \ + && git ls-files --others --exclude-standard -z -- \ + > "$UNTRACKED_LIST" 2>/dev/null; then + { + printf 'committed\0' + command cat "$RAW_COMMITTED" + printf 'index\0' + command cat "$RAW_INDEX" + printf 'worktree\0' + command cat "$RAW_WORKTREE" + printf 'untracked\0' + command cat "$UNTRACKED_LIST" + } > "$DIFF_BYTES" + DIFF_DIGEST=$(hash_file "$DIFF_BYTES") || DIFF_DIGEST=unavailable + DIFF_AVAILABLE=true +fi + +CHANGE_PATHS=() +CHANGE_STATUSES=() +CHANGE_OLD_MODES=() +CHANGE_NEW_MODES=() +PARSE_UNSAFE=false +PARSE_STRUCTURAL=false + +parse_raw_file() { + local raw_file=$1 meta first newmode status oldmode path oldpath + while IFS= read -r -d '' meta <&3; do + IFS=' ' read -r first newmode _ _ status < "$base_file" 2>/dev/null \ + || return 1 + count=$(grep -Fxc -- "$TRIGGER_LINE" "$base_file" 2>/dev/null) || count=0 + [ "$count" -eq 1 ] || return 1 + line=$(grep -Fnx -- "$TRIGGER_LINE" "$base_file" | cut -d: -f1) + base_lines=$(wc -l < "$base_file" | tr -d ' ') + awk -v line="$line" 'NR != line { print }' "$base_file" > "$base_without" || return 1 + git show "$HEAD_SHA:AGENTS.md" > "$head_file" 2>/dev/null || return 1 + git show :AGENTS.md > "$index_file" 2>/dev/null || return 1 + command cp AGENTS.md "$worktree_file" || return 1 + for state_file in "$head_file" "$index_file" "$worktree_file"; do + state_lines=$(wc -l < "$state_file" | tr -d ' ') + [ "$base_lines" = "$state_lines" ] || return 1 + state_without="$state_file.without" + awk -v line="$line" 'NR != line { print }' "$state_file" > "$state_without" \ + || return 1 + cmp -s "$base_without" "$state_without" || return 1 + done + return 0 +} + +classify_diff() { + local i path status oldmode newmode allowed_count=0 skill_seen=false + local agents_seen=false test_seen=false disallowed_reason= + if ! "$DIFF_AVAILABLE"; then + REASON=diff-unavailable + return 1 + fi + if ! parse_diff; then + REASON=diff-parse-uncertain + return 1 + fi + if "$PARSE_UNSAFE"; then + REASON=invalid-path + return 1 + fi + if "$PARSE_STRUCTURAL"; then + REASON=structural-change + return 1 + fi + if [ "$DIFF_COUNT" -eq 0 ]; then + CLASSIFICATION=no-change + REASON=no-change + LOCAL_PLAN=no-change + return 0 + fi + + i=0 + while [ "$i" -lt "${#CHANGE_PATHS[@]}" ]; do + path=${CHANGE_PATHS[$i]} + status=${CHANGE_STATUSES[$i]} + oldmode=${CHANGE_OLD_MODES[$i]} + newmode=${CHANGE_NEW_MODES[$i]} + case "$path" in + .agents/skills/scout-implementation-contract/SKILL.md) + if [ "$status" = M ] && [ "$oldmode" = 100644 ] \ + && [ "$newmode" = 100644 ] && [ -f "$path" ] && [ ! -L "$path" ] \ + && [ ! -x "$path" ]; then + skill_seen=true + allowed_count=$((allowed_count + 1)) + else + disallowed_reason=structural-change + fi + ;; + AGENTS.md) + if [ "$status" = M ] && [ "$oldmode" = 100644 ] \ + && [ "$newmode" = 100644 ] && [ -f "$path" ] && [ ! -L "$path" ] \ + && [ ! -x "$path" ] && agents_trigger_only; then + agents_seen=true + allowed_count=$((allowed_count + 1)) + else + disallowed_reason=agents-outside-trigger-region + fi + ;; + tests/fm-scout-implementation-contract.test.sh) + if { [ "$status" = M ] || [ "$status" = A ] || [ "$status" = '?' ]; } \ + && { [ "$oldmode" = 100755 ] || [ "$oldmode" = 000000 ]; } \ + && { [ "$newmode" = 100755 ] || [ "$newmode" = untracked ]; } \ + && [ -f "$path" ] && [ ! -L "$path" ] && [ -x "$path" ]; then + test_seen=true + allowed_count=$((allowed_count + 1)) + else + disallowed_reason=structural-change + fi + ;; + bin/fm-test-select.sh|tests/fm-test-select.test.sh) + disallowed_reason=selector-self-change + ;; + tests/lib.sh|tests/*-helpers.sh|tests/*-helper.sh) + disallowed_reason=shared-test-infra-change + ;; + .agents/*|CLAUDE.md|CONTRIBUTING.md|README.md|docs/*|skills/*) + disallowed_reason=unknown-instruction-change + ;; + bin/backends/*|bin/fm-backend.sh|bin/fm-spawn.sh|bin/fm-watch.sh|bin/fm-teardown.sh|state/*|config/*) + disallowed_reason=mixed-runtime-change + ;; + *) + disallowed_reason=unmapped-change + ;; + esac + i=$((i + 1)) + done + + if [ -n "$disallowed_reason" ] || [ "$allowed_count" -ne "$DIFF_COUNT" ]; then + REASON=${disallowed_reason:-mixed-change} + return 1 + fi + if "$test_seen" && [ "$CONTEXT" = gate-shadow ]; then + REASON=gate-mapped-test-change + return 1 + fi + + CLASSIFICATION=instruction-scout-contract-v1 + LOCAL_PLAN=focus + if "$skill_seen" && ! "$agents_seen" && ! "$test_seen"; then + REASON=eligible-skill + elif "$agents_seen" && ! "$skill_seen" && ! "$test_seen"; then + REASON=eligible-trigger + elif "$test_seen" && ! "$skill_seen" && ! "$agents_seen"; then + REASON=eligible-test-edit + else + REASON=eligible-contract-bundle + fi + return 0 +} + +if [ "$CONTEXT" = full ]; then + CLASSIFICATION=complete + REASON=explicit-full + LOCAL_PLAN=full +elif ! "$FOCUS_TRUSTED"; then + CLASSIFICATION=complete + LOCAL_PLAN=full + [ "$REASON" != unavailable ] || REASON='target-trust-unavailable' +else + classify_diff || { + CLASSIFICATION=complete + LOCAL_PLAN=full + } +fi + +if [ "$CONTEXT" = local ] && [ "$CLASSIFICATION" = no-change ]; then + ORDERED_TESTS=none + RESULT=no-change + FINAL_SNAPSHOT="$TMP_ROOT/final-snapshot" + snapshot_signature "$FINAL_SNAPSHOT" || { + SNAPSHOT_STABILITY=changed + RESULT=error + REASON=concurrent-snapshot-change + emit_receipt + exit 75 + } + if cmp -s "$INITIAL_SNAPSHOT" "$FINAL_SNAPSHOT"; then + SNAPSHOT_STABILITY=stable + emit_receipt + exit 0 + fi + SNAPSHOT_STABILITY=changed + RESULT=error + REASON=concurrent-snapshot-change + emit_receipt + exit 75 +fi + +FOCUS_CODES= +FULL_FOCUS_CODES= +FOCUS_FAILED=0 +FULL_FAILED=0 +FULL_PASSED=0 + +run_focus() { + local test rc + FOCUS_CODES= + FOCUS_FAILED=0 + if ! command -v bash >/dev/null 2>&1; then + FOCUS_EXECUTION=blocked + FOCUS_RESULTS=unavailable + FOCUS_FAILED=${#FOCUS_TESTS[@]} + return 70 + fi + for test in "${FOCUS_TESTS[@]}"; do + if [ ! -f "$test" ] || [ -L "$test" ]; then + rc=70 + else + printf '== %s ==\n' "$test" + rc=0 + bash "$test" || rc=$? + fi + if [ -n "$FOCUS_CODES" ]; then + FOCUS_CODES="$FOCUS_CODES,$rc" + else + FOCUS_CODES=$rc + fi + [ "$rc" -eq 0 ] || FOCUS_FAILED=$((FOCUS_FAILED + 1)) + done + FOCUS_RESULTS=$FOCUS_CODES + if [ "$FOCUS_FAILED" -eq 0 ]; then + FOCUS_EXECUTION=pass + else + FOCUS_EXECUTION=fail + fi +} + +run_full() { + local test rc + FULL_FAILED=0 + FULL_PASSED=0 + FULL_FOCUS_CODES= + if ! command -v bash >/dev/null 2>&1 || ! command -v tmux >/dev/null 2>&1; then + FULL_EXECUTION=blocked + FULL_RESULTS=unavailable + return 70 + fi + tmux -V || { + FULL_EXECUTION=blocked + FULL_RESULTS=unavailable + return 70 + } + for test in "${TESTS[@]}"; do + printf '== %s ==\n' "$test" + rc=0 + bash "$test" || rc=$? + if [ "$rc" -eq 0 ]; then + FULL_PASSED=$((FULL_PASSED + 1)) + else + FULL_FAILED=$((FULL_FAILED + 1)) + fi + case "$test" in + tests/fm-instruction-owners.test.sh|tests/fm-scout-implementation-contract.test.sh) + if [ -n "$FULL_FOCUS_CODES" ]; then + FULL_FOCUS_CODES="$FULL_FOCUS_CODES,$rc" + else + FULL_FOCUS_CODES=$rc + fi + ;; + esac + done + FULL_RESULTS="passed:$FULL_PASSED,failed:$FULL_FAILED" + if [ "$FULL_FAILED" -eq 0 ]; then + FULL_EXECUTION=pass + return 0 + fi + FULL_EXECUTION=fail + return 1 +} + +TEST_EXIT=0 +if [ "$CLASSIFICATION" = instruction-scout-contract-v1 ]; then + focus_rc=0 + run_focus || focus_rc=$? + ORDERED_TESTS="focus[$(join_encoded_tests "${FOCUS_TESTS[@]}")]" + if [ "$focus_rc" -eq 70 ]; then + TEST_EXIT=70 + REASON=bash-unavailable + elif [ "$FOCUS_FAILED" -ne 0 ]; then + TEST_EXIT=1 + fi +fi + +if [ "$CONTEXT" = gate-shadow ] || [ "$LOCAL_PLAN" = full ] || [ "$CONTEXT" = full ]; then + if [ "$ORDERED_TESTS" = none ]; then + ORDERED_TESTS="full[$FULL_TEST_LIST]" + else + ORDERED_TESTS="$ORDERED_TESTS;full[$FULL_TEST_LIST]" + fi + full_rc=0 + run_full || full_rc=$? + if [ "$full_rc" -eq 70 ]; then + TEST_EXIT=70 + REASON=complete-tools-unavailable + elif [ "$full_rc" -ne 0 ]; then + TEST_EXIT=1 + fi +fi + +if [ "$CONTEXT" = gate-shadow ] && [ "$FOCUS_EXECUTION" != skipped ]; then + if [ "$FOCUS_CODES" = "$FULL_FOCUS_CODES" ]; then + COMPARISON=match + else + COMPARISON=mismatch + TEST_EXIT=1 + fi +fi + +# Recheck the remote target independently from the temporary fetched ref. +TARGET_CHANGED=false +if "$TARGET_FETCHED"; then + latest_target=$(git ls-remote --heads "$ORIGIN_URL" refs/heads/main 2>/dev/null \ + | awk 'NR == 1 { print $1 }') || latest_target= + if [ -z "$latest_target" ] || [ "$latest_target" != "$TARGET_BASE_TIP" ]; then + TARGET_CHANGED=true + fi +fi + +FINAL_SNAPSHOT="$TMP_ROOT/final-snapshot" +if ! snapshot_signature "$FINAL_SNAPSHOT" \ + || ! cmp -s "$INITIAL_SNAPSHOT" "$FINAL_SNAPSHOT" \ + || "$TARGET_CHANGED"; then + SNAPSHOT_STABILITY=changed + RESULT=error + REASON=concurrent-snapshot-change + emit_receipt + exit 75 +fi +SNAPSHOT_STABILITY=stable + +case "$TEST_EXIT" in + 0) + RESULT=pass + emit_receipt + exit 0 + ;; + 1) + RESULT=fail + emit_receipt + exit 1 + ;; + 70) + RESULT=error + emit_receipt + exit 70 + ;; + *) + RESULT=error + REASON=execution-error + emit_receipt + exit 70 + ;; +esac diff --git a/tests/fm-test-select.test.sh b/tests/fm-test-select.test.sh new file mode 100755 index 000000000..dc0c88550 --- /dev/null +++ b/tests/fm-test-select.test.sh @@ -0,0 +1,660 @@ +#!/usr/bin/env bash +# Contract and behavior tests for fail-closed Firstmate test selection. +# +# Each behavior case runs the real selector against an isolated repository with +# a bare origin. The fixture retains the exact landed scout-contract dependency +# tree while reducing unrelated complete-suite entries to deterministic stubs. +# shellcheck disable=SC2016 +set -u + +# shellcheck source=tests/lib.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" + +SELECTOR="$ROOT/bin/fm-test-select.sh" +CONTRACT_BASE=54bb9c9d94ac3cc89d7e7d379a9a6eac95b01131 +TRIGGER_LINE='- `scout-implementation-contract` - load before briefing a scout whose output may scope implementation, and before dispatching or promoting implementation from such a report.' + +TMP= +REMOTE= +WORK= +OUT= +RC=0 +trap fm_test_cleanup EXIT + +usage() { + cat <<'EOF' +usage: fm-test-select.test.sh [case-name|--help] + +With no case name, run every selector test. A named case runs only that case. +EOF +} + +write_pass_test() { + local path=$1 label=$2 + cat > "$path" <> "\$FM_SELECTOR_TEST_LOG" +fi +printf 'ok - fixture $label\n' +EOF + chmod +x "$path" +} + +write_fail_test() { + local path=$1 label=$2 + cat > "$path" <> "\$FM_SELECTOR_TEST_LOG" +fi +printf 'not ok - fixture $label\n' >&2 +exit 1 +EOF + chmod +x "$path" +} + +setup_fixture() { + local test_file + if [ -n "$TMP" ]; then + rm -rf "$TMP" + fi + TMP=$(mktemp -d "${TMPDIR:-/tmp}/fm-test-select.XXXXXX") + FM_TEST_CLEANUP_DIRS=("$TMP") + REMOTE="$TMP/origin.git" + WORK="$TMP/work" + + git clone --quiet --bare "$ROOT" "$REMOTE" + git --git-dir="$REMOTE" update-ref refs/heads/main "$CONTRACT_BASE" + git --git-dir="$REMOTE" symbolic-ref HEAD refs/heads/main + git clone --quiet --branch main "$REMOTE" "$WORK" + fm_git_identity 'Selector Tests' 'selector@example.invalid' + + for test_file in "$WORK"/tests/*.test.sh; do + case "${test_file#"$WORK"/}" in + tests/fm-instruction-owners.test.sh|tests/fm-scout-implementation-contract.test.sh) ;; + *) git -C "$WORK" rm -q "${test_file#"$WORK"/}" ;; + esac + done + write_pass_test "$WORK/tests/a-pass.test.sh" a-pass + write_pass_test "$WORK/tests/z-pass.test.sh" z-pass + git -C "$WORK" add tests + git -C "$WORK" commit -qm 'test: slim selector fixture inventory' + git -C "$WORK" push -q origin main + git -C "$WORK" checkout -qb feature +} + +commit_all() { + git -C "$WORK" add -A + git -C "$WORK" commit -qm "${1:-test change}" +} + +run_selector() { + local mode=$1 + shift + RC=0 + OUT=$(cd "$WORK" && env "$@" "$SELECTOR" "$mode" 2>&1) || RC=$? +} + +receipt() { + printf '%s\n' "$OUT" | grep '^firstmate\.test-selection\.v1 ' | tail -1 +} + +assert_one_terminal_receipt() { + local count last + count=$(printf '%s\n' "$OUT" | grep -c '^firstmate\.test-selection\.v1 ') + [ "$count" -eq 1 ] || fail "expected one selector receipt, found $count"$'\n'"$OUT" + last=$(printf '%s\n' "$OUT" | tail -1) + case "$last" in + firstmate.test-selection.v1\ *) ;; + *) fail "selector receipt is not terminal"$'\n'"$OUT" ;; + esac +} + +assert_receipt_field() { + local field=$1 value=$2 line + line=$(receipt) + case " $line " in + *" $field=$value "*) ;; + *) fail "receipt missing $field=$value"$'\n'"$OUT" ;; + esac +} + +replace_trigger_line() { + local input="$WORK/AGENTS.md" output="$WORK/AGENTS.md.new" + awk -v trigger="$TRIGGER_LINE" ' + $0 == trigger { print trigger " changed"; next } + { print } + ' "$input" > "$output" + mv "$output" "$input" +} + +advance_target_with_full_failure() { + git -C "$WORK" checkout -q main + write_fail_test "$WORK/tests/a-pass.test.sh" a-pass + git -C "$WORK" add tests/a-pass.test.sh + git -C "$WORK" commit -qm 'test: target full failure' + git -C "$WORK" push -q origin main + git -C "$WORK" checkout -qB feature main +} + +advance_target_with_concurrent_test() { + git -C "$WORK" checkout -q main + cat > "$WORK/tests/z-pass.test.sh" <<'SH' +#!/usr/bin/env bash +set -eu +printf 'concurrent mutation\n' >> README.md +printf 'ok - fixture concurrent mutation\n' +SH + chmod +x "$WORK/tests/z-pass.test.sh" + git -C "$WORK" add tests/z-pass.test.sh + git -C "$WORK" commit -qm 'test: target concurrent mutation fixture' + git -C "$WORK" push -q origin main + git -C "$WORK" checkout -qB feature main +} + +test_eligible_skill() { + setup_fixture + printf '\n\n' >> "$WORK/.agents/skills/scout-implementation-contract/SKILL.md" + run_selector local + expect_code 0 "$RC" eligible-skill + assert_one_terminal_receipt + assert_receipt_field classification instruction-scout-contract-v1 + assert_receipt_field reason eligible-skill + assert_receipt_field local_plan focus + assert_receipt_field focus_execution pass + assert_receipt_field full_execution skipped + pass "eligible-skill" +} + +test_eligible_trigger() { + setup_fixture + replace_trigger_line + run_selector local + expect_code 0 "$RC" eligible-trigger + assert_one_terminal_receipt + assert_receipt_field classification instruction-scout-contract-v1 + assert_receipt_field reason eligible-trigger + assert_receipt_field focus_execution pass + assert_receipt_field full_execution skipped + pass "eligible-trigger" +} + +test_eligible_test_edit() { + setup_fixture + printf '\n# harmless selector fixture\n' >> "$WORK/tests/fm-scout-implementation-contract.test.sh" + run_selector local + expect_code 0 "$RC" eligible-test-edit-local + assert_receipt_field reason eligible-test-edit + assert_receipt_field focus_execution pass + commit_all 'test: edit mapped contract test' + run_selector gate-shadow + expect_code 0 "$RC" eligible-test-edit-gate + assert_receipt_field classification complete + assert_receipt_field reason gate-mapped-test-change + assert_receipt_field focus_execution skipped + assert_receipt_field full_execution pass + pass "eligible-test-edit" +} + +test_gate_shadow_match() { + setup_fixture + printf '\n\n' >> "$WORK/.agents/skills/scout-implementation-contract/SKILL.md" + commit_all 'test: eligible skill change' + run_selector gate-shadow + expect_code 0 "$RC" gate-shadow-match + assert_one_terminal_receipt + assert_receipt_field focus_execution pass + assert_receipt_field full_execution pass + assert_receipt_field comparison match + assert_receipt_field gate_plan full + pass "gate-shadow-match" +} + +test_explicit_full() { + setup_fixture + run_selector full + expect_code 0 "$RC" explicit-full + assert_one_terminal_receipt + assert_receipt_field classification complete + assert_receipt_field reason explicit-full + assert_receipt_field focus_execution skipped + assert_receipt_field full_execution pass + assert_contains "$OUT" "tmux " "explicit full did not print tmux -V" + pass "explicit-full" +} + +test_empty_diff() { + setup_fixture + run_selector local + expect_code 0 "$RC" empty-diff + assert_one_terminal_receipt + assert_receipt_field classification no-change + assert_receipt_field local_plan no-change + assert_receipt_field result no-change + assert_receipt_field focus_execution skipped + assert_receipt_field full_execution skipped + pass "empty-diff" +} + +test_local_dirty_inventory() { + setup_fixture + printf '\n\n' >> "$WORK/.agents/skills/scout-implementation-contract/SKILL.md" + commit_all 'test: committed delta' + printf '\nselector staged\n' >> "$WORK/README.md" + git -C "$WORK" add README.md + git -C "$WORK" show HEAD:README.md > "$WORK/README.md" + printf '\nselector worktree\n' >> "$WORK/CONTRIBUTING.md" + printf 'selector untracked\n' > "$WORK/local-note.txt" + run_selector local + expect_code 0 "$RC" local-dirty-inventory + assert_receipt_field worktree_state dirty + assert_receipt_field classification complete + assert_receipt_field full_execution pass + pass "local-dirty-inventory" +} + +test_stale_ancestry() { + local sibling tree + setup_fixture + tree=$(git -C "$WORK" rev-parse 'origin/main^{tree}') + sibling=$(printf 'sibling\n' | git -C "$WORK" commit-tree "$tree" -p "$CONTRACT_BASE") + git -C "$WORK" checkout -q --detach "$sibling" + run_selector local + expect_code 0 "$RC" stale-ancestry + assert_receipt_field classification complete + assert_receipt_field reason target-contract-unproven + assert_receipt_field full_execution pass + pass "stale-ancestry" +} + +test_unusual_safe_path() { + setup_fixture + printf 'safe unusual path\n' > "$WORK/unusual safe name.txt" + run_selector local + expect_code 0 "$RC" unusual-safe-path + assert_receipt_field classification complete + assert_receipt_field reason unmapped-change + assert_receipt_field full_execution pass + pass "unusual-safe-path" +} + +test_reject_caller_policy() { + local out rc + setup_fixture + rc=0 + out=$(cd "$WORK" && "$SELECTOR" local --base HEAD 2>&1) || rc=$? + expect_code 64 "$rc" reject-caller-base + assert_contains "$out" "usage:" "caller base input was not rejected with usage" + rc=0 + out=$(cd "$WORK" && "$SELECTOR" --test tests/a-pass.test.sh 2>&1) || rc=$? + expect_code 64 "$rc" reject-caller-test + rc=0 + out=$(cd "$WORK" && "$SELECTOR" -h 2>&1) || rc=$? + expect_code 64 "$rc" reject-undocumented-help-alias + pass "reject-caller-policy" +} + +test_agents_outside_region() { + setup_fixture + printf '\nOutside the one allowed trigger region.\n' >> "$WORK/AGENTS.md" + run_selector local + expect_code 0 "$RC" agents-outside-region + assert_receipt_field classification complete + assert_receipt_field reason agents-outside-trigger-region + assert_receipt_field full_execution pass + pass "agents-outside-region" +} + +test_mixed_runtime() { + setup_fixture + printf '\n\n' >> "$WORK/.agents/skills/scout-implementation-contract/SKILL.md" + printf '\n# selector runtime fixture\n' >> "$WORK/bin/fm-backend.sh" + run_selector local + expect_code 0 "$RC" mixed-runtime + assert_receipt_field classification complete + assert_receipt_field reason mixed-runtime-change + assert_receipt_field full_execution pass + pass "mixed-runtime" +} + +test_unknown_instruction() { + setup_fixture + printf '\nUnknown instruction change.\n' >> "$WORK/README.md" + run_selector local + expect_code 0 "$RC" unknown-instruction + assert_receipt_field classification complete + assert_receipt_field reason unknown-instruction-change + pass "unknown-instruction" +} + +test_shared_test_infra() { + setup_fixture + printf '\n# selector shared helper fixture\n' >> "$WORK/tests/lib.sh" + run_selector local + expect_code 0 "$RC" shared-test-infra + assert_receipt_field classification complete + assert_receipt_field reason shared-test-infra-change + assert_receipt_field full_execution pass + pass "shared-test-infra" +} + +test_selector_self_change() { + setup_fixture + cp "$SELECTOR" "$WORK/bin/fm-test-select.sh" + chmod +x "$WORK/bin/fm-test-select.sh" + run_selector local + expect_code 0 "$RC" selector-self-change + assert_receipt_field classification complete + assert_receipt_field reason selector-self-change + assert_receipt_field full_execution pass + pass "selector-self-change" +} + +test_structural_changes() { + setup_fixture + chmod -x "$WORK/tests/fm-scout-implementation-contract.test.sh" + run_selector local + expect_code 0 "$RC" structural-changes + assert_receipt_field classification complete + assert_receipt_field reason structural-change + assert_receipt_field full_execution pass + pass "structural-changes" +} + +test_invalid_paths() { + local invalid + setup_fixture + invalid=$'invalid\npath.txt' + printf 'invalid path fixture\n' > "$WORK/$invalid" + run_selector local + expect_code 0 "$RC" invalid-paths + assert_receipt_field classification complete + assert_receipt_field reason invalid-path + assert_receipt_field full_execution pass + assert_not_contains "$OUT" "invalid path.txt" "unsafe raw path leaked into output" + pass "invalid-paths" +} + +test_inventory_coverage() { + local log a_line b_line z_line count + setup_fixture + log="$TMP/order.log" + write_pass_test "$WORK/tests/b extra.test.sh" b-extra + run_selector local FM_SELECTOR_TEST_LOG="$log" + expect_code 0 "$RC" inventory-coverage + assert_receipt_field classification complete + assert_receipt_field full_execution pass + assert_contains "$(receipt)" 'tests/b%20extra.test.sh' "receipt did not encode the unusual safe test path" + count=$(wc -l < "$log" | tr -d ' ') + [ "$count" -eq 3 ] || fail "expected three logging fixture tests exactly once, got $count" + a_line=$(grep -n '^a-pass$' "$log" | cut -d: -f1) + b_line=$(grep -n '^b-extra$' "$log" | cut -d: -f1) + z_line=$(grep -n '^z-pass$' "$log" | cut -d: -f1) + [ "$a_line" -lt "$b_line" ] && [ "$b_line" -lt "$z_line" ] \ + || fail "complete inventory was not lexical"$'\n'"$(cat "$log")" + pass "inventory-coverage" +} + +test_gate_owner() { + local config="$ROOT/.no-mistakes.yaml" + assert_grep "disable_project_settings: true" "$config" "gate owner lost project-settings refusal" + assert_grep "lint: 'bin/fm-lint.sh'" "$config" "gate owner lost lint owner" + assert_grep "store_in_repo: false" "$config" "gate owner enabled repository evidence" + assert_grep "https://github.com/xLabs-OS/firstmate" "$config" "gate owner does not fetch authoritative target main" + assert_grep "show HEAD:bin/fm-test-select.sh" "$config" "gate owner does not read trusted selector content" + assert_grep 'bash "$tmp_dir/selector" gate-shadow' "$config" "gate owner does not invoke gate-shadow" + assert_no_grep "allow_repo_commands" "$config" "gate owner enabled branch repository commands" + pass "gate-owner" +} + +test_local_owner() { + local contributing="$ROOT/CONTRIBUTING.md" + assert_grep "bin/fm-test-select.sh local" "$contributing" "local guidance does not invoke selector" + assert_grep "Focused selection is local feedback" "$contributing" "local guidance overstates focused authority" + assert_grep "no-mistakes still runs the complete suite" "$contributing" "local guidance lost the complete gate backstop" + pass "local-owner" +} + +test_receipt_schema() { + local line field + setup_fixture + run_selector full + expect_code 0 "$RC" receipt-schema + assert_one_terminal_receipt + line=$(receipt) + for field in policy_version context target_base_ref target_base_tip merge_base head \ + worktree_state diff_digest diff_count classification reason local_plan gate_plan \ + ordered_tests focus_execution focus_results full_execution full_results comparison \ + snapshot_stability result; do + case " $line " in + *" $field="*) ;; + *) fail "receipt schema is missing $field"$'\n'"$line" ;; + esac + done + printf '%s\n' "$line" | grep -Eq ' target_base_tip=([0-9a-f]{40,64}|unavailable) ' \ + || fail "target base tip is not lowercase hex or unavailable" + printf '%s\n' "$line" | grep -Eq ' diff_digest=([0-9a-f]{40,64}|unavailable) ' \ + || fail "diff digest is not lowercase hex or unavailable" + assert_receipt_field snapshot_stability stable + assert_receipt_field result pass + pass "receipt-schema" +} + +test_idempotent() { + local first second + setup_fixture + run_selector full + expect_code 0 "$RC" idempotent-first + first=$(receipt) + run_selector full + expect_code 0 "$RC" idempotent-second + second=$(receipt) + [ "$first" = "$second" ] || fail "same snapshot produced different receipts"$'\n'"$first"$'\n'"$second" + pass "idempotent" +} + +test_full_order_and_aggregate() { + local log a_line z_line + setup_fixture + log="$TMP/order.log" + write_fail_test "$WORK/tests/a-pass.test.sh" a-pass + run_selector full FM_SELECTOR_TEST_LOG="$log" + expect_code 1 "$RC" full-order-and-aggregate + assert_receipt_field full_execution fail + assert_receipt_field result fail + assert_grep "a-pass" "$log" "failing first test did not run" + assert_grep "z-pass" "$log" "suite stopped before the later test" + a_line=$(grep -n '^a-pass$' "$log" | cut -d: -f1) + z_line=$(grep -n '^z-pass$' "$log" | cut -d: -f1) + [ "$a_line" -lt "$z_line" ] || fail "complete suite order was not lexical" + pass "full-order-and-aggregate" +} + +test_tmux_required_for_full() { + setup_fixture + run_selector full PATH=/usr/bin:/bin + expect_code 70 "$RC" tmux-required-for-full + assert_receipt_field full_execution blocked + assert_receipt_field reason complete-tools-unavailable + assert_receipt_field result error + pass "tmux-required-for-full" +} + +test_unsafe_full_inventory() { + setup_fixture + ln -s a-pass.test.sh "$WORK/tests/unsafe.test.sh" + run_selector full + expect_code 70 "$RC" unsafe-full-inventory + assert_receipt_field reason unsafe-full-inventory + assert_receipt_field full_execution skipped + pass "unsafe-full-inventory" +} + +test_fetch_failure() { + setup_fixture + git -C "$WORK" remote set-url origin "$TMP/missing-origin.git" + run_selector local + expect_code 0 "$RC" fetch-failure + assert_receipt_field target_base_tip unavailable + assert_receipt_field classification complete + assert_receipt_field reason target-fetch-unavailable + assert_receipt_field full_execution pass + pass "fetch-failure" +} + +test_shallow_no_base() { + local orphan tree + setup_fixture + tree=$(git -C "$WORK" rev-parse 'origin/main^{tree}') + orphan=$(printf 'orphan target\n' | git -C "$WORK" commit-tree "$tree") + git -C "$WORK" push -q --force origin "$orphan:refs/heads/main" + run_selector local + expect_code 0 "$RC" shallow-no-base + assert_receipt_field classification complete + assert_receipt_field reason target-contract-unproven + assert_receipt_field full_execution pass + pass "shallow-no-base" +} + +test_trusted_selector_missing() { + local config="$ROOT/.no-mistakes.yaml" + assert_grep 'git -C "$tmp_dir/target" show HEAD:bin/fm-test-select.sh > "$tmp_dir/selector"' "$config" \ + "trusted loader does not fail when target main lacks the selector" + assert_no_grep 'git show HEAD:bin/fm-test-select.sh' "$config" \ + "trusted loader can fall back to branch selector content" + pass "trusted-selector-missing" +} + +test_shadow_full_fails() { + setup_fixture + advance_target_with_full_failure + printf '\n\n' >> "$WORK/.agents/skills/scout-implementation-contract/SKILL.md" + commit_all 'test: eligible skill over failing target full suite' + run_selector gate-shadow + expect_code 1 "$RC" shadow-full-fails + assert_receipt_field focus_execution pass + assert_receipt_field full_execution fail + assert_receipt_field comparison match + assert_receipt_field result fail + pass "shadow-full-fails" +} + +test_shadow_focus_fails() { + setup_fixture + awk '!/single owner of Firstmate.s hardened scout report and implementation-packet contract/' \ + "$WORK/.agents/skills/scout-implementation-contract/SKILL.md" \ + > "$WORK/.agents/skills/scout-implementation-contract/SKILL.md.new" + mv "$WORK/.agents/skills/scout-implementation-contract/SKILL.md.new" \ + "$WORK/.agents/skills/scout-implementation-contract/SKILL.md" + commit_all 'test: eligible failing skill change' + run_selector gate-shadow + expect_code 1 "$RC" shadow-focus-fails + assert_receipt_field focus_execution fail + assert_receipt_field full_execution fail + assert_receipt_field comparison match + assert_receipt_field result fail + pass "shadow-focus-fails" +} + +test_concurrent_change() { + setup_fixture + advance_target_with_concurrent_test + run_selector full + expect_code 75 "$RC" concurrent-change + assert_one_terminal_receipt + assert_receipt_field snapshot_stability changed + assert_receipt_field reason concurrent-snapshot-change + assert_receipt_field result error + pass "concurrent-change" +} + +CASES=( + eligible-skill + eligible-trigger + eligible-test-edit + gate-shadow-match + explicit-full + empty-diff + local-dirty-inventory + stale-ancestry + unusual-safe-path + reject-caller-policy + agents-outside-region + mixed-runtime + unknown-instruction + shared-test-infra + selector-self-change + structural-changes + invalid-paths + inventory-coverage + gate-owner + local-owner + receipt-schema + idempotent + full-order-and-aggregate + tmux-required-for-full + unsafe-full-inventory + fetch-failure + shallow-no-base + trusted-selector-missing + shadow-full-fails + shadow-focus-fails + concurrent-change +) + +run_case() { + case "$1" in + eligible-skill) test_eligible_skill ;; + eligible-trigger) test_eligible_trigger ;; + eligible-test-edit) test_eligible_test_edit ;; + gate-shadow-match) test_gate_shadow_match ;; + explicit-full) test_explicit_full ;; + empty-diff) test_empty_diff ;; + local-dirty-inventory) test_local_dirty_inventory ;; + stale-ancestry) test_stale_ancestry ;; + unusual-safe-path) test_unusual_safe_path ;; + reject-caller-policy) test_reject_caller_policy ;; + agents-outside-region) test_agents_outside_region ;; + mixed-runtime) test_mixed_runtime ;; + unknown-instruction) test_unknown_instruction ;; + shared-test-infra) test_shared_test_infra ;; + selector-self-change) test_selector_self_change ;; + structural-changes) test_structural_changes ;; + invalid-paths) test_invalid_paths ;; + inventory-coverage) test_inventory_coverage ;; + gate-owner) test_gate_owner ;; + local-owner) test_local_owner ;; + receipt-schema) test_receipt_schema ;; + idempotent) test_idempotent ;; + full-order-and-aggregate) test_full_order_and_aggregate ;; + tmux-required-for-full) test_tmux_required_for_full ;; + unsafe-full-inventory) test_unsafe_full_inventory ;; + fetch-failure) test_fetch_failure ;; + shallow-no-base) test_shallow_no_base ;; + trusted-selector-missing) test_trusted_selector_missing ;; + shadow-full-fails) test_shadow_full_fails ;; + shadow-focus-fails) test_shadow_focus_fails ;; + concurrent-change) test_concurrent_change ;; + *) fail "unknown selector test case: $1" ;; + esac +} + +if [ "${1:-}" = --help ] || [ "${1:-}" = -h ]; then + usage + exit 0 +fi +if [ "$#" -gt 1 ]; then + usage >&2 + exit 64 +fi +if [ "$#" -eq 1 ]; then + run_case "$1" + exit 0 +fi + +for case_name in "${CASES[@]}"; do + run_case "$case_name" +done From aa1312f93d93e917ee266c9a50809860b9b46112 Mon Sep 17 00:00:00 2001 From: Eli Nimad Date: Wed, 15 Jul 2026 21:54:32 -0700 Subject: [PATCH 2/7] fix: harden test selector fail-closed gates --- .no-mistakes.yaml | 2 +- bin/fm-test-select.sh | 175 +++++++++++++++++++++++-------- tests/fm-test-select.test.sh | 195 ++++++++++++++++++++++++++++++++++- 3 files changed, 326 insertions(+), 46 deletions(-) diff --git a/.no-mistakes.yaml b/.no-mistakes.yaml index df2980dfd..11e19b448 100644 --- a/.no-mistakes.yaml +++ b/.no-mistakes.yaml @@ -24,7 +24,7 @@ disable_project_settings: true # need tmux on PATH, which the firstmate environment provides. commands: lint: 'bin/fm-lint.sh' - test: 'umask 077; tmp_dir=$(mktemp -d "${TMPDIR:-/tmp}/fm-test-select-gate.XXXXXX") || exit 70; trap "rm -rf \"$tmp_dir\"" EXIT; trap "exit 70" HUP INT TERM; git clone --quiet --filter=blob:none --no-checkout --depth=1 --branch main https://github.com/xLabs-OS/firstmate "$tmp_dir/target" || exit 70; git -C "$tmp_dir/target" show HEAD:bin/fm-test-select.sh > "$tmp_dir/selector" || exit 70; chmod 500 "$tmp_dir/selector" || exit 70; bash "$tmp_dir/selector" gate-shadow' + test: 'umask 077; tmp_dir=$(mktemp -d "${TMPDIR:-/tmp}/fm-test-select-gate.XXXXXX") || exit 70; cleanup() { rm -rf -- "$tmp_dir"; }; signal_exit() { exit 70; }; trap cleanup EXIT; trap signal_exit HUP INT TERM; git clone --quiet --filter=blob:none --no-checkout --depth=1 --branch main https://github.com/xLabs-OS/firstmate "$tmp_dir/target" || exit 70; git -C "$tmp_dir/target" show HEAD:bin/fm-test-select.sh > "$tmp_dir/selector" || exit 70; chmod 500 "$tmp_dir/selector" || exit 70; bash "$tmp_dir/selector" gate-shadow' # Keep test evidence out of this repo; it stays in a temp dir instead. test: diff --git a/bin/fm-test-select.sh b/bin/fm-test-select.sh index 6a4746204..ad935fb63 100755 --- a/bin/fm-test-select.sh +++ b/bin/fm-test-select.sh @@ -75,6 +75,12 @@ case "$1" in ;; esac +SELECTOR_SOURCE=${BASH_SOURCE[0]} +case "$SELECTOR_SOURCE" in + /*) ;; + *) SELECTOR_SOURCE="$(pwd -P)/$SELECTOR_SOURCE" ;; +esac + # Receipt fields default to documented unavailable/skipped values so failures # never imply that an observation or a test run occurred. TARGET_BASE_TIP=unavailable @@ -82,7 +88,7 @@ MERGE_BASE=unavailable HEAD_SHA=unavailable WORKTREE_STATE=unavailable DIFF_DIGEST=unavailable -DIFF_COUNT=0 +DIFF_COUNT=unavailable CLASSIFICATION=complete REASON=unavailable LOCAL_PLAN=full @@ -95,12 +101,37 @@ FULL_RESULTS=unavailable COMPARISON=not-applicable SNAPSHOT_STABILITY=unavailable RESULT=error +RECEIPT_EMITTED=false emit_receipt() { + if "$RECEIPT_EMITTED"; then + return 0 + fi + RECEIPT_EMITTED=true printf '%s\n' \ "firstmate.test-selection.v1 policy_version=$POLICY_VERSION context=$CONTEXT target_base_ref=$TARGET_BASE_REF target_base_tip=$TARGET_BASE_TIP merge_base=$MERGE_BASE head=$HEAD_SHA worktree_state=$WORKTREE_STATE diff_digest=$DIFF_DIGEST diff_count=$DIFF_COUNT classification=$CLASSIFICATION reason=$REASON local_plan=$LOCAL_PLAN gate_plan=$GATE_PLAN ordered_tests=$ORDERED_TESTS focus_execution=$FOCUS_EXECUTION focus_results=$FOCUS_RESULTS full_execution=$FULL_EXECUTION full_results=$FULL_RESULTS comparison=$COMPARISON snapshot_stability=$SNAPSHOT_STABILITY result=$RESULT" } +# shellcheck disable=SC2329 # Invoked by the signal trap below. +handle_signal() { + trap - HUP INT TERM + REASON=terminated-by-signal + RESULT=error + SNAPSHOT_STABILITY=unavailable + if [ "$FOCUS_EXECUTION" = running ]; then + FOCUS_EXECUTION=interrupted + FOCUS_RESULTS=unavailable + fi + if [ "$FULL_EXECUTION" = running ]; then + FULL_EXECUTION=interrupted + FULL_RESULTS=unavailable + fi + emit_receipt + exit 70 +} + +trap handle_signal HUP INT TERM + coverage_error() { REASON=$1 printf 'fm-test-select: %s\n' "$2" >&2 @@ -143,8 +174,11 @@ chmod 700 "$TMP_ROOT" 2>/dev/null || { rm -rf "$TMP_ROOT" coverage_error temporary-storage-unavailable "cannot restrict temporary storage" } -trap 'rm -rf "$TMP_ROOT"' EXIT -trap 'exit 70' HUP INT TERM +# shellcheck disable=SC2329 # Invoked by the EXIT trap below. +cleanup_selector() { + rm -rf -- "$TMP_ROOT" +} +trap cleanup_selector EXIT hash_file() { git hash-object --stdin < "$1" 2>/dev/null @@ -225,6 +259,27 @@ if ! fetch_target; then REASON='target-fetch-unavailable' fi +verify_gate_selector_identity() { + local target_selector="$TMP_ROOT/target-selector" running_hash target_hash + [ "$CONTEXT" = gate-shadow ] || return 0 + "$TARGET_FETCHED" \ + || coverage_error trusted-selector-unavailable "gate-shadow cannot verify target-main selector identity" + [ -f "$SELECTOR_SOURCE" ] && [ ! -L "$SELECTOR_SOURCE" ] \ + || coverage_error trusted-selector-unavailable "the running gate selector is not a regular file" + git -C "$TARGET_GIT" show \ + "$TARGET_BASE_TIP:bin/fm-test-select.sh" > "$target_selector" 2>/dev/null \ + || coverage_error trusted-selector-unavailable "target main does not contain the trusted selector" + running_hash=$(git hash-object --stdin < "$SELECTOR_SOURCE" 2>/dev/null) \ + || coverage_error trusted-selector-unavailable "cannot hash the running gate selector" + target_hash=$(git hash-object --stdin < "$target_selector" 2>/dev/null) \ + || coverage_error trusted-selector-unavailable "cannot hash the target-main selector" + if [ "$running_hash" != "$target_hash" ]; then + coverage_error trusted-selector-mismatch "the running gate selector does not match target main" + fi +} + +verify_gate_selector_identity + # Inventory is the lexical filesystem set, not a caller-provided list. Every # member must be a safe regular non-symlink before complete execution is safe. TESTS=() @@ -403,6 +458,7 @@ CHANGE_OLD_MODES=() CHANGE_NEW_MODES=() PARSE_UNSAFE=false PARSE_STRUCTURAL=false +DIFF_PARSED=false parse_raw_file() { local raw_file=$1 meta first newmode status oldmode path oldpath @@ -484,7 +540,7 @@ classify_diff() { REASON=diff-unavailable return 1 fi - if ! parse_diff; then + if ! "$DIFF_PARSED"; then REASON=diff-parse-uncertain return 1 fi @@ -583,6 +639,74 @@ classify_diff() { return 0 } +if "$DIFF_AVAILABLE" && parse_diff; then + DIFF_PARSED=true +else + DIFF_COUNT=unavailable +fi + +verify_final_stability() { + local final_snapshot="$TMP_ROOT/final-snapshot" target_recheck="$TMP_ROOT/target-recheck" + local latest_target='' latest_ref='' extra='' line_count=0 + local row_target row_ref row_extra + if ! snapshot_signature "$final_snapshot" \ + || ! cmp -s "$INITIAL_SNAPSHOT" "$final_snapshot"; then + SNAPSHOT_STABILITY=changed + RESULT=error + REASON=concurrent-snapshot-change + emit_receipt + exit 75 + fi + if "$TARGET_FETCHED"; then + if ! git ls-remote --heads "$ORIGIN_URL" refs/heads/main \ + > "$target_recheck" 2>/dev/null; then + SNAPSHOT_STABILITY=unavailable + RESULT=error + REASON='target-recheck-unavailable' + emit_receipt + exit 70 + fi + while IFS=$'\t' read -r row_target row_ref row_extra; do + line_count=$((line_count + 1)) + latest_target=$row_target + latest_ref=$row_ref + extra=$row_extra + done < "$target_recheck" + if [ "$line_count" -eq 0 ]; then + SNAPSHOT_STABILITY=changed + RESULT=error + REASON=concurrent-snapshot-change + emit_receipt + exit 75 + fi + if [ "$latest_ref" != refs/heads/main ] || [ -n "$extra" ] \ + || [ "$line_count" -ne 1 ]; then + SNAPSHOT_STABILITY=unavailable + RESULT=error + REASON='target-recheck-unavailable' + emit_receipt + exit 70 + fi + case "$latest_target" in + *[!0-9a-f]*|'') + SNAPSHOT_STABILITY=unavailable + RESULT=error + REASON='target-recheck-unavailable' + emit_receipt + exit 70 + ;; + esac + if [ "$latest_target" != "$TARGET_BASE_TIP" ]; then + SNAPSHOT_STABILITY=changed + RESULT=error + REASON=concurrent-snapshot-change + emit_receipt + exit 75 + fi + fi + SNAPSHOT_STABILITY=stable +} + if [ "$CONTEXT" = full ]; then CLASSIFICATION=complete REASON=explicit-full @@ -600,25 +724,10 @@ fi if [ "$CONTEXT" = local ] && [ "$CLASSIFICATION" = no-change ]; then ORDERED_TESTS=none + verify_final_stability RESULT=no-change - FINAL_SNAPSHOT="$TMP_ROOT/final-snapshot" - snapshot_signature "$FINAL_SNAPSHOT" || { - SNAPSHOT_STABILITY=changed - RESULT=error - REASON=concurrent-snapshot-change - emit_receipt - exit 75 - } - if cmp -s "$INITIAL_SNAPSHOT" "$FINAL_SNAPSHOT"; then - SNAPSHOT_STABILITY=stable - emit_receipt - exit 0 - fi - SNAPSHOT_STABILITY=changed - RESULT=error - REASON=concurrent-snapshot-change emit_receipt - exit 75 + exit 0 fi FOCUS_CODES= @@ -631,6 +740,7 @@ run_focus() { local test rc FOCUS_CODES= FOCUS_FAILED=0 + FOCUS_EXECUTION=running if ! command -v bash >/dev/null 2>&1; then FOCUS_EXECUTION=blocked FOCUS_RESULTS=unavailable @@ -665,6 +775,7 @@ run_full() { FULL_FAILED=0 FULL_PASSED=0 FULL_FOCUS_CODES= + FULL_EXECUTION=running if ! command -v bash >/dev/null 2>&1 || ! command -v tmux >/dev/null 2>&1; then FULL_EXECUTION=blocked FULL_RESULTS=unavailable @@ -741,27 +852,7 @@ if [ "$CONTEXT" = gate-shadow ] && [ "$FOCUS_EXECUTION" != skipped ]; then fi fi -# Recheck the remote target independently from the temporary fetched ref. -TARGET_CHANGED=false -if "$TARGET_FETCHED"; then - latest_target=$(git ls-remote --heads "$ORIGIN_URL" refs/heads/main 2>/dev/null \ - | awk 'NR == 1 { print $1 }') || latest_target= - if [ -z "$latest_target" ] || [ "$latest_target" != "$TARGET_BASE_TIP" ]; then - TARGET_CHANGED=true - fi -fi - -FINAL_SNAPSHOT="$TMP_ROOT/final-snapshot" -if ! snapshot_signature "$FINAL_SNAPSHOT" \ - || ! cmp -s "$INITIAL_SNAPSHOT" "$FINAL_SNAPSHOT" \ - || "$TARGET_CHANGED"; then - SNAPSHOT_STABILITY=changed - RESULT=error - REASON=concurrent-snapshot-change - emit_receipt - exit 75 -fi -SNAPSHOT_STABILITY=stable +verify_final_stability case "$TEST_EXIT" in 0) diff --git a/tests/fm-test-select.test.sh b/tests/fm-test-select.test.sh index dc0c88550..de864a113 100755 --- a/tests/fm-test-select.test.sh +++ b/tests/fm-test-select.test.sh @@ -80,7 +80,9 @@ setup_fixture() { done write_pass_test "$WORK/tests/a-pass.test.sh" a-pass write_pass_test "$WORK/tests/z-pass.test.sh" z-pass - git -C "$WORK" add tests + command cp "$SELECTOR" "$WORK/bin/fm-test-select.sh" + chmod +x "$WORK/bin/fm-test-select.sh" + git -C "$WORK" add tests bin/fm-test-select.sh git -C "$WORK" commit -qm 'test: slim selector fixture inventory' git -C "$WORK" push -q origin main git -C "$WORK" checkout -qb feature @@ -155,6 +157,45 @@ SH git -C "$WORK" checkout -qB feature main } +advance_target_selector_identity() { + git -C "$WORK" checkout -q main + printf '\n# changed target selector identity\n' >> "$WORK/bin/fm-test-select.sh" + git -C "$WORK" add bin/fm-test-select.sh + git -C "$WORK" commit -qm 'test: change target selector identity' + git -C "$WORK" push -q origin main + git -C "$WORK" checkout -qB feature main +} + +advance_target_with_final_marker() { + git -C "$WORK" checkout -q main + cat > "$WORK/tests/z-pass.test.sh" <<'SH' +#!/usr/bin/env bash +set -eu +: > "${FM_SELECTOR_FINAL_MARKER:?}" +printf 'ok - fixture final snapshot marker\n' +SH + chmod +x "$WORK/tests/z-pass.test.sh" + git -C "$WORK" add tests/z-pass.test.sh + git -C "$WORK" commit -qm 'test: add final snapshot marker' + git -C "$WORK" push -q origin main + git -C "$WORK" checkout -qB feature main +} + +advance_target_with_signal_test() { + git -C "$WORK" checkout -q main + cat > "$WORK/tests/00-signal.test.sh" <<'SH' +#!/usr/bin/env bash +set -eu +kill -TERM "$PPID" +sleep 1 +SH + chmod +x "$WORK/tests/00-signal.test.sh" + git -C "$WORK" add tests/00-signal.test.sh + git -C "$WORK" commit -qm 'test: add selector signal fixture' + git -C "$WORK" push -q origin main + git -C "$WORK" checkout -qB feature main +} + test_eligible_skill() { setup_fixture printf '\n\n' >> "$WORK/.agents/skills/scout-implementation-contract/SKILL.md" @@ -343,8 +384,7 @@ test_shared_test_infra() { test_selector_self_change() { setup_fixture - cp "$SELECTOR" "$WORK/bin/fm-test-select.sh" - chmod +x "$WORK/bin/fm-test-select.sh" + printf '\n# selector self-change fixture\n' >> "$WORK/bin/fm-test-select.sh" run_selector local expect_code 0 "$RC" selector-self-change assert_receipt_field classification complete @@ -571,6 +611,145 @@ test_concurrent_change() { pass "concurrent-change" } +test_gate_selector_identity() { + setup_fixture + advance_target_selector_identity + printf '\n\n' \ + >> "$WORK/.agents/skills/scout-implementation-contract/SKILL.md" + commit_all 'test: eligible change over new target selector' + run_selector gate-shadow + expect_code 70 "$RC" gate-selector-identity + assert_one_terminal_receipt + assert_receipt_field reason trusted-selector-mismatch + assert_receipt_field focus_execution skipped + assert_receipt_field full_execution skipped + assert_receipt_field result error + pass "gate-selector-identity" +} + +test_target_moves_during_final_snapshot() { + local marker moved target_tip tree mover_pid mover_rc=0 current i=1 + setup_fixture + advance_target_with_final_marker + mkdir -p "$WORK/bulk-final-snapshot" + while [ "$i" -le 400 ]; do + printf 'final snapshot inventory %04d\n' "$i" \ + > "$WORK/bulk-final-snapshot/item-$i.txt" + i=$((i + 1)) + done + target_tip=$(git -C "$WORK" rev-parse main) + tree=$(git -C "$WORK" rev-parse 'main^{tree}') + moved=$(printf 'moved target\n' | git -C "$WORK" commit-tree "$tree" -p "$target_tip") + git -C "$WORK" push -q origin "$moved:refs/heads/selector-moved" + marker="$TMP/final-snapshot.marker" + ( + attempts=0 + while [ ! -e "$marker" ] && [ "$attempts" -lt 500 ]; do + sleep 0.01 + attempts=$((attempts + 1)) + done + [ -e "$marker" ] || exit 1 + sleep 0.1 + git --git-dir="$REMOTE" update-ref refs/heads/main "$moved" + ) & + mover_pid=$! + run_selector full FM_SELECTOR_FINAL_MARKER="$marker" + wait "$mover_pid" || mover_rc=$? + [ "$mover_rc" -eq 0 ] || fail "target mover did not complete" + current=$(git --git-dir="$REMOTE" rev-parse refs/heads/main) + [ "$current" = "$moved" ] || fail "target main did not move during the fixture" + expect_code 75 "$RC" target-moves-during-final-snapshot + assert_one_terminal_receipt + assert_receipt_field snapshot_stability changed + assert_receipt_field reason concurrent-snapshot-change + assert_receipt_field result error + assert_not_contains "$(receipt)" "result=pass" "target movement emitted a pass receipt" + pass "target-moves-during-final-snapshot" +} + +test_signal_receipt_cleanup() { + local selector_tmp remaining + setup_fixture + advance_target_with_signal_test + selector_tmp="$TMP/selector temp [safe]" + mkdir -p "$selector_tmp" + run_selector full TMPDIR="$selector_tmp" + expect_code 70 "$RC" signal-receipt-cleanup + assert_one_terminal_receipt + assert_receipt_field reason terminated-by-signal + assert_receipt_field full_execution interrupted + assert_receipt_field snapshot_stability unavailable + assert_receipt_field result error + remaining=$(find "$selector_tmp" -mindepth 1 -print -quit) + [ -z "$remaining" ] || fail "signal exit left selector temporary state behind" + pass "signal-receipt-cleanup" +} + +test_explicit_full_dirty_count() { + setup_fixture + printf '\n\n' \ + >> "$WORK/.agents/skills/scout-implementation-contract/SKILL.md" + commit_all 'test: explicit full committed layer' + printf '\nexplicit full index layer\n' >> "$WORK/README.md" + git -C "$WORK" add README.md + printf '\nexplicit full worktree layer\n' >> "$WORK/CONTRIBUTING.md" + printf 'explicit full untracked layer\n' > "$WORK/explicit-full-untracked.txt" + run_selector full + expect_code 0 "$RC" explicit-full-dirty-count + assert_one_terminal_receipt + assert_receipt_field worktree_state dirty + assert_receipt_field classification complete + assert_receipt_field reason explicit-full + assert_receipt_field diff_count 4 + assert_receipt_field full_execution pass + assert_receipt_field result pass + pass "explicit-full-dirty-count" +} + +test_loader_cleanup_safety() { + local config="$ROOT/.no-mistakes.yaml" loader fakebin loader_root keep rc=0 remaining + setup_fixture + assert_grep 'cleanup() { rm -rf -- "$tmp_dir"; }' "$config" \ + "trusted loader cleanup is not a quoted static function" + assert_grep 'trap cleanup EXIT' "$config" "trusted loader does not install the static cleanup trap" + assert_no_grep 'trap "rm -rf' "$config" "trusted loader reparses an expanded cleanup path" + loader=$(sed -n "s/^ test: '\(.*\)'$/\1/p" "$config") + [ -n "$loader" ] || fail "could not extract trusted loader command" + fakebin="$TMP/loader-fakebin" + loader_root="$TMP/nontrivial safe [loader root]" + keep="$TMP/keep me" + mkdir -p "$fakebin" "$loader_root" "$keep" + cat > "$fakebin/git" <<'SH' +#!/bin/bash +set -eu +case "${1:-}" in + clone) + last= + for arg in "$@"; do + last=$arg + done + mkdir -p "$last" + ;; + -C) + [ "${3:-}" = show ] || exit 2 + printf '#!/usr/bin/env bash\nexit 0\n' + ;; + *) exit 2 ;; +esac +SH + cat > "$fakebin/bash" <<'SH' +#!/bin/bash +exit 17 +SH + chmod +x "$fakebin/git" "$fakebin/bash" + PATH="$fakebin:$PATH" TMPDIR="$loader_root" /bin/sh -c "$loader" || rc=$? + expect_code 17 "$rc" loader-child-status + assert_present "$keep" "trusted loader cleanup removed a sibling path" + remaining=$(find "$loader_root" -mindepth 1 -print -quit) + [ -z "$remaining" ] || fail "trusted loader did not remove only its created directory" + pass "loader-cleanup-safety" +} + CASES=( eligible-skill eligible-trigger @@ -603,6 +782,11 @@ CASES=( shadow-full-fails shadow-focus-fails concurrent-change + gate-selector-identity + target-moves-during-final-snapshot + signal-receipt-cleanup + explicit-full-dirty-count + loader-cleanup-safety ) run_case() { @@ -638,6 +822,11 @@ run_case() { shadow-full-fails) test_shadow_full_fails ;; shadow-focus-fails) test_shadow_focus_fails ;; concurrent-change) test_concurrent_change ;; + gate-selector-identity) test_gate_selector_identity ;; + target-moves-during-final-snapshot) test_target_moves_during_final_snapshot ;; + signal-receipt-cleanup) test_signal_receipt_cleanup ;; + explicit-full-dirty-count) test_explicit_full_dirty_count ;; + loader-cleanup-safety) test_loader_cleanup_safety ;; *) fail "unknown selector test case: $1" ;; esac } From 240e51308decfe27b7dabe28135bd9c3c23086aa Mon Sep 17 00:00:00 2001 From: Eli Nimad Date: Wed, 15 Jul 2026 22:41:43 -0700 Subject: [PATCH 3/7] no-mistakes(test): Fix selector bootstrap gate --- .no-mistakes.yaml | 8 +-- tests/fm-test-select.test.sh | 94 ++++++++++++++++++++++++++++++++++-- 2 files changed, 94 insertions(+), 8 deletions(-) diff --git a/.no-mistakes.yaml b/.no-mistakes.yaml index 11e19b448..f799f762f 100644 --- a/.no-mistakes.yaml +++ b/.no-mistakes.yaml @@ -20,11 +20,13 @@ disable_project_settings: true # definition that .github/workflows/ci.yml also invokes, so local can never # diverge from CI again (parity asserted by tests/fm-lint.test.sh). # The target-main selector may shadow an exact focused mapping, but gate-shadow -# always runs the complete suite once and propagates every failure. The e2e tests -# need tmux on PATH, which the firstmate environment provides. +# always runs the complete suite once and propagates every failure. +# A selector-introducing change has no target-main selector to load yet, so its +# bootstrap path runs this fixed complete-suite command once instead. +# The e2e tests need tmux on PATH, which the firstmate environment provides. commands: lint: 'bin/fm-lint.sh' - test: 'umask 077; tmp_dir=$(mktemp -d "${TMPDIR:-/tmp}/fm-test-select-gate.XXXXXX") || exit 70; cleanup() { rm -rf -- "$tmp_dir"; }; signal_exit() { exit 70; }; trap cleanup EXIT; trap signal_exit HUP INT TERM; git clone --quiet --filter=blob:none --no-checkout --depth=1 --branch main https://github.com/xLabs-OS/firstmate "$tmp_dir/target" || exit 70; git -C "$tmp_dir/target" show HEAD:bin/fm-test-select.sh > "$tmp_dir/selector" || exit 70; chmod 500 "$tmp_dir/selector" || exit 70; bash "$tmp_dir/selector" gate-shadow' + test: 'umask 077; tmp_dir=$(mktemp -d "${TMPDIR:-/tmp}/fm-test-select-gate.XXXXXX") || exit 70; cleanup() { rm -rf -- "$tmp_dir"; }; signal_exit() { exit 70; }; trap cleanup EXIT; trap signal_exit HUP INT TERM; git clone --quiet --filter=blob:none --no-checkout --depth=1 --branch main https://github.com/xLabs-OS/firstmate "$tmp_dir/target" || exit 70; target_selector=$(git -C "$tmp_dir/target" ls-tree -r --name-only HEAD -- bin/fm-test-select.sh) || exit 70; if [ "$target_selector" = bin/fm-test-select.sh ]; then git -C "$tmp_dir/target" show HEAD:bin/fm-test-select.sh > "$tmp_dir/selector" || exit 70; chmod 500 "$tmp_dir/selector" || exit 70; bash "$tmp_dir/selector" gate-shadow; elif [ -z "$target_selector" ]; then printf "%s\\n" "fm-test-select: target main has no selector; running complete bootstrap suite" >&2; command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"; else exit 70; fi' # Keep test evidence out of this repo; it stays in a temp dir instead. test: diff --git a/tests/fm-test-select.test.sh b/tests/fm-test-select.test.sh index de864a113..fc722b84d 100755 --- a/tests/fm-test-select.test.sh +++ b/tests/fm-test-select.test.sh @@ -444,8 +444,12 @@ test_gate_owner() { assert_grep "lint: 'bin/fm-lint.sh'" "$config" "gate owner lost lint owner" assert_grep "store_in_repo: false" "$config" "gate owner enabled repository evidence" assert_grep "https://github.com/xLabs-OS/firstmate" "$config" "gate owner does not fetch authoritative target main" + assert_grep 'ls-tree -r --name-only HEAD -- bin/fm-test-select.sh' "$config" \ + "gate owner does not distinguish a missing selector from target read failure" assert_grep "show HEAD:bin/fm-test-select.sh" "$config" "gate owner does not read trusted selector content" assert_grep 'bash "$tmp_dir/selector" gate-shadow' "$config" "gate owner does not invoke gate-shadow" + assert_grep 'target main has no selector; running complete bootstrap suite' "$config" \ + "gate owner lacks the selector-bootstrap complete-suite fallback" assert_no_grep "allow_repo_commands" "$config" "gate owner enabled branch repository commands" pass "gate-owner" } @@ -560,14 +564,89 @@ test_shallow_no_base() { } test_trusted_selector_missing() { - local config="$ROOT/.no-mistakes.yaml" - assert_grep 'git -C "$tmp_dir/target" show HEAD:bin/fm-test-select.sh > "$tmp_dir/selector"' "$config" \ - "trusted loader does not fail when target main lacks the selector" + local config="$ROOT/.no-mistakes.yaml" loader fakebin fixture marker rc=0 + setup_fixture + loader=$(sed -n "s/^ test: '\(.*\)'$/\1/p" "$config") + [ -n "$loader" ] || fail "could not extract trusted loader command" + fakebin="$TMP/selector-missing-fakebin" + fixture="$TMP/selector missing bootstrap fixture" + marker="$TMP/selector-missing.marker" + mkdir -p "$fakebin" "$fixture/tests" + write_pass_test "$fixture/tests/bootstrap.test.sh" bootstrap + cat > "$fakebin/git" <<'SH' +#!/bin/bash +set -eu +case "${1:-}" in + clone) + last= + for arg in "$@"; do + last=$arg + done + mkdir -p "$last" + ;; + -C) + [ "${3:-}" = ls-tree ] || exit 2 + exit 0 + ;; + *) exit 2 ;; +esac +SH + cat > "$fakebin/tmux" <<'SH' +#!/bin/bash +set -eu +[ "${1:-}" = -V ] || exit 2 +printf 'tmux fixture\n' +SH + chmod +x "$fakebin/git" "$fakebin/tmux" + (cd "$fixture" && FM_SELECTOR_TEST_LOG="$marker" PATH="$fakebin:$PATH" /bin/sh -c "$loader") || rc=$? + expect_code 0 "$rc" trusted-selector-missing + assert_present "$marker" "bootstrap fallback did not run the complete suite" + assert_grep 'bootstrap' "$marker" "bootstrap fallback did not run its test" + assert_grep 'for t in tests/*.test.sh' "$config" \ + "bootstrap fallback does not own the complete test inventory" assert_no_grep 'git show HEAD:bin/fm-test-select.sh' "$config" \ "trusted loader can fall back to branch selector content" pass "trusted-selector-missing" } +test_trusted_selector_read_failure() { + local config="$ROOT/.no-mistakes.yaml" loader fakebin fixture marker rc=0 + setup_fixture + loader=$(sed -n "s/^ test: '\(.*\)'$/\1/p" "$config") + [ -n "$loader" ] || fail "could not extract trusted loader command" + fakebin="$TMP/selector-read-failure-fakebin" + fixture="$TMP/selector read failure fixture" + marker="$TMP/selector-read-failure.marker" + mkdir -p "$fakebin" "$fixture/tests" + write_pass_test "$fixture/tests/should-not-run.test.sh" should-not-run + cat > "$fakebin/git" <<'SH' +#!/bin/bash +set -eu +case "${1:-}" in + clone) + last= + for arg in "$@"; do + last=$arg + done + mkdir -p "$last" + ;; + -C) + case "${3:-}" in + ls-tree) printf 'bin/fm-test-select.sh\n' ;; + show) exit 1 ;; + *) exit 2 ;; + esac + ;; + *) exit 2 ;; +esac +SH + chmod +x "$fakebin/git" + (cd "$fixture" && FM_SELECTOR_TEST_LOG="$marker" PATH="$fakebin:$PATH" /bin/sh -c "$loader") || rc=$? + expect_code 70 "$rc" trusted-selector-read-failure + assert_absent "$marker" "selector read failure ran the bootstrap suite" + pass "trusted-selector-read-failure" +} + test_shadow_full_fails() { setup_fixture advance_target_with_full_failure @@ -731,8 +810,11 @@ case "${1:-}" in mkdir -p "$last" ;; -C) - [ "${3:-}" = show ] || exit 2 - printf '#!/usr/bin/env bash\nexit 0\n' + case "${3:-}" in + ls-tree) printf 'bin/fm-test-select.sh\n' ;; + show) printf '#!/usr/bin/env bash\nexit 0\n' ;; + *) exit 2 ;; + esac ;; *) exit 2 ;; esac @@ -779,6 +861,7 @@ CASES=( fetch-failure shallow-no-base trusted-selector-missing + trusted-selector-read-failure shadow-full-fails shadow-focus-fails concurrent-change @@ -819,6 +902,7 @@ run_case() { fetch-failure) test_fetch_failure ;; shallow-no-base) test_shallow_no_base ;; trusted-selector-missing) test_trusted_selector_missing ;; + trusted-selector-read-failure) test_trusted_selector_read_failure ;; shadow-full-fails) test_shadow_full_fails ;; shadow-focus-fails) test_shadow_focus_fails ;; concurrent-change) test_concurrent_change ;; From 80bc0354161fcd467bc99a75c7dbbf9a42768f2b Mon Sep 17 00:00:00 2001 From: Eli Nimad Date: Wed, 15 Jul 2026 23:12:44 -0700 Subject: [PATCH 4/7] no-mistakes(document): Document trusted gate test selection --- docs/configuration.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/docs/configuration.md b/docs/configuration.md index d289c4c15..a85ae3117 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -103,10 +103,13 @@ See [`wedge-alarm.md`](wedge-alarm.md) for the channel reference and macOS verif ## Gate defaults (.no-mistakes.yaml) -The tracked `.no-mistakes.yaml` keeps test evidence outside the repo and defines `commands.test` so no-mistakes runs firstmate's bash behavior suite directly. +The tracked `.no-mistakes.yaml` keeps test evidence outside the repo, pins lint to `bin/fm-lint.sh`, and defines `commands.test` as a trusted target-main test-selector loader. That evidence policy is specific to the firstmate repo: target projects may legitimately commit `.no-mistakes/evidence/` from their own no-mistakes pipeline, but firstmate keeps `.no-mistakes/` local and CI rejects tracked entries under that path. -That command requires `tmux` on `PATH`, prints `tmux -V`, runs every `tests/*.test.sh` with `bash`, and fails if any script exits non-zero. -It intentionally mirrors the behavior-test baseline in [`.github/workflows/ci.yml`](../.github/workflows/ci.yml) instead of delegating the test step to an agent. +The loader clones `main`, reads `bin/fm-test-select.sh` only from that target revision, and runs its `gate-shadow` command. +The selector may shadow its one exact focused mapping for local feedback parity, but it always runs the complete lexical `tests/*.test.sh` inventory and propagates every failure before delivery. +Any unavailable or mismatched trusted selector blocks the gate rather than running branch-controlled policy; only when target `main` has no selector does the loader use its fixed complete-suite bootstrap fallback. +Complete execution requires `tmux` on `PATH`, prints `tmux -V`, and runs every inventory member with `bash` without stopping at the first failure. +CI retains the behavior-test baseline and additional platform-specific compatibility lanes; the gate does not delegate test-policy selection to an agent or the branch under review. ## Captain preferences (data/captain.md) From 7c7637eff266ce9e6356f23c2ec6423af909d390 Mon Sep 17 00:00:00 2001 From: Eli Nimad Date: Wed, 15 Jul 2026 23:26:23 -0700 Subject: [PATCH 5/7] fix: restore approved test selector scope --- docs/configuration.md | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/docs/configuration.md b/docs/configuration.md index a85ae3117..d289c4c15 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -103,13 +103,10 @@ See [`wedge-alarm.md`](wedge-alarm.md) for the channel reference and macOS verif ## Gate defaults (.no-mistakes.yaml) -The tracked `.no-mistakes.yaml` keeps test evidence outside the repo, pins lint to `bin/fm-lint.sh`, and defines `commands.test` as a trusted target-main test-selector loader. +The tracked `.no-mistakes.yaml` keeps test evidence outside the repo and defines `commands.test` so no-mistakes runs firstmate's bash behavior suite directly. That evidence policy is specific to the firstmate repo: target projects may legitimately commit `.no-mistakes/evidence/` from their own no-mistakes pipeline, but firstmate keeps `.no-mistakes/` local and CI rejects tracked entries under that path. -The loader clones `main`, reads `bin/fm-test-select.sh` only from that target revision, and runs its `gate-shadow` command. -The selector may shadow its one exact focused mapping for local feedback parity, but it always runs the complete lexical `tests/*.test.sh` inventory and propagates every failure before delivery. -Any unavailable or mismatched trusted selector blocks the gate rather than running branch-controlled policy; only when target `main` has no selector does the loader use its fixed complete-suite bootstrap fallback. -Complete execution requires `tmux` on `PATH`, prints `tmux -V`, and runs every inventory member with `bash` without stopping at the first failure. -CI retains the behavior-test baseline and additional platform-specific compatibility lanes; the gate does not delegate test-policy selection to an agent or the branch under review. +That command requires `tmux` on `PATH`, prints `tmux -V`, runs every `tests/*.test.sh` with `bash`, and fails if any script exits non-zero. +It intentionally mirrors the behavior-test baseline in [`.github/workflows/ci.yml`](../.github/workflows/ci.yml) instead of delegating the test step to an agent. ## Captain preferences (data/captain.md) From 681fbb07d6c1941fb85ccd6deddd94e7df87f7df Mon Sep 17 00:00:00 2001 From: Eli Nimad Date: Thu, 16 Jul 2026 00:58:49 -0700 Subject: [PATCH 6/7] no-mistakes(review): Captain: clarify fail-closed gate documentation --- docs/configuration.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/configuration.md b/docs/configuration.md index d289c4c15..c9c405536 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -103,9 +103,11 @@ See [`wedge-alarm.md`](wedge-alarm.md) for the channel reference and macOS verif ## Gate defaults (.no-mistakes.yaml) -The tracked `.no-mistakes.yaml` keeps test evidence outside the repo and defines `commands.test` so no-mistakes runs firstmate's bash behavior suite directly. +The tracked `.no-mistakes.yaml` keeps test evidence outside the repo and defines `commands.test` to load the selector committed on target `main` from a restricted temporary clone. +The gate fails closed when that target fetch or the selector's target-main identity check cannot establish trusted policy. +The trusted selector runs `gate-shadow`, which may shadow an exact focused plan but always runs the complete bash behavior suite once before delivery. That evidence policy is specific to the firstmate repo: target projects may legitimately commit `.no-mistakes/evidence/` from their own no-mistakes pipeline, but firstmate keeps `.no-mistakes/` local and CI rejects tracked entries under that path. -That command requires `tmux` on `PATH`, prints `tmux -V`, runs every `tests/*.test.sh` with `bash`, and fails if any script exits non-zero. +The complete suite requires `tmux` on `PATH`, prints `tmux -V`, runs every `tests/*.test.sh` with `bash`, and fails if any script exits non-zero. It intentionally mirrors the behavior-test baseline in [`.github/workflows/ci.yml`](../.github/workflows/ci.yml) instead of delegating the test step to an agent. ## Captain preferences (data/captain.md) From 687729b730231c81277c3da801c07ab43edad0be Mon Sep 17 00:00:00 2001 From: Eli Nimad Date: Thu, 16 Jul 2026 01:31:03 -0700 Subject: [PATCH 7/7] no-mistakes(document): Document path-aware test selector --- docs/scripts.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/scripts.md b/docs/scripts.md index db420026e..29c161a26 100644 --- a/docs/scripts.md +++ b/docs/scripts.md @@ -52,6 +52,7 @@ The shared no-mistakes gate refusal used by `fm-spawn.sh`, `fm-send.sh`, and `fm | `fm-crew-state.sh` | Print one deterministic current-state line for a crew | | `fm-tangle-lib.sh` | Shared default-branch resolution and primary-checkout tangle classification | | `fm-supervision-lib.sh` | Shared in-flight-work-without-fresh-watcher-beacon predicate | +| `fm-test-select.sh` | Fail-closed path-aware local test selection and complete gate shadow | | `fm-ff-lib.sh` | Shared guarded fast-forward helper for origin pulls and local secondmate syncs | | `fm-lock-lib.sh` | Shared "is this git lock provably abandoned?" proof used by teardown and fleet-sync | | `fm-config-inherit-lib.sh` | Shared primary-to-secondmate inheritable-config propagation |