chore(ci): fail when a test suite is executed by nothing (DEBT-031)

.github/workflows/ci-unit.yml

@@ -197,11 +197,40 @@ jobs:
       - name: Run unit tests
         run: uv run pytest vektra-app/tests/ -v --tb=short -m "not integration"
 
+  # The structural guards, and the only job here that no path filter can skip. That is
+  # the whole point of it. Every other job is gated on dorny/paths-filter, so a PR that
+  # only edits the Makefile, only edits a workflow, or adds a brand-new vektra-foo/
+  # package matches no filter and skips them all — and those are exactly the three
+  # changes a guard over the runner lists exists to catch. A guard a path filter can
+  # skip is not a guard (DEBT-031), which also applies to the DEBT-029 guard that until
+  # now ran only when vektra-shared itself changed.
+  #
+  # It parses the Makefile and the workflows; it runs no suite and needs no services.
+  test-structure:
+    name: Structural guards (suite execution, env isolation)
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+        with:
+          # Runs PR-authored test code; don't leave the token in .git/config
+          persist-credentials: false
+      - uses: astral-sh/setup-uv@v7
+        with:
+          enable-cache: true
+          cache-dependency-glob: "uv.lock"
+      - run: uv sync --dev --frozen --package vektra-shared
+      - name: Check every suite is executed, and every package is env-isolated
+        run: >
+          uv run pytest
+          vektra-shared/tests/test_suite_execution_coverage.py
+          vektra-shared/tests/test_env_isolation_coverage.py
+          -v --tb=short
+
   # Aggregator job used as the single required status check in branch protection.
   # Passes if no upstream job failed (skipped jobs are treated as passing).
   ci-gate:
     name: CI gate
-    needs: [detect-changes, test-shared, test-core, test-ingest, test-index, test-admin, test-analytics, test-learn, test-app]
+    needs: [detect-changes, test-shared, test-core, test-ingest, test-index, test-admin, test-analytics, test-learn, test-app, test-structure]
     runs-on: ubuntu-latest
     if: always()
     steps:

.github/workflows/integration.yml

@@ -72,10 +72,14 @@ jobs:
           echo "startup_ms=$ELAPSED_MS" >> "$GITHUB_OUTPUT"
           echo "::notice::Container startup: ${ELAPSED_MS}ms (NFR-004 limit: 60000ms)"
 
-      - name: Run integration tests (REQ-032)
+      # test_startup.py (ARCH-057) asserts the 11 startup steps against the logs of the
+      # stack this job has just brought up. Its docstring said it ran here; it did not,
+      # because this step named tests/integration/ and nothing named tests/ (DEBT-031).
+      # It is the same startup surface BUG-024 broke, and it was watching nothing.
+      - name: Run integration tests (REQ-032) and startup validation (ARCH-057)
         id: integration
         run: |
-          uv run pytest tests/integration/ \
+          uv run pytest tests/integration/ tests/test_startup.py \
             -v --tb=short \
             --junitxml=test-results/integration-${{ matrix.vector_store }}.xml
         timeout-minutes: 10
@@ -118,16 +122,21 @@ jobs:
           path: test-results/
           retention-days: 14
 
-  # The ARCH-057 startup sequence and the REQ-011/NFR-009 error contract. These
-  # tests existed but no workflow ran them (DEBT-030), which is the surface a
-  # provider-wiring defect broke in BUG-024 — and, unrun, they had themselves rotted:
-  # they masked the container password and failed at setup for anyone who tried.
-  # They bring up their own postgres via testcontainers, so they need Docker but not
-  # the compose stack.
-  app-integration:
-    name: App integration tests (startup + error contract)
+  # Every `integration`-marked test in every package, vektra-app included (its own
+  # suite is the ARCH-057 startup sequence and the REQ-011/NFR-009 error contract).
+  # DEBT-030 wired up vektra-app by hand and left the identical hole open next door:
+  # the integration suites of vektra-admin, vektra-index and vektra-ingest are excluded
+  # from every unit run by `-m "not integration"` and were named by no workflow, so they
+  # had never been executed once (DEBT-031).
+  #
+  # The target is a glob deliberately. A hand-written package list is the thing that
+  # keeps failing here — twice now — so a package added tomorrow is picked up with no
+  # edit to this file. These suites bring up their own postgres via testcontainers, so
+  # they need Docker but not the compose stack.
+  package-integration:
+    name: Package integration tests (all packages)
     runs-on: ubuntu-latest
-    timeout-minutes: 20
+    timeout-minutes: 30
 
     steps:
       - uses: actions/checkout@v7
@@ -146,41 +155,42 @@ jobs:
       - name: Prepare test results directory
         run: mkdir -p test-results
 
-      - name: Run vektra-app integration tests
+      - name: Run every package's integration tests
         run: |
-          uv run pytest vektra-app/tests/ -m integration \
-            -v --tb=short --junitxml=test-results/app-integration.xml
+          uv run pytest vektra-*/tests -m integration \
+            -v --tb=short --junitxml=test-results/package-integration.xml
 
       - name: Upload test results
         if: always()
         uses: actions/upload-artifact@v7
         with:
-          name: app-integration-test-results
+          name: package-integration-test-results
           path: test-results/
           retention-days: 14
 
   # Aggregator, and the single required status check in branch protection. The
   # matrix reports one check per provider ("Integration tests (pgvector)", "…
   # (qdrant)"), so without a job carrying the required name the check would never
   # report and every PR would sit BLOCKED. Same pattern as ci-gate in ci-unit.yml:
-  # adding a provider to the matrix, or a job like app-integration, then needs no
-  # change to branch protection.
+  # adding a provider to the matrix, or a job like package-integration, then needs no
+  # change to branch protection. The name below is load-bearing: change it and every PR
+  # sits BLOCKED with all checks green.
   integration-gate:
     name: Integration tests + NFR gates
-    needs: [integration, app-integration]
+    needs: [integration, package-integration]
     runs-on: ubuntu-latest
     if: always()
     steps:
       - name: Check every integration job passed
         run: |
           MATRIX="${{ needs.integration.result }}"
-          APP="${{ needs.app-integration.result }}"
+          PACKAGES="${{ needs.package-integration.result }}"
           if [[ "$MATRIX" != "success" ]]; then
             echo "Integration matrix did not pass (result: $MATRIX)."
             exit 1
           fi
-          if [[ "$APP" != "success" ]]; then
-            echo "App integration tests did not pass (result: $APP)."
+          if [[ "$PACKAGES" != "success" ]]; then
+            echo "Package integration tests did not pass (result: $PACKAGES)."
             exit 1
           fi
-          echo "Integration tests passed on every vector store provider, and the app suite passed."
+          echo "Integration tests passed on every vector store provider, and every package's integration suite passed."

.s2s/BACKLOG.md

@@ -163,9 +163,40 @@ They need Docker, and they now carry the `integration` marker, so the unit runs
 
 ---
 
-### DEBT-031: nothing guarantees a test suite is actually executed
+### BUG-025: a startup failure prints a raw Python traceback next to the structured error
 
 **Status**: planned | **Priority**: medium | **Created**: 2026-07-14
+**Origin**: DEBT-031 (2026-07-14). Surfaced by running `tests/test_startup.py` for the first time.
+
+**Context**: ARCH-057 exists so that a misconfiguration is reported clearly (REQ-011, NFR-009): a structured, remediable error rather than a stack dump. Step 1 does emit exactly that —
+
+```
+{"event": "startup_failed", "error": "[STARTUP ERROR] Step: config_validation\n  Detail: ...\n  Remediation: Set all required environment variables. At minimum: VEKTRA_LLM_PROVIDER ..."}
+```
+
+— and then `raise SystemExit(1)` propagates out of the ASGI lifespan into uvicorn, which logs the exception with `Traceback (most recent call last)` before "Application startup failed. Exiting." So the operator who typo'd an env var gets the good message *and* a Python traceback. `tests/test_startup.py` has asserted the absence of that traceback since the day it was written; nothing ever ran it (DEBT-031), so nobody found out.
+
+This is not cosmetic: the traceback is the failure mode NFR-009 names, and it applies to **every** one of the 11 steps, not only config validation.
+
+**Proposed approach** (recommended): validate before serving rather than inside the lifespan. The container runs `exec uvicorn vektra_app.main:app` (`docker/entrypoint.sh`), so the 11-step sequence necessarily runs as ASGI startup and any abort must travel through uvicorn. Replacing the uvicorn CLI with a `main()` that runs the validation and only then calls `uvicorn.run()` lets a failed step print its structured error and `sys.exit(1)` with no ASGI involvement and no traceback.
+
+Alternatives considered: suppressing uvicorn's exception logging (fragile, and hides real errors); a preflight that validates only step 1 in the entrypoint (leaves steps 2-11 still leaking a traceback, so it fixes the test rather than the contract).
+
+**Caution**: this touches the boot path, which is the surface BUG-024 broke. `vektra-app/tests/test_app.py::test_missing_llm_provider_aborts_startup` drives the lifespan in-process and expects `SystemExit`; a change to a hard exit there would kill the test runner.
+
+**Acceptance criteria**:
+- [ ] A misconfigured env var produces the structured `[STARTUP ERROR]` block and **no** `Traceback (most recent call last)` in container output
+- [ ] Holds for a step other than config validation (e.g. database connectivity), not just step 1
+- [ ] The container still exits non-zero
+- [ ] The `xfail(strict=True)` on `tests/test_startup.py::test_no_raw_traceback_on_startup_failure` is removed; strict mode makes the suite go red if the fix lands and the marker is left behind
+
+**Traceability**: REQ-011, NFR-009, ARCH-057, DEBT-031 (found by), BUG-024 (same surface)
+
+---
+
+### DEBT-031: nothing guarantees a test suite is actually executed
+
+**Status**: done (2026-07-14) | **Priority**: medium | **Created**: 2026-07-14
 **Origin**: DEBT-029/030 (2026-07-14). The lesson the fix left behind, rather than a defect the fix left behind.
 
 **Context**: BUG-024 (a startup blocker) shipped because `vektra-app/tests/` was executed by nothing — neither `make test` nor CI. DEBT-030 wired that one package in, and the two files in it turned out to have **never worked at all** (`str(make_url(...))` masks the password as `***`, so alembic authenticated with `***` and every test died in setup). A test nobody runs rots.
@@ -177,10 +208,20 @@ That is the exact shape of the hole BUG-024 fell through, still open one level u
 **Proposed approach**: extend the structural test (or add a sibling) so that every `vektra-*/tests` directory, plus `tests/integration` and `tests/nfr`, is referenced by the `make test` target **and** by a CI job. Parsing the Makefile and the workflow YAML is enough; it does not need to run them. Consider also asserting that a package's `integration`-marked tests are named in some workflow, which is the specific gap DEBT-030 closed by hand.
 
 **Acceptance criteria**:
-- [ ] A test fails when a `vektra-*/tests` directory exists that no CI job runs
-- [ ] A test fails when such a directory is missing from the `make test` target, so the local gate and CI cannot drift apart (they are two independent hand-maintained lists today)
-- [ ] It fails for the unit path and the integration path independently (an `integration`-marked suite excluded from unit runs and named in no workflow is the DEBT-030 case, and must be caught)
-- [ ] Verified by deleting a package from the workflow, and separately from the Makefile, and watching the test go red each time
+- [x] A test fails when a `vektra-*/tests` directory exists that no CI job runs
+- [x] A test fails when such a directory is missing from the `make test` target, so the local gate and CI cannot drift apart (they are two independent hand-maintained lists today)
+- [x] It fails for the unit path and the integration path independently (an `integration`-marked suite excluded from unit runs and named in no workflow is the DEBT-030 case, and must be caught)
+- [x] Verified by deleting a package from the workflow, and separately from the Makefile, and watching the test go red each time
+
+**Resolution**: `vektra-shared/tests/test_suite_execution_coverage.py` parses the `test:` recipe and every workflow's `pytest` invocations, and asserts each test file would actually be *collected* — honouring the `-m` filter, not just the paths. That distinction is the whole thing: a directory-level check would have passed, because `vektra-admin/tests/` **is** named in both lists; what nobody ran was the `integration`-marked file inside it. YAML is parsed, not grepped, so the commented-out `pytest` line in `integration.yml` grants no coverage. The guard runs in a `test-structure` job with **no path filter** — every other unit job is gated on `paths-filter`, so a Makefile-only edit, a workflow-only edit, or a new package skips them all, which are exactly the three cases the guard is for. The DEBT-029 guard now runs there too.
+
+Proven by breaking it five ways, each going red on the right assertion and green on restore: package dropped from the workflow; package dropped from the Makefile; the integration glob narrowed back to `vektra-app` (the DEBT-030 regression); a `vektra-foo/tests/` nobody wired up; and the allowlist emptied, which correctly re-flags the commented-out line as no coverage.
+
+**Found by the guard, previously unknown** (see CHANGELOG): five suites executed by nothing. `tests/test_startup.py` (ARCH-057 startup validation — its docstring assumed integration.yml ran it; integration.yml names `tests/integration/`, never `tests/`), and the `integration`-marked suites of vektra-admin, vektra-index (×2) and vektra-ingest, which DEBT-030 left behind when it wired up vektra-app by hand. All now run. The `app-integration` job becomes `package-integration` over a `vektra-*/tests` glob so the list stops being hand-maintained.
+
+**And, on cue, one of them had rotted — and it was hiding two product bugs.** The four package suites passed, but `tests/test_startup.py` went red in CI the first time it ever ran: `test_graceful_failure_on_missing_config` hung until timeout. It sets `-e VEKTRA_LLM_PROVIDER=`, believing that *unsets* the variable; it sets it to the **empty string**. `llm_provider` was a bare required `str`, and `""` is a valid `str`, so pydantic never raised, step 1 never fired, and **the stack booted and served with an empty LLM provider** — deferring the misconfiguration to the first query, which is precisely what ARCH-057 exists to prevent. Fixed at the source (`min_length=1`), not by weakening the test. With that fixed, the same test surfaced **BUG-025**: the structured error is emitted and a raw uvicorn traceback follows it, where NFR-009 asks for the one instead of the other. That one needs the boot path restructured (validate before `uvicorn.run()`), which is the surface BUG-024 broke, so it is filed rather than rushed in here; the assertion is `xfail(strict=True)` so it cannot rot in turn.
+
+Third time in this repo that a suite turned out to be broken the moment it was executed. The difference is that this one was also concealing live defects — which is the argument for the guard, made better than the guard's own docstring makes it.
 
 **Traceability**: BUG-024 (root cause), DEBT-029, DEBT-030