acceptance: Update testing guidance and backfill best practices

.agent/rules/testing.md

@@ -103,6 +103,24 @@ If the only reason for divergence is a server-side default that one engine sets
 - Run tests on cloud: `deco env run -i -n aws-prod-ucws -- <go test command>` (requires `deco` tool and access to test env).
 - `script.prepare` files from parent directories are concatenated into the test script. Use them for shared bash helpers.
 
+### Built-in shell helpers
+
+`acceptance/script.prepare` defines shell helpers that are in scope for every `script`. Prefer them over hand-rolled equivalents — they keep `output.txt` consistent across tests and make intent obvious to the next reader.
+
+- `trace CMD...`: print `>>> CMD` to stderr, then run CMD. Wrap commands whose invocation should appear in `output.txt` so the captured output explains what produced it. Leading `KEY=value` arguments are exported for the command (e.g. `trace FOO=bar $CLI ...`).
+- `title "TEXT"`: print a `=== TEXT` section header. Use `\n` escapes for spacing. Use it to label the phases of a multi-step script.
+- `errcode CMD...`: run CMD; if it exits non-zero, append `Exit code: N` to the output but let the script continue (scripts run under `bash -e`, which would otherwise abort). Use when a command is *allowed* to fail and later steps must still run.
+- `musterr CMD...`: run CMD and fail the whole test if it *succeeds*; on the expected failure the script continues. Use to assert a command must error.
+- `withdir DIR CMD...`: run CMD with the working directory set to DIR, restoring it afterwards.
+- `git-repo-init`: initialize a deterministic git repo (fixed user/email, no hooks) and commit `databricks.yml`.
+- `uuid`, `sethome DIR`, `venv_activate`, `as-test-sp CMD...`, `readplanarg FILE`, `envsubst`: see `acceptance/script.prepare` for the full list and exact semantics.
+
+**RULE: Wrap commands whose invocation should be visible in `output.txt` with `trace`.** The `>>> ...` line ties each block of output to the command that produced it; without it the output is an unlabeled wall of text.
+
+**RULE: Assert an expected failure with `musterr`, not `! cmd` or a bare `errcode`.** Only `musterr` fails the test when the command unexpectedly *succeeds*. `! cmd` is exempt from `set -e` and silently passes on success; `errcode` is for *tolerated* failures, not required ones.
+
+**RULE: Capture a tolerated non-zero exit with `errcode`, not `cmd || true`.** `errcode` records `Exit code: N` in the output so the failure stays visible and asserted; `|| true` hides it entirely.
+
 ### Helper scripts
 
 **RULE: Use the `acceptance/bin/` helpers before reaching for inline `jq` or `grep` pipelines.** When a test needs to filter recorded requests, assert a substring is or isn't present, or register a dynamic replacement, the helpers handle sorting, URL query normalization, redaction hooks, and cross-platform path issues. Inline `jq` in an acceptance script is brittle and hard to read.
@@ -124,7 +142,7 @@ BAD:
 Available on `PATH` during test execution (from `acceptance/bin/`):
 
 - `contains.py SUBSTR [!SUBSTR_NOT]`: passthrough filter (stdin→stdout) that checks substrings are present (or absent with `!` prefix). Errors are reported on stderr.
-- `print_requests.py //path [^//exclude] [--get] [--sort] [--keep]`: print recorded HTTP requests matching path filters. Requires `RecordRequests=true` in `test.toml`. Clears `out.requests.txt` afterwards unless `--keep`. Use `--get` to include GET requests (excluded by default). Use `^` prefix to exclude paths.
+- `print_requests.py //path [^//exclude] [--get] [--sort] [--unique] [--oneline] [--keep]`: print recorded HTTP requests matching path filters. Requires `RecordRequests = true` in `test.toml`. Excludes GET by default (`--get` includes them); clears `out.requests.txt` afterwards (`--keep` retains it). `^` prefix excludes a path; multiple positive filters are OR'd together. `--sort` orders output deterministically (use when the request set is order-independent), `--unique` collapses consecutive duplicates (e.g. repeated polls), `--oneline` prints one request per line.
 - `replace_ids.py [-t TARGET]`: read deployment state and add `[NAME_ID]` replacements for all resource IDs.
 - `read_id.py [-t TARGET] NAME`: read ID of a single resource from state, print it, and add a `[NAME_ID]` replacement.
 - `add_repl.py VALUE REPLACEMENT`: add a custom replacement (VALUE will be replaced with `[REPLACEMENT]` in output).
@@ -140,8 +158,45 @@ Available on `PATH` during test execution (from `acceptance/bin/`):
 
 **RULE: Don't pass `--keep` to `print_requests.py` if a later `print_requests.py` call follows.** The buffer accumulates, so the second call double-prints the earlier requests.
 
+**RULE: Filter recorded requests with `print_requests.py`, never with a hand-written `jq 'select(...)' out.requests.txt` pipeline** — inline, or hidden inside a local `print_requests()` shell function. The helper already excludes GET, normalizes query strings, optionally sorts, and deletes `out.requests.txt` afterwards. A copy-pasted `jq` wrapper reimplements all of that, drifts from the canonical output format, and is the single most common acceptance-test anti-pattern in this repo. Wrapping `print_requests.py` *itself* in a local function is fine — e.g. to send each variant's output to its own `out.requests.<name>.json`. Reach for `jq` on `out.requests.txt` only for what `print_requests.py` genuinely can't express: filtering on request *body* content, or deleting noisy body fields (prefer a `Repls` entry in `test.toml` even then).
+
+GOOD:
+
+```bash
+trace print_requests.py //pipelines --sort
+```
+
+BAD:
+
+```bash
+print_requests() {
+    jq --sort-keys 'select(.method != "GET" and (.path | contains("/pipelines")))' < out.requests.txt
+    rm out.requests.txt
+}
+trace print_requests
+```
+
 **RULE: Route noisy or non-deterministic command output to `LOG.<name>` instead of `output.txt` or `/dev/null`.** `LOG.*` files are visible under `go test -v` but excluded from the diff — see `acceptance/selftest/log/`. Use `&> LOG.<name>` to capture both streams (then `contains.py` to assert invariants like `'!panic' '!internal error'`), or `2>>LOG.<name>` for cleanup-step stderr you'd otherwise drop to `/dev/null`.
 
+### Test server
+
+Acceptance tests run against an in-process fake of the Databricks API in `libs/testserver/` (`FakeWorkspace` and the per-service handler files). The fake keeps real in-memory state and returns the same errors the backend does: 404 on a missing parent, 409 on a duplicate create, 400 on a missing required field, and so on. `test.toml` can also stub a single route with a canned response:
+
+```toml
+[[Server]]
+Pattern = "POST /api/2.2/jobs/create"
+Response.StatusCode = 400
+Response.Body = '''{"error_code": "INVALID_PARAMETER_VALUE", "message": "..."}'''
+```
+
+**RULE: Model API behavior in `libs/testserver/`, not in per-test `[[Server]]` response stubs.** When a test needs the fake server to validate input or return an error, add or extend the handler in `libs/testserver/` so the behavior is stateful and shared by every test. A `[[Server]]` stub hijacks the route with a static response that ignores request state, diverges from the real API, and only helps the one test that declares it — so the next test re-stubs the same error and the fake never converges on the real contract.
+
+GOOD: teach the create handler in `libs/testserver/postgres.go` to return 404 when the referenced role does not exist, so every test that creates a database against a missing role observes the real error.
+
+BAD: add `[[Server]]` with `Pattern = "POST .../databases"` and `Response.StatusCode = 404` to a single test's `test.toml` to fake that same error.
+
+Reserve `[[Server]]` for routes the testserver does not model at all (a one-off endpoint exercised by a single test) and for injecting a response a stateful handler genuinely can't express (for transient faults and forced disconnects, prefer the `fault.py` / kill helpers instead).
+
 ### Update workflow
 
 **RULE: Run `./task test-update` to regenerate outputs, then `./task fmt` and `./task lint`.** If fmt or lint modify files in `acceptance/`, there's an issue in the source files. Fix the source, regenerate, and verify fmt/lint pass cleanly.

acceptance/README.md

@@ -15,9 +15,13 @@ The scripts are run with `bash -e` so any errors will be propagated. They are ca
 
 For more complex tests one can also use:
 - `errcode` helper: if the command fails with non-zero code, it appends `Exit code: N` to the output but returns success to caller (bash), allowing continuation of script.
+- `musterr` helper: runs the command and fails the test if it *succeeds*; on the expected failure the script continues. Use it to assert a command must error.
 - `trace` helper: prints the arguments before executing the command.
+- `title` helper: prints a `=== <text>` section header to label a phase of the script.
 - custom output files: redirect output to custom file (it must start with `out`), e.g. `$CLI bundle validate > out.txt 2> out.error.txt`.
 
+The complete set of shell helpers (the above plus `withdir`, `git-repo-init`, `uuid`, `sethome`, and others) is defined in `acceptance/script.prepare`.
+
 Any file starting with "LOG" will be logged to test log (visible with go test -v).
 
 See [selftest](./selftest) for more examples.

acceptance/apps/deploy/no-bundle-no-args/output.txt

@@ -5,5 +5,3 @@ Usage: databricks apps deploy APP_NAME
 APP_NAME is the name of the Databricks app to operate on.
 Alternatively, run this command from a project directory containing
 databricks.yml to auto-detect the app name.
-
-Exit code: 1

acceptance/apps/deploy/no-bundle-no-args/script

@@ -1,3 +1,3 @@
 # Test: apps deploy without databricks.yml and no APP_NAME
 # Expected: Error about missing argument
-errcode $CLI apps deploy
+musterr $CLI apps deploy

acceptance/bundle/artifacts/nil_artifacts/output.txt

@@ -12,5 +12,3 @@ Workspace:
   Path: /Workspace/Users/[USERNAME]/.bundle/nil_artifacts_test/default
 
 Found 1 error
-
-Exit code: 1

acceptance/bundle/artifacts/nil_artifacts/script

@@ -1 +1 @@
-errcode trace $CLI bundle validate
+musterr trace $CLI bundle validate

acceptance/bundle/generate/alert_existing_id_not_found/output.txt

@@ -1,5 +1,3 @@
 
->>> errcode [CLI] bundle generate alert --existing-id f00dcafe
+>>> musterr [CLI] bundle generate alert --existing-id f00dcafe
 Error: alert with ID f00dcafe not found
-
-Exit code: 1

acceptance/bundle/generate/alert_existing_id_not_found/script

@@ -1,2 +1,2 @@
 # Test that bundle generate alert fails when the existing ID is not found
-trace errcode $CLI bundle generate alert --existing-id f00dcafe
+trace musterr $CLI bundle generate alert --existing-id f00dcafe

acceptance/bundle/multi_profile/no_workspace_profiles/output.txt

@@ -8,5 +8,3 @@ Workspace:
   Host: [DATABRICKS_URL]
 
 Found 1 error
-
-Exit code: 1

acceptance/bundle/multi_profile/no_workspace_profiles/script

@@ -14,4 +14,4 @@ unset DATABRICKS_HOST
 unset DATABRICKS_TOKEN
 
 # No workspace-compatible profiles → original multi-profile error returned.
-errcode trace $CLI bundle validate
+musterr trace $CLI bundle validate

acceptance/bundle/multi_profile/non_interactive_error/output.txt

@@ -19,5 +19,3 @@ Workspace:
   Host: [DATABRICKS_URL]
 
 Found 1 error
-
-Exit code: 1

acceptance/bundle/multi_profile/non_interactive_error/script

@@ -14,4 +14,4 @@ unset DATABRICKS_HOST
 unset DATABRICKS_TOKEN
 
 # Multiple workspace profiles, non-interactive → error with guidance.
-errcode trace $CLI bundle validate
+musterr trace $CLI bundle validate

acceptance/bundle/paths/pipeline_root_path_doesnotexist/output.txt

@@ -9,5 +9,3 @@ Workspace:
   Path: /Workspace/Users/[USERNAME]/.bundle/test-bundle/default
 
 Found 1 error
-
-Exit code: 1

acceptance/bundle/paths/pipeline_root_path_doesnotexist/script

@@ -1 +1 @@
-errcode trace $CLI bundle validate
+musterr trace $CLI bundle validate

acceptance/bundle/plan/no_upload/output.txt

@@ -4,4 +4,4 @@ create jobs.my_job
 
 Plan: 1 to add, 0 to change, 0 to delete, 0 unchanged
 
->>> jq -s .[] | select(.method != "GET") out.requests.txt
+>>> print_requests.py

acceptance/bundle/plan/no_upload/script

@@ -1,6 +1,4 @@
 trace $CLI bundle plan
 
 # Expect no non-GET requests
-trace jq -s '.[] | select(.method != "GET")' out.requests.txt
-
-rm out.requests.txt
+trace print_requests.py

acceptance/bundle/resources/alerts/with_file_variable_interpolation_error/output.txt

@@ -13,5 +13,3 @@ Workspace:
   Path: /Workspace/Users/[USERNAME]/.bundle/alerts-variable-error/default
 
 Found 1 error
-
-Exit code: 1

acceptance/bundle/resources/alerts/with_file_variable_interpolation_error/script

@@ -1 +1 @@
-errcode trace $CLI bundle validate
+musterr trace $CLI bundle validate

acceptance/bundle/resources/apps/lifecycle-started-omitted/output.txt

@@ -8,14 +8,14 @@ Deployment complete!
 
 >>> print_app_requests
 {
-  "body": {
-    "description": "my_app_description",
-    "name": "[UNIQUE_NAME]"
-  },
   "method": "POST",
   "path": "/api/2.0/apps",
   "q": {
     "no_compute": "true"
+  },
+  "body": {
+    "description": "my_app_description",
+    "name": "[UNIQUE_NAME]"
   }
 }
 
@@ -46,17 +46,17 @@ Deployment complete!
 
 >>> print_app_requests
 {
-  "body": {},
   "method": "POST",
-  "path": "/api/2.0/apps/[UNIQUE_NAME]/start"
+  "path": "/api/2.0/apps/[UNIQUE_NAME]/start",
+  "body": {}
 }
 {
+  "method": "POST",
+  "path": "/api/2.0/apps/[UNIQUE_NAME]/deployments",
   "body": {
     "mode": "SNAPSHOT",
     "source_code_path": "/Workspace/Users/[USERNAME]/.bundle/lifecycle-started-omitted-[UNIQUE_NAME]/default/files/app"
-  },
-  "method": "POST",
-  "path": "/api/2.0/apps/[UNIQUE_NAME]/deployments"
+  }
 }
 
 === started: true -> (started omitted) -> deploy: no start/stop requests (compute stays as-is)

acceptance/bundle/resources/apps/lifecycle-started-omitted/script

@@ -7,8 +7,7 @@ cleanup() {
 trap cleanup EXIT
 
 print_app_requests() {
-    jq --sort-keys 'select(.method != "GET" and (.path | contains("/apps")))' < out.requests.txt
-    rm out.requests.txt
+    print_requests.py //apps
 }
 
 title "(started omitted) -> deploy: app created with no_compute=true, no start/stop requests"

acceptance/bundle/resources/permissions/genie_spaces/current_can_manage/script

@@ -4,8 +4,7 @@ rm out.requests.txt
 $CLI bundle plan -o json > out.plan.$DATABRICKS_BUNDLE_ENGINE.json
 
 print_requests() {
-    jq -c < out.requests.txt | jq 'select(.method != "GET" and (.path | contains("permissions")))'
-    rm out.requests.txt
+    print_requests.py //permissions
 }
 
 rm out.requests.txt

acceptance/bundle/resources/permissions/jobs/deleted_remotely/script

@@ -1,6 +1,5 @@
 print_requests() {
-    jq 'select(.path | contains("/jobs/")) | select(.method != "GET")' < out.requests.txt
-    rm out.requests.txt
+    print_requests.py //jobs/
 }
 
 trace $CLI bundle plan -o json > out.plan_create.$DATABRICKS_BUNDLE_ENGINE.json

acceptance/bundle/resources/permissions/jobs/update/script

@@ -1,6 +1,5 @@
 print_requests() {
-    jq 'select(.path | contains("/jobs/")) | select(.method != "GET")' < out.requests.txt
-    rm out.requests.txt
+    print_requests.py //jobs/
 }
 
 cp databricks.yml databricks.yml.saved

acceptance/bundle/resources/permissions/models/current_can_manage/script

@@ -4,8 +4,7 @@ rm out.requests.txt
 $CLI bundle plan -o json > out.plan.$DATABRICKS_BUNDLE_ENGINE.json
 
 print_requests() {
-    jq -c < out.requests.txt | jq 'select(.method != "GET" and (.path | contains("permissions")))'
-    rm out.requests.txt
+    print_requests.py //permissions
 }
 
 rm out.requests.txt

acceptance/bundle/resources/permissions/pipelines/update/script

@@ -1,11 +1,9 @@
 print_requests() {
-    jq 'select(.path | contains("/pipeline")) | select(.method != "GET")' < out.requests.txt
-    rm out.requests.txt
+    print_requests.py //pipeline
 }
 
 print_sorted_requests() {
-    jq -c < out.requests.txt | sort | jq --sort-keys 'select(.path | contains("/pipeline")) | select(.method != "GET")'
-    rm out.requests.txt
+    print_requests.py //pipeline --sort
 }
 
 

acceptance/bundle/resources/permissions/target_permissions/script

@@ -1,11 +1,9 @@
 print_requests() {
-    jq 'select(.path | contains("/jobs/")) | select(.method != "GET")' < out.requests.txt
-    rm out.requests.txt
+    print_requests.py //jobs/
 }
 
 print_sorted_requests() {
-    jq -c < out.requests.txt | sort | jq --sort-keys 'select(.path | contains("/jobs/")) | select(.method != "GET")'
-    rm out.requests.txt
+    print_requests.py //jobs/ --sort
 }
 
 trace $CLI bundle plan

acceptance/bundle/resources/pipelines/allow-duplicate-names/output.txt

@@ -7,6 +7,8 @@ Deployment complete!
 
 >>> print_requests
 {
+  "method": "POST",
+  "path": "/api/2.0/pipelines",
   "body": {
     "libraries": [
       {
@@ -16,11 +18,11 @@ Deployment complete!
       }
     ],
     "name": "test-pipeline-same-name-[UNIQUE_NAME]"
-  },
-  "method": "POST",
-  "path": "/api/2.0/pipelines"
+  }
 }
 {
+  "method": "POST",
+  "path": "/api/2.0/pipelines",
   "body": {
     "allow_duplicate_names": true,
     "channel": "CURRENT",
@@ -37,9 +39,7 @@ Deployment complete!
       }
     ],
     "name": "test-pipeline-same-name-[UNIQUE_NAME]"
-  },
-  "method": "POST",
-  "path": "/api/2.0/pipelines"
+  }
 }
 
 >>> [CLI] bundle destroy --auto-approve

acceptance/bundle/resources/pipelines/allow-duplicate-names/script

@@ -17,6 +17,6 @@ export PIPELINE_ID
 trace $CLI bundle deploy
 
 print_requests() {
-    jq --sort-keys 'select(.method != "GET" and (.path | contains("/pipelines")))' < out.requests.txt
+    print_requests.py //pipelines
 }
 trace print_requests

acceptance/bundle/resources/pipelines/update/output.txt

@@ -7,6 +7,8 @@ Deployment complete!
 
 >>> print_requests
 {
+  "method": "POST",
+  "path": "/api/2.0/pipelines",
   "body": {
     "channel": "CURRENT",
     "deployment": {
@@ -22,9 +24,7 @@ Deployment complete!
       }
     ],
     "name": "test-pipeline-[UNIQUE_NAME]"
-  },
-  "method": "POST",
-  "path": "/api/2.0/pipelines"
+  }
 }
 pipelines my id='[MY_ID]' name='test-pipeline-[UNIQUE_NAME]'
 

acceptance/bundle/resources/pipelines/update/script

@@ -4,8 +4,7 @@ touch bar.py
 trace $CLI bundle deploy
 
 print_requests() {
-    jq --sort-keys 'select(.method != "GET" and (.path | contains("/pipelines")))' < out.requests.txt
-    rm out.requests.txt
+    print_requests.py //pipelines
     read_state.py pipelines my id name
 }
 

acceptance/bundle/resources/postgres_projects/recreate/script

@@ -23,6 +23,11 @@ project_id_1=`read_id.py my_project`
 
 print_requests() {
     local name=$1
+    # Keep jq here, not print_requests.py: the bundle name contains "postgres",
+    # so the state-file paths under .bundle/deploy-postgres-recreate/ also match
+    # a bare "postgres" filter. jq's contains("/postgres") matches only the API
+    # paths, and unlike print_requests.py's //postgres arg it isn't subject to
+    # Git-Bash/MSYS slash mangling on Windows. Matches the sibling recreate tests.
     jq --sort-keys 'select(.method != "GET" and (.path | contains("/postgres")))' < out.requests.txt > out.requests.${name}.txt
     rm -f out.requests.txt
 }

acceptance/bundle/resources/schemas/recreate/script

@@ -2,8 +2,7 @@ echo "*" > .gitignore
 trace $CLI bundle deploy
 
 print_requests() {
-    jq 'select(.method != "GET" and (.path | contains("/unity")))' < out.requests.txt
-    rm out.requests.txt
+    print_requests.py //unity
 }
 
 trace print_requests