crowdsecurity · sabban · May 22, 2026 · May 22, 2026 · May 22, 2026 · May 22, 2026
diff --git a/.github/workflows/validate.yml b/.github/workflows/validate.yml
@@ -30,3 +30,9 @@ jobs:
         run: |
           npm install -g @anthropic-ai/claude-code
           claude plugin validate .
+
+      # Reports per-doc verification coverage (date / version / env) and flags
+      # stale or missing records. Gaps and staleness are informational and do
+      # not fail the build; a malformed `verified:` block does.
+      - name: Verification coverage report
+        run: python3 skills/crowdsec/scripts/check-verification.py
diff --git a/CLAUDE.md b/CLAUDE.md
@@ -10,9 +10,9 @@ Conventions for authoring this skill. This governs how skill content is **writte
 - **Write only the final, correct information.** Never record self-corrections, dead ends, or
   "actually, it turned out…" narration discovered while authoring. The reader gets the conclusion,
   not the journey. This applies equally to inline expected-output hints: state the correct
-  outcome, never the wrong-then-fixed version. Do not annotate content as *verified* (no
-  "(verified)", "Verified on…", "verified gotcha") — verification is guaranteed by this file,
-  not restated per-doc.
+  outcome, never the wrong-then-fixed version. Do not annotate verification *inline in prose* (no
+  "(verified)", "Verified on…", "verified gotcha"). Record verification in the per-file `verified:`
+  frontmatter block instead — see **Verification tracking** below.
 - **Keep `SKILL.md` a router.** It is an index and decision layer that points to `references/`.
   Depth and recipes belong in the reference docs, not in `SKILL.md`.
 - **Cover every environment.** Each command or recipe carries its systemd / docker / k8s variant,
@@ -34,4 +34,35 @@ Conventions for authoring this skill. This governs how skill content is **writte
   behavior as confirmed.
 - **Avoid duplicates** by checking in the existing skill documentation if the information is already present. If it is, ensure it's properly linked/routed, but don't duplicate instruction, code blocks or configurations.
 
+## Verification tracking
+
+Verification is recorded **per reference doc**, in a `verified:` YAML frontmatter block, so freshness
+is queryable and drift is visible. A doc with no `verified:` block has never been confirmed against a
+real environment.
+
+```yaml
+---
+verified:
+  - date: 2026-05-22      # ISO 8601 (YYYY-MM-DD); the day the recipes were run
+    version: "1.6.5"      # CrowdSec engine version, from `cscli version`
+    env: docker           # systemd | docker | k8s (free-form allowed, e.g. opnsense)
+    notes: "deploy + reload path only"   # optional, short scope note
+---
+```
+
+Rules:
+
+- **One entry per env.** Re-verifying the same env updates that entry in place (bump `date` and
+  `version`); verifying a new env appends a new entry. This keeps "cover every environment"
+  auditable.
+- **Stamp it when you verify it.** When you run a doc's commands against the real test environment
+  (see Testing) and observe them work, add or update that doc's `verified:` entry in the same change:
+  `date` = today, `version` = the output of `cscli version`, `env` = the detected environment. The
+  record and the verification happen together — never stamp from memory or inference.
+- Use ISO 8601 dates so the checker can sort and age them.
+- Don't pre-seed unverified docs with empty blocks; absence *is* the "never verified" signal.
+
+`skills/crowdsec/scripts/check-verification.py` (stdlib-only, safe to run locally) reports coverage,
+lists docs with no block, flags entries older than 180 days, and fails on a malformed block.
+
 
diff --git a/skills/crowdsec/references/appsec/configure.md b/skills/crowdsec/references/appsec/configure.md
@@ -1,3 +1,11 @@
+---
+verified:
+  - date: 2026-05-22
+    version: "1.7.8"
+    env: systemd
+    notes: "appsec-configs/rules list+inspect, metrics rules table; fixed eval-time claim"
+---
+
 # AppSec — Configure
 
 Canonical docs: <https://docs.crowdsec.net/docs/next/appsec/configuration> · rule management <https://docs.crowdsec.net/docs/next/appsec/configuration_rule_management> · hooks <https://docs.crowdsec.net/docs/next/appsec/hooks> · alerts & scenarios <https://docs.crowdsec.net/docs/next/appsec/alerts_and_scenarios> · API validation <https://docs.crowdsec.net/docs/next/appsec/api_validation>
@@ -132,7 +140,7 @@ For captcha responses, configuration lives on the **bouncer** (captcha provider
 ## Performance levers
 
 - `request_body_limit` (engine config) caps how much of the request body AppSec processes — defaults are usually fine; raise for APIs with large legitimate payloads, lower for static-only fronts.
-- Rule load order is automatic; per-rule eval time appears in `cscli metrics show appsec` once you generate traffic.
+- Rule load order is automatic; per-rule **trigger counts** appear in `cscli metrics show appsec` once you generate traffic (the Rules Metrics table) — useful for spotting hot rules, though `cscli` does not report per-rule eval time.
 - Inband evaluation adds latency on the request path. Out-of-band is asynchronous and does not.
 - Move expensive rules (large regex, body inspection) to out-of-band if latency matters more than per-request blocking.
 

diff --git a/skills/crowdsec/references/appsec/deploy.md b/skills/crowdsec/references/appsec/deploy.md
@@ -1,3 +1,15 @@
+---
+verified:
+  - date: 2026-05-21
+    version: "1.7.8"
+    env: systemd
+    notes: "nginx acquisition + AppSec deploy"
+  - date: 2026-05-22
+    version: "1.7.8"
+    env: k8s
+    notes: "k8s with traefik + AppSec"
+---
+
 # AppSec — Deploy
 
 Canonical docs: <https://docs.crowdsec.net/docs/next/appsec/intro> · quickstart <https://docs.crowdsec.net/docs/next/appsec/quickstart/general> · rules deploy <https://docs.crowdsec.net/docs/next/appsec/rules_deploy> · advanced deployments <https://docs.crowdsec.net/docs/next/appsec/advanced_deployments>
@@ -112,7 +124,7 @@ The smoke test above proves the WAF works. For production you point a real bounc
 | Bouncer | Where to set the AppSec endpoint |
 |---|---|
 | `crowdsec-nginx-bouncer` (lua module) | `APPSEC_URL=http://127.0.0.1:7422` in `/etc/crowdsec/bouncers/crowdsec-nginx-bouncer.conf` (shell-style `KEY=VALUE`, empty by default = WAF off). The self-registered `API_KEY` already serves AppSec — reuse it. |
-| Traefik (`maxlerebourg/crowdsec-bouncer-traefik-plugin`) | Flat plugin options: `crowdsecAppsecEnabled: true` (default false), `crowdsecAppsecHost: crowdsec:7422` (host:port, no scheme), and the bouncer key in `crowdsecLapiKey`. Full recipe in [../configure/bouncers/web-servers.md](../configure/bouncers/web-servers.md) § Traefik. |
+| Traefik (`maxlerebourg/crowdsec-bouncer-traefik-plugin`) | Flat plugin options: `crowdsecAppsecEnabled: true` (default false), `crowdsecAppsecHost: <host>:<port>` (no scheme — `crowdsec:7422` in Docker Compose where the container is named `crowdsec`; `<release>-appsec-service.<namespace>.svc.cluster.local:7422` in Kubernetes), and the bouncer key in `crowdsecLapiKey`. In Kubernetes the Middleware `spec.plugin.<key>` must match the key registered in Traefik's `experimental.plugins.<key>` — NOT the module name `crowdsec-bouncer-traefik-plugin`. Full recipe in [../configure/bouncers/web-servers.md](../configure/bouncers/web-servers.md) § Traefik. |
 | `github.com/hslatman/caddy-crowdsec-bouncer` (Caddy module) | Two handlers required in the Caddy route — **`appsec` AND `crowdsec`** (see critical note below). The `appsec_url` field goes in the top-level `crowdsec` app config block. |
 | Any other AppSec-aware bouncer | Look for an `appsec_url` / `appsec.url` field; auth is always the bouncer's existing API key. |
 
@@ -130,7 +142,7 @@ See also: [../configure/bouncers/web-servers.md](../configure/bouncers/web-serve
 | **systemd / bare-metal** | The recipe above as-is. |
 | **OPNsense / FreeBSD** | Config root is `/usr/local/etc/crowdsec/`; drop acquisition in `acquis.d/appsec.yaml`. The `os-crowdsec` plugin manages the engine — reload with `service crowdsec reload`. No Lua module in the OPNsense nginx package: use the Caddy-based bouncer instead (see [../configure/bouncers/web-servers.md](../configure/bouncers/web-servers.md) § Caddy). Note the LAPI port conflict below. |
 | **Docker / compose** | AppSec runs inside the crowdsec container and must `listen_addr: 0.0.0.0:7422`. Bouncer containers reach it via the service name + internal port (`appsec_url: http://crowdsec:7422`), not the published port. The acquisition file is mounted from the host or baked into a customised image. `docker compose exec crowdsec cscli appsec-*` for management commands. **Containerized lua bouncers need a DNS `resolver` — see [../install/docker.md](../install/docker.md) § Bouncer key bootstrap.** |
-| **Kubernetes / Helm** | The official chart has `appsec.enabled: true` plus values for `appsec.listen_addr`, `appsec.config`, and a separate `appsec` Service. Bouncers target the AppSec Service DNS name. Required collections/rules can be listed in the chart's hub config. |
+| **Kubernetes / Helm** | Set `appsec.enabled: true`, `appsec.acquisitions` (list, not scalar — use `appsec_configs:` plural), and `appsec.env` with `COLLECTIONS`. **Also required:** either `agent.enabled: false` or a populated `agent.acquisition` — the chart template rejects values with no agent acquisition at all. Use `helm upgrade --install` (plain `helm upgrade` fails on a fresh cluster with no prior release). The chart creates a service named `<release>-appsec-service`; bouncers reach AppSec at `<release>-appsec-service.<namespace>.svc.cluster.local:7422`. For the Traefik plugin: Traefik needs a writable `/plugins-storage` volume (add an `emptyDir` via `deployment.additionalVolumes` + `additionalVolumeMounts`) and the plugin must be registered under `experimental.plugins.<key>` in Traefik Helm values — the `spec.plugin.<key>` in the Middleware must match that same key name. Create the bouncer key with `kubectl exec -n crowdsec deploy/crowdsec-lapi -- cscli bouncers add <name>`. Cross-namespace Middleware references are disabled by default — either put the Middleware in the same namespace as the IngressRoute, or enable `providers.kubernetesCRD.allowCrossNamespace: true` in Traefik. |
 
 ### OPNsense / FreeBSD: LAPI port conflict
 

diff --git a/skills/crowdsec/references/appsec/overview.md b/skills/crowdsec/references/appsec/overview.md
@@ -1,3 +1,11 @@
+---
+verified:
+  - date: 2026-05-22
+    version: "1.7.8"
+    env: systemd
+    notes: "listener 7422, configs/rules paths, out-of-band scenario→ban path; conceptual doc"
+---
+
 # AppSec — Overview
 
 Canonical docs: <https://docs.crowdsec.net/docs/next/appsec/intro> · request lifecycle <https://docs.crowdsec.net/docs/next/appsec/request-lifecycle> · protocol <https://docs.crowdsec.net/docs/next/appsec/protocol> · FAQ <https://docs.crowdsec.net/docs/next/appsec/faq>

diff --git a/skills/crowdsec/references/appsec/troubleshoot.md b/skills/crowdsec/references/appsec/troubleshoot.md
@@ -1,3 +1,11 @@
+---
+verified:
+  - date: 2026-05-22
+    version: "1.7.8"
+    env: systemd
+    notes: "metrics show appsec, listener on 7422, appsec-rules/configs inspect; fixed eval-time claim"
+---
+
 # AppSec — Troubleshoot
 
 Canonical docs: <https://docs.crowdsec.net/docs/next/appsec/troubleshooting> · FAQ <https://docs.crowdsec.net/docs/next/appsec/faq> · benchmark <https://docs.crowdsec.net/docs/next/appsec/benchmark>
@@ -159,7 +167,7 @@ Inspect the bouncer's own log for `captcha` lines; it should announce when it's
 sudo cscli metrics show appsec
 ```
 
-Per-rule eval time shows up in the rules table. Suspects:
+The Rules Metrics table shows a per-rule **Triggered** count (how often each rule matched), not timing — use it to spot a hot rule worth moving out-of-band. `cscli` does not expose per-rule eval time; for actual latency numbers use the benchmark methodology below. Suspects:
 
 - **Expensive regex** in a rule body — move to out-of-band, or replace with a cheaper match.
 - **Large request bodies** — raise / lower `request_body_limit` in the engine config; the default is conservative.

diff --git a/skills/crowdsec/references/configure/acquisition.md b/skills/crowdsec/references/configure/acquisition.md
@@ -1,3 +1,11 @@
+---
+verified:
+  - date: 2026-05-22
+    version: "1.7.8"
+    env: systemd
+    notes: "config keys, crowdsec -t (clean + FATAL bad-source), appsec listener"
+---
+
 # Configure — Acquisition (log sources)
 
 Canonical docs: <https://docs.crowdsec.net/docs/next/getting_started/post_installation/acquisition> · datasources index <https://docs.crowdsec.net/docs/next/data_sources/intro>

diff --git a/skills/crowdsec/references/configure/allowlists.md b/skills/crowdsec/references/configure/allowlists.md
@@ -1,3 +1,11 @@
+---
+verified:
+  - date: 2026-05-20
+    version: "1.7.8"
+    env: systemd
+    notes: "allowlists add/check/list"
+---
+
 # Configure — Allowlists, whitelist parsers, and postoverflows
 
 Canonical docs: <https://docs.crowdsec.net/docs/next/local_api/centralized_allowlists/>

diff --git a/skills/crowdsec/references/configure/bouncers/firewall.md b/skills/crowdsec/references/configure/bouncers/firewall.md
@@ -1,3 +1,11 @@
+---
+verified:
+  - date: 2026-05-22
+    version: "1.7.8"
+    env: systemd
+    notes: "nftables bouncer v0.0.34 install+register, nft sets/chains (priority filter-10), ban→cscli set→drop; added stale-key recovery note"
+---
+
 # Bouncers — Firewall (nftables / iptables / ipset)
 
 Canonical docs: <https://docs.crowdsec.net/u/bouncers/firewall>
@@ -49,6 +57,15 @@ If you *also* run `cscli bouncers add` you create a second, unused key — skip
 Only register manually when the bouncer runs on a **different host** than LAPI
 (then set `api_url` to the remote LAPI and paste the manual key into the yaml).
 
+> **Gotcha — reinstall over a stale key.** If a previous
+> `/etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml` survives a purge/reinstall,
+> the postinst may not rewrite the `api_key`, and the service then fails to start with
+> `API error: access forbidden` / `bouncer stream halted` in
+> `/var/log/crowdsec-firewall-bouncer.log` (and the dpkg `--configure` step errors).
+> Re-register: `cscli bouncers delete <name>`, `KEY=$(cscli bouncers add fw-local -o raw)`,
+> write it into the yaml's `api_key:`, `systemctl restart crowdsec-firewall-bouncer`.
+> See [../../debug/bouncer-not-blocking.md](../../debug/bouncer-not-blocking.md) § 3.
+
 ## 3 — What it creates in nftables
 
 The bouncer builds its **own** tables, isolated from your existing ruleset:

diff --git a/skills/crowdsec/references/configure/bouncers/web-servers.md b/skills/crowdsec/references/configure/bouncers/web-servers.md
@@ -1,3 +1,15 @@
+---
+verified:
+  - date: 2026-05-21
+    version: "1.7.8"
+    env: systemd
+    notes: "nginx bouncer path only"
+  - date: 2026-05-22
+    version: "1.7.8"
+    env: k8s
+    notes: "k8s with traefik + AppSec"
+---
+
 # Bouncers — Web servers (nginx, haproxy, apache, Traefik, Caddy)
 
 Canonical docs: <https://docs.crowdsec.net/u/bouncers/intro> (per-bouncer pages: nginx, haproxy, apache, traefik, caddy)
@@ -330,6 +342,88 @@ docker exec crowdsec cscli metrics show appsec     # Processed/Blocked increment
 - **`stream` lag:** a fresh ban lands within `updateIntervalSeconds`; immediate ban-then-curl
   looks like a failure. (See [../../debug/bouncer-not-blocking.md](../../debug/bouncer-not-blocking.md).)
 
+### Kubernetes (Helm) — extra gotchas
+
+Deploying the plugin on the official Traefik Helm chart requires several steps the upstream docs omit:
+
+**1. Writable plugin storage**
+
+The Traefik Helm chart mounts a read-only root filesystem by default. Traefik crashes at startup with `"unable to create directory /plugins-storage/sources: read-only file system"` unless you give it a writable volume:
+
+```yaml
+# traefik values.yaml
+experimental:
+  plugins:
+    bouncer:
+      moduleName: "github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin"
+      version: "v1.4.4"
+
+deployment:
+  additionalVolumes:
+    - name: plugins-storage
+      emptyDir: {}
+
+additionalVolumeMounts:
+  - name: plugins-storage
+    mountPath: /plugins-storage
+```
+
+Use the `experimental.plugins` section (not `additionalArguments`) — the chart handles the correct flag format.
+
+**2. Middleware must use the plugin key name, not the module name**
+
+The Middleware `spec.plugin.<key>` must match the key in `experimental.plugins.<key>` (here: `bouncer`), **not** the module name `crowdsec-bouncer-traefik-plugin`. Error if wrong: `"plugin: unknown plugin type: crowdsec-bouncer-traefik-plugin"`.
+
+```yaml
+# Correct Kubernetes Middleware
+apiVersion: traefik.io/v1alpha1
+kind: Middleware
+metadata:
+  name: crowdsec
+  namespace: traefik
+spec:
+  plugin:
+    bouncer:                        # <-- key from experimental.plugins.bouncer
+      enabled: true
+      crowdsecMode: stream
+      crowdsecLapiScheme: http
+      crowdsecLapiHost: crowdsec-service.crowdsec.svc.cluster.local:8080
+      crowdsecLapiKey: <bouncer-key>
+      httpTimeoutSeconds: 10
+      crowdsecAppsecEnabled: true   # WAF off by default — must set true
+      crowdsecAppsecHost: crowdsec-appsec-service.crowdsec.svc.cluster.local:7422
+      crowdsecAppsecFailureBlock: true
+      crowdsecAppsecUnreachableBlock: true
+      forwardedHeadersTrustedIPs:
+        - 10.0.0.0/8
+        - 192.168.0.0/16
+```
+
+`crowdsecAppsecHost` must be the full Kubernetes service DNS name (`<release>-appsec-service.<namespace>.svc.cluster.local:7422`), not the Docker Compose shortname `crowdsec:7422`.
+
+**3. Cross-namespace Middleware is blocked by default**
+
+If your IngressRoute lives in a different namespace than the Middleware, Traefik rejects the reference: `"allowCrossNamespace is disabled, cross-namespace are disallowed"`. Either:
+- put the Middleware in the same namespace as the IngressRoute, or
+- enable in Traefik values: `providers.kubernetesCRD.allowCrossNamespace: true` (and accept the warning logged on startup).
+
+**4. Bouncer key for Kubernetes**
+
+There is no auto-registration in the Helm chart. Mint the key manually:
+
+```bash
+kubectl exec -n crowdsec deploy/crowdsec-lapi -- cscli bouncers add traefik-bouncer
+# outputs the key — copy it into the Middleware crowdsecLapiKey
+```
+
+**5. AppSec Helm values (what actually works)**
+
+The CrowdSec AppSec hostname in the Traefik Middleware (`crowdsec-appsec-service`) comes from the chart's release name. With `helm install crowdsec crowdsec/crowdsec -n crowdsec ...`, the service is:
+- LAPI: `crowdsec-service.crowdsec.svc.cluster.local:8080`
+- AppSec: `crowdsec-appsec-service.crowdsec.svc.cluster.local:7422`
+
+After a `kubectl rollout restart deployment/traefik -n traefik`, confirm the plugin connects: `cscli bouncers list` (via `kubectl exec`) should show `traefik-bouncer` with a recent `Last API pull`.
+
 ## Caddy — `github.com/hslatman/caddy-crowdsec-bouncer`
 
 WAF-capable Caddy module ([`hslatman/caddy-crowdsec-bouncer`](https://github.com/hslatman/caddy-crowdsec-bouncer)).

diff --git a/skills/crowdsec/references/configure/hub.md b/skills/crowdsec/references/configure/hub.md
@@ -1,3 +1,11 @@
+---
+verified:
+  - date: 2026-05-22
+    version: "1.7.8"
+    env: systemd
+    notes: "update/upgrade, list -o raw, collections install+remove, inspect tainted fields"
+---
+
 # Configure — Hub management
 
 Canonical docs: <https://docs.crowdsec.net/docs/next/getting_started/post_installation/console_hub> · `cscli hub` reference <https://docs.crowdsec.net/docs/next/cscli/cscli_hub>
@@ -13,9 +21,11 @@ Install a **collection** and it pulls every item it depends on. Installing
 
 ```bash
 sudo cscli collections install crowdsecurity/wordpress
-#  scenarios: crowdsecurity/http-bf-wordpress_bf, crowdsecurity/http-wordpress_user-enum,
-#             crowdsecurity/http-wordpress_wpconfig
-#  collections: crowdsecurity/wordpress
+#  enabling scenarios:crowdsecurity/http-bf-wordpress_bf
+#  enabling scenarios:crowdsecurity/http-wordpress_wpconfig
+#  enabling scenarios:crowdsecurity/http-wordpress_user-enum
+#  enabling collections:crowdsecurity/wordpress
+#
 #  Run 'sudo systemctl reload crowdsec' for the new configuration to be effective.
 ```
 

diff --git a/skills/crowdsec/references/configure/profiles.md b/skills/crowdsec/references/configure/profiles.md
@@ -1,3 +1,11 @@
+---
+verified:
+  - date: 2026-05-22
+    version: "1.7.8"
+    env: systemd
+    notes: "profiles.yaml structure, no cscli profiles cmd, simulation status, decision add/list/delete"
+---
+
 # Configure — Profiles (decisions, durations, simulation)
 
 Canonical docs: <https://docs.crowdsec.net/docs/next/local_api/profiles> · post-install profiles <https://docs.crowdsec.net/docs/next/getting_started/post_installation/profiles>

diff --git a/skills/crowdsec/references/debug/bouncer-not-blocking.md b/skills/crowdsec/references/debug/bouncer-not-blocking.md
@@ -1,3 +1,11 @@
+---
+verified:
+  - date: 2026-05-22
+    version: "1.7.8"
+    env: systemd
+    notes: "allowlists/decisions/bouncers ladder, LAPI curl 200, firewall nft sets/chains/counter"
+---
+
 # Debug — Decisions exist but bouncer not blocking
 
 Canonical docs: <https://docs.crowdsec.net/docs/next/troubleshooting/intro> · bouncers index <https://docs.crowdsec.net/u/bouncers/intro>

diff --git a/skills/crowdsec/references/debug/common-errors.md b/skills/crowdsec/references/debug/common-errors.md
@@ -1,3 +1,11 @@
+---
+verified:
+  - date: 2026-05-22
+    version: "1.7.8"
+    env: systemd
+    notes: "diagnostic commands + ExecStartPre/-t, machines, geoip path; error strings are catalog"
+---
+
 # Debug — Common errors (string → cause catalog)
 
 Canonical docs: <https://docs.crowdsec.net/docs/next/troubleshooting/intro>