docs: CHANGELOG audit findings for v3.x — completeness, wording, and process gaps

[membphis]

Current State

The CHANGELOG for the 3.x series (21 versions, ~1100 PRs spanning 2022-10 → 2026-04) has systemic issues that go beyond individual PR omissions. The most impactful problems fall into recurring patterns:

1. Four patch tags are entirely missing from master CHANGELOG.md

   • 3.2.2, 3.4.1, 3.8.1, 3.9.1 were tagged and released, with their sections written on release/X.Y branches, but never cherry-picked back to master.
   • These sections contain real security-impact fixes that master readers cannot see today, including jwt-auth authentication bypass (fix: upgrade api7-lua-resty-jwt to 0.2.5 #9837) and forward-auth POST header leak (fix: add post request headers only if auth request method is post #11184).
   • 3.2.1 exists in master only as placeholder text ("This is an LTS maintenance release...") with no actual bugfix entries.

2. Silent rockspec dependency upgrades (15+ instances)
   Every minor release upgrades dependencies with user-visible behavior or compatibility impact, but the CHANGELOG is silent. Most impactful examples:

   • 3.15.0: lua-resty-session 3.10 → 4.1.5 — default switches to AES-256-GCM; all session cookies issued by 3.14.x become invalid, forcing every OIDC user to re-authenticate after upgrade. Not mentioned.
   • 3.15.0: lua-resty-openidc 1.7.6 → 1.8.0; net-url 0.9 → 1.2 (major bump).
   • 3.16.0: api7-lua-resty-http 0.2.2 → 0.2.3 (rejects control chars in headers, affects every plugin using resty.http).
   • 3.8.0: luasec 0.9 → 1.3.2 (TLS library cross-major upgrade).
   • 3.4.0: lua-resty-healthcheck-api7 2.2.3 → 3.0.0 (major bump).
   • 3.9.0: lua-resty-radixtree 2.8.2 → 2.9.1 (route ordering fix chore: upgrade radixtree version #10861 hidden behind a chore title).

3. Security fixes are not consistently flagged

   • jwt-auth auth bypass (fix: upgrade api7-lua-resty-jwt to 0.2.5 #9837) appears in three release lines (3.2.2, 3.4.1, 3.5.0) and is flagged as security in none of them.
   • OpenResty CVE upgrade in 3.10.0 (chore: upgrade openresty version to 1.25.3.2 #11419) is listed without CVE annotation.
   • No ### Security section convention exists today.

4. Breaking changes mis-classified as Bugfixes / missing :warning:

   • 3.14.0 fix: only trust X-Forwarded-* headers from trusted_addresses #12551: X-Forwarded-* trusted-source whitelist change (a real breaking change) was placed under "Bugfixes".
   • 3.12.0 feat: set default value of ssl_trusted_certificate to system #11993: ssl_trusted_certificate default changed from "not configured = no verification" to "system CA bundle" — wording is one line, with no upgrade impact note. OIDC and outbound HTTPS calls may newly fail TLS handshake after upgrade.
   • 3.11.0 feat: implement credential #11601: credential resource silently dropped auth check and injects 3 upstream headers — no :warning:.
   • 3.10.0 feat: move config-default.yaml to hardcoded lua file #11343 / feat: move tinyyaml to lyaml #11312: config-default.yaml physically removed; YAML parser changed from tinyyaml → lyaml (stricter) — both can break IaC templates and existing apisix.yaml on startup, no :warning:.
   • 3.6.0 feat: strictly validate the input of core resources #10233: "Enforce strict schema validation" wording does not mention that additionalProperties=false was added to route / service / upstream / consumer / ssl / plugin_config / global_rule / stream_route / proto — this is the single largest silent-upgrade source in 3.6.0.

5. Wording from implementation side rather than user side

   • 3.7.0 OpenTelemetry change(opentelemetry): make span name and attributes follow the standard spec #10393: span name changed from URI to {method} {route_name} — breaks existing tracing dashboards. CHANGELOG only says "replace route with apisix.route_name".
   • 3.11.0 hmac-auth feat: rewrite hmac-auth plugin for usability #11581: field rename (access_key → key_id), 4 fields removed, single Authorization header — every existing signing client must change. CHANGELOG says "rewrite hmac-auth plugin for usability".
   • 3.16.0 Backward-compatible multi-tenant Keycloak integration - preserving previous token endpoint #12940 (use apisix_request_id only in http subsystem log format): default access_log_format silently changes.

6. Within-cycle intermediate changes not folded

   • 3.13.0 server-info: removed (change: delete the server-info plugin  #12218) → reverted (revert: "change: delete the server-info plugin" #12225) → marked deprecated (change: mark server-info plugin as deprecated #12244) within the same cycle — narrative is incomplete in CHANGELOG.
   • 3.9.0 quic → http3 rename: 3 PRs, users should only see final option enable_http3.
   • 3.15.0 rate-limit redis keepalive field rename (feat: rate limiting plugins support setting keepalive for redis policy #12861 → chore: add redis prefix to keepalive settings #12955).

7. Same PR listed twice in Change and Bugfixes (same release)

   • 3.12.0: feat: set default value of ssl_trusted_certificate to system #11993 / fix(grpc-web): response contains two trailer chunks #11988 / fix(ai-proxy): remove model options' stream default value #12013 / fix(ai-proxy): remove passthrough #12014 / fix(jwt-auth): disallow empty key configuration attributes #11852 appear in both sections.
   • 3.10.0: fix(request.header): core.request.header return string instead of table #11127 / fix(cors): avoid overwriting Access-Control-Expose-Headers response header #11136 / fix(request.headers): no longer returns all http headers, but limited to the default 100.  #11140 appear in both Change (:warning:) and Bugfix.

8. Markdown / numbering / typos

   • 3.14.0 and 3.14.1: ## Doc improvements uses H2 (breaks TOC, looks like a new release).
   • 3.5.0: PR #9813 referenced — actual PR is #9622 (google-cloud-logging client_email).
   • 3.11.0: plugin name typo attach-consmer-label.
   • 3.12.0 chore: upgrade openresty version to v1.27.11 #11936: OpenResty version typo v1.27.11 — actual 1.27.1.1.
   • 3.0.0: APSISIX typo on the enable_cpu_affinity entry.

9. Large omission batches in older versions

   • 3.13.0: 10 user-visible PRs not listed, including the new ai-request-rewrite plugin.
   • 3.2.0: ~17 user-facing PRs not listed; jwt-auth Vault entry says "Deprecated" but the change is actually a removal (change(jwt-auth): unify apisix/core/vault.lua and apisix/secret/vault.lua #8660).
   • 3.1.0: upstream type switched from required to optional (feat: add dafault value roundrobin for upstream_schema #8180), /etc/hosts prioritized resolution (feat: add method that parse ip from /etc/hosts in high priority #8270), GM crypto plugin (feat: add GM support #8389), etcd IPv6 (feat: support connect etcd with ipv6 address #8245), DNS client switched to api7-lua-resty-dns-client fork — none mentioned.

Desired State

A. Backfill missing patch sections into master CHANGELOG.md

• Cherry-pick 3.2.2, 3.4.1, 3.8.1, 3.9.1 sections from their release/X.Y branches into master CHANGELOG.md. Add ### Security annotation for jwt-auth auth bypass (fix: upgrade api7-lua-resty-jwt to 0.2.5 #9837) and forward-auth POST header leak (fix: add post request headers only if auth request method is post #11184).
• Replace the 3.2.1 placeholder paragraph with the 5–6 real bugfixes from release/3.2 (including the missing #9309 proxy-rewrite ngx.var.uri fix).

B. Six release-process additions to prevent regression

1. Rockspec diff gate — release pipeline runs git diff <prev>..<this> -- '*.rockspec'; any non-patch dependency bump requires a corresponding CHANGELOG entry. Closes the 15+ silent dep-bump pattern.
2. Patch-release CHANGELOG cherry-pick rule — any CHANGELOG entry written on a release/X.Y branch must be paired with a cherry-pick PR back to master. Prevents the four-tag pattern from recurring.
3. Breaking-change scan gate — scan apisix/schema_def.lua, apisix/admin/*.lua, apisix/plugins/*.lua for changes touching default | required | enum | encrypt_fields | deprecated | remove | rename; require a :warning: CHANGELOG line for each. Covers feat: set default value of ssl_trusted_certificate to system #11993, feat: implement credential #11601, feat: strictly validate the input of core resources #10233, fix: only trust X-Forwarded-* headers from trusted_addresses #12551 patterns.
4. PR-number cross-check — grep -oE '#[0-9]+' CHANGELOG.md | sort -u vs commit-derived PR list to catch transcription errors like #9622 → #9813.
5. Markdown lint — every ## heading in CHANGELOG must match a release version; ## Doc improvements style mistakes flagged.
6. ### Security convention — any CVE / auth-bypass / privilege-escalation fix gets a dedicated subsection with :warning:, so operators can quickly identify must-upgrade versions.

C. Per-version revision lists

Detailed per-PR evidence (CHANGELOG diff, bug label state, code path, proposed rewording) is available offline and can be split into separate revision PRs per version. Highest-priority near-term:

• 3.16.0: 4 entries (missing chore: header related fixes #12961; :warning: rewrite for fix(fw-auth): disallow control characters in headers #13057 and feat(standalone): reject configurations when configured with unknown plugin #13046; rewrite fix: use apisix_request_id only in http subsystem log format #13006 → Change).
• 3.15.0: 9 entries (must :warning: chore: update dependencies #12862 lua-resty-session 4.x; suggested :warning: for fix(performance): move the ipv6 check to schema validation #12714, fix: load full data during init_worker phase require a new apisix-runtime #12678; 3 missing fixes; 1 feat→fix re-classification; single rockspec aggregate line).
• 3.14.0: move fix: only trust X-Forwarded-* headers from trusted_addresses #12551 from Bugfixes to Change with :warning: + add a pre-upgrade trusted_addresses configuration hint.
• 3.12.0: expand feat: set default value of ssl_trusted_certificate to system #11993 wording to "old default = no verification, new default = system CA bundle"; fix chore: upgrade openresty version to v1.27.11 #11936 OpenResty version typo.

Earlier versions (3.0.0 → 3.13.0) can be batched per still-maintained LTS branch.