Develop

.env.example

@@ -280,13 +280,40 @@ NATS_URL=nats://localhost:4222
 # default.
 # AI_GATEWAY_AUDIT_SPOOL_DIR=/var/lib/nexus/audit-spool
 
-# In-heap audit record-buffer cap (overflow → durable spill above). Each queued
-# record pins its pooled ~50 KB body until marshaled, so this bound is the primary
-# control over the audit side-path's gw heap: 10000 (default) holds the body pool
-# near ~1 GB under a slow-publish burst vs ~5 GB at the former 50000, same spill
-# rate. Raise on a memory-rich box, lower on a constrained one. 0/unset → 10000.
+# In-heap audit record-buffer cap (overflow → durable spill above). Bounds the
+# queue's POINTER count only — the memory bound is AI_GATEWAY_AUDIT_MEM_MAX_BYTES
+# below, which accounts each record's REAL body bytes. 0/unset keeps the default.
 # AI_GATEWAY_AUDIT_MAX_QUEUED_RECORDS=10000
 
+# In-memory audit byte budget — the memory half of the bounded audit queue. The
+# audit writer reserves each record's REAL captured body bytes against this budget
+# at admission; on a full budget the no-loss modes back-pressure the request path
+# (RPS throttles to the audit drain rate, nothing dropped) and the lossy modes
+# shed with a counted drop. Semantics mirror NEXUS_EVENTS_MAX_BYTES: unset/"auto"
+# auto-sizes to ~15% of available RAM (2 GiB fallback off-Linux); an explicit
+# human size ("8GB", "2048MB", raw bytes) pins it. A bigger budget buys BURST
+# absorption (audit work deferred to quieter moments), not sustained throughput —
+# under sustained overload the drain rate is the ceiling regardless. SIZING: the
+# budget accounts raw body bytes only; REAL process memory runs ~2x the budget
+# (Go GC headroom + marshal/compression copies), and the NATS memory stream holds
+# another ~15% of RAM on the same box — keep a pin <=20% of RAM (a 10GB pin on a
+# 32GB box OOM-killed the gateway at 15.9GB RSS in rig validation; a pin >25%
+# logs a WARN). Observability: nexus_audit_mem_backpressure_total counts
+# budget-full events; the resolved value is logged at startup.
+# AI_GATEWAY_AUDIT_MEM_MAX_BYTES=auto
+
+# In-flight request admission cap. Bounds concurrent proxy requests so overload
+# degrades into fast retryable 429s (Retry-After: 1, OpenAI-shaped error body)
+# instead of unbounded in-heap queueing toward the GOMEMLIMIT collapse. Health,
+# metrics, and admin endpoints are never gated. Shed requests are counted on
+# nexus_ai_gateway_admission_shed_total (no audit record — auditing a shed storm
+# would itself be load). DEFAULT unset/"auto" = 1024 x GOMAXPROCS — scaled by
+# core count and sized generously so SSE streams (which hold a slot for the
+# stream's full duration) do not false-shed a healthy streaming deployment; the
+# bound caps heap growth under overload, not normal traffic. 0/negative
+# disables; a non-numeric value falls back to auto with a warning.
+# AI_GATEWAY_MAX_INFLIGHT=auto
+
 # Audit overflow policy. Durable audit is a product promise + compliance
 # requirement. DEFAULT "spill" is spill-defer: the request path never
 # back-pressures — overflow goes to the durable on-disk spool and the
@@ -323,6 +350,20 @@ NATS_URL=nats://localhost:4222
 # AI_GATEWAY_AUDIT_SPILL_RECOVERY_INTERVAL_MS=2000
 # AI_GATEWAY_AUDIT_SPILL_RECOVERY_PACE_MS=50
 
+# ---------------------------------------------------------------------------
+# BENCHMARK / ABLATION SWITCHES — NOT FOR PRODUCTION
+# These exist only to measure throughput against forward-only gateways.
+# Do NOT copy these into any production or customer environment file.
+# ---------------------------------------------------------------------------
+# NEXUS_PERF_PURE_FORWARD — BENCHMARK ONLY. Set to 1 to make the AI Gateway a
+#   pure forwarding proxy: it SKIPS the entire audit tail, so NO traffic_event
+#   rows are written. This DESTROYS the audit trail (GDPR / audit-retention
+#   loss) and must NEVER be set in any production or customer environment.
+#   Default (unset) = audit stored, normal operation. Detection when active:
+#   the startup WARN banner "PURE-FORWARD BENCHMARK MODE ACTIVE" and the
+#   Prometheus gauge nexus_ai_gateway_pure_forward_mode == 1.
+#   Intentionally has NO assignable line here so it is never accidentally shipped.
+
 # Service discovery — co-located services on localhost; on prod each is a
 # domain or LB. Every URL below is bare-named (no service prefix) because
 # it identifies a shared environment-level entity, not a service-private

CHANGELOG.md

@@ -6,6 +6,64 @@ All notable changes to this project are documented here. The format follows
 
 ## [Unreleased]
 
+### Changed — overload now degrades into retryable 429s (in-flight admission gate)
+
+- The AI Gateway bounds concurrent in-flight proxy requests (default
+  `1024 × GOMAXPROCS`; `AI_GATEWAY_MAX_INFLIGHT` overrides, `0` disables). At
+  arrival rates beyond the box's capacity, excess requests are rejected fast
+  with **429 + `Retry-After: 1`** in the caller's ingress error shape (OpenAI /
+  Anthropic / Gemini envelopes) instead of queueing in-heap until the Go memory
+  limit collapses throughput (measured pre-fix: 15.9s p99 at 1.5× capacity;
+  the pre-GOMEMLIMIT failure mode was an OOM kill). 429 was already part of the
+  data-plane contract (per-key rate limits and quota denials); SDK retry logic
+  engages unchanged. Health, metrics, and admin endpoints are never gated. Shed
+  requests are counted on `nexus_ai_gateway_admission_shed_total`.
+
+### Fixed — hook-config reload stampede at high load
+
+- Hook configuration freshness is now push-driven with a background TTL-backstop
+  ticker; the request path never loads configuration. Previously a TTL-stale
+  check on the request path could fan out one full rule-pack database load per
+  in-flight request while a slow load was running, collapsing the gateway at
+  high request rates (measured: p99 120s at 16k req/s with content hooks on;
+  fixed: p99 27ms at the same rate). Rule-pack install ordering also gained a
+  deterministic tiebreaker so no-change config reloads can no longer churn the
+  compiled matchers.
+
+### Performance — content-hook path allocation and CPU
+
+- Bodies-off deployments no longer allocate a fresh request-body buffer per
+  request (the pooled buffer is returned at request end; previously measured at
+  52% of all gateway allocation under content-scan load).
+- Redact-action rule packs skip re-localization entirely on benign traffic
+  (zero matches on a complete scan).
+- Config snapshot loads expose `nexus_configcache_load_failures_total` and
+  `nexus_configcache_last_success_timestamp_seconds` for alerting on a frozen
+  config plane.
+
+### Fixed — Request/Response hook timing
+
+- **Streamed responses now record response-hook timing, exactly once per hook.**
+  The streaming response pipeline runs the response stage at every checkpoint, so
+  the live audit-only path previously recorded nothing (`response_hooks_ms` NULL)
+  while the chunked_async path recorded the same hook once per checkpoint (N
+  duplicate rows, an N×-inflated aggregate — observed as a "RESPONSE PIPELINE (63)"
+  list of identical rows). The trace is now folded to one record per hook (summed
+  latency, latest decision) across the ai-gateway live + Model A paths and the
+  shared compliance-proxy/agent path. The audit drawer also collapses any residual
+  duplicates (historical rows) into a single `×N` card.
+
+### Added — microsecond-precision hook timing (additive, backward compatible)
+
+- Per-hook latency is now measured in **microseconds** (`latencyUs`) alongside the
+  existing truncated-millisecond `latencyMs`, with new aggregate columns
+  `request_hooks_us` / `response_hooks_us` beside the unchanged `_ms` columns.
+  Hooks run at microsecond scale, so the millisecond aggregates floored a
+  sub-millisecond hook to `0`; the µs fields carry the real value, surfaced
+  precisely per hook in the control-plane audit drawer. The `_ms` columns / wire
+  ids / values are unchanged. The new binwire field ids are forward-incompatible,
+  so the deploy order is **schema → Hub → producers**.
+
 ### Changed (BREAKING — major version bump)
 - **Hook `onMatch` collapses to a single `action` (approve | redact | block).**
   The orthogonal `onMatch.inflightAction` (approve / block-hard / block-soft /
@@ -40,6 +98,46 @@ All notable changes to this project are documented here. The format follows
   as a 403 reject. The Agent signals a block by dropping the
   connection (no rich error body); the proxies return an attributed 403 whose
   response-stage reason carries rule-ID labels only, never the upstream value.
+### Fixed — co-firing redact + soft-block no longer drops the redaction (security)
+
+- **A redact hook co-firing with a soft-block hook now masks-and-delivers instead of
+  leaking or failing closed.** When a redact hook (`Modify` + masked content/spans) and
+  a soft-block hook fired on the same request or response, the pipeline aggregator
+  promoted the reported `Decision` to `BlockSoft` (the strictest) but DROPPED the redact
+  hook's replacement content, leaving spans without content. Downstream this produced a
+  no-op rewrite that, depending on the path, either failed closed (canonical response)
+  or replayed/forwarded the ORIGINAL unredacted body — a PII leak on the shared buffer
+  pipeline (compliance-proxy appliance included), the agent Model A wire, and both
+  request stages. `mergeResults` now carries the redact's `ModifiedContent`
+  unconditionally, and every redaction consumer gates on the new
+  `decision.CompliancePipelineResult.CarriesRedaction()` predicate (Modify OR a
+  BlockSoft masking a co-firing redact) rather than `Decision==Modify`, so the masked
+  body is applied and delivered on all paths. The audit row stamps the disposition
+  `action=redact` even when the (soft-block) `Decision` ceiling is `BlockSoft`. No
+  config or schema change; behavior is compliance-safe (a hard `block`/`RejectHard`
+  still rejects; a standalone soft-block still delivers-with-warning). The no-redactor
+  buffer degrade is now posture-aware (appliance fail-closed, agent fail-open).
+
+### Changed — three-end streaming-compliance parity via a shared Model A engine
+
+- **The Model A streaming-compliance algorithm is now a single shared engine driving
+  three ends.** The prescan-gated real-time streaming path (bounded tail-hold +
+  union prescan + confirm + escalate-to-buffer redaction) for a redact-scope
+  `chunked_async` stream is extracted into a substrate-agnostic engine
+  (`shared/transport/streaming/modela`). The AI Gateway drives it with a canonical
+  substrate (fail-closed) and the transparent proxy used by the Agent +
+  Compliance Proxy drives it with a raw-SSE-wire substrate (fail-open, NE
+  host-packet safety) — so hooks/compliance behave identically across all three
+  ends while each keeps its own ingress and delivery. The transparent-proxy live
+  path becomes **audit-only** (real-time write-through, observe-only checkpoints,
+  never blocks/rewrites): scope-derived routing sends a `block` scope to buffer and
+  a `redact` scope to Model A (or buffer), so only non-enforcing traffic reaches
+  live. The adoption also closed two latent PII-leak paths in the shipped AI Gateway
+  Model A (a redact masked behind a co-firing soft-block; a memory-pressure eviction
+  of an incomplete content unit). No config or contract change; behavior is
+  compliance-safe (a sub-window value is never delivered raw; storage never persists
+  a raw prefix on an enforcing outcome).
+
 ### Changed — normalized projection is now fully view-time (no migration required)
 
 - **The normalized traffic projection is no longer written on the hot path; it is

README.md

@@ -1,7 +1,7 @@
 # Nexus Gateway
 
-[![CI](https://github.com/AlphaBitCore/nexus-gateway/actions/workflows/ci.yml/badge.svg?branch=main)](.github/workflows/ci.yml)
-[![Go CI](https://github.com/AlphaBitCore/nexus-gateway/actions/workflows/go-ci.yml/badge.svg?branch=main)](.github/workflows/go-ci.yml)
+[![CI](https://github.com/itechchoice/abc-nexus-gateway/actions/workflows/ci.yml/badge.svg?branch=main)](.github/workflows/ci.yml)
+[![Go CI](https://github.com/itechchoice/abc-nexus-gateway/actions/workflows/go-ci.yml/badge.svg?branch=main)](.github/workflows/go-ci.yml)
 [![Coverage gate](https://img.shields.io/badge/coverage-%E2%89%A595%25%20per%20package-brightgreen)](./scripts/check-go-coverage.sh)
 [![Status: 1.1.0](https://img.shields.io/badge/status-1.1.0-brightgreen)](./CHANGELOG.md)
 [![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](./LICENSE)
@@ -102,52 +102,6 @@ Nexus is designed so that the compliance layer adds as little latency as possibl
 
 **Quota accounting is write-behind.** Per-request quota costs accumulate in-process and flush to Redis on a 250ms interval, removing the synchronous Redis round-trip from the hot path. Configurable back to synchronous mode (`NEXUS_QUOTA_WRITE_BEHIND=0`) for strict accounting requirements.
 
-### Benchmarking & load-testing toolkit
-
-The numbers above are produced and re-verified by a three-repo toolkit, each maintained as its own standalone repository. The load generator was previously the in-tree `tools/loadtest` and was extracted to `nexus-loadtest` (see [CHANGELOG](./CHANGELOG.md)).
-
-| Repo | Role | Key docs |
-|---|---|---|
-| **[llm-gateway-benchmark](https://github.com/AlphaBitCore/llm-gateway-benchmark)** | On-demand AWS rig (CloudFormation + Ansible) that benchmarks Nexus head-to-head against 5 other gateways (Bifrost, LiteLLM, Kong, Portkey, TensorZero) — each isolated on its own box, all hitting one shared mock upstream. Deploy to run, `delete-stack` to tear down. | [ARCHITECTURE](https://github.com/AlphaBitCore/llm-gateway-benchmark/blob/main/ARCHITECTURE.md) · [LOADTEST-RUNBOOK](https://github.com/AlphaBitCore/llm-gateway-benchmark/blob/main/docs/LOADTEST-RUNBOOK.md) · [CONTROL-BOX-RUNBOOK](https://github.com/AlphaBitCore/llm-gateway-benchmark/blob/main/docs/CONTROL-BOX-RUNBOOK.md) |
-| **[nexus-mock-provider](https://github.com/AlphaBitCore/nexus-mock-provider)** | High-performance mock upstream that speaks the real OpenAI / Gemini / Anthropic wire formats (streaming + non-streaming) and echoes requests back with plausible token usage. Removes the real, paid, rate-limited provider from the measurement so you benchmark the gateway, not the model. Listens on `:3062`. | [README](https://github.com/AlphaBitCore/nexus-mock-provider/blob/main/README.md) · [CONFIGURE-NEXUS](https://github.com/AlphaBitCore/nexus-mock-provider/blob/main/CONFIGURE-NEXUS.md) |
-| **[nexus-loadtest](https://github.com/AlphaBitCore/nexus-loadtest)** | Scenario-driven load generator for any OpenAI- or Anthropic-compatible endpoint. Simulates realistic, weighted, multi-turn traffic and reports TTFT, inter-token latency, and token throughput. Scales to tens of thousands of concurrent virtual users from one host. | [README](https://github.com/AlphaBitCore/nexus-loadtest/blob/main/README.md) · [DESIGN](https://github.com/AlphaBitCore/nexus-loadtest/blob/main/DESIGN.md) |
-
-#### Run a quick local benchmark (single gateway)
-
-Measure your local AI Gateway (`:3050`) against the mock upstream — no real provider, no cost:
-
-1. **Start the mock upstream** (in a `nexus-mock-provider` checkout):
-   ```bash
-   make run            # serves :3062, all three specs (OpenAI / Gemini / Anthropic)
-   ```
-2. **Point Nexus at the mock.** Add a provider credential / routing rule whose upstream base URL is `http://localhost:3062` (the mock accepts any API key). See [`CONFIGURE-NEXUS.md`](https://github.com/AlphaBitCore/nexus-mock-provider/blob/main/CONFIGURE-NEXUS.md) for the exact routing setup.
-3. **Run the load generator** (in a `nexus-loadtest` checkout) against the gateway, authenticating with a Nexus virtual key:
-   ```bash
-   go run ./cmd/loadtest -config profiles/realistic.json \
-     -target http://localhost:3050 -vk <your-virtual-key> -out runs/
-   ```
-4. **Read the report** at `runs/<run-id>/` — TTFT, inter-token latency, throughput, and per-tier breakdown. Use `-compare` against an earlier `summary.json` to gate regressions.
-
-#### Run the full head-to-head matrix (AWS)
-
-To compare Nexus against the other gateways on isolated boxes, use the `llm-gateway-benchmark` rig (binaries ship prebuilt in `artifacts/` — nothing compiles on-box):
-
-```bash
-# 1. Deploy infra
-aws cloudformation deploy --stack-name nexus-perf-matrix \
-  --template-file cloudformation/perf-matrix-stack.yaml --capabilities CAPABILITY_IAM \
-  --parameter-overrides KeyName=<your-key> AdminCidr=<your.ip>/32
-# 2. Provision every box (host-native)
-scripts/gen-inventory.sh nexus-perf-matrix ~/.ssh/<your-key>.pem <region>
-cd ansible && ansible-playbook -i inventory.ini site.yml
-# 3. Run the benchmark (one gateway, all tiers → a report each)
-GATEWAY=nexus scripts/bench/run-tiers.sh
-# 4. Tear down when idle (on-demand, cost control)
-aws cloudformation delete-stack --stack-name nexus-perf-matrix
-```
-
-Compare gateways by **TTFT delta** — the shared mock's latency cancels out, so the difference is the gateway's own overhead. Full procedure and the report-validity gate are in the rig's [LOADTEST-RUNBOOK](https://github.com/AlphaBitCore/llm-gateway-benchmark/blob/main/docs/LOADTEST-RUNBOOK.md).
-
 ---
 
 ## Architecture in one minute