Skip to content

feat: harden ingestion api with rate limiting, backpressure, gzip and self-metrics#108

Merged
AKogut merged 1 commit into
mainfrom
feat/101-api-hardening
Jul 18, 2026
Merged

feat: harden ingestion api with rate limiting, backpressure, gzip and self-metrics#108
AKogut merged 1 commit into
mainfrom
feat/101-api-hardening

Conversation

@AKogut

@AKogut AKogut commented Jul 17, 2026

Copy link
Copy Markdown
Owner

Closes #101

Summary

Closes the #15/#16 hardening scope items that were missing: rate limiting, queue backpressure, gzip request decompression, structured logging, and OTel self-instrumentation.

Changes

  • Rate limiting: a small fixed-window limiter (rate-limit.ts, injectable clock) keyed by project token; returns 429 with Retry-After when exceeded.
  • Backpressure: FLAKEMETRY_MAX_QUEUE_DEPTH sheds load with 503 + Retry-After once pending jobs reach the threshold (uses the existing queue.depth()).
  • gzip request decompression: a preParsing hook inflates gzip request bodies (dropping the stale content-length/encoding), unblocking OTLP compression. The SDK exporter gains an opt-in compression flag; the reporter enables it via FLAKEMETRY_COMPRESSION=gzip.
  • Structured logging: pino logger wired in index.ts with LOG_LEVEL and authorization redaction.
  • Self-instrumentation (telemetry.ts): OTel counters (runs/executions accepted, rate-limited, backpressured), a request-duration histogram, and a queue-depth observable gauge. No-ops until FLAKEMETRY_SELF_OTEL_ENDPOINT registers an OTLP metrics exporter — the platform dogfoods OTel.

Testing

  • api: 23 tests — rate limiter unit (window, reset, key isolation) and hardening over HTTP (429 rate limit, 503 backpressure, gzip → 200) against real Postgres
  • full turbo pipeline green, 38/38 tasks
  • new env vars documented in docs/configuration.md

@AKogut AKogut self-assigned this Jul 18, 2026
@AKogut
AKogut merged commit f073199 into main Jul 18, 2026
2 checks passed
@AKogut
AKogut deleted the feat/101-api-hardening branch July 18, 2026 10:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

API hardening: rate limiting, structured logging, self-instrumentation, backpressure, gzip

1 participant