No chaos/failure testing for circuit breaker and resilience features

[yairfalse]

Problem

The circuit breaker, rate limiter, and retry logic are unit tested, but never validated under realistic failure scenarios:

• What happens when a backend dies mid-request?
• Does the circuit breaker actually trip under sustained failures?
• Do retries work correctly with real network errors?

Current State

• ✅ Unit tests for circuit breaker state machine
• ✅ Unit tests for rate limiter token bucket
• ❌ No integration tests with actual failing backends
• ❌ No chaos engineering (pod kills, network partitions)

Suggested Fix

Integration Tests

1. Deploy backend that returns 500s on demand
2. Verify circuit breaker opens after threshold
3. Verify traffic fails fast when circuit is open
4. Verify half-open state allows probe requests

Chaos Testing (optional)

1. Use Chaos Mesh or LitmusChaos in CI
2. Kill backend pods during load test
3. Inject network latency
4. Verify graceful degradation

Files

• control/src/proxy/circuit_breaker.rs
• control/src/proxy/rate_limiter.rs
• tests/integration/scenarios/ (add new scenarios)

[yairfalse]

Still valid. The diagnostics engine (7 rules) and oracle (21 cases) test correctness, but not resilience under failure. This ties into the eBPF vision (ADR-003) — when we add kernel-level TCP health sensors, chaos testing becomes the validation layer. For now, the circuit breaker and rate limiter have unit tests with concurrent access, but no integration-level failure injection.