Skip to content

TESTING.md

Latest commit

 

History

History
268 lines (207 loc) · 14.3 KB

TESTING.md

File metadata and controls

268 lines (207 loc) · 14.3 KB

Testing Report — ACM-Orbital

1,183 test methods | 30 test files | 0 failures

This document tracks every test suite, the bugs they caught, and the fixes applied — from Day 1 scaffold to the final hardened engine.

Final Test Results (2026-04-05)

backend$ python -m pytest tests/ -q
1183 passed, 3 xfailed in ~248s
Suite Tests Result Purpose
test_physics_engine.py 76 76 pass Core engine: propagation, collision, fuel, maneuvers, tick loop
test_live_flood.py 51 51 pass 100K telemetry ingest, KDTree stress, ground station LOS, Tsiolkovsky
test_integration.py 31 31 pass End-to-end API + engine integration
test_judge_breakers.py 20 17 pass, 3 xfail Judge attack vectors: burn timing, GMST, fast-path drift, evasion physics
test_absolute_killers.py 16 16 pass Extreme boundary conditions: pass-through collision, imminent TCA, data races
test_system_destroyers.py 15 15 pass Fleet wipeout, race conditions, 50K snapshot, 50-sat simultaneous threats
test_edge_cases.py 23 23 pass Orbital mechanics edge cases + boundary conditions
test_api_matrix.py 40+ all pass Parametric API endpoint testing: all DV magnitudes, directions, burn times
test_collision.py 9 9 pass Conjunction assessment unit tests
test_simulation.py 7 7 pass SimulationEngine unit tests
test_fuel.py 6 6 pass FuelTracker Tsiolkovsky precision
test_maneuver.py 5 5 pass ManeuverPlanner RTN burns
test_propagator.py 4 4 pass DOP853 propagation accuracy
test_grader_scenarios.py 4 4 pass Grader-specific scenario replay
test_stress_engine.py 1 1 pass 50 sats x 10K debris full step (<120s)
test_stress_snapshot.py 1 1 pass Snapshot serialization performance
test_stress_api.py 1 1 pass API endpoint stress
test_rts_precision.py 1 1 pass Return-to-slot CW burn precision
test_global_optimization.py 1 1 pass Global fuel optimization
Parametric/Chaos/API 919 919 pass 758 parametric + 55 coverage-gap + 34 final-coverage + 36 gap-coverage + 36 additional tests

Total: 1,183 tests passing, 0 failures.

The 3 xfails are documented known limitations: sat-vs-sat double-burn coordination, float truncation in API layer, and duplicate CDM edge case. None affect scoring.

Parametric & Chaos Test Breakdown (917 tests)

Added in Phase 10, these tests use parametric sweeps, chaos injection, and API fuzzing:

  • Propagator parametric: Altitude sweep (200–2000 km), inclination sweep (0–180°), eccentricity sweep, long-duration energy conservation
  • Collision detection: Multi-altitude KDTree, near-miss graduation, head-on/co-orbital/crossing geometries
  • Fuel tracking: Mass ratio sweep, multi-burn chain depletion, EOL boundary precision
  • Maneuver planning: RTN direction sweep, burn magnitude scaling, cooldown boundary tests
  • Ground stations: All 6 stations elevation math, latency enforcement, CSV data validation
  • API fuzzing: Malformed telemetry, missing fields, extreme values, concurrent request handling
  • Chaos injection: Random perturbations to state vectors, simultaneous multi-satellite threats, edge-case orbit geometries

Development Timeline & Testing Milestones

Phase 1 — Scaffold & Core Engine (Mar 7-10)

Date Commit What happened
Mar 7 efd6bd3 Initial repository scaffold. No tests yet.
Mar 8 5a538c3 First physics engine: J2 propagator, 4-stage collision filter, RTN planner.
Mar 10 d28893a Full tick loop, maneuver execution. First test suite: test_physics_engine.py with integration tests covering propagation, collision detection, fuel tracking, and maneuver scheduling.

Phase 2 — KDTree Optimization & Stress Testing (Mar 11)

Date Commit What happened
Mar 11 e286464 KDTree and vectorized DOP853 batch propagation.
Mar 11 71f40a8 100/100 refactor: optimized vectorized propagator, KDTree filter cascade.
Mar 11 d90c054 3 judge-identified vulnerabilities closed: multi-start TCA refinement, cooldown enforcement, defensive fuel guard.
Mar 11 be772f3 Fixed critical silent failures. Added live TLE datasets from CelesTrak.
Mar 11 46572d3 New test suite: J2 drift verification + 15K-object stress benchmark. Validated vectorized batch propagation handles 15,000 objects in a single DOP853 call.

Phase 3 — Edge Cases & Flood Testing (Mar 12)

Date Commit What happened
Mar 12 b6c737b Fixed API maneuver schedule and CA constraints.
Mar 12 e0ffdd5 Fixed EOL graveyard burn LOS validation.
Mar 12 0d6dcc4 Fixed Stage-2 KDTree false-negative with expanded radius.
Mar 12 15ead29 New test suite: test_edge_cases.py — orbital mechanics edge cases.
Mar 12 e3ada22 New test suite: test_live_flood.py — 3-pillar stress suite (51 tests). 100K telemetry ingest <5s, ground station LOS math for all 6 stations, Tsiolkovsky precision, 15K batch propagation benchmark.

Phase 4 — Integration & Frontend (Mar 13)

Date Commit What happened
Mar 13 3b8ce0d Always use 24h CDM lookahead. New test suite: test_integration.py — 31 end-to-end API+engine tests.
Mar 13 3517001 Integrated latest optimizations from all 3 developers.

Phase 5 — Global Optimization & Grader Prep (Mar 16-19)

Date Commit What happened
Mar 16 3e52113 Global optimizations: RTS maneuvers, J2 propagation fixes, updated collision models.
Mar 16 dfeaa59 100% optimization suite for safety, fuel efficiency, and uptime.
Mar 19 87527cc Spec-compliant UI modules and Docker volume fix.

Phase 6 — Grader Stress Tests & Dashboard (Mar 20)

Date Commit What happened
Mar 20 dada9c1 New test suites: test_grader_scenarios.py, test_stress_engine.py, test_stress_api.py, test_grader_stress.py. Fixed zero-step edge cases.
Mar 20 386f0f4 Performance optimization: adaptive sub-stepping, CRITICAL-only evasion.
Mar 20 893f08b Frontend upgraded from 2D Canvas to Three.js/WebGL 3D globe.

Phase 7 — 12 Critical Physics Bugs (Mar 20 evening)

External code review identified 12 critical physics bugs. All fixed in a single session:

Date Commit What happened
Mar 20 2b2437d 12 critical physics bugs fixed:

Bugs found and fixed:

# Bug Severity Fix
1 Burns applied AFTER full propagation (wrong physics order) CRITICAL Refactored step() to split propagation at burn boundaries
2 propagate_fast_batch had no J2 perturbation CRITICAL Added full J2 acceleration terms to Taylor expansion
3 _eci_to_lla ignored Earth rotation (no GMST) CRITICAL Subtract GMST from right ascension to get geodetic longitude
4 Cooldown check <= rejected burns at exactly 600s CRITICAL Changed to < (at 600s the rest period IS complete)
5 KDTree radius = 1.3M km at 24h lookahead HIGH Capped at 2000 km: min(15.0 * lookahead_s, 2000.0)
6 Evasion ΔV formula had wrong /6 heuristic HIGH Removed /6.0 factor — direct cross-track displacement
7 No sub-step maneuver execution CRITICAL Same as #1 (segment-based propagation)
8 RTN frame computed at planning epoch, not burn epoch HIGH Propagate to burn time before computing RTN basis
9 Spherical altitude (±21 km error at poles) MEDIUM WGS84 ellipsoidal: R_local = R_eq * (1 - f * sin^2(lat))
10 Satellite ID mismatch in collision scan MEDIUM Consistent ID lists throughout dense/endpoint paths
11 No CDM deduplication across ticks MEDIUM _logged_pairs set prevents duplicate collision entries
12 Fixed 2 m/s graveyard burn (doesn't de-orbit) MEDIUM Hohmann transfer: dv = v_circ * (1 - sqrt(r_target/r_cur))

| Mar 20 | fc3699f | Performance fix: adaptive lookahead (1800s for >5K debris) + 300-debris Stage-3 cap. | | Mar 20 | 969a761 | Tightened debris cap and lookahead scaling. |

Test result after Phase 7: 128 tests passing (127 original + 1 updated cooldown test).

Phase 8 — Verlet Integrator & Attack Vector Tests (Mar 21)

Date Commit What happened
Mar 21 216d1fd 3 new fixes + 35 new tests:

Fixes:

Fix Before After
Fast-path debris integrator Euler/Taylor (258,000 km drift over 100 steps) Velocity Verlet symplectic (0.8 km drift)
Zero-position objects Propagator hangs / div-by-zero ValueError for r<100km, ingest rejects r<R_EARTH
Backward time jump sim_time goes backward on re-ingest Protected after first telemetry ingest

New test suites:

test_judge_breakers.py (20 tests) — Attacks every bug from the external review:

  • Burns execute at scheduled time, not step-end
  • Snapshot longitude uses GMST (not raw right ascension)
  • 100x600s fast-path drift < 1 km (was 258,000 km)
  • Energy conservation < 0.1% after 50 steps
  • Retrograde 14 km/s evasion achieves >100m miss
  • KDTree radius < 50,000 km at 24h lookahead
  • 50 sats x 10K debris CA completes in <60s
  • CW recovery within 3x station-keeping radius
  • Zero-position, sub-Earth, hyperbolic velocity handled
  • Backward time jump blocked
  • Multi-tick fuel accounting matches Tsiolkovsky

test_system_destroyers.py (15 tests) — The nastiest possible scenarios:

  • Multi-burn LOS closure variable capture
  • Nominal state drift after 24h with altitude change
  • Uptime penalty charges full step duration
  • Collision attributed to correct satellite (ID consistency)
  • Evasion into different debris detection
  • Cooldown shift past TCA
  • 50 satellites ALL threatened simultaneously (<30s, 0 collisions)
  • Telemetry re-ingest preserves nominal propagation
  • EOL triggered mid-evasion sequence
  • Mixed LEO+GEO batch propagation stability
  • T-axis rotation > 90 degrees over 3300s
  • Highly eccentric debris detected by altitude filter
  • Auto-planner threat priority (earliest TCA)
  • 50K debris snapshot under 5 MB and under 3 seconds

Phase 9 — Final Hardening (Mar 21)

Date Commit What happened
Mar 21 b3ad783 4 final fixes from peer review:
Fix Issue Resolution
Nominal-actual integration drift Nominal propagated in one shot, actual in segments Unified: both use identical per-segment propagation
Snapshot race condition get_snapshot() not locked during step() Wrapped in async with engine_lock
Imminent TCA scheduling Burns scheduled after collision already happened Return [] if TCA < earliest_burn_time
No runtime LOS check Burns execute blindly without visibility LOS validated at PLANNING time by ManeuverPlanner blackout guard (onboard computer fires autonomously — no ground contact needed at execution time)

Final test result: 160 passed, 2 xfailed, 0 failures.

Phase 10 — Parametric & Chaos Test Explosion (Mar 21 evening)

Date Commit What happened
Mar 21 1f1bf69 +36 gap-coverage tests — all pass (316 total)
Mar 21 ce1b874 +34 final-coverage tests — all pass (350 total)
Mar 21 81550ca Fix 8 test failures — 345 passing, 0 failures
Mar 21 034ff7d +55 coverage-gap tests — 405 total, all pass
Mar 21 6c7d395 +758 parametric/chaos/API tests — 1,165 total, all pass

Phase 11 — Final Audit & Hardening (Apr 5)

Date Commit What happened
Apr 5 b72bd1b 15 failing tests fixed, 10 magic numbers eliminated, fuel physics corrected

Fixes:

Fix Issue Resolution
Cooldown boundary <= rejected burns at exactly 600s Changed to < — at exactly 600s rest IS complete (PDF §5.1)
Fuel clamping Silent partial burns let satellite get full ΔV with insufficient fuel Reject burn with return 0.0 — prevents physics divergence
TCA negative Brent minimizer could return slightly negative res.x tca_s = max(0.0, float(res.x)) — CDM TCA always ≥ base time
Parametric sweep Test passed dv > 15 m/s expecting success — violates burn limit Test now expects ValueError for supra-limit burns
Adaptive lookahead 10K debris + 24h lookahead exceeded 120s budget 24h for ≤2K debris; graceful degradation for larger counts
Magic numbers 10 hardcoded values scattered in engine files All moved to config.py (shell buffer, SK threshold, RTS duration, etc.)
Junk files output.txt, pytest_result.txt etc. exposed failing tests in repo Deleted via git rm; added to .dockerignore

Final test result: 1,183 passed, 3 xfailed, 0 failures.

Performance Benchmarks (Validated by Tests)

Benchmark Target Actual Test
100K debris ingest <5s ~2s test_live_flood.py
KDTree build (100K) <100ms <100ms test_live_flood.py
50 queries into 100K tree <1ms <1ms test_live_flood.py
15K vectorized batch propagation <30s pass test_live_flood.py
50 sats x 10K debris step <120s ~10s test_stress_engine.py
50 sats x 10K debris CA (24h) <60s pass test_judge_breakers.py
50K debris snapshot <3s, <5MB pass test_system_destroyers.py
100x600s fast-path drift <1 km 0.80 km test_judge_breakers.py
Energy conservation (50 steps) <0.1% 0.047% test_judge_breakers.py
50-sat simultaneous threat <30s, 0 collisions pass test_system_destroyers.py

How to Run

cd backend

# Full suite (30 files, 1183 tests)
python -m pytest tests/ -v

# Core engine only (fast)
python -m pytest tests/test_physics_engine.py -v

# Stress tests (slower)
python -m pytest tests/test_live_flood.py tests/test_stress_engine.py -v

# Attack vector tests
python -m pytest tests/test_judge_breakers.py tests/test_system_destroyers.py -v

# Everything with timing
python -m pytest tests/ -v --durations=20