Gazebo flight-core revival + rotor-out recovery that actually holds (v1.113–v1.114) - #254
Merged
Conversation
…ransport-rs The `gazebo` feature (real gz-transport bridge) had bit-rotted: 28 E0277/E0599 errors, all "gz protobuf type does not impl prost::Message". Root cause = a prost version skew — falcon-sitl-gz pinned prost 0.14 (generating the gz message types with prost 0.14's Message), but gz-transport-rs 0.1 uses prost 0.13, so its Subscriber::recv / Publisher::publish wanted prost 0.13's Message. Pin prost to 0.13 to match. `cargo build --features gazebo` now compiles clean. First step of reviving the gz video/flight path (broken since the last recording ~v1.60). Optional-dep-only change; the default (campaign) build is unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…v1.113) Close the orphaned-core gap in the Gazebo SITL bench. Every run_* scenario re-implemented the flight cascade by hand against the example's Physics plant, so the sim exercised a PARALLEL controller, not the one that ships. - SitlBackend adapter (src/flightcore.rs): implements falcon-core's FlightBackend seam on a &mut dyn Physics plant, so the same byte-for-byte verified FlightCore (IEKF -> geo-SE(3) -> ADRC -> mixer, + rotor-out FDI + degraded-allocator recovery) flies the sim. Advances the plant once per control tick inside write_motors -> causal sense->decide->actuate ordering. - --scenario=flightcore + run_flightcore: altitude hold via the production core. Same PASS bar as the hand-rolled scenario (settle <0.5m, RMS <1.0m); measured final 0.04m, steady RMS 0.06m. - MockPhysics correctness fix: report a faithful specific-force accelerometer (thrust - drag rotated to body) instead of a constant [0,0,-g]. A constant starves the full IEKF's vertical-velocity estimate (process model says "not accelerating" while GNSS says "climbing" -> runaway, 704m before the fix). Backward-compatible at hover/level (reduces to [0,0,-g]); existing IMU tests + near-level scenarios unchanged. - test production_flightcore_holds_altitude_through_sitl_plant (28/28 green). - FV-FALCON-SITLCORE-001 verifies SWREQ-FALCON-HAL-P01 (same core, swappable backend). rivet validate PASS. Scope: the mock plant responds to collective thrust only; horizontal translation + the rotor-out RECOVERY demo need the full-physics --features gazebo backend (recordable --scenario=flightcore rotor-out = next slice). Local-verified (runner down): no CI/PR/tag yet. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
….113) The production FlightCore's single-rotor-out FDI was INERT in SITL: the SitlBackend adapter returned read_motor_rpm() -> None, so no ESC telemetry ever reached the detector. A rotor loss went unnoticed — the plumbing a recordable rotor-out flight depends on was missing. - Physics seam gains motor_rpm() (per-rotor ESC RPM, default None) and fail_rotor() (stage a single-rotor loss, default no-op). Non-breaking: existing plants/scenarios are unaffected. - MockPhysics reports rpm = achieved-throttle x ESC_RPM_FULL (8000, matching falcon-core); a failed rotor delivers 0 thrust and reads 0 RPM. Guarded by failed_rotor: Option, default None -> nominal flight byte-for-byte unchanged. - SitlBackend forwards read_motor_rpm to the plant + a fail_rotor passthrough. - --scenario=flightcore-rotorout: hover, then lose rotor 0 at the midpoint. - test production_flightcore_fdi_isolates_failed_rotor_through_sitl_seam: the core's CUSUM (relay-iekf RotorFaultDetector) isolates EXACTLY the failed rotor through the full chain (plant RPM -> adapter -> commanded-vs-achieved residual -> latch). Negative control: nominal flight reports isolated=None (no spurious firing), rotor-out reports isolated=Some(rotor). 29/29 green. Honest scope: the mock has no differential-motor torque, so it proves FDI ISOLATION (the RPM residual), NOT the 3D tilt-recovery or a post-failure altitude hold. Those need the full-physics --features gazebo plant (real per-rotor torque) — the recordable --scenario=flightcore-rotorout flight, now reachable via this seam, is the next slice. FV-FALCON-SITLCORE-001 extended: also verifies SWREQ-FALCON-FAULT-P02 (detect + isolate). rivet validate PASS. Local-verified (runner down). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…RPM (v1.113) Extend the real GazeboPhysics bridge so --scenario=flightcore-rotorout drives the production FDI + reduced-attitude recovery against the REAL 6-DOF plant (the mock proves isolation only; gz has the per-rotor torque the recovery needs). - GazeboPhysics::fail_rotor(i): stage a single-rotor loss (AtomicI32, -1=none). - step(): the failed rotor is commanded to 0 rad/s, so the real MulticopterMotorModel stops it and the plant loses its thrust + reaction torque — a genuine attitude upset. `achieved` (post-failure) is recorded. - motor_rpm(): reports achieved-throttle x ESC_RPM_FULL (8000, matching falcon-core); the failed rotor reads 0 -> the production commanded-vs-achieved FDI residual fires. Was returning the trait default None -> FDI inert on gz. Non-breaking: failed_rotor defaults -1 (no failure), so nominal gz flight is unchanged. Compile-verified (cargo build --features gazebo, Finished clean); default build + 29/29 tests unaffected (all additions gz-gated or trait-default). Runtime gz verification is pending: gz-transport-rs 0.1.0 joins discovery multicast on the OS-default interface (en0) and ignores GZ_IP, but this host's 239.255.0.7 route is en0 where multicast is dead — the C++ tools only work forced onto loopback (GZ_IP=127.0.0.1). Fix = route 239.255.0.7 to lo0 (needs sudo). Diagnosed, not a code defect in the bridge. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…s; runtime pending) The pure-Rust gz-transport-rs 0.1.0 cannot complete the subscription handshake with Harmonic (gz-transport13) — it discovers but hangs, no subscriber ever registers. Swap the GazeboPhysics transport to the C++-backed `gz-transport` 0.10 crate (links the installed gz-transport13 via gz-transport-sys + pkg-config; `gz-msgs` supplies the protobuf message types). VALIDATED: - gz-transport 0.10 + harmonic LINKS against gz-transport13 (build Finished). - The crate does in-process pub/sub with the running server: a standalone probe received 2944 IMU msgs in 3s (977 Hz) — exactly what gz-transport-rs couldn't. - This full rewrite COMPILES (cargo build --features gazebo, Finished clean). - Default (non-gazebo) build + tests UNCHANGED: FlightCore tests 2/2, suite green (all changes are inside the gazebo-gated gz_real module). Rewrite shape: GazeboPhysics holds a Node + crossbeam Receivers (subscribe_channel) + an Actuators Publisher; pump() drains the channels into latest-value caches each measure(); step() publishes gz.msgs.Actuators; rotor-out fail_rotor/motor_rpm carry over. No tokio, no prost, no local message defs. NOT YET RUNTIME-VALIDATED (honest): the full bridge flight could not be confirmed end-to-end. After dozens of gz server/client kill-relaunch cycles this session, the gz-transport Rust wrapper's Node::new/subscribe began hanging non-deterministically (the SAME probe that got 2944 msgs later hung 3/3 while C++ `gz topic` stayed healthy) — stale local-discovery/IPC state from churn, not a known code defect. Needs a CLEAN environment (fresh boot / no churn), one server + one bridge, to validate — ideally the user's display setup for recording. Build requires PKG_CONFIG_PATH to include the brew pkgconfig dir. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…nt (v1.113) At arm/spin-up the achieved rotor state (real ESC RPM, or a sim reporting the plant's actual rotor speed) lags the commanded throttle, so the FDI's commanded-vs-achieved effectiveness residual spikes on EVERY rotor for the first fraction of a second. The CUSUM would latch a PHANTOM single-rotor failure and drop a healthy vehicle into the degraded 3-rotor reduced-attitude law — a real robustness bug for hardware AND sim. Hold the FDI off for ~0.2 s (step_count < fdi_warmup_steps = loop_hz*0.2, ≥10), after which the residual reflects real faults, not the spin-up jump. A latched failure still drives the degraded path unchanged. Caught flying the production FlightCore against the real Gazebo plant, where the bridge's achieved-throttle starts at 0: nominal flight false-isolated rotor 0 (isolated=Some(0)); with the guard it is correctly isolated=None. falcon-core 43/43 green; SITL rotor-out isolation (injected past warmup) still fires; altitude hold unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…g (v1.113) Flying the production FlightCore against the real gz plant surfaced two frame bugs, each isolated by a per-tick estimator trace (FC_DEBUG=1, added here): 1. HOME OFFSET (fixed): the bridge used Home::ORIGIN (alt=0) while falcon-quad.sdf sits at 488 m MSL (Zürich), so the raw NavSat altitude projected to a constant −488 m NED — the vehicle looked like a 488 m "runaway" when it was sitting still. connect_with_home now AUTO-CAPTURES the first NavSat fix as the launch datum (standard relative-nav convention) when no explicit --home is given. Verified: true_z→0, est_z tracks, est_vz→0, no drift. 2. MAGNETOMETER FRAME (disabled): the gz-mag→NED-body conversion is UNVALIDATED (its own note: "confirmed at one heading; a yaw-sweep oracle would pin it"). Feeding it drove the IEKF yaw to ~0.88 rad (50°) at rest → the geometric controller fought a phantom yaw error → the attitude-priority mixer saturated to [1,0,1,0] (pure yaw, zero collective) → the quad never left the ground. mag_body_ned now returns None (IEKF holds yaw at its initial 0 — correct for a hover demo); re-enable once the frame is pinned with a yaw-sweep oracle. FC_DEBUG=1 env-gates a per-tick trace (true/est pos+vel, yaw, quat, accel, gyro, all 4 motors) in run_flightcore — the instrument that found both bugs. Off by default (zero normal-path cost); gazebo-gated changes leave the default build + mock tests unaffected. REMAINING (documented, not yet fixed): with home+mag corrected the yaw starts at 0, but the attitude loop is still unstable against the gz plant — within 0.5 s the estimate tumbles (accel_z flips −9.8→+9.8) while the real quad sits still. This is the geometric-SE(3)+ADRC frame/gain reconciliation vs the gz ENU dynamics (the old hand-cascade needed frame_correct_torque for exactly these axes). Next arc. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ht (v1.113) Flying the production core against the gz plant, the rootcause of the "tumble" turned out to be the FDI itself: a phantom rotor-out (its residual spiking during a transient) drops a healthy quad into the degraded 3-rotor reduced-attitude law, which is what actually destabilizes it. Proven by disabling ESC telemetry (FC_NO_RPM) — with the FDI inert the attitude loop is STABLE (stays level, no roll/pitch), isolating the real remaining issue to yaw observability. - FDI now gated on fdi_steady = tilt < ~26° (R[2][2] > 0.90) AND body rate < 1 rad/s, on top of the spin-up warmup. You cannot diagnose a dead rotor while the airframe is tilting/spinning — the controller commands large asymmetric motors there, so |commanded − achieved| spikes on the saturated rotors and false-trips the CUSUM. Only run the detector in steady flight. - FC_NO_RPM=1 (bridge diagnostic): withhold ESC telemetry so the FDI is fully inert — the isolation knob that pinned the cause. falcon-core 43/43 (the FDI-isolation test's backend is level+still → gate open, still isolates); mock rotor-out isolation (injected at t=12 s, level) still fires; altitude hold unaffected. 2/2 sitl tests green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…lity, v1.113) Flying the production core against gz with the mag disabled, yaw was unobservable → the vehicle wandered and the controller (targeting North=0) burned authority on a phantom yaw error, saturating the mixer so it never lifted. Fix the yaw observability + reference properly: - FlightBackend gains read_heading() -> Option<f32> (default None): a DIRECT absolute yaw (fused compass / GNSS heading / sim truth), fed straight to the IEKF's update_yaw — the clean alternative to inferring yaw from a raw magnetometer field. SitlBackend forwards the gz Pose_V truth heading. - FlightCore holds its LAUNCH heading, not North: yaw_setpoint tracks the heading during the warmup (no yaw error while gz init + the estimate settle) then freezes — capturing the first raw sample instead locked in a gz init transient (1.57 rad vs the true 0.94 → a frozen error that saturated yaw). desired_rate now targets yaw_setpoint. Backward-compatible: backends without a heading source keep yaw_setpoint=0 (North), unchanged behaviour — falcon-core 43/43, mock flightcore tests 2/2. RESULT + remaining (documented): yaw is now OBSERVABLE and held, and with the FDI/home/mag fixes the vehicle no longer tumbles or runs away. The last blocker is now cleanly isolated: the YAW CONTROL AXIS is wrong-signed against the gz plant — once yaw is actively held, the controller commands yaw the wrong way → a steady spin (gyro_z~4.4). frame_correct_torque is identity and the hand-cascade hovered with the same gyro/mixer, so roll/pitch are right; this is a yaw-convention mismatch between enu_quat_to_ned_yaw / the IEKF yaw / the geometric controller. NEXT: a frame_check-style per-axis oracle on the FlightCore yaw path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…t pitch (v1.113) --scenario=yaw-probe: an OPEN-LOOP yaw-sign oracle. Commands a constant +yaw torque (no attitude control) while running the IEKF alongside, and logs all four links of the yaw chain each 0.5 s so their signs can be compared in ONE run: commanded torque → gz TRUTH heading → sensed gyro_z → IEKF est-yaw. RESULT (built to test the "yaw is wrong-signed" hypothesis — and it REFUTED it): - est_yaw TRACKS the truth heading exactly (t=0.5: 0.95 vs 0.85; t=1.0: -0.42 vs -0.45; same sign, ~0.05 rad off) → the heading→estimator path is CORRECT. - gyro_z sign matches (frame-yaw AGREES) → gyro convention CORRECT. So the closed-loop "yaw spin" is NOT a yaw-sign inversion in any link. The real anomaly, from frame_check across axes: PITCH produces ~0 rate response to a pitch torque (roll +3.80, yaw +1.49, pitch -0.0002 rad/s). An uncontrollable pitch axis → the geometric controller can't stabilize pitch → the airframe tips → the heading wraps chaotically (read as a "yaw spin"). Yaw was the symptom; PITCH AUTHORITY is the disease — a QuadMixer pitch-column vs falcon-quad.sdf rotor-layout mismatch. Next: verify the rotor positions / motorNumber mapping in the SDF against MIXER_X. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… open) (v1.113) The gz "tumble/won't-lift" was a chain of gain/mixer/reference issues, not signs or geometry (frame_check + yaw-probe + the SDF read cleared those). Measured the gz plant's constants (2 kg, J=[0.0217,0.0217,0.04], motorConstant 8.55e-6, arm 0.088) and reconciled the inner loop: - ADRC b0 (control effectiveness ≈ τ_max/J) raised 30→60 (roll/pitch), 6→18 (yaw): the gz falcon-quad is ~2× more responsive than the analytic SimBackend the ADRC was matched to, so b0=30 left the rate-loop gain ~2× too high → tumble. FIXED roll/pitch: gyro→0, dead level. The ESO absorbs the residual b0 shift for the analytic plant too (falcon-core 43/43, relay-adrc 11/11). - mixer.mix → mix_thrust_floor (thrust-priority, MIX-P05): attitude-priority mix sacrificed COLLECTIVE near saturation → the quad never left the ground. Thrust- priority preserves the mean (=thrust); it now LIFTS and tracks altitude. - yaw setpoint tracks the ESTIMATE's yaw (relay_math::atan2f on est.q), not the raw gz heading — the raw heading's init transient (1.57 vs a lagging est 0.14) had commanded an initial yaw torque that spun it up. Removes the startup spin. RESULT: roll/pitch STABLE + the vehicle LIFTS + holds heading through startup, all analytic/mock tests green (43/43, 11/11, 2/2). REMAINING (OPEN, documented): the YAW rate loop is unstable in closed loop — even with zero yaw error (setpoint = estimate) the yaw spins up to ±44 rad/s, +yaw torque producing MORE spin. This CONTRADICTS the open-loop frame_check (+torque → +rate AGREE) and yaw-probe (est-yaw tracks truth) — so the open-loop yaw sign number is likely as unreliable as the pitch=0 one, or there's an ADRC/geo yaw coupling. Needs careful analysis (instrument omega_d + torque[2] vs the yaw rate), not more rebuild-loop guessing. Altitude also overshoots (2→4 m) — secondary. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ver (v1.113) THE yaw fix. Instrumenting the yaw loop (omega_d[2], torque[2] vs gyro_z) showed the geo commanded ~0 desired yaw rate (correct) but the ADRC torque grew unbounded (+2.6→+121 N·m) fighting a spin going the OTHER way: +yaw torque produced −yaw rate. The yaw torque→rate sign was INVERTED in gz. Root cause: the SDF rotor spin directions were inverted vs MIXER_X. MIXER_X assumes M0 CW / M1 CCW / M2 CW / M3 CCW; falcon-quad.sdf had rotor_0 CCW / rotor_1 CW / ... — exactly opposite. Yaw reaction torque flips with spin direction, so the mixer's yaw column fought the plant (positive feedback), masked until mix_thrust_floor stopped clamping it. The open-loop frame_check "+yaw AGREE" was unreliable (tip coupling), like pitch=0. FIX: swap the four <turningDirection> in the SDF to match MIXER_X (runtime config, no code). RESULT: yaw now LOCKS and holds (gyro_z≈0, torque≈0); with the earlier b0 (roll/ pitch) + thrust-priority mix + heading-hold, ALL THREE ATTITUDE AXES ARE STABLE on the real gz plant — the vehicle lifts and holds a steady hover. Also: gz hover_thrust 0.72→0.57 (real gz hover: ω≈757/1000 via the √pwm map) so the P-D altitude loop has ~0 steady-state error (the altitude integral wound into a slow runaway — removed). Instrumentation kept: FlightCore last_omega_d()/last_torque() getters + the FC_DEBUG yaw-loop trace. Analytic 43/43, adrc 11/11, mock 2/2 green. REMAINING (outer loop, not attitude): altitude settles ~1.7 m vs 2 m and a marginal long-run (>20 s) altitude/estimator drift (est_z diverges from true_z) — next is altitude-loop + estimator-vz tuning against the gz plant. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…entation (v1.113) Diagnosed the long-run altitude drift (est_z freezing while true_z drifts, the occasional runaway). Traced the estimator layer: est_z tracked true_z PERFECTLY for ~17-20 s, then froze while GNSS kept flowing. Mechanism: FlightCore's default pos_var (0.01 = 1 cm²) over-trusts the gz NavSat. On a long static hover the position covariance COLLAPSES; then the IEKF's NIS outlier gate (pos_gate_nis) rejects the now-large-looking (but correct) fixes, so the estimate goes DEAF and the true altitude drifts unbounded (down here, up = the 40 m runaway) — the control flying on a frozen estimate. - set_pos_var(0.25) for the gz backend (metre-class, realistic for the NavSat vs the cm default) keeps P alive and the fixes accepted: tracking now holds tight to ~22 s (was ~17). Mock unchanged (uses the noiseless default). - Instrumentation kept: SitlBackend::counters() passthrough + the FC_DEBUG altitude/estimator trace (true_z, est_z, err, est_vz, mot_mean, navsat). Mock tests 2/2 (pos_var change is gz-gated). This is the attitude-stable-hover milestone: roll/pitch/yaw rock-solid + a ~20 s recordable hover near 2 m. REMAINING (verified-IEKF follow-up, not attitude): the covariance still starves on a >20 s dead-still hover — the real fix is a process-noise floor / covariance lower bound in relay-iekf so P never collapses (careful: the estimator campaigns' NEES bounds). Altitude also settles ~1.7 m vs 2 m (hover feedforward slightly low). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…n (v1.113) The gz long-hover estimator freeze (est_z locks while true drifts → runaway) is a covariance STARVATION: on a dead-still hover the adaptive Q is ~0, the base q_accel is small, and tight fixes shrink the δv/δp covariance toward 0 → the Kalman gain with it → the NIS gate then rejects the (correct) fixes → deaf filter. - Iekf gains q_vel_extra / q_pos_extra (variance/s), added to the δv/δp diagonal each propagate, via set_process_floor(vel, pos). Default 0 = OFF, so the estimator campaigns are provably unchanged (relay-iekf 26/26, falcon-core 43/43). KEY: ADDITIVE process noise, not a diagonal clamp — the propagation turns the sustained δv uncertainty into a live δv↔δp CORRELATION, which is what lets a GNSS *position* fix correct VELOCITY. A first attempt clamped the diagonal; that left the correlation collapsed and the velocity free-ran (est_vz → −1.6, worse). - FlightCore::set_process_floor delegates to the IEKF; the gz path sets vel=0.05 (~5× q_accel), pos=0.01. Result: the gz hover now tracks tight to ~25 s (was ~17-22). Attitude remains rock-stable; a ~25 s attitude-perfect hover near 2 m is recordable. REMAINING (coupled, documented): a >25 s divergence persists — it's not pure covariance starvation but a compound of (a) thrust marginally below hover → slow true descent, (b) estimator lag during the descent, (c) a gentle altitude loop (kp_alt 0.05) that doesn't arrest it. Fully nulling it needs joint estimator-Q + altitude-gain + hover-thrust tuning against the gz plant. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Closes the coupled outer-loop tuning. With the attitude solved and the estimator kept alive (additive Q), the >25 s divergence was an underdamped altitude loop oscillating the vehicle down into the ground. Isolated and tuned each factor: - Estimator: set_process_floor(0.30, 0.05) — enough extra δv/δp process noise that the estimate tracks the true altitude for the whole flight (err < 0.2 m). - Altitude loop: FlightCore::set_altitude_gains (new; analytic defaults 0.05/0.30 unchanged) → gz uses kp=0.15, kd=1.00. The gentle default was underdamped on the gz plant (a ~25 s-period climb/overshoot/descend limit cycle); kd=1.0 damps it to a firm hold. - Feedforward + trim: hover_thrust 0.57→0.585 (the observed hover mot_mean) and a small altitude integral (0.03) to null the residual settle offset. Safe now (well-damped loop + tracking estimate) where it wound up before. RESULT: a STABLE, REPEATABLE hover — 3/3 fresh 28 s runs PASS (final_dist 0.29-0.41 m, rms_steady 0.15 m, no runaway/crash). The production FlightCore flies the real gz plant: attitude rock-stable + altitude held near 2 m + estimator tracking, end to end. Analytic 43/43, mock 2/2, relay-iekf 26/26 all green (changes default-off or gz-path-only). This completes the gz-flight arc: transport → FDI → home → mag → yaw-observability → roll/pitch gains → mixer priority → yaw sign (SDF spin dirs) → hover thrust → GNSS var → IEKF anti-starvation → altitude loop. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…3 allocation + full-loop oracle (v1.114) The single-rotor-out recovery flipped to ~180° on the real gz plant. Root cause, proven mechanically (no sim): mix_rotor_out reused the 4-rotor MIXER_X rows with the failed rotor zeroed, which breaks the zero-sum of each torque column — so the collective THRUST leaks into roll/pitch, a parasitic body moment ≈ collective toward the dead corner (at hover: roll +0.585 / pitch −0.585 at ZERO commanded torque). The reduced-attitude law can only fight it from a growing tilt; once the altitude loop raises thrust to replace lost lift the parasitic grows too → runaway → flip. Why it hid: FV-FALCON-FAULT-002 verified recovery only in an idealised attitude-only harness (constant thrust, true-state feedback, level b3_d) — there the parasitic just leans < 80°. The full production loop (IEKF + altitude + FDI) was never tested end to end. Fix — rank-3 allocation (MIX-P08): mix_rotor_out now SOLVES the 3×3 system for the healthy-rotor commands producing exactly (collective, roll, pitch), yaw free, via a collective base + zero-collective moment delta scaled by scale_to_fit (moment direction + collective preserved; a clamp can't flip a sign). Kills the diagonal-opposite rotor, drives the through-CoM pair: pure lift, zero parasitic tilt, controlled yaw spin. Floor → 0 (that rotor rests at 0); TAU_BOUND clamp keeps the Cramer solve below f32 overflow. - relay-mix-quad: 3×3 solve3/det3, rank-3 mix_rotor_out; new tests (zero-parasitic, moment-direction, small-command-exact). MIX-P08 Kani verify_mix_rotor_out_bound still SUCCESSFUL (0/377). - falcon-core: SimBackend gains fail_rotor + read_motor_rpm + quadratic rotational aero drag (rot_drag) so the SAME rigid-body plant exercises the FDI + recovery (a real rotor-out yaw spin is drag-bounded; the analytic plant omitted it, letting the spin run away and corrupt the estimator). ROTOR_OUT_FLOOR 0.15 → 0. - NEW full-loop oracle survives_single_rotor_failure_without_flipping: flies the PRODUCTION FlightCore through a rotor-0 loss — FDI isolates in a step, thrust axis stays near-level (peak < 0.5 rad), yaw spin bounded (< 10 rad/s), lands near-upright. The first test to grade the real loop, not the idealised one. - FV-FALCON-FAULT-003 traces it (verifies FAULT-P02, refines FAULT-002). Verified locally: relay-mix-quad 26 tests + MIX-P08 Kani; falcon-core 44 tests; gz fault_tolerance_chain still green; clippy clean; rivet PASS. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…MC blind spot (v1.114) The existing motor-out Monte-Carlo campaign disperses over the SAME idealised attitude-only sim as the point-test it mirrors (constant thrust, TRUE-state feedback, no estimator, no rotational drag) — its own header says so. That fidelity ceiling is why thousands of trials never caught the v1.114 rotor-out flip: the failure is a coupling (parasitic lean → descent → estimator divergence + thrust collapse → tumble) whose three ingredients (altitude loop, IEKF, drag-bounded spin) none exist in that harness. Dispersion is breadth, not fidelity. New run_fullloop_motor_out_campaign disperses 200 trials over the FULL production FlightCore (IEKF + altitude + FDI + rank-3 allocation, rot_drag plant — the survives_single_rotor_failure_without_flipping harness) across failed-rotor, hover-altitude, rotational-drag, and sensor-noise axes, asserting no-flip + bounded-spin + correct-FDI + no-NaN. Measured worst case: peak tilt 0.274 rad, yaw 5.7 rad/s, FDI detect 5 steps, 0/200 failures. A parasitic-moment regression drives peak tilt toward π and trips this guard — the guard the idealised campaign could not provide. (Altitude after the loss is the FlightSupervisor's LAND job — reported, not gated.) Runs under the existing closed-loop-sim CI gate (cargo test -p falcon-sitl-gz). Also — floor consistency: the idealised motor-out/maneuver campaigns and the fault_tolerance_chain point-test still passed FLOOR=0.15 to the rank-3 mix_rotor_out, which pins the diagonal-opposite rotor up and reinjects the parasitic it removes. Aligned all callers to floor 0 (matching production ROTOR_OUT_FLOOR) + the MIX-P08 healthy-bound checks. The idealised motor-out recovery worst peak tilt drops 0.393 → 0.040 rad as a result. - falcon-core: SimBackend::reseed for independent per-trial noise. - FV-FALCON-FAULT-003: dispersed-guard evidence added. Verified: falcon-sitl-gz 30 tests pass (release); full-loop campaign 0/200; all idealised campaigns green under the new allocation; gated-crate clippy clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bundles the gazebo-revival arc: a stable production-FlightCore hover on the real gz plant (v1.113) and a rotor-out recovery that genuinely holds — with the Monte-Carlo fidelity gap that hid its bug now closed (v1.114).
v1.113 — production FlightCore flies the real gz plant
Revived the C++-backed gz-transport bridge, flew the verified IEKF→SE(3)→ADRC→mixer cascade through the SITL seam, and reconciled control/estimator tuning for the higher-fidelity plant (yaw torque-sign from SDF spin dirs, ADRC b0 for the 2×-responsive plant, additive IEKF process noise vs covariance starvation, altitude-loop damping). Result: stable repeatable hover (final_dist 0.29–0.41 m).
v1.114 — rotor-out recovery actually holds
Root cause (proven mechanically):
mix_rotor_outreused the 4-rotor mixer rows with the failed rotor zeroed, breaking each torque column's zero-sum → the collective leaked into roll/pitch, a parasitic moment ≈ collective toward the dead corner (roll +0.585 / pitch −0.585 at zero command). It flipped the vehicle to ~180°.Why it hid: the "verified" recovery (
FV-FALCON-FAULT-002) and the motor-out Monte-Carlo campaign both ran an idealised attitude-only sim — constant thrust, true-state feedback, no estimator, no rotational drag. There the parasitic only leans <80°. Dispersion gave breadth, not fidelity.Fix: rank-3 allocation — solve the 3×3 for the healthy-rotor commands producing exactly (collective, roll, pitch), yaw free, moment-direction-preserving desaturation. Kills the diagonal-opposite rotor, drives the through-CoM pair: zero parasitic tilt, controlled spin-descent.
New guards:
survives_single_rotor_failure_without_flipping— full production loop (real IEKF + altitude + FDI) through a rotor loss: bounded tilt (<0.5 rad), bounded spin (<10 rad/s), lands upright.run_fullloop_motor_out_campaign— 200 dispersed full-loop trials; worst peak tilt 0.274 rad, 0/200 failures. The fidelity guard the idealised campaign could not provide.verify_mix_rotor_out_boundstill SUCCESSFUL;SimBackendgains rotational drag + fail_rotor + ESC RPM.Traced by
FV-FALCON-FAULT-003(verifies FAULT-P02, refines FAULT-002).Falsification
This recovery is wrong if a single-rotor-out from a settled hover is observed to exceed ~45° tilt, spin past ~10 rad/s, or fail to reach a near-upright landing — on the analytic plant or gz.
Verified locally
relay-mix-quad 26 + MIX-P08 Kani (0/377); falcon-core 44; falcon-sitl-gz 30 (all campaigns green); gated-crate clippy clean; rivet PASS.
🤖 Generated with Claude Code