Port tools/dgx to combined driving_supercombo.onnx

openpilot/tools/dgx/HARDWARE_TEST_CHECKLIST.md

@@ -61,12 +61,11 @@ print(f'EON: {HARDWARE.get_device_type()}')
 ```bash
 python openpilot/tools/dgx/benchmark_tensorrt.py --runs 50 --warmup 10
 ```
-Expected results (DGX Spark GB10):
-- [ ] driving_policy: < 0.5ms (>2000 FPS)
-- [ ] driving_vision: < 2ms (>500 FPS)
-- [ ] dmonitoring: < 1ms (>1000 FPS)
-- [ ] Combined: < 3ms (>300 FPS)
-- [ ] No CUDA errors or warnings
+Expected results (DGX Spark GB10; measured 2026-07-03, TensorRT 10.14):
+- [x] driving_supercombo: < 2.5ms (measured 1.07ms, 937 FPS)
+- [x] dmonitoring: < 1ms (measured 0.26ms, 3,891 FPS)
+- [x] Combined: < 3.5ms (measured 1.32ms, 755 FPS)
+- [x] No CUDA errors (two benign FP16 clamp precision warnings during engine build)
 
 ### 2.2 tinygrad CUDA Benchmark
 ```bash
@@ -84,8 +83,7 @@ Device.DEFAULT = 'CUDA'
 # Attempt to load each model
 import onnx
 models = [
-    'openpilot/selfdrive/modeld/models/driving_policy.onnx',
-    'openpilot/selfdrive/modeld/models/driving_vision.onnx',
+    'openpilot/selfdrive/modeld/models/driving_supercombo.onnx',
     'openpilot/selfdrive/modeld/models/dmonitoring_model.onnx',
 ]
 for m in models:
@@ -98,6 +96,11 @@ for m in models:
 ```
 - [ ] All models load successfully
 - [ ] No ONNX parsing errors
+- [ ] Embedded supercombo metadata (input_shapes, output_slices) is readable via
+      `openpilot.selfdrive.modeld.get_model_metadata.make_metadata_dict(model_path)`
+
+Note: `big_driving_supercombo.onnx` (~1.75GB) is only used with comma's USB AMD GPU
+and is not part of this checklist.
 
 ## 3. GPU Monitoring
 

openpilot/tools/dgx/README.md

@@ -162,7 +162,7 @@ ssh -i ~/.ssh/dgx_key user@dgx-spark-ip
 ### Slow tinygrad performance
 tinygrad CUDA is not optimized for Blackwell architecture. Use TensorRT:
 ```bash
-# TensorRT: ~800 FPS
+# TensorRT: ~800 FPS (historical split-model measurement)
 python openpilot/tools/dgx/benchmark_tensorrt.py
 
 # tinygrad: ~2 FPS (not optimized)
@@ -214,29 +214,37 @@ Options:
 - `--warmup N`: Number of warmup runs (default: 5)
 - `--beam N`: BEAM optimization level (default: 0, disabled)
 
-### Benchmark Results (2026-01)
+### Benchmark Results
 
-Tested on DGX Spark (GB10, Blackwell, compute 12.1):
+Tested on DGX Spark (GB10, Blackwell, compute 12.1).
 
 #### TensorRT FP16 (Recommended)
 
-| Model | Inference | FPS | vs tinygrad |
-|-------|-----------|-----|-------------|
-| driving_policy | 0.09ms | 11,355 | 659x faster |
-| driving_vision | 0.85ms | 1,181 | 432x faster |
-| dmonitoring | 0.28ms | 3,584 | 1,175x faster |
-| **Combined** | **1.21ms** | **824** | **620x faster** |
+Measured 2026-07-03 (TensorRT 10.14, driver 580.159, combined model):
 
-**TensorRT is 41x faster than comma 3X** (1.2ms vs ~50ms).
+| Model | Inference | FPS |
+|-------|-----------|-----|
+| driving_supercombo | 1.07ms ± 0.10ms | 937 |
+| dmonitoring | 0.26ms ± 0.03ms | 3,891 |
+| **Combined pipeline** | **1.32ms** | **755** |
+
+Engine build time: 23s (supercombo), 8s (dmonitoring). **TensorRT is ~38x
+faster than the comma 3X real-time budget** (1.3ms vs ~50ms).
+
+Historical split-model measurements (2026-01, pre-restructure): driving_policy
+0.09ms / driving_vision 0.85ms / combined 1.21ms (824 FPS, 620x over tinygrad).
 
 #### tinygrad CUDA (Not Optimized for Blackwell)
 
+Historical measurements (2026-01, pre-July-2026 split models):
+
 | Model | Inference | FPS |
 |-------|-----------|-----|
-| driving_policy | 58ms | 17.2 |
-| driving_vision | 366ms | 2.7 |
+| driving_policy (historical) | 58ms | 17.2 |
+| driving_vision (historical) | 366ms | 2.7 |
 | dmonitoring | 328ms | 3.0 |
-| **Combined** | **514ms** | **1.9** |
+| **Combined (historical)** | **514ms** | **1.9** |
+| driving_supercombo | not yet measured (tinygrad's CPU-side compiler needs `clang` installed) |
 
 tinygrad's CUDA backend is not yet optimized for Blackwell architecture.
 Use TensorRT for production-level performance.
@@ -249,9 +257,16 @@ The DGX Spark is ideal for fine-tuning openpilot models using DoRA (Weight-Decom
 
 - **Parameter Efficient**: Only ~2-5% of parameters are trained
 - **Preserves Base Model**: Original weights are frozen, preventing catastrophic forgetting
-- **Fast Training**: With TensorRT teacher, pseudo-label generation runs at 800+ FPS
+- **Fast Training**: With TensorRT teacher, pseudo-label generation runs at 937 FPS (supercombo, measured on GB10 2026-07)
 - **Easy Deployment**: DoRA weights can be merged back into the base model for inference
 
+> **Known limitation (2026-07)**: the student-model path is currently blocked —
+> neither onnx2pytorch nor onnx2torch can convert the opset-20
+> `driving_supercombo.onnx` to PyTorch (missing Cast v19, Gelu v20, Reshape
+> `allowzero`, and axes-as-input Reduce* v18 support). The TensorRT **teacher**
+> works fully. Until a converter catches up: fine-tune in tinygrad (the model
+> already runs there), or pin the last split-model release for torch experiments.
+
 ### Quick Start Training
 
 ```bash
@@ -273,7 +288,7 @@ python openpilot/tools/dgx/training/train.py --data comma_car_segments --epochs
 ```bash
 python openpilot/tools/dgx/training/train.py \
   --data /path/to/segments \    # Training data path
-  --model openpilot/selfdrive/modeld/models/driving_policy.onnx \
+  --model openpilot/selfdrive/modeld/models/driving_supercombo.onnx \
   --epochs 10 \                 # Number of training epochs
   --batch-size 32 \             # Batch size
   --lr 1e-4 \                   # Learning rate

openpilot/tools/dgx/TRAINING.md

@@ -51,33 +51,61 @@ route_id/
 
 **Extracted data per frame**:
 - Camera frame (YUV420 → RGB → normalized)
-- Desire vector (8-dim: lane changes, turns)
+- Desire vector (8-dim per frame: lane changes, turns; the model consumes a 25-frame `desire_pulse` history, shape (25, 8))
 - Traffic convention (2-dim: left/right hand drive)
 - GPS coordinates (for filtering/validation)
 - Vehicle state (speed, steering angle, etc.)
 
 ### 2. Teacher Model (TensorRT)
 
-Uses pre-trained comma models to generate pseudo-labels:
+Uses the pre-trained comma supercombo model to generate pseudo-labels. Since the
+July 2026 upstream restructure there is no separate vision stage: the combined
+`driving_supercombo.onnx` does vision + policy in a single forward pass.
 
 ```python
-# Teacher generates targets at 800+ FPS
-teacher_vision = TensorRTEngine("driving_vision.onnx")
-teacher_policy = TensorRTEngine("driving_policy.onnx")
-
-# For each batch of frames
-features = teacher_vision(frames)           # Visual features
-targets = teacher_policy(features, desire)  # Path predictions
+# Teacher generates targets (937 FPS measured on GB10, TensorRT 10.14, 2026-07)
+teacher = TensorRTEngine("driving_supercombo.onnx")
+
+# Model metadata (input_shapes, output_shapes, output_slices, model_checkpoint)
+# is embedded in the ONNX metadata_props -- there is no sidecar *_metadata.pkl
+from openpilot.selfdrive.modeld.get_model_metadata import make_metadata_dict
+metadata = make_metadata_dict("driving_supercombo.onnx")
+output_slices = metadata["output_slices"]
+
+# For each frame: one forward pass with the 6 supercombo inputs
+outputs = teacher(
+    img=img,                          # (1, 12, 128, 256) uint8, road camera
+    big_img=big_img,                  # (1, 12, 128, 256) uint8, wide camera
+    features_buffer=features_buffer,  # (1, 24, 512) float, rolling recurrent state
+    desire_pulse=desire_pulse,        # (1, 25, 8) float
+    traffic_convention=traffic,       # (1, 2) float
+    action_t=action_t,                # (1, 2) float
+)  # -> single flat output vector (1, 2576)
+
+# Roll the recurrent state: this frame's hidden_state (512) feeds the next frame
+hidden = outputs[:, output_slices["hidden_state"]]
+features_buffer = np.concatenate([features_buffer[:, 1:], hidden[:, None]], axis=1)
 ```
 
-**Outputs (pseudo-labels)**:
+**Outputs (pseudo-labels)**: the flat `outputs` vector is decomposed with
+`output_slices` (meta, desire_pred, pose, wide_from_device_euler, road_transform,
+lane_lines, lane_lines_prob, road_edges, lead, lead_prob, hidden_state, plan,
+desire_state, pad) and parsed with `Parser().parse_outputs()` from
+`openpilot.selfdrive.modeld.parse_model_outputs`:
 - Path predictions (33 future timestamps × 2 coords)
 - Lane line positions
 - Road edge positions
 - Lead vehicle detection
 
 ### 3. DoRA Fine-Tuning
 
+> **Known limitation (validated on DGX Spark, 2026-07)**: loading the student
+> currently fails — neither onnx2pytorch nor onnx2torch can convert the
+> opset-20 `driving_supercombo.onnx` to PyTorch (missing Cast v19, Gelu v20,
+> Reshape `allowzero`, and axes-as-input Reduce* v18). The TensorRT teacher
+> below works fully. Alternatives: fine-tune in tinygrad, or pin the last
+> split-model release for torch experiments.
+
 **DoRA** (Weight-Decomposed Low-Rank Adaptation):
 - Decomposes weights into magnitude and direction
 - Only trains low-rank direction updates
@@ -103,7 +131,7 @@ class DoRALayer(nn.Module):
 ```
 
 **Target layers for adaptation**:
-- Final dense layers in policy network
+- Final dense layers in the supercombo policy head
 - GRU hidden state projections
 - Output heads (paths, lanes, edges)
 
@@ -140,27 +168,39 @@ def train_epoch(student, teacher, dataloader, optimizer, device):
     student.train()
 
     for batch in dataloader:
-        frames = batch['frames'].to(device)        # (B, T, C, H, W)
-        desire = batch['desire'].to(device)        # (B, T, 8)
-        traffic = batch['traffic'].to(device)      # (B, 2)
-
-        # Teacher generates pseudo-labels (no grad)
+        frames = batch['frames'].to(device)              # (B, T, C, H, W) road camera
+        big_frames = batch['big_frames'].to(device)      # (B, T, C, H, W) wide camera
+        desire_pulse = batch['desire_pulse'].to(device)  # (B, T, 25, 8)
+        traffic = batch['traffic'].to(device)            # (B, 2)
+        action_t = batch['action_t'].to(device)          # (B, 2)
+
+        # Teacher generates pseudo-labels (no grad): one supercombo pass per
+        # frame, rolling the previous frames' hidden_state through features_buffer
         with torch.no_grad():
-            features = teacher.vision(frames)
-            targets = teacher.policy(features, desire, traffic)
-
-        # Student forward pass
-        hidden = None
+            feat_buf = torch.zeros(frames.shape[0], 24, 512, device=device)
+            targets = []
+            for t in range(frames.shape[1]):
+                out = teacher(frames[:, t], big_frames[:, t], feat_buf,
+                              desire_pulse[:, t], traffic, action_t)
+                targets.append(out)
+                hidden = out[:, output_slices["hidden_state"]]  # (B, 512)
+                feat_buf = torch.cat([feat_buf[:, 1:], hidden.unsqueeze(1)], dim=1)
+
+        # Student forward pass (same rolling recurrent state)
+        student_buf = torch.zeros(frames.shape[0], 24, 512, device=device)
         total_loss = 0
 
         for t in range(frames.shape[1]):
             pred, hidden = student(
                 frames[:, t],
-                desire[:, t],
+                big_frames[:, t],
+                student_buf,
+                desire_pulse[:, t],
                 traffic,
-                hidden
+                action_t,
             )
-            loss = compute_loss(pred, targets[:, t])
+            student_buf = torch.cat([student_buf[:, 1:], hidden.unsqueeze(1)], dim=1)
+            loss = compute_loss(pred, targets[t])
             total_loss += loss
 
         # Backward pass (only DoRA params have gradients)
@@ -188,10 +228,12 @@ def merge_dora_weights(model):
 def export_to_onnx(model, path):
     """Export merged model to ONNX for comma device."""
     dummy_inputs = {
-        'img': torch.randn(1, 12, 128, 256),
-        'big_img': torch.randn(1, 12, 128, 256),
-        'desire': torch.randn(1, 8),
+        'img': torch.randint(0, 256, (1, 12, 128, 256), dtype=torch.uint8),
+        'big_img': torch.randint(0, 256, (1, 12, 128, 256), dtype=torch.uint8),
+        'features_buffer': torch.randn(1, 24, 512),
+        'desire_pulse': torch.randn(1, 25, 8),
         'traffic_convention': torch.randn(1, 2),
+        'action_t': torch.randn(1, 2),
     }
     torch.onnx.export(model, dummy_inputs, path, opset_version=14)
 ```
@@ -225,7 +267,7 @@ openpilot/tools/dgx/
 
 DGX Spark advantages:
 - 128GB unified memory = large batch sizes
-- TensorRT teacher at 800+ FPS = fast pseudo-label generation
+- TensorRT teacher at 937 FPS (supercombo, measured on GB10 2026-07) = fast pseudo-label generation
 - Single device = no distributed training complexity
 
 ## Usage
@@ -245,7 +287,7 @@ python openpilot/tools/dgx/training/train.py \
 # 3. Export to ONNX
 python openpilot/tools/dgx/training/export.py \
     --checkpoint best_model.pt \
-    --output custom_driving_policy.onnx
+    --output custom_driving_supercombo.onnx
 ```
 
 ## Next Steps

openpilot/tools/dgx/benchmark_inference.py

@@ -43,77 +43,39 @@ def main():
 
   if args.beam > 0:
     os.environ["BEAM"] = str(args.beam)
+  # device must be chosen via env before tinygrad import (Device.DEFAULT is read-only now)
+  os.environ.setdefault("DEV", "CUDA")
 
   from tinygrad import Device, Tensor
-  from tinygrad.frontend.onnx import OnnxRunner  # type: ignore[import-not-found]
+  from tinygrad.nn.onnx import OnnxRunner  # type: ignore[import-not-found]
 
-  Device.DEFAULT = "CUDA"
   print(f"Device: {Device.DEFAULT}")
   print(f"BEAM: {os.environ.get('BEAM', 'disabled')}")
   print("=" * 60)
 
   models_dir = "openpilot/selfdrive/modeld/models"
 
-  # driving_policy
-  print("\n[driving_policy.onnx]")
-  policy = OnnxRunner(f"{models_dir}/driving_policy.onnx")
-  policy_inputs = {
-    "desire_pulse": Tensor(np.random.randn(1, 25, 8).astype(np.float16)),
-    "traffic_convention": Tensor(np.random.randn(1, 2).astype(np.float16)),
-    "features_buffer": Tensor(np.random.randn(1, 25, 512).astype(np.float16)),
-  }
-  stats = benchmark_model(policy, policy_inputs, args.warmup, args.runs)
-  print(f"  {stats['mean_ms']:.2f}ms +/- {stats['std_ms']:.2f}ms ({stats['fps']:.1f} FPS)")
-
-  # driving_vision
-  print("\n[driving_vision.onnx]")
-  vision = OnnxRunner(f"{models_dir}/driving_vision.onnx")
-  vision_inputs = {
-    "img": Tensor(np.random.randint(0, 255, (1, 12, 128, 256), dtype=np.uint8)),
-    "big_img": Tensor(np.random.randint(0, 255, (1, 12, 128, 256), dtype=np.uint8)),
-  }
-  stats = benchmark_model(vision, vision_inputs, args.warmup, args.runs)
+  def random_inputs(runner) -> dict:
+    inputs = {}
+    for k, v in runner.get_empty_input_data().items():
+      if "float" in str(v.dtype):
+        inputs[k] = Tensor(np.random.randn(*v.shape).astype(np.float32))
+      else:
+        inputs[k] = Tensor(np.random.randint(0, 255, v.shape, dtype=np.uint8))
+    return inputs
+
+  # driving_supercombo: merged vision+policy graph, one pass does everything
+  print("\n[driving_supercombo.onnx]")
+  supercombo = OnnxRunner(f"{models_dir}/driving_supercombo.onnx")
+  stats = benchmark_model(supercombo, random_inputs(supercombo), args.warmup, args.runs)
   print(f"  {stats['mean_ms']:.2f}ms +/- {stats['std_ms']:.2f}ms ({stats['fps']:.1f} FPS)")
 
   # dmonitoring_model
   print("\n[dmonitoring_model.onnx]")
   dmon = OnnxRunner(f"{models_dir}/dmonitoring_model.onnx")
-  dmon_empty = dmon.get_empty_input_data()
-  dmon_inputs = {}
-  for k, v in dmon_empty.items():
-    if "float" in str(v.dtype):
-      dmon_inputs[k] = Tensor(np.random.randn(*v.shape).astype(np.float32))
-    else:
-      dmon_inputs[k] = Tensor(np.random.randint(0, 255, v.shape, dtype=np.uint8))
-  stats = benchmark_model(dmon, dmon_inputs, args.warmup, args.runs)
+  stats = benchmark_model(dmon, random_inputs(dmon), args.warmup, args.runs)
   print(f"  {stats['mean_ms']:.2f}ms +/- {stats['std_ms']:.2f}ms ({stats['fps']:.1f} FPS)")
 
-  # Combined pipeline
-  print("\n[Combined Vision + Policy Pipeline]")
-  times = []
-  for _ in range(args.warmup):
-    vout = vision(vision_inputs)
-    for v in vout.values():
-      v.realize()
-    pout = policy(policy_inputs)
-    for v in pout.values():
-      v.realize()
-
-  for _ in range(args.runs):
-    start = time.perf_counter()
-    vout = vision(vision_inputs)
-    for v in vout.values():
-      v.realize()
-    pout = policy(policy_inputs)
-    for v in pout.values():
-      v.realize()
-    times.append(time.perf_counter() - start)
-
-  mean_ms = np.mean(times) * 1000
-  std_ms = np.std(times) * 1000
-  fps = 1 / np.mean(times)
-  print(f"  {mean_ms:.2f}ms +/- {std_ms:.2f}ms ({fps:.1f} FPS)")
-
   print("\n" + "=" * 60)
   print("Benchmark complete!")