Port tools/dgx to combined driving_supercombo.onnx - #49
Conversation
Upstream deleted the split driving_vision.onnx/driving_policy.onnx in the
July 2026 restructure; the merged graph has 6 inputs (img, big_img,
features_buffer (1,24,512), desire_pulse, traffic_convention, action_t) and
one flat (1,2576) output, with metadata embedded in the ONNX instead of a
sidecar pkl.
- teacher.py: single supercombo TensorRT engine; real output parsing via
embedded output_slices + modeld's Parser (replaces the placeholder that
returned zeros); emits path_mean/std from the plan position MDN plus all
parsed outputs and hidden_state features
- train.py: student runs the full 6-input graph (batch-1 ONNX, per-sample);
differentiable plan extraction feeds the Laplacian NLL — the old wiring
passed a {raw: ...} dict the loss ignored, so total was always 0; dry-run
now works (DummyTeacher instead of None); route-dataloader batch keys and
6->12 channel duplication handled
- model_runner: dummy inputs from ONNX-embedded metadata (uint8 imgs),
supercombo fallback spec
- benchmarks: single supercombo pass replaces the two-stage pipeline; stale
split-model tinygrad baselines dropped (numbers pending DGX re-validation)
- tests: contract tests for the embedded metadata, output_slices coverage,
Parser output shapes, and torch-vs-numpy plan extraction consistency
- docs: ported to the combined flow, old benchmark numbers labeled historical
Verified on CPU: metadata parse against the real ONNX, torch extraction
matches Parser exactly, full training step produces finite non-zero loss
with gradient flow. TensorRT/DoRA runs on DGX hardware still pending.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
📝 WalkthroughWalkthroughThis PR consolidates the DGX tooling from separate driving_policy/driving_vision models to a single driving_supercombo.onnx model. Benchmark scripts, model runner, teacher training pipeline, tests, and documentation are updated accordingly, including new metadata-driven input handling and recurrent state support. ChangesSupercombo Model Consolidation
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant TrainLoop as train_epoch
participant Teacher as TeacherModel
participant Engine as TensorRTEngine
participant Parser as parse_supercombo_outputs
TrainLoop->>Teacher: generate_labels(desire, features_buffer, action_t)
Teacher->>Engine: run(driving_supercombo.onnx inputs)
Engine-->>Teacher: raw_outputs
Teacher->>Parser: parse_supercombo_outputs(raw_outputs, output_slices)
Parser-->>Teacher: parsed outputs (plan, hidden_state, desire_state)
Teacher-->>TrainLoop: features, path_mean, path_std, path_prob
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Process replay diff reportReplays driving segments through this PR and compares the behavior to master. ✅ 0 changed, 66 passed, 0 errors |
tinygrad moved OnnxRunner to tinygrad.nn.onnx (matching upstream's compile_modeld.py); tinygrad.frontend.onnx no longer exists. Found running the benchmark on the DGX Spark. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Found running the benchmark on the DGX Spark. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Measured on GB10 (TensorRT 10.14, driver 580.159, 2026-07-03): - driving_supercombo: 1.07ms (937 FPS), engine builds in 23s - dmonitoring: 0.26ms (3,891 FPS); combined pipeline 1.32ms (755 FPS) - teacher generate_labels validated end-to-end on GPU (all parsed outputs, recurrent feature feedback roundtrip) - tinygrad sees CUDA (SConscript build-time selection would pick it); tinygrad baseline still needs clang installed on the box Known limitation, validated in practice: neither onnx2pytorch nor onnx2torch can convert the opset-20 supercombo to PyTorch (Cast v19, Gelu v20, Reshape allowzero, axes-as-input Reduce* v18 all unsupported), so the DoRA student path fails at model load. load_student_model now raises an actionable error instead of a KeyError four layers deep; README/TRAINING document alternatives (tinygrad fine-tuning or pinning the last split-model release). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
DGX Spark hardware validation (2026-07-03)Ran on spark-personal (GB10, driver 580.159, TensorRT 10.14): Validated working:
Found & fixed during validation (commits
Known limitation, now documented ( Not measured: tinygrad CUDA baseline — needs 🤖 Generated with Claude Code |
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (1)
openpilot/tools/dgx/model_runner.py (1)
339-344: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winLog the swallowed exception when embedded-metadata parsing fails.
except Exception: passsilently discards the failure reason before falling back to_metadata.pkl/hardcoded shapes. A genuine metadata bug or unparsable ONNX file would go unnoticed here, unlike the test fixture for the same code path which surfaces the exception viapytest.skip(f"...{e}").♻️ Suggested fix
try: from openpilot.selfdrive.modeld.get_model_metadata import make_metadata_dict self.input_shapes = make_metadata_dict(self.model_path)["input_shapes"] - except Exception: # no embedded metadata / unparsable — fall through - pass + except Exception as e: # no embedded metadata / unparsable — fall through + print(f"warning: could not parse embedded metadata for {self.model_path}: {e}")Static analysis also flags the
pickle.loadon the_metadata.pklfallback (lines 349-350) as deserialization of untrusted data — worth a quick sanity check that this sidecar is always produced by a trusted pipeline step, though the path itself (derived fromself.model_path) isn't attacker-controlled here.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@openpilot/tools/dgx/model_runner.py` around lines 339 - 344, The embedded-metadata parsing fallback in `model_runner.py` is swallowing the real failure with a bare `except Exception: pass`; update that `try`/`except` around `make_metadata_dict(self.model_path)` in the model initialization path to log the caught exception before falling back to `_metadata.pkl` or hardcoded shapes. Keep the fallback behavior, but include the exception details in the existing logging for `self.input_shapes` resolution so genuine ONNX/metadata parsing issues are visible.Source: Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@openpilot/tools/dgx/README.md`:
- Around line 241-247: The table in the README has a malformed row for
driving_supercombo because it only provides two cells while the Model,
Inference, and FPS columns require three. Update the driving_supercombo row in
the markdown table so it includes an explicit FPS cell, matching the other rows
and satisfying markdownlint MD056.
In `@openpilot/tools/dgx/TRAINING.md`:
- Around line 90-98: Update the Outputs description in TRAINING.md to match the
current supercombo plan layout: the path prediction output is 33 future
timestamps with 3 coordinates, not 2. Use the existing model contract from
Parser().parse_outputs(), Plan.POSITION, and extract_path_distribution as the
reference so the documentation reflects the x/y/z path representation.
- Around line 171-203: The training example in TRAINING.md shows a sequential
hidden-state rollout that does not match the current cold-start single-frame
flow in train.py. Update the example around the teacher/student loop to reflect
that features_buffer/action_t start zeroed and the hidden-state output is not
reused across timesteps; locate the teacher() and student() calls and remove the
rolling feat_buf/student_buf behavior from the docs example so it matches the
actual training path.
In `@openpilot/tools/dgx/training/train.py`:
- Around line 49-56: The ConvertModel error handling in train.py only wraps
KeyError and NotImplementedError, so other unsupported conversion failures can
leak as raw tracebacks. Broaden the exception filter around
ConvertModel(onnx_model) to also catch the additional converter failures
mentioned in the review, and keep raising the same RuntimeError with the
existing contextual message built from onnx_path and onnx_model.opset_import.
---
Nitpick comments:
In `@openpilot/tools/dgx/model_runner.py`:
- Around line 339-344: The embedded-metadata parsing fallback in
`model_runner.py` is swallowing the real failure with a bare `except Exception:
pass`; update that `try`/`except` around `make_metadata_dict(self.model_path)`
in the model initialization path to log the caught exception before falling back
to `_metadata.pkl` or hardcoded shapes. Keep the fallback behavior, but include
the exception details in the existing logging for `self.input_shapes` resolution
so genuine ONNX/metadata parsing issues are visible.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 579d92a9-8166-4a71-a592-f60b806d80b9
📒 Files selected for processing (11)
openpilot/tools/dgx/HARDWARE_TEST_CHECKLIST.mdopenpilot/tools/dgx/README.mdopenpilot/tools/dgx/TRAINING.mdopenpilot/tools/dgx/benchmark_inference.pyopenpilot/tools/dgx/benchmark_tensorrt.pyopenpilot/tools/dgx/model_runner.pyopenpilot/tools/dgx/tests/test_model_inference.pyopenpilot/tools/dgx/training/dataloader.pyopenpilot/tools/dgx/training/losses.pyopenpilot/tools/dgx/training/teacher.pyopenpilot/tools/dgx/training/train.py
| | Model | Inference | FPS | | ||
| |-------|-----------|-----| | ||
| | driving_policy | 58ms | 17.2 | | ||
| | driving_vision | 366ms | 2.7 | | ||
| | driving_policy (historical) | 58ms | 17.2 | | ||
| | driving_vision (historical) | 366ms | 2.7 | | ||
| | dmonitoring | 328ms | 3.0 | | ||
| | **Combined** | **514ms** | **1.9** | | ||
| | **Combined (historical)** | **514ms** | **1.9** | | ||
| | driving_supercombo | not yet measured (tinygrad's CPU-side compiler needs `clang` installed) | |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Malformed table row: missing FPS column for driving_supercombo.
The row | driving_supercombo | not yet measured (tinygrad's CPU-side compiler needs \clang` installed) | has only 2 cells but the table header (Model | Inference | FPS`) expects 3, as flagged by markdownlint (MD056).
📝 Fix the row cell count
-| driving_supercombo | not yet measured (tinygrad's CPU-side compiler needs `clang` installed) |
+| driving_supercombo | not yet measured (needs `clang`) | — |📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| | Model | Inference | FPS | | |
| |-------|-----------|-----| | |
| | driving_policy | 58ms | 17.2 | | |
| | driving_vision | 366ms | 2.7 | | |
| | driving_policy (historical) | 58ms | 17.2 | | |
| | driving_vision (historical) | 366ms | 2.7 | | |
| | dmonitoring | 328ms | 3.0 | | |
| | **Combined** | **514ms** | **1.9** | | |
| | **Combined (historical)** | **514ms** | **1.9** | | |
| | driving_supercombo | not yet measured (tinygrad's CPU-side compiler needs `clang` installed) | | |
| | Model | Inference | FPS | | |
| |-------|-----------|-----| | |
| | driving_policy (historical) | 58ms | 17.2 | | |
| | driving_vision (historical) | 366ms | 2.7 | | |
| | dmonitoring | 328ms | 3.0 | | |
| | **Combined (historical)** | **514ms** | **1.9** | | |
| | driving_supercombo | not yet measured (needs `clang`) | — | |
🧰 Tools
🪛 markdownlint-cli2 (0.22.1)
[warning] 247-247: Table column count
Expected: 3; Actual: 2; Too few cells, row will be missing data
(MD056, table-column-count)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@openpilot/tools/dgx/README.md` around lines 241 - 247, The table in the
README has a malformed row for driving_supercombo because it only provides two
cells while the Model, Inference, and FPS columns require three. Update the
driving_supercombo row in the markdown table so it includes an explicit FPS
cell, matching the other rows and satisfying markdownlint MD056.
Source: Linters/SAST tools
| **Outputs (pseudo-labels)**: the flat `outputs` vector is decomposed with | ||
| `output_slices` (meta, desire_pred, pose, wide_from_device_euler, road_transform, | ||
| lane_lines, lane_lines_prob, road_edges, lead, lead_prob, hidden_state, plan, | ||
| desire_state, pad) and parsed with `Parser().parse_outputs()` from | ||
| `openpilot.selfdrive.modeld.parse_model_outputs`: | ||
| - Path predictions (33 future timestamps × 2 coords) | ||
| - Lane line positions | ||
| - Road edge positions | ||
| - Lead vehicle detection |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
"2 coords" is stale relative to the new supercombo plan layout.
Plan.POSITION in openpilot/selfdrive/modeld/constants.py is slice(0, 3), and extract_path_distribution's own test (test_extract_path_distribution_matches_parser) asserts mean.shape == (2, 1, 33, 3) — position is 3D (x, y, z), not 2D. This line, added in this diff, still says "33 future timestamps × 2 coords," which is inconsistent with the actual output contract.
📝 Fix the coordinate count
-- Path predictions (33 future timestamps × 2 coords)
+- Path predictions (33 future timestamps × 3 coords: x, y, z)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| **Outputs (pseudo-labels)**: the flat `outputs` vector is decomposed with | |
| `output_slices` (meta, desire_pred, pose, wide_from_device_euler, road_transform, | |
| lane_lines, lane_lines_prob, road_edges, lead, lead_prob, hidden_state, plan, | |
| desire_state, pad) and parsed with `Parser().parse_outputs()` from | |
| `openpilot.selfdrive.modeld.parse_model_outputs`: | |
| - Path predictions (33 future timestamps × 2 coords) | |
| - Lane line positions | |
| - Road edge positions | |
| - Lead vehicle detection | |
| **Outputs (pseudo-labels)**: the flat `outputs` vector is decomposed with | |
| `output_slices` (meta, desire_pred, pose, wide_from_device_euler, road_transform, | |
| lane_lines, lane_lines_prob, road_edges, lead, lead_prob, hidden_state, plan, | |
| desire_state, pad) and parsed with `Parser().parse_outputs()` from | |
| `openpilot.selfdrive.modeld.parse_model_outputs`: | |
| - Path predictions (33 future timestamps × 3 coords: x, y, z) | |
| - Lane line positions | |
| - Road edge positions | |
| - Lead vehicle detection |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@openpilot/tools/dgx/TRAINING.md` around lines 90 - 98, Update the Outputs
description in TRAINING.md to match the current supercombo plan layout: the path
prediction output is 33 future timestamps with 3 coordinates, not 2. Use the
existing model contract from Parser().parse_outputs(), Plan.POSITION, and
extract_path_distribution as the reference so the documentation reflects the
x/y/z path representation.
| frames = batch['frames'].to(device) # (B, T, C, H, W) road camera | ||
| big_frames = batch['big_frames'].to(device) # (B, T, C, H, W) wide camera | ||
| desire_pulse = batch['desire_pulse'].to(device) # (B, T, 25, 8) | ||
| traffic = batch['traffic'].to(device) # (B, 2) | ||
| action_t = batch['action_t'].to(device) # (B, 2) | ||
|
|
||
| # Teacher generates pseudo-labels (no grad): one supercombo pass per | ||
| # frame, rolling the previous frames' hidden_state through features_buffer | ||
| with torch.no_grad(): | ||
| features = teacher.vision(frames) | ||
| targets = teacher.policy(features, desire, traffic) | ||
|
|
||
| # Student forward pass | ||
| hidden = None | ||
| feat_buf = torch.zeros(frames.shape[0], 24, 512, device=device) | ||
| targets = [] | ||
| for t in range(frames.shape[1]): | ||
| out = teacher(frames[:, t], big_frames[:, t], feat_buf, | ||
| desire_pulse[:, t], traffic, action_t) | ||
| targets.append(out) | ||
| hidden = out[:, output_slices["hidden_state"]] # (B, 512) | ||
| feat_buf = torch.cat([feat_buf[:, 1:], hidden.unsqueeze(1)], dim=1) | ||
|
|
||
| # Student forward pass (same rolling recurrent state) | ||
| student_buf = torch.zeros(frames.shape[0], 24, 512, device=device) | ||
| total_loss = 0 | ||
|
|
||
| for t in range(frames.shape[1]): | ||
| pred, hidden = student( | ||
| frames[:, t], | ||
| desire[:, t], | ||
| big_frames[:, t], | ||
| student_buf, | ||
| desire_pulse[:, t], | ||
| traffic, | ||
| hidden | ||
| action_t, | ||
| ) | ||
| loss = compute_loss(pred, targets[:, t]) | ||
| student_buf = torch.cat([student_buf[:, 1:], hidden.unsqueeze(1)], dim=1) | ||
| loss = compute_loss(pred, targets[t]) |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Locate the relevant docs and implementation
git ls-files openpilot/tools/dgx/TRAINING.md openpilot/tools/dgx/train.py
echo '--- TRAINING.md around cited lines ---'
sed -n '150,230p' openpilot/tools/dgx/TRAINING.md
echo '--- train.py candidates ---'
rg -n "student_forward|features_buffer|action_t|cold start|hidden_state|isinstance\(out, \(list, tuple\)\)" openpilot/tools/dgx/train.py -n -A4 -B4
echo '--- file size ---'
wc -l openpilot/tools/dgx/train.pyRepository: montge/openpilot
Length of output: 3425
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- files under openpilot/tools/dgx ---'
fd -t f . openpilot/tools/dgx
echo '--- search for student_forward / features_buffer / hidden_state / action_t anywhere ---'
rg -n "student_forward|features_buffer|hidden_state|action_t|cold start|DoRA|train_epoch|compute_loss" openpilot/tools/dgx openpilot -g '!**/node_modules/**'
echo '--- outline likely training files ---'
for f in $(fd -t f 'train.*|TRAINING\.md|.*\.py$' openpilot/tools/dgx); do
echo "### $f"
wc -l "$f"
doneRepository: montge/openpilot
Length of output: 50372
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- training/train.py around student_forward and train_epoch ---'
sed -n '140,280p' openpilot/tools/dgx/training/train.py
echo '--- training/teacher.py around buffer handling ---'
sed -n '160,240p' openpilot/tools/dgx/training/teacher.py
echo '--- TRAINING.md around the training loop ---'
sed -n '167,206p' openpilot/tools/dgx/TRAINING.mdRepository: montge/openpilot
Length of output: 10734
Clarify this example’s training flow — openpilot/tools/dgx/training/train.py currently runs cold-start single-frame inputs (features_buffer/action_t are zeroed and the hidden-state output is dropped), so the docs should not show sequential hidden-state rollout here.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@openpilot/tools/dgx/TRAINING.md` around lines 171 - 203, The training example
in TRAINING.md shows a sequential hidden-state rollout that does not match the
current cold-start single-frame flow in train.py. Update the example around the
teacher/student loop to reflect that features_buffer/action_t start zeroed and
the hidden-state output is not reused across timesteps; locate the teacher() and
student() calls and remove the rolling feat_buf/student_buf behavior from the
docs example so it matches the actual training path.
| try: | ||
| pytorch_model = ConvertModel(onnx_model) | ||
| except (KeyError, NotImplementedError) as e: | ||
| opsets = [(o.domain or "ai.onnx", o.version) for o in onnx_model.opset_import] | ||
| msg = f"onnx2pytorch cannot convert {onnx_path} (opsets {opsets}): {e!r}." | ||
| msg += " The combined driving_supercombo.onnx uses opset 20, which current ONNX->PyTorch converters do not support" | ||
| msg += " (Cast v19, Gelu v20, Reshape allowzero, Reduce* v18). See the load_student_model docstring for options." | ||
| raise RuntimeError(msg) from e |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
What exceptions does onnx2pytorch 0.5.3 ConvertModel raise for unsupported ONNX ops like Cast, Gelu, or Reduce with axes as input?
💡 Result:
The onnx2pytorch library (v0.5.3) does not explicitly define custom, user-facing exception types for unsupported ONNX operators. When ConvertModel encounters an operator it does not recognize or support during the graph conversion process, it typically raises standard Python exceptions—most commonly a KeyError, AttributeError, or NotImplementedError—depending on where the failure occurs in the conversion logic [1]. Because onnx2pytorch maps ONNX operations to specific PyTorch modules or functions, an unsupported operator generally results in an error when the converter attempts to look up or instantiate the corresponding implementation [1]. Specifically: 1. KeyError or AttributeError: If the converter attempts to access a translation function or class for a missing operator type, Python will raise these standard exceptions when the mapping lookup fails [1]. 2. NotImplementedError: This is used within the library's internal logic to signal that certain features or operator variants are not supported (e.g., the library explicitly raises a NotImplementedError if an input has a batch size greater than 1) [1]. 3. Operation-specific limitations: While operators like Cast, Gelu, or various Reduce ops (e.g., ReduceMean, ReduceSum) are standard, support depends on the specific implementation provided within the library's convert sub-package. If an operator requires specific attributes (such as dynamic axes) that the library's conversion function is not programmed to handle, the code will likely fail with a standard Python runtime error (like a TypeError or ValueError) when the internal PyTorch function is called with incompatible arguments [1]. Users attempting to debug such issues are encouraged to set the debug=True flag in the ConvertModel constructor [2][3]. This feature helps identify exactly where in the ONNX graph the conversion process fails by comparing the intermediate activations produced by the converted PyTorch model against those produced by ONNX Runtime [2][3]. There is no specialized mechanism to catch "unsupported op" errors other than standard Python exception handling.
Citations:
- 1: https://github.com/BBuf/onnx2nn/blob/master/onnx2pytorch/convert/model.py
- 2: https://github.com/Talmaj/onnx2pytorch
- 3: https://github.com/ToriML/onnx2pytorch
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the relevant section of the file with line numbers.
sed -n '1,140p' openpilot/tools/dgx/training/train.py | cat -n
# Find any local references to onnx2pytorch usage or exception handling.
rg -n "onnx2pytorch|ConvertModel|load_student_model|NotImplementedError|AttributeError|KeyError" openpilot/tools/dgx/training -SRepository: montge/openpilot
Length of output: 7381
🌐 Web query:
onnx2pytorch 0.5.3 ConvertModel unsupported op KeyError NotImplementedError AttributeError TypeError ValueError source code GitHub
💡 Result:
When encountering NotImplementedError, TypeError, or KeyError during conversion with onnx2pytorch 0.5.3, it generally indicates that the library lacks an implementation for specific ONNX operations found in your model or that there is an incompatibility with the underlying PyTorch version [1][2][3][4]. Common Causes and Troubleshooting: 1. NotImplementedError: This is the most frequent error when using onnx2pytorch [1][3][4]. It signifies that a specific ONNX operation type (e.g., LRN, HardSigmoid, RandomUniformLike) or an operation attribute (e.g., auto_pad=VALID or SAME_UPPER) has not been mapped to a corresponding PyTorch layer [1][3][5][4]. Because onnx2pytorch is a static mapping tool, if an operation is not explicitly handled in the source code, it cannot convert the model [1]. 2. Troubleshooting via Debugging: You can enable debug mode when initializing the converter to pinpoint where the conversion fails or where activations deviate. Use: ConvertModel(onnx_model, debug=True) [6]. 3. Environment and Incompatibilities: Errors like ImportError or TypeError (e.g., related to BatchNormalization or module signatures) often stem from internal changes in newer PyTorch versions that conflict with older library code [2][7]. Ensure your environment's PyTorch version is compatible with the version of onnx2pytorch installed. 4. Community Resources: Since the library relies on community contributions for operator coverage, check the GitHub repository's issue tracker to see if the unsupported operation has already been reported or if a pull request exists to add support [8][2][9]. If the specific operation required for your model is not supported, you may need to: - Simplify your ONNX model to use only standard, supported operations before conversion. - Manually implement the missing operation in the library's source code and submit a pull request. - Consider alternative conversion tools or frameworks if your model architecture is complex [9][6].
Citations:
- 1: NotImplementedError: auto_pad=VALID functionality not implemented. Talmaj/onnx2pytorch#10
- 2: cannot import name '_LazyBatchNorm' from 'torch.nn.modules.batchnorm' Talmaj/onnx2pytorch#25
- 3: NotImplementedError: Conversion not implemented for op_type=LRN. Talmaj/onnx2pytorch#53
- 4: NotImplementedError: Conversion not implemented for op_type=HardSigmoid. Talmaj/onnx2pytorch#65
- 5: auto_pad=SAME_UPPER functionality not implemented. Talmaj/onnx2pytorch#58
- 6: https://github.com/Talmaj/onnx2pytorch/blob/master/README.md
- 7: https://discuss.pytorch.org/t/convert-onnx-to-pytorch-typeerror-conv2d-init-missing-2-required-positional-arguments-in-channels-and-out-channels/215169
- 8: https://github.com/Talmaj/onnx2pytorch/issues
- 9: https://github.com/Talmaj/onnx2pytorch
🌐 Web query:
site:github.com/Talmaj/onnx2pytorch ConvertModel unsupported operator raises NotImplementedError KeyError TypeError source
💡 Result:
Errors such as NotImplementedError, KeyError, or TypeError when using the ConvertModel class in the onnx2pytorch library typically indicate missing operator support, incompatible API usage, or mismatches between ONNX and PyTorch operations [1][2][3]. These errors generally arise from the following causes: 1. NotImplementedError: This error is frequently raised when the conversion library encounters an ONNX operator or specific attribute configuration (e.g., certain auto_pad modes) that has not yet been implemented in the library's mapping logic [1]. Because ONNX is a broad specification, third-party converters often lack full coverage for every operator or attribute combination [4]. 2. TypeError: This occurs when the translated operation in PyTorch receives arguments that do not match the expected signature [2][3]. For example, if the library maps an ONNX operation to a PyTorch function that has evolved (such as changes in how reduction dimensions are handled by torch.max versus torch.amax), the call may fail due to an incorrect number or type of arguments [3]. 3. KeyError: These often occur when the conversion process attempts to look up an operator or node attribute that does not exist in the internal dictionary of supported operations or the specific graph metadata. To diagnose and potentially resolve these issues: - Use the debug mode: When initializing ConvertModel, set debug=True (e.g., ConvertModel(onnx_model, debug=True)). This enables the library to compare activations between the converted PyTorch model and the original ONNX runtime, helping identify exactly which operator or layer is causing the discrepancy [5][6]. - Check for library updates: Many of these issues are resolved in newer releases of the library as operator coverage expands [1][4]. - Inspect the ONNX graph: Use tools like Netron to visualize the model and identify if the problematic operator is a standard, widely supported operation or a custom/less common one. - Community resources: Search the issues tracker of the specific library you are using (e.g., Talmaj/onnx2pytorch) for the specific error message, as many users encounter similar operator-level incompatibilities and often provide patches or workarounds [1][2][3].
Citations:
- 1: NotImplementedError: auto_pad=VALID functionality not implemented. Talmaj/onnx2pytorch#10
- 2: GRU support Talmaj/onnx2pytorch#70
- 3: change torch.max to torch.amax when reducing on multiple dimensions Talmaj/onnx2pytorch#34
- 4: https://github.com/Talmaj/onnx2pytorch/releases
- 5: https://github.com/Talmaj/onnx2pytorch
- 6: https://github.com/Talmaj/onnx2pytorch/blob/master/README.md
🌐 Web query:
onnx2pytorch 0.5.3 Cast Gelu Reduce allowzero NotImplementedError KeyError TypeError AttributeError GitHub issues source code
💡 Result:
The onnx2pytorch library version 0.5.3 [1][2] is known to have limitations regarding specific ONNX operator implementations and conversion edge cases. Below is a synthesis of the issues associated with the keywords provided: NotImplementedError Commonly encountered when encountering specific attributes or operation modes not supported by the library. For example, historical issues noted NotImplementedError for auto_pad options like VALID or SAME_UPPER [3]. If you encounter this, it often requires manual intervention by replacing the module in the converted PyTorch model [4]. Reduce Operations (ReduceMax, ReduceSum) A known issue exists where ReduceMax (and similar reduction ops) fails when reducing on multiple dimensions because the library historically mapped these to torch.max, which only supports single-dimension reduction [5]. Users have suggested patching the operation to use torch.amax instead [5], and recent community patches have surfaced to address these limitations for specific model exports [4]. Cast, Gelu, and Other Ops While Cast and Gelu are common, they are often sources of conversion errors if the opset version of the ONNX model is newer than what the library expects, or if attributes are missing during the conversion phase [6]. allowzero The term allowzero is primarily associated with ONNX Reshape operations in PyTorch exports (introduced around Opset 14 to clarify behavior for zero-value dimensions) [7]. It is generally a warning or an export-time attribute, not a core functional bug within onnx2pytorch itself, though serialization issues involving boolean-vs-integer representation of this attribute have been reported in newer PyTorch/ONNX toolchains [8]. KeyError / TypeError / AttributeError These are frequently caused by: - Node name mismatches: A KeyError during conversion often indicates the library cannot locate an initializer or a node output in the internal graph representation, which can sometimes be resolved by ensuring node categories (e.g., Constant vs. Initializer) are correctly set in the ONNX file [9]. - Attribute Mismatch: A TypeError or AttributeError during the conversion of nodes often stems from the extract_attributes function failing to parse or map ONNX attributes to the expected PyTorch arguments [6]. Recommendation If you are encountering these errors, check your model for unsupported opset features. Many users resolve these by manually overriding the problematic layers in the generated PyTorch object: pytorch_model = ConvertModel(onnx_model) pytorch_model.LayerName = CustomReplacementModule Refer to community PRs [4] for recent patches addressing activation retention and operation mapping bugs.
Citations:
- 1: Talmaj/onnx2pytorch@v0.5.2...v0.5.3
- 2: https://github.com/Talmaj/onnx2pytorch/releases
- 3: NotImplementedError: auto_pad=VALID functionality not implemented. Talmaj/onnx2pytorch#10
- 4: patch reduce_max and ConvertModel.forward retention of output activations Talmaj/onnx2pytorch#79
- 5: change torch.max to torch.amax when reducing on multiple dimensions Talmaj/onnx2pytorch#34
- 6: https://github.com/BBuf/onnx2nn/blob/master/onnx2pytorch/convert/attribute.py
- 7: I get an error when exporting to onnx but cannot find what it means pytorch/pytorch#73129
- 8: PyTorch 2.8.0 ONNX export generates boolean values for integer-only allowzero attribute pytorch/pytorch#161941
- 9: KeyError: 'onnx::Conv_197' on converting torchvision resnet18 ENOT-AutoDL/onnx2torch#156
Broaden the ConvertModel exception filter. onnx2pytorch can also fail with TypeError or AttributeError on unsupported conversions, so catching only KeyError/NotImplementedError still lets some converter failures escape as raw tracebacks. Catch the full failure surface you want to wrap as RuntimeError.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@openpilot/tools/dgx/training/train.py` around lines 49 - 56, The ConvertModel
error handling in train.py only wraps KeyError and NotImplementedError, so other
unsupported conversion failures can leak as raw tracebacks. Broaden the
exception filter around ConvertModel(onnx_model) to also catch the additional
converter failures mentioned in the review, and keep raising the same
RuntimeError with the existing contextual message built from onnx_path and
onnx_model.opset_import.
Ports the DGX Spark tooling (TensorRT benchmarks + DoRA fine-tuning pipeline) from the split
driving_vision.onnx/driving_policy.onnx(deleted by the July 2026 upstream restructure) to the combineddriving_supercombo.onnx.Model contract
6 inputs —
img/big_img(1,12,128,256) uint8,features_buffer(1,24,512),desire_pulse(1,25,8),traffic_convention(1,2),action_t(1,2) — one flatoutputs(1,2576), sliced via ONNX-embedded metadata (get_model_metadata.make_metadata_dict; no sidecar pkl) and parsed with modeld'sParser.Highlights
path_mean/path_stdfrom the (now single-hypothesis) plan MDN plus all parsed outputs.{raw: ...}dict it ignored (total was always 0), and--dry-runcrashed onteacher=None.Verification
CPU-verified: metadata parse against the real ONNX, torch extraction matches
Parserexactly, full training step yields finite non-zero loss with gradient flow. TensorRT engine build, real FPS numbers, DoRAtarget_modules, and onnx2pytorch conversion of the merged graph still need validation on the DGX box.🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
supercombomodel for benchmarking and training.Bug Fixes
Documentation
Tests