Skip to content

Add ONNX and TensorRT export for LT-DETR instance segmentation - #861

Merged
yutong-xiang-97 merged 9 commits into
mainfrom
yutong-trn-2165-add-onnx-and-tensorrt-export-for
Jul 13, 2026
Merged

Add ONNX and TensorRT export for LT-DETR instance segmentation#861
yutong-xiang-97 merged 9 commits into
mainfrom
yutong-trn-2165-add-onnx-and-tensorrt-export-for

Conversation

@yutong-xiang-97

@yutong-xiang-97 yutong-xiang-97 commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

What has changed and why?

Add ONNX and TensorRT export support for LT-DETR instance segmentation.

Note: it now exports both images and orig_target_size inputs so predicted boxes and masks can be scaled to the caller-provided image size - something listed as TODO in object detection. The TensorRT helper now configures matching optimization profiles for additional inputs with dynamic batch dimensions, such as orig_target_size, while rejecting unsupported dynamic non-batch dimensions.

How has it been tested?

  1. Unit tests covering
  • static and dynamic batch export
  • FP16
  • simplification
  • opset selection (ranging from 16 to 20 on CI)
  • model aliases and non-square inputs
  • prediction parity
  1. A vibe-coded verification script that checks:
  • Both inputs have a dynamic batch dimension.
  • TensorRT accepts configured batch sizes, resolves output shapes, and produces finite outputs with the expected dimensions.
  • TensorRT and ONNX Runtime produce compatible detections for identical random inputs.
    Detections are aligned order-independently using labels, boxes, and scores.
  • Boxes and scores are numerically close after alignment.
  • Labels match exactly.
  • Raw mask-logit error statistics are reported.
  • Thresholded masks are compared using pixel agreement, IoU, and agreement away from the zero-logit boundary.

Did you update CHANGELOG.md?

  • Yes
  • Not needed (internal change)

Did you update the documentation?

  • Yes
  • Not needed (internal change without effects for user)

@yutong-xiang-97
yutong-xiang-97 marked this pull request as ready for review July 11, 2026 15:39
@yutong-xiang-97

Copy link
Copy Markdown
Contributor Author

/review

@liopeer liopeer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! In the future we should not have orig_target_size in the exported model anyway. The model graph should only forward the image and yield class logits and box coordinates, working on this in #860, but this will take some time (and likely a design doc as well).

@yutong-xiang-97
yutong-xiang-97 enabled auto-merge (squash) July 13, 2026 12:54

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3fa4dc6f1c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/lightly_train/_task_models/ltdetr_instance_segmentation/task_model.py Outdated
Comment thread src/lightly_train/_task_models/ltdetr_instance_segmentation/task_model.py Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9d144269d8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@yutong-xiang-97
yutong-xiang-97 force-pushed the yutong-trn-2165-add-onnx-and-tensorrt-export-for branch from 9d14426 to ae8e2dd Compare July 13, 2026 13:51

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d0b96bcac2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@yutong-xiang-97
yutong-xiang-97 force-pushed the yutong-trn-2165-add-onnx-and-tensorrt-export-for branch from d0b96bc to a4b5977 Compare July 13, 2026 14:30
@yutong-xiang-97
yutong-xiang-97 merged commit 12a73ee into main Jul 13, 2026
12 checks passed
@yutong-xiang-97
yutong-xiang-97 deleted the yutong-trn-2165-add-onnx-and-tensorrt-export-for branch July 13, 2026 16:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants