TensorRT convert and Test

[BlackLee68]

hi @wenbowen123 , thanks for the great work!

I tested the model on AGX orin 32G(Jetpack 6.2), I found the 20-30-48 and the 23-36-67 shows different results when using TRT model for testing, I use the same convert code and command to generate TRT engine, but the final results seemed quite different when using run_demo_tensorrt.py, below is the res:

1, 23-36-67 Checkpoint with 4 iters, the result looks good:

2, 20-30-48 Checkpoint with 4 iters, the result looks a little strange:

I checked the post_runner, found the input is different between 23-36-67 and 20-30-48, the "features_left_16" is not used in 20-30-48 TRT engine, is this normal or i missed something when converting the model ?

Thanks a lot！

[te-shi]

Hi, I encountered the same issue. From my testing, if you are using the 20-26-39 or 20-30-48 checkpoints, you need to set normalize=False in the build_gwc_volume_triton function. Hope this helps.

[wenbowen123]

Hi, this should have been fixed in latest update.

[BlackLee68]

thanks, that helps. @wenbowen123 any plan to release training code?

[runner-9]

Hello, may I ask if your jetpack is 6.2.2? What version of tensorrt is there? The model operation result I obtained by converting with origin jetpack6.2.2 and tensorrt 10.3 is completely abnormal.

[Iridium771110]

@runner-9 same issue，jetpack 6 with cuda 12.6 & tensorrt 10.3, the result is chaos. Checked the output of feature-runner from run_demo_tensorrt semms different from the result from run_demo.

[Iridium771110]

@runner-9 I have solved it by my side.
It is not a bug from nvidia, just try img0 = img0.contiguous() and img1 = img1.contiguous() after permute operation.
Since in pytorch permutation may do not change the data in memory, but for tensorrt, it directly manages the memory.

[Iridium771110]

@wenbowen123
and one more question. As far as I know, cudagraph is a method for acceleration in inference, so use --useCudaGraph while building engine will not help for load this engine and inference use API individually, unless capture cudagraph and enqueue it use API after load engine. I wonder whether I am right.

[Mikasal]

@Iridium771110 Hello, can you run on the orin platform in real time? Although the result of my run was fine, the FPS (frames per second) was very low.

[Iridium771110]

@Mikasal my orin is AGX-64G，with MAXN-mode and jetson_clocks activated, it can run in real time.
for 23-36-67, 8-iters for post, 100ms per frame excluding img-reading, while 20-30-48 and 20-26-39 take 80ms and 90ms respectively. Since img-reading is a cpu-process, can be paralleled with gpu-process.
so 10FPS can be reached in my sight

[BlackLee68]

@Mikasal my orin is AGX-64G，with MAXN-mode and jetson_clocks activated, it can run in real time. for 23-36-67, 8-iters for post, 100ms per frame excluding img-reading, while 20-30-48 and 20-26-39 take 80ms and 90ms respectively. Since img-reading is a cpu-process, can be paralleled with gpu-process. so 10FPS can be reached in my sight

basically the same, orin AGX-32G, with MAXN-mode, for 23-36-67/8iters, using cudagraph, 11FPS can be reached

[jinyqjinyq]

@Iridium771110 my orin is also AGX-64G，with MAXN-mode and jetson_clocks activated, but i can't run in real time. for 23-36-67, 8-iters for post, 500ms-600ms per frame. The tensorrt version is 8.5.2, what can i try to get the same fps with you?

[DylanCoene]

How long did TRT conversion take on AGX-32G? I'm planning to run this next to Foundation Pose, and was able to get the .pt version running but this maxed out my RAM. I did not yet manage to convert to TRT, because this takes hours on Jetpack 6.1.