推理速度的问题

[lyxlplhy]

你好，我看您的视频内small的推理速度大约为一秒10帧，但我转换tiny模型后使用cuda加速后只有2帧，这和onnx的版本有关系吗

[Aimol-l]

你好，我看您的视频内small的推理速度大约为一秒10帧，但我转换tiny模型后使用cuda加速后只有2帧，这和onnx的版本有关系吗

我们硬件不一样哦。我用tiny推理速度会是10+帧

[lyxlplhy]

你好，我看您的视频内small的推理速度大约为一秒10帧，但我转换tiny模型后使用cuda加速后只有2帧，这和onnx的版本有关系吗

我们硬件不一样哦。我用tiny推理速度会是10+帧

您的硬件是什么呢，我的3060慢这么多吗

[tppqt]

还不如用flash-atten的，我测试了可以达到13fps，small 4090上面; 还有这个tensorrt代码有python推理的版本吗，我看只有C++的

[chinshou]

怎么启用flash-attn?我这里用3050+tiny model，只能达到2s一帧. C++的程序都没有python直接跑的快

[chinshou]

找了个2080ti做了一下测试，每秒5帧的处理速度

[whittenator]

Anyway to convert this to TensorRT and get a speedup? Tiny model is taking over 300ms per frame on my GPU.

[whittenator]

Anyway to convert this to TensorRT and get a speedup? Tiny model is taking over 300ms per frame on my GPU.

I converted all 4 Small ONNX files to TensorRT for an Ada 2000 Laptop GPU at FP16 precision and here are my Performance Summaries.
For reference I used TensorRT 8.6.1, upgrading to TensorRT 10 could show significant speed gains.

At a High level I am getting the following:

1. image_encoder: ~72.9ms
2. image_decoder: ~4.3ms
3. memory_attention: ~24.7ms
4. memory_encoder: ~2.4ms

With that said, the total time for my Adap 2000 Laptop GPU to run through these 4 models is going to take ~104.3ms per frame. Unless some optimizations are done this is going to take a beefy GPU to be able to run in realtime.

Here are the detailed TensorRT performance outputs:

image_encoder.engine

[01/23/2025-18:34:46] [I] === Performance summary ===
[01/23/2025-18:34:46] [I] Throughput: 13.666 qps
[01/23/2025-18:34:46] [I] Latency: min = 66.6482 ms, max = 84.8353 ms, mean = 72.9738 ms, median = 70.98 ms, percentile(90%) = 81.2166 ms, percentile(95%) = 82.41 ms, percentile(99%) = 84.8353 ms
[01/23/2025-18:34:46] [I] Enqueue Time: min = 1.73981 ms, max = 3.02232 ms, mean = 2.10232 ms, median = 2.07886 ms, percentile(90%) = 2.28784 ms, percentile(95%) = 2.42529 ms, percentile(99%) = 3.02232 ms
[01/23/2025-18:34:46] [I] H2D Latency: min = 1.02588 ms, max = 3.24902 ms, mean = 1.17779 ms, median = 1.0354 ms, percentile(90%) = 1.51224 ms, percentile(95%) = 2.28247 ms, percentile(99%) = 3.24902 ms
[01/23/2025-18:34:46] [I] GPU Compute Time: min = 63.6693 ms, max = 80.4639 ms, mean = 69.6234 ms, median = 67.7079 ms, percentile(90%) = 78.2488 ms, percentile(95%) = 78.676 ms, percentile(99%) = 80.4639 ms
[01/23/2025-18:34:46] [I] D2H Latency: min = 1.91895 ms, max = 3.94867 ms, mean = 2.17258 ms, median = 1.92163 ms, percentile(90%) = 3.38379 ms, percentile(95%) = 3.57373 ms, percentile(99%) = 3.94867 ms
[01/23/2025-18:34:46] [I] Total Host Walltime: 3.1465 s
[01/23/2025-18:34:46] [I] Total GPU Compute Time: 2.99381 s

image_decoder.engine

[01/23/2025-18:30:40] [I] === Performance summary ===
[01/23/2025-18:30:40] [I] Throughput: 224.69 qps
[01/23/2025-18:30:40] [I] Latency: min = 3.38501 ms, max = 15.7097 ms, mean = 4.31412 ms, median = 3.55927 ms, percentile(90%) = 7.06482 ms, percentile(95%) = 10.0282 ms, percentile(99%) = 13.3904 ms
[01/23/2025-18:30:40] [I] Enqueue Time: min = 0.409424 ms, max = 1.83032 ms, mean = 0.650047 ms, median = 0.614105 ms, percentile(90%) = 0.862305 ms, percentile(95%) = 0.960449 ms, percentile(99%) = 1.06946 ms
[01/23/2025-18:30:40] [I] H2D Latency: min = 1.35327 ms, max = 6.5921 ms, mean = 1.54517 ms, median = 1.36456 ms, percentile(90%) = 1.40454 ms, percentile(95%) = 2.4353 ms, percentile(99%) = 6.1015 ms
[01/23/2025-18:30:40] [I] GPU Compute Time: min = 1.70386 ms, max = 13.2527 ms, mean = 2.41677 ms, median = 1.87073 ms, percentile(90%) = 4.68372 ms, percentile(95%) = 6.65625 ms, percentile(99%) = 8.62 ms
[01/23/2025-18:30:40] [I] D2H Latency: min = 0.323242 ms, max = 3.99805 ms, mean = 0.352183 ms, median = 0.32373 ms, percentile(90%) = 0.326904 ms, percentile(95%) = 0.334229 ms, percentile(99%) = 0.95166 ms
[01/23/2025-18:30:40] [I] Total Host Walltime: 3.00859 s
[01/23/2025-18:30:40] [I] Total GPU Compute Time: 1.63373 s

memory_attention.engine

[01/23/2025-18:44:54] [I] === Performance summary ===
[01/23/2025-18:44:54] [I] Throughput: 40.1462 qps
[01/23/2025-18:44:54] [I] Latency: min = 17.2607 ms, max = 36.7759 ms, mean = 24.7423 ms, median = 20.802 ms, percentile(90%) = 34.8137 ms, percentile(95%) = 35.4905 ms, percentile(99%) = 36.2314 ms
[01/23/2025-18:44:54] [I] Enqueue Time: min = 1.38403 ms, max = 3.66284 ms, mean = 1.91261 ms, median = 1.81812 ms, percentile(90%) = 2.30518 ms, percentile(95%) = 2.70312 ms, percentile(99%) = 3.17578 ms
[01/23/2025-18:44:54] [I] H2D Latency: min = 0.775879 ms, max = 5.31812 ms, mean = 0.927753 ms, median = 0.782227 ms, percentile(90%) = 0.806152 ms, percentile(95%) = 1.5238 ms, percentile(99%) = 4.57162 ms
[01/23/2025-18:44:54] [I] GPU Compute Time: min = 16.1567 ms, max = 35.3474 ms, mean = 23.4404 ms, median = 19.3157 ms, percentile(90%) = 33.5995 ms, percentile(95%) = 34.302 ms, percentile(99%) = 35.1335 ms
[01/23/2025-18:44:54] [I] D2H Latency: min = 0.321045 ms, max = 4.0564 ms, mean = 0.374119 ms, median = 0.321472 ms, percentile(90%) = 0.325623 ms, percentile(95%) = 0.329102 ms, percentile(99%) = 2.3999 ms
[01/23/2025-18:44:54] [I] Total Host Walltime: 3.0638 s
[01/23/2025-18:44:54] [I] Total GPU Compute Time: 2.88317 s

memory_encoder.engine

[01/23/2025-18:47:47] [I] === Performance summary ===
[01/23/2025-18:47:47] [I] Throughput: 396.61 qps
[01/23/2025-18:47:47] [I] Latency: min = 1.98547 ms, max = 8.68701 ms, mean = 2.39439 ms, median = 2.05225 ms, percentile(90%) = 4.2207 ms, percentile(95%) = 4.43848 ms, percentile(99%) = 4.90359 ms
[01/23/2025-18:47:47] [I] Enqueue Time: min = 0.204102 ms, max = 0.885132 ms, mean = 0.312146 ms, median = 0.286743 ms, percentile(90%) = 0.404175 ms, percentile(95%) = 0.458008 ms, percentile(99%) = 0.601807 ms
[01/23/2025-18:47:47] [I] H2D Latency: min = 0.682739 ms, max = 2.26224 ms, mean = 0.736502 ms, median = 0.686646 ms, percentile(90%) = 0.69751 ms, percentile(95%) = 1.10449 ms, percentile(99%) = 1.82202 ms
[01/23/2025-18:47:47] [I] GPU Compute Time: min = 1.12842 ms, max = 4.76672 ms, mean = 1.48406 ms, median = 1.19995 ms, percentile(90%) = 2.83337 ms, percentile(95%) = 3.46021 ms, percentile(99%) = 3.62195 ms
[01/23/2025-18:47:47] [I] D2H Latency: min = 0.165283 ms, max = 3.96094 ms, mean = 0.173831 ms, median = 0.165771 ms, percentile(90%) = 0.166748 ms, percentile(95%) = 0.168945 ms, percentile(99%) = 0.311279 ms
[01/23/2025-18:47:47] [I] Total Host Walltime: 3.00295 s
[01/23/2025-18:47:47] [I] Total GPU Compute Time: 1.76751 s

[8utterf1y]

Anyway to convert this to TensorRT and get a speedup? Tiny model is taking over 300ms per frame on my GPU.

请问您memory attention部分的代码是如何写的，是否能提供memory attention部分的代码，我在编写过程遇到了一些问题。