SAM2 decoder部分疑问

[kywish]

你好，我在测试本项目sam2的过程中，发现得到的mask结果和原SAM2的推理结果不一致:
在小目标时，mask经常偏大且超出prompt框，并且在传播几帧后容易分割到背景上导致跟踪失败。

另外，注意到代码中parms.prompt_box = min_dis_rect;使用上一帧的结果框box作为下一帧的prompt，这一点和原SAM2设计是否不一致？

我的测试视频为B站飞鸟，box为【1226, 666, 18, 20】
效果图对比：
sam2:
OrtInference:

[liyihao76]

我也有相同的疑问，目前正在尝试使用TensorRT部署，可以加个微信相互交流一下吗：18291895929 ，非常感谢。

[Anivie]

请问问题解决了吗？

[kywish]

解决了，但还没完全解决。
目前进度：

• c++onnxruntime调用全流程：cpu版，单目标
• 多目标
• fill_hole
• gpu版本

[okideal]

@kywish
我看官方后续跟踪是用point 0,0 label -1的，也就说和原博主实现不一致，是完全依赖memory中的数据来做目标跟踪
请问老哥是怎么解决这个问题的呢？mask稳定了么

[kywish]

主要是参考这个 https://github.com/axinc-ai/segment-anything-2
将image encoder、mask decoder, memory encoder和memory attention都拆分出来，需要自己控制下ptr和feature的缓存，也不是很复杂。
效果基本和原版一致，但测试下来cpu onnxruntime太慢，最近又比较忙，gpu版本还没空搞😂。
至于prompt，有用户输入的帧，label即为指定的0/1/2/3，无即为-1。

[weiyaokun]

解决了，但还没完全解决。 目前进度：

* [x]  c++onnxruntime调用全流程：cpu版，单目标

* [x]  多目标

* [ ]  fill_hole

* [ ]  gpu版本

请问多目标是怎么解决的？ 需要修改哪的代码呢？

[kywish]

https://github.com/axinc-ai/segment-anything-2
这个仓库基本上是支持多目标的，把旋转矩阵编码这部分改回6dims版本即可

[weiyaokun]

https://github.com/axinc-ai/segment-anything-2
这个仓库基本上是支持多目标的，把旋转矩阵编码这部分改回6dims版本即可

这个好像是python 版本 想跑c++版本的

[kywish]

最近比较忙，过完年有空整理一下发出来

[8utterf1y]

最近比较忙，过完年有空整理一下发布来

请问您可以分享一下吗

[kywish]

快忘了，马上开搞！
不过有个问题，就是onnxruntime对导出的SDPA会分化在CPU上执行拷贝，导致虽指定CUDAExecutionProvider但仍然很慢，我在考虑将其导出成attention算子。
还有个就是各模块的结果涉及到多维矩阵运算，暂时还没找到好用的GPU库，目前最佳选择是libtorch，但引入libtorch的话就没必要使用onnx了，这块还没规划好

[MattLiutt]

快忘了，马上开搞！ 不过有个问题，就是onnxruntime对导出的SDPA会分化在CPU上执行拷贝，导致虽指定CUDAExecutionProvider但仍然很慢，我在考虑将其导出成attention算子。 还有个就是各模块的结果涉及到多维矩阵运算，暂时还没找到好用的GPU库，目前最佳选择是libtorch，但引入libtorch的话就没必要使用onnx了，这块还没规划好

I also tried the repo you mentioned, but it's quite slow based on CPU onnxruntime. Did you manage to export in cuda version? You also mentioned MOT is possible to be done. I'm wondering if you can insert new points/box in the middle of processing? Thanks!

[kywish]

I also tried the repo you mentioned, but it's quite slow based on CPU onnxruntime. Did you manage to export in cuda version? You also mentioned MOT is possible to be done. I'm wondering if you can insert new points/box in the middle of processing? Thanks!

The CUDA version , qkv, is exported as a complete MultiHeadAttention operator.
SAM2 uses the first frame and history as input, so inserting a new target halfway is also supported, which only needs to be implemented at the software call level.

[MattLiutt]

The CUDA version , qkv, is exported as a complete MultiHeadAttention operator. SAM2 uses the first frame and history as input, so inserting a new target halfway is also supported, which only needs to be implemented at the software call level.

Hi thanks for reply! I've tried to add prompt in the middle myself. However, the fps gain seems not really impressive which is basically similar to what I've tried using original SAM2 script. May I know if got to try out with the speed test?

[kywish]

Hi thanks for reply! I've tried to add prompt in the middle myself. However, the fps gain seems not really impressive which is basically similar to what I've tried using original SAM2 script. May I know if got to try out with the speed test?

A prompt inserted midway should be treated as a new target for tracking, since its shape is inconsistent with the input structure formed by the historical cache of other targets. Therefore, its processing time should be measured separately.