MoT or not?

[armanakbari]

Thanks for the great work. In the paper you mentioned that you used MoT architecture so you can have action and video in the same space. In MoT architecture, different modalities have different qkv and they share the attention mechanism.

But when I check the layers in the released code base, both action and video share the same qkv. Am I wrong somewhere?

[TURING132]

It seems the MoT version hasn't been released yet; only the shared version is currently available.

[armanakbari]

It seems the MoT version hasn't been released yet; only the shared version is currently available.

I see. But I can already reproduce the results with this shared backbone version. So what does the MoT version bring to the table?!

[Hrren]

How did you reproduce the results in the LingBot paper? I was unable to finetune the official base model to achieve the results reported in the paper, neither on Libero nor with the public RobotWin config and data.

[armanakbari]

How did you reproduce the results in the LingBot paper? I was unable to finetune the official base model to achieve the results reported in the paper, neither on Libero nor with the public RobotWin config and data.

I did not fine tune. By reproducing I meant to evaluate the checpoint on HF on Robotwin

[myendless1]

How did you reproduce the results in the LingBot paper? I was unable to finetune the official base model to achieve the results reported in the paper, neither on Libero nor with the public RobotWin config and data.

hello Hrren, I'm also doing finetuning on robotwin data provided by this repo. However I continue post-training by loading the provided post-trained robotwin checkpoint, I found the post training process make the model performance even worse, did you see similar phenomenon?

[Hrren]

How did you reproduce the results in the LingBot paper? I was unable to finetune the official base model to achieve the results reported in the paper, neither on Libero nor with the public RobotWin config and data.

hello Hrren, I'm also doing finetuning on robotwin data provided by this repo. However I continue post-training by loading the provided post-trained robotwin checkpoint, I found the post training process make the model performance even worse, did you see similar phenomenon?

yes, also even worse, now work at all.

[myendless1]

yes, also even worse, now work at all.

Ah, that makes me feel better. I thought it was just me doing something wrong!