Few questions about model code

[jingweim]

Hi authors,

Great work on this project! I have a few questions regarding the Tracktention implementation in your code:

1. Feature Output vs. Residuals: The module currently outputs the track-smoothed features directly (updated_feature_map), rather than a residual that modifies the original feature as described in the Tracktention paper. Is there a specific reason for implementing this way?
2. Initialization in Forward Pass: I noticed a nn.Linear layer being initialized inside the forward function. Is this a bug?
3. Single-sided RoPE: Rotary Positional Embeddings (RoPE) are only applied to Q during splatting and only to K during sampling. Usually, Q and K are rotated together to maintain relative positional information; what is the intuition behind using single-sided RoPE here?

Thanks for your help!

[honge-gg]

@Smirkkkk @XuweiyiChen Hi authors, thank you for the great work. I also have similar questions so +1-ing the post above by jingweim.

[Smirkkkk]

Thank you for your acknowledgement.

1. Yes, our initial thought was to take the track-smoothed features as a residual, but we found that this features had already integrated information like dino and depth features, so to make the model more space-efficient, we decide to use the track-smoothed features directly.
2. I'll fix this.
3. This is a good question. In track splatting and sampling, we use this 2D RoPE on tokens on the image to get better alignment with the 2D position in the image, but for track tokens, their dimension is decoupled from the image and already contains temporal information. As the tokens for Q and K are from different modalities, we found this single-sided RoPE more effective.

It's my own implementation of tracktention since the original paper has not been public, sorry for the confusions.