How is Compact Side Network implemented in code?

[Rhtoric]

Thanks to the author for his contribution. I am confused as to how the Compact Side Network mentioned in the paper is implemented in the code. Please answer my question.

[heekhero]

Hi, thanks for your attention!
We implement the Compact Side Network within the definition of vision_transformer.py and swin_transformer.py. For example,

DTL/models/vision_transformer.py

Line 362 in bf6c51e

self.lora_dict = nn.ModuleDict({i: LoRALinear(embed_dim, embed_dim, r=r) for i in self.lora_before_blocks})

defines the CSN with a dict of LoRALinear layers (simple lora A+B), and it's used in the forward pass of ViT model (see Line 427-442 of vision_transformer.py).