[FEAT] JEPA framework

[Alighieri1231]

Are there any plans to add the capacity to pretrain/distill JEPA models as V-JEPA or I-JEPA where the loss is MSE but in latent space?

[liopeer]

Hi @Alighieri1231,
With the next release (this or next week) you'll be able to not only distill to any model, but it will also be possible to use any model as the teacher, so the distillation use cases should be covered by that. For pretraining, we're definitely looking more into JEPA models, but still debating whether it makes more sense to include them here or rather only in LightlySSL: https://github.com/lightly-ai/lightly
Would you prefer to have it just running with a single command (like here with LightlyTrain), or would you prefer to have customizable components (like with LightlySSL)?

[Alighieri1231]

I get it, thanks! I think lightly train is amazing so reducing the complexity of adding a new predictor (JEPA) in a single command would be great. I have not tested yet LightlySSL, thanks for the link, I will check it.