Discriminative learning rates for FineTuningTask

[vreis]

🚀 Feature

Right now we only support fine tuning by freezing the trunk weights, or training all weights together. Discriminative learning rates means we can apply different learning rates for different parts of the model, which usually leads to better performance.

Motivation

https://arxiv.org/pdf/1801.06146.pdf introduced discriminative fine-tuning in NLP. Since then it's been found to be useful in computer vision as well.

Pitch

This could be implemented in either FineTuningTask or ClassyModel. I'd rather keep ClassyModel as simple as possible and move this type of logic to the task level.

Alternatives

N/A

Additional context

N/A