In this project, I have compared full fine-tuning in which the whole parameters of the model would be changed by retraining the model on the new dataset v.s. the LoRA fine-tuning method in which using the low rank factorization only the parameters partially would be optimised.
LoRA prevents catastrophic forgetting in model adaptation while preserving a manageable memory footprint for the model training.