KL loss in validation

[lizekai-richard]

Hi, may I ask why the KL loss is used during validation? This doesn't match equation 9 in the paper which is a cross-entropy loss.

[Jiacheng8]

Hi, I think it's basically the same, you can think this KL-divergence as a cross-entropy function to multiple soft labels.

[lizekai-richard]

@Jiacheng8
Are there any ablation results? How does using KL-loss compare with using cross-entropy loss?

[Jiacheng8]

The two are equivalent except for a constant term, but of course the optimisation will be a little different for the lowest value, I will try to do an ablation result on this.

[lizekai-richard]

@Jiacheng8 Thanks. My major concern is that using kl-loss results in stronger knowledge distillation enhancement, especially when temperature is also used in your case. I'm wondering if adopting this evaluation strategy is the main reason for performance improvement instead of the dataset itself having a higher quality. Yet, the paper didn't present ablation on this.