kl loss explode

[zhehengluoK]

Hi, I was trying to employ the same vae way to guide a text generation task, but during my training, the kl loss always exploded and I got a nan Z after several thousands of steps. Could you please share some ideas of how to jointly train a model like this?

[ImKeTT]

Did you use kl annealing method?

Or may be try adversarial (wasserstein) distance in the latent space instead, which may help to reduce kl variation..