Training stability: Continue training even if a data batch was hit which causes OOM#113

Closed

RuntimeRacer wants to merge 4 commits intolifeiteng:mainfrom

RuntimeRacer:skip-broken-batches

Commits on May 2, 2023

Commits on May 3, 2023

try explicit cleanup of everything related to the batch to prevent training being stale.
RuntimeRacer
committed

Commits on May 4, 2023

avoid error in case we hit OOM already on forward pass
RuntimeRacer
committed