test features#61
Conversation
forklady42
left a comment
forklady42
left a comment
There was a problem hiding this comment.
Thanks for tackling this! Having a held out test set will be useful. I've been wondering if we're overfitting, and this will give us a better sense of how we're exhausting our current data.
| pred_i = preds[i].numpy() | ||
| np.save(out_dir / f"{idx}.npy", pred_i) | ||
|
|
||
| self.test_outputs.append(outputs) |
There was a problem hiding this comment.
Appending all of the outputs could cause this to blow up and OOM. While the test set is small, we can leave this, but noting that we'll probably need to switch to only saving aggregated metrics or writing preds incremently to disk when we have a larger test set.
There was a problem hiding this comment.
Would you prefer I make this change now, or wait until we encounter the OOM issue?
There was a problem hiding this comment.
Is there a reason you want all the preds saved for now? If so, fine to wait until we run into OOMs. If not, let's go ahead and switch to saving only aggregated metrics.
There was a problem hiding this comment.
I would like to visualize some of the output density maps to see how they qualitatively look compared to that of labels. In any case, I have now set up save_pred as a configurable parameter in #62 to avoid the potential OOM issue for large test sets.
There was a problem hiding this comment.
Ok, there seems to be some spill over between these two PRs. I added this comment on 62 because there's additional work we should avoid when not writing preds.
… in train and test modules
hanaol
left a comment
hanaol
left a comment
There was a problem hiding this comment.
addressed the comments.
Handles logging/saving the performance metric across multiple ranks. --------- Co-authored-by: Hananeh Oliaei <ho0950@della-vis1.princeton.edu> Co-authored-by: Betsy Cannon <betsy@openathena.ai>
2 participants
Problem
Solution
Note: The
log_dirandout_dirkeys in the configuration should be set to specify where performance metrics and model outputs are saved, respectively.