Reproducing Test AUCs

[geoffreyangus]

Hi,

Thank you all for providing this repository for public use. I am trying to reproduce the results from the paper, namely the test AUC given for the 50-bag, 10-instance experiment.

I've run the implementation in this repository with the following command:

python main.py --num_bags_train 50 --num_bags_test 1000

Doing so actually overshoots the result given in the paper, by a substantial margin (0.768 (paper) vs. 0.898 (repo)). I understand that there are differences in the repository vs. the implementation in the paper (i.e. no validation set, no early stopping). However, given that the sample count in the training bags is so small, I am not convinced that such a large difference is due to the data split, and I am printing the AUC at each step for both the train and test set, which should allow me to reason about the early stopping difference. Is there something else that I am missing? Let me know, thanks!

[max-ilse]

Hey, I recently committed the original dataloader mnist_bags_loader.py you would have to use that instead of dataloader.py

…

On Tue, May 19, 2020, 4:34 AM Geoffrey Angus ***@***.***> wrote: Hi, Thank you all for providing this repository for public use. I am trying to reproduce the results from the paper, namely the test AUC given for the 50-bag, 10-instance experiment. I've run the implementation in this repository with the following command: python main.py --num_bags_train 50 --num_bags_test 1000 Doing so actually *overshoots* the result given in the paper, by a substantial margin (0.768 (paper) vs. 0.898 (repo)). I understand that there are differences in the repository vs. the implementation in the paper (i.e. no validation set, no early stopping). However, given that the sample count in the training bags is so small, I am not convinced that such a large difference is due to the data split, and I am printing the AUC at each step for both the train and test set, which should allow me to reason about the early stopping difference. Is there something else that I am missing? Let me know, thanks! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#14>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACNXTIRDEWDOEMZQ27NJ3LTRSHV3JANCNFSM4NETAD2A> .

[binli123]

Hey, I recently committed the original dataloader mnist_bags_loader.py you would have to use that instead of dataloader.py
…
On Tue, May 19, 2020, 4:34 AM Geoffrey Angus @.***> wrote: Hi, Thank you all for providing this repository for public use. I am trying to reproduce the results from the paper, namely the test AUC given for the 50-bag, 10-instance experiment. I've run the implementation in this repository with the following command: python main.py --num_bags_train 50 --num_bags_test 1000 Doing so actually overshoots the result given in the paper, by a substantial margin (0.768 (paper) vs. 0.898 (repo)). I understand that there are differences in the repository vs. the implementation in the paper (i.e. no validation set, no early stopping). However, given that the sample count in the training bags is so small, I am not convinced that such a large difference is due to the data split, and I am printing the AUC at each step for both the train and test set, which should allow me to reason about the early stopping difference. Is there something else that I am missing? Let me know, thanks! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#14>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACNXTIRDEWDOEMZQ27NJ3LTRSHV3JANCNFSM4NETAD2A .

Hi Max, I am also getting much higher AUC values around 0.89. I returned the value Y_prob, and used all truth labels and Y_prob values to compute ROC and AUC, was there anything I did wrong? I used the newly uploaded dataloader mnist_bags_loader.py. Thanks.