Runtime for MIMIC III Semi-synthetic Simulator

[yilingmialiu]

Hello,

Thank you so much for this well-written package!

I am using the MIMIC III Semi-synthetic Simulator and have encountered an issue during the data processing stage. The process has been running for over 5 hours with no progress.

Is this expected? How long does it typically take to process the data? Are there any potential optimizations or troubleshooting steps I can follow to resolve this?

Also, do we need to reprocess the data every time we run the simulation, or the script will cache the processed data for future runs?

Below is a screenshot of where the process seems to be stuck:

Any advice would be appreciated. Thanks in advance!

[Valentyn1997]

Dear @yilingmialiu,

Sorry for the delayed answer!

Indeed, the semi-synthetic dataset generation is a computationally intensive task as every trajectory has to be handled separately in a for-loop (I haven't had time to tensorize everything). In our experiments for the ICML 2022 paper, we parallelized this for-loop by using a server with ~40 CPUs.

If you don't have 40 CPUs, I can suggest shortening the max sequence length max_seq_length or setting a small max_number.

Hope that helps!

Best, Valentyn

[A-l-Mahi]

@Valentyn1997 Since you're currently occupied, I'd like to assist by tensorizing the package. This will make it accessible to users with limited computing resources.

[Valentyn1997]

@A-l-Mahi that would be wonderful! You can make a merge request.

[A-l-Mahi]

@A-l-Mahi that would be wonderful! You can make a merge request.

my pleasure