During the conversion of pandas to hf.Dataset the Dataset.from_pandas builds an in-memory Arrow table, and the conversion is redone on every call. We could cache the converted dataset with save_to_disk() next to the extracted archive (keyed by the archive checksum, which the code already has for schema caching), then load_from_disk() on subsequent calls. So its memory-mapped, so RAM usage drops and reloads become near-instant.
During the conversion of
pandastohf.DatasettheDataset.from_pandasbuilds an in-memory Arrow table, and the conversion is redone on every call. We could cache the converted dataset withsave_to_disk()next to the extracted archive (keyed by the archive checksum, which the code already has for schema caching), thenload_from_disk()on subsequent calls. So its memory-mapped, so RAM usage drops and reloads become near-instant.