H5 and Loggers fixes#257
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces major improvements to the durability and reliability of signal history logging and HDF5 array storage. The logger now supports on-disk persistence via DuckDB, ensuring experiment signal histories are durably stored and checkpointed alongside model weights. The HDF5 array store is made robust against concurrent read/write issues that previously led to file corruption. The old JSON-based logger snapshot system is removed in favor of this new, more efficient approach.
Signal history logging and checkpointing:
Added on-disk DuckDB persistence for the signal-history logger (
LoggerQueue), including automatic migration from in-memory DB and reliable checkpointing during experiment saves. The logger now checkpoints its history to disk instead of saving JSON snapshots, and the checkpoint manager coordinates this process.Removed the old
save_logger_snapshotmethod and all related JSON/zstd snapshotting logic, simplifying logger state management.Concurrency and corruption fixes for HDF5 array store:
Improved
save_arrayandsave_arrays_batchinH5ArrayStoreto acquire an exclusive write lock before opening the HDF5 file in write mode, preventing concurrent read/write access that could corrupt the file.Enhanced error handling and backup logic in
save_arrays_batchto distinguish between cases where the main file was actually modified (and needs rollback) versus untouched (where backup can be safely deleted).These changes significantly improve experiment reproducibility, logger durability, and data integrity for concurrent workloads.