Check devices matching during load random seed for device. - #48
Merged
Conversation
hzhangxyz
force-pushed
the
dev/fix-seed-for-different-device
branch
from
June 18, 2025 09:03
96056c8 to
4576b6e
Compare
hzhangxyz
force-pushed
the
dev/fix-seed-for-different-device
branch
from
June 18, 2025 09:09
4576b6e to
16f0021
Compare
There was a problem hiding this comment.
Pull Request Overview
This PR centralizes random engine state handling by removing per-module dumps and adding device-type checks when loading checkpoints.
- Removed redundant
dump_random_engine_statecalls invmc.py,rldiag.py, andimag.py. - Added random state serialization (including device type) in
common.save. - Added a check in
common.mainto only load device state when the saved device type matches the current one.
Reviewed Changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| qmb/vmc.py | Removed manual random state dump in the closure (now handled by common.save). |
| qmb/rldiag.py | Removed manual random state dump in the checkpoint logic (delegated to common). |
| qmb/imag.py | Removed manual random state dump in the closure (delegated to common). |
| qmb/common.py | Imported both dump/load functions, added random state saving with device type, and added a conditional load in main. |
Comments suppressed due to low confidence (2)
qmb/common.py:155
- The new branch that skips loading when device types differ should be covered by tests to ensure it behaves as expected without raising errors. Please add unit tests simulating mismatched device states.
if data["random"]["device_type"] == self.device.type:
qmb/common.py:78
- The docstring for
save()should be updated to mention that it now also serializes the random engine state fields (host,device, anddevice_type).
Save data to checkpoint.
| """ | ||
| Save data to checkpoint. | ||
| """ | ||
| data["random"] = {"host": torch.get_rng_state(), "device": dump_random_engine_state(self.device), "device_type": self.device.type} |
There was a problem hiding this comment.
Storing only the device type may lead to mismatches on multi-GPU setups (e.g., cuda:0 vs cuda:1). Consider saving the full device spec (such as str(self.device) or the device index) to ensure the random state is loaded onto the correct device.
Suggested change
| data["random"] = {"host": torch.get_rng_state(), "device": dump_random_engine_state(self.device), "device_type": self.device.type} | |
| data["random"] = { | |
| "host": torch.get_rng_state(), | |
| "device": dump_random_engine_state(self.device), | |
| "device_spec": str(self.device) | |
| } |
BrevityD
approved these changes
Jun 19, 2025
| """ | ||
| Save data to checkpoint. | ||
| """ | ||
| data["random"] = {"host": torch.get_rng_state(), "device": dump_random_engine_state(self.device), "device_type": self.device.type} |
Member
Author
|
@BrevityD 一开始在cpu上跑然后切换到gpu上继续跑的时候会出问题。 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
When dumping random state for one device but loading it from another, the format of random state differs so program will raise error, we need to check whether the device type is unchanged before loading it.
Checklist: