Skip to content

Check devices matching during load random seed for device. - #48

Merged
hzhangxyz merged 1 commit into
mainfrom
dev/fix-seed-for-different-device
Jun 19, 2025
Merged

Check devices matching during load random seed for device.#48
hzhangxyz merged 1 commit into
mainfrom
dev/fix-seed-for-different-device

Conversation

@hzhangxyz

Copy link
Copy Markdown
Member

Description

When dumping random state for one device but loading it from another, the format of random state differs so program will raise error, we need to check whether the device type is unchanged before loading it.

Checklist:

Copilot AI review requested due to automatic review settings June 18, 2025 09:01

This comment was marked as outdated.

@hzhangxyz
hzhangxyz force-pushed the dev/fix-seed-for-different-device branch from 96056c8 to 4576b6e Compare June 18, 2025 09:03
@hzhangxyz
hzhangxyz requested a review from Copilot June 18, 2025 09:06

This comment was marked as outdated.

@hzhangxyz
hzhangxyz requested a review from CuSO4Deposit June 18, 2025 09:07
@hzhangxyz
hzhangxyz force-pushed the dev/fix-seed-for-different-device branch from 4576b6e to 16f0021 Compare June 18, 2025 09:09
@hzhangxyz
hzhangxyz requested a review from Copilot June 18, 2025 09:10

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR centralizes random engine state handling by removing per-module dumps and adding device-type checks when loading checkpoints.

  • Removed redundant dump_random_engine_state calls in vmc.py, rldiag.py, and imag.py.
  • Added random state serialization (including device type) in common.save.
  • Added a check in common.main to only load device state when the saved device type matches the current one.

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File Description
qmb/vmc.py Removed manual random state dump in the closure (now handled by common.save).
qmb/rldiag.py Removed manual random state dump in the checkpoint logic (delegated to common).
qmb/imag.py Removed manual random state dump in the closure (delegated to common).
qmb/common.py Imported both dump/load functions, added random state saving with device type, and added a conditional load in main.
Comments suppressed due to low confidence (2)

qmb/common.py:155

  • The new branch that skips loading when device types differ should be covered by tests to ensure it behaves as expected without raising errors. Please add unit tests simulating mismatched device states.
            if data["random"]["device_type"] == self.device.type:

qmb/common.py:78

  • The docstring for save() should be updated to mention that it now also serializes the random engine state fields (host, device, and device_type).
        Save data to checkpoint.

Comment thread qmb/common.py
"""
Save data to checkpoint.
"""
data["random"] = {"host": torch.get_rng_state(), "device": dump_random_engine_state(self.device), "device_type": self.device.type}

Copilot AI Jun 18, 2025

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Storing only the device type may lead to mismatches on multi-GPU setups (e.g., cuda:0 vs cuda:1). Consider saving the full device spec (such as str(self.device) or the device index) to ensure the random state is loaded onto the correct device.

Suggested change
data["random"] = {"host": torch.get_rng_state(), "device": dump_random_engine_state(self.device), "device_type": self.device.type}
data["random"] = {
"host": torch.get_rng_state(),
"device": dump_random_engine_state(self.device),
"device_spec": str(self.device)
}

Copilot uses AI. Check for mistakes.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

多GPU情况下会出现问题吗?我怎么觉得不会?

@hzhangxyz
hzhangxyz requested review from stevapple and removed request for CuSO4Deposit June 19, 2025 01:33
Comment thread qmb/common.py
"""
Save data to checkpoint.
"""
data["random"] = {"host": torch.get_rng_state(), "device": dump_random_engine_state(self.device), "device_type": self.device.type}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

多GPU情况下会出现问题吗?我怎么觉得不会?

@hzhangxyz

Copy link
Copy Markdown
Member Author

@BrevityD 一开始在cpu上跑然后切换到gpu上继续跑的时候会出问题。

@hzhangxyz
hzhangxyz merged commit e782ca5 into main Jun 19, 2025
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants