[fix] Fix UT of YuanrongStorageClient
#12
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Evelynn-V <liwenlin0223l@gmail.com>
Signed-off-by: Evelynn-V <liwenlin0223l@gmail.com>
Signed-off-by: Evelynn-V <liwenlin0223l@gmail.com>
Evelynn-V
changed the title
0oshowero0
reviewed
Jan 26, 2026
| run: | | ||
| python -m pip install --upgrade pip | ||
| python -m pip install flake8 pytest build pytest_asyncio | ||
| python -m pip install flake8 pytest build pytest_asyncio pytest-mock openyuanrong-datasystem |
Collaborator
There was a problem hiding this comment.
You may place yuanrong and test dependencies in https://github.com/Ascend/TransferQueue/blob/main/pyproject.toml
Collaborator
There was a problem hiding this comment.
I can do it in next PR.
0oshowero0
approved these changes
Jan 26, 2026
0oshowero0
changed the title
YuanrongStorageClient
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fix
Added comments to mset_zcopy and mget_zcopy.
Fix the UT
tests/test_yuanrong_storage_client.pyto pass the CI.==================================================================================
Summary
When connecting to the backend of the YuanrongStorageClient, zero-copy is activated to enhance the transmission speed.
Change
Modified the
transfer_queue/storage/clients/yuanrong_client.pyto call the zero-copy interface, and performed operations such as serialization and pack.Add mget and mset UT:
tests/test_yuanrong_storage_client.py.Testing
Test on CPU:
pytest tests/test_yuanrong_storage_client.pyResult
When transmitting 512 pieces of data, each 32 MB in size, with a total data volume of 16GB:
End-to-end Get took 10s and the bandwidth was 1.6 GB/s. The time spent calling the YuanrongStorageClient interface was 2.27s with a bandwidth of 7.05 GB/s.
End-to-end Put took 3.42s and the bandwidth was 4.68 GB/s. The time spent calling the YuanrongStorageClient interface was 3.32s with a bandwidth of 4.83 GB/s.
Related Links
Pending fix for the gitcode pull request [feat] Introduce Zero-Copy to use YuanrongStorageClient for transmitting CPU Tensors
Previous issues can be viewed: [Feat]: Try zero-copy serialize objects that can be converted to memoryview
Yuanrong Datasystem PR: https://atomgit.com/openeuler/yuanrong-datasystem/pull/141