Use dedicated temporary directory for all rsync pulls#4662
Use dedicated temporary directory for all rsync pulls#4662
Conversation
There was a problem hiding this comment.
Code Review
This pull request adds the --temp-dir option to rsync for pulls and local syncs. To fully resolve potential race conditions, also apply this to the push method in tmt/guest/__init__.py. Use the self.mkdtemp() context manager to create a temporary directory on the guest.
Note: Security Review did not run due to the size of the PR.
LecrisUT
left a comment
There was a problem hiding this comment.
Interesting. Would be nice if this could be handled more centrally at the TransferOptions.to_rsync, but should be good for now
thrix
left a comment
There was a problem hiding this comment.
The fix correctly targets all 3 rsync invocations in the codebase (verified via ag). push() is correctly NOT modified since --temp-dir applies to the receiving side which is remote for push.
One concern: tempfile.TemporaryDirectory() creates in /tmp (tmpfs) while tmt workdirs are under /var/tmp/tmt/ (disk) - different filesystems. This makes rsync fall back to copy-in-place instead of atomic rename, introducing a brief window where destination files could have truncated data. See inline comment for details.
For small files (typical in tmt) this window is negligible and the fix is a clear net improvement over the original race condition.
1bae78f to
bce2f71
Compare
thrix
left a comment
There was a problem hiding this comment.
Cross-filesystem concern addressed in bce2f718. All 3 rsync invocations covered. Clean fix for the race condition.
This should prevent rsync process from creating temporary files in the destination directory tree. In case multiple rsync commands operate on the same directory structure, they may run into race condition: * rsync A is pull into the tree, creates a temporary file to hold partially transfered file; * rsync B is pushing the tree elsewhere, notices the file, and intends to push it; * rsync A completes the transfer, removes the temporary file; * rsync B sees the file is gone, and reports an error. Isolating temporary files between rsync processes and from the manipulated tree should prevent this conflict.
bce2f71 to
3ffb5b3
Compare
This should prevent rsync process from creating temporary files in the destination directory tree. In case multiple rsync commands operate on the same directory structure, they may run into race condition:
Isolating temporary files between rsync processes and from the manipulated tree should prevent this conflict.
The patch comes with new "tmpdir" attribute to provide run-specific temporary directories in a way compatible with
tempfile.TemporaryDirectory.Pull Request Checklist