Use native, leaveOpen tar streaming to decompress the tar without copy#54757
Draft
nagilson wants to merge 1 commit into
Draft
Conversation
… copy Attempt to resolve the following issue in CI (We're on a new branch but want to fix a different CI flake issue with dotnetup or the install scripts around dotnetup. Issue 1 is that the install script fallback failed because it got an empty architecture string: ''. This is likely an issue in origin/dotnet/sdk main branch wit hthe engineering scripts. We dont necessarily have the right copy of that script on this branch so you might need to fix that in a worktree. Issue 2 is that the archive failed - I wonder if this is a race condition, a file deleition, missing file, or other issue with dotnetup extraction. There is only 1 instance of dotnetup running at a time. [Pipelines - Run 20260612.115 logs](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1462304&view=logs&jobId=2709a726-7db6-5829-ca7b-958b9d664f9e&j=2709a726-7db6-5829-ca7b-958b9d664f9e&t=cbd5e4d2-ff07-56e0-f071-f4be5ccd6b94) - fix this issue in this branch or investigate potential causes Installing .NET SDK 11.0.100-preview.5.26227.104 to D:\a\_work\1\s\.dotnet... ⚠ Daily builds are not code-signed. Only the SHA-512 hash is verified. Installing .NET SDK 11.0.100-preview.5.26227.104: 0% Installing .NET SDK 11.0.100-preview.5.26227.104: 48% Installing .NET SDK 11.0.100-preview.5.26227.104: 83% Installed at D:\a\_work\1\s\.dotnet: .NET SDK 11.0.100-preview.5.26227.104 Downloading package Microsoft.DotNet.Arcade.Sdk, version 11.0.0-beta.26277.111. Package 'Microsoft.DotNet.Arcade.Sdk' (11.0.0-beta.26277.111) successfully downloaded to 'D:\a\_work\1\s\.packages\'. Downloading vswhere 3.1.7 INFO: Tests will run against full MSBuild in C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Current\Bin\msbuild.exe dotnetup binary is less than 24 hours old; skipping re-download. Installing dotnet (runtime) 6.0.36, dotnet (runtime) 7.0.20, dotnet (runtime) 8.0.28, dotnet (runtime) 9.0.17, dotnet (runtime) 10.0.9, dotnet (runtime) 11.0.0-preview.6.26277.111, aspnet (runtime) 11.0.0-preview.6.26277.111 to D:\a\_work\1\s\.dotnet... ⚠ Daily builds are not code-signed. Only the SHA-512 hash is verified. Installing dotnet (runtime) 10.0.9: 59% Installing dotnet (runtime) 11.0.0-preview.6.26277.111: 0% Installing dotnet (runtime) 9.0.17: 58% Installed at D:\a\_work\1\s\.dotnet: dotnet (runtime) 6.0.36 dotnet (runtime) 10.0.9 dotnet (runtime) 8.0.28 dotnet (runtime) 7.0.20 dotnet (runtime) 9.0.17 aspnet (runtime) 11.0.0-preview.6.26277.111 The following installs failed: dotnet (runtime) 11.0.0-preview.6.26277.111: Failed to extract .NET archive for version 11.0.0-preview.6.26277.111: Could not find a part of the path 'C:\Users\cloudtest\AppData\Local\Temp\ecl2kgt3q\dotnet-62f51424-63c9-4e6a-b09b-380303c69000.tar'. Error: Failed to extract .NET archive for version 11.0.0-preview.6.26277.111: Could not find a part of the path 'C:\Users\cloudtest\AppData\Local\Temp\ecl2kgt3q\dotnet-62f51424-63c9-4e6a-b09b-380303c69000.tar'. Failed to install shared frameworks (6.0, 7.0, 8.0, 9.0, 10.0, 11.0.0-preview.6.26277.111, aspnetcore@11.0.0-preview.6.26277.111) to 'D:\a\_work\1\s\.dotnet' using dotnetup (exit code '1'); falling back to dotnet install script. GET [https://builds.dotnet.microsoft.com/dotnet/scripts/v1/dotnet-install.ps1](https://builds.dotnet.microsoft.com/dotnet/scripts/v1/dotnet-install.ps1) dotnet-install: Remote file [https://builds.dotnet.microsoft.com/dotnet/Runtime/6.0.0/dotnet-runtime-6.0.0-win-x64.zip](https://builds.dotnet.microsoft.com/dotnet/Runtime/6.0.0/dotnet-runtime-6.0.0-win-x64.zip) size is 32516454 bytes. dotnet-install: Downloaded file [https://builds.dotnet.microsoft.com/dotnet/Runtime/6.0.0/dotnet-runtime-6.0.0-win-x64.zip](https://builds.dotnet.microsoft.com/dotnet/Runtime/6.0.0/dotnet-runtime-6.0.0-win-x64.zip) size is 32516454 bytes. dotnet-install: The remote and local file sizes are equal. dotnet-install: Extracting the archive. dotnet-install: Adding to current process PATH: "D:\a\_work\1\s\.dotnet\". Note: This change will not be visible if PowerShell was run as a child process. dotnet-install: Note that the script does not ensure your Windows version is supported during the installation. dotnet-install: To check the list of supported versions, go to [https://learn.microsoft.com/dotnet/core/install/windows#supported-versions](https://learn.microsoft.com/dotnet/core/install/windows#supported-versions) dotnet-install: Installed version is 6.0.0 dotnet-install: Installation finished at InstallDotNetSharedFrameworksWithInstallScript, D:\a\_work\1\s\eng\restore-toolset.ps1: line 249 at InstallDotNetSharedFrameworks, D:\a\_work\1\s\eng\restore-toolset.ps1: line 226 at InitializeCustomSDKToolset, D:\a\_work\1\s\eng\restore-toolset.ps1: line 47 at <ScriptBlock>, D:\a\_work\1\s\eng\restore-toolset.ps1: line 285 at InitializeCustomToolset, D:\a\_work\1\s\eng\common\build.ps1: line 99 at Build, D:\a\_work\1\s\eng\common\build.ps1: line 105 at <ScriptBlock>, D:\a\_work\1\s\eng\common\build.ps1: line 190 at <ScriptBlock>, D:\a\_work\_temp\3095a923-09e9-43ab-8e1e-bde4534a48b5.ps1: line 4 at <ScriptBlock>, <No file>: line 1 ##[error](InitializeToolset) Failed to install shared framework 6.0 to 'D:\a\_work\1\s\.dotnet' using dotnet install script for architecture '' (exit code '1'). ##[error]PowerShell exited with code '1'.) exhibited with `dotnetup` in the full framework win leg: ``` Downloading package Microsoft.DotNet.Arcade.Sdk, version 11.0.0-beta.26277.111. Package 'Microsoft.DotNet.Arcade.Sdk' (11.0.0-beta.26277.111) successfully downloaded to 'D:\a_work\1\s.packages'. Downloading vswhere 3.1.7 INFO: Tests will run against full MSBuild in C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Current\Bin\msbuild.exe dotnetup binary is less than 24 hours old; skipping re-download. Installing dotnet (runtime) 6.0.36, dotnet (runtime) 7.0.20, dotnet (runtime) 8.0.28, dotnet (runtime) 9.0.17, dotnet (runtime) 10.0.9, dotnet (runtime) 11.0.0-preview.6.26277.111, aspnet (runtime) 11.0.0-preview.6.26277.111 to D:\a_work\1\s.dotnet... ⚠ Daily builds are not code-signed. Only the SHA-512 hash is verified. Installing dotnet (runtime) 10.0.9: 59% Installing dotnet (runtime) 11.0.0-preview.6.26277.111: 0% Installing dotnet (runtime) 9.0.17: 58% Installed at D:\a_work\1\s.dotnet: dotnet (runtime) 6.0.36 dotnet (runtime) 10.0.9 dotnet (runtime) 8.0.28 dotnet (runtime) 7.0.20 dotnet (runtime) 9.0.17 aspnet (runtime) 11.0.0-preview.6.26277.111 The following installs failed: dotnet (runtime) 11.0.0-preview.6.26277.111: Failed to extract .NET archive for version 11.0.0-preview.6.26277.111: Could not find a part of the path 'C:\Users\cloudtest\AppData\Local\Temp\ecl2kgt3q\dotnet-62f51424-63c9-4e6a-b09b-380303c69000.tar'. Error: Failed to extract .NET archive for version 11.0.0-preview.6.26277.111: Could not find a part of the path 'C:\Users\cloudtest\AppData\Local\Temp\ecl2kgt3q\dotnet-62f51424-63c9-4e6a-b09b-380303c69000.tar'. ``` Windows: FileShare.Read (no FileShare.Delete) means nothing can delete the archive while we hold it open, and a directory can't be removed while it still contains an undeletable file. So the scratch dir can't be reaped mid-extract. Unix: even if the path is unlinked, our open fd keeps the data readable for both passes. Each pass wraps the shared stream in a non-owning GZipStream/TarReader (leaveOpen: true), and only the gzip wrapper is disposed — the shared handle stays open the entire time. This closes the create/close/reopen-by-path window that produced the original DirectoryNotFoundException. The test-only ExtractTarContents(string tarPath, ...) overload is preserved (it opens its own FileShare.Read handle and delegates), so existing tests are unchanged. Tests: 12 total, 0 failed, 2 skipped; the two tar.gz tests pass individually.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue 2 exhibited in CI, #54752, is that the archive extract of a .tar.gz failed.
Changes to try to remedy this:
We no longer create a temp compy to decompress the tar on disk. This closes the create/close/reopen-by-path window that produced the original
DirectoryNotFoundException.In addition:
Windows: Use
FileShare.Read(no FileShare.Delete) - try to prevent others from deleting the archive while we hold it open. In Unix even if the path is unlinked, our open fd keeps the data readable.