From bc0b69deb5a96eeb41f2cab8870f7702831fe71c Mon Sep 17 00:00:00 2001 From: Noah Gilson Date: Fri, 12 Jun 2026 16:20:16 -0700 Subject: [PATCH] Use native, `leaveOpen` tar streaming to decompress the tar without a copy MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Attempt to resolve the following issue in CI (We're on a new branch but want to fix a different CI flake issue with dotnetup or the install scripts around dotnetup. Issue 1 is that the install script fallback failed because it got an empty architecture string: ''. This is likely an issue in origin/dotnet/sdk main branch wit hthe engineering scripts. We dont necessarily have the right copy of that script on this branch so you might need to fix that in a worktree. Issue 2 is that the archive failed - I wonder if this is a race condition, a file deleition, missing file, or other issue with dotnetup extraction. There is only 1 instance of dotnetup running at a time. [Pipelines - Run 20260612.115 logs](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1462304&view=logs&jobId=2709a726-7db6-5829-ca7b-958b9d664f9e&j=2709a726-7db6-5829-ca7b-958b9d664f9e&t=cbd5e4d2-ff07-56e0-f071-f4be5ccd6b94) - fix this issue in this branch or investigate potential causes Installing .NET SDK 11.0.100-preview.5.26227.104 to D:\a\_work\1\s\.dotnet... ⚠ Daily builds are not code-signed. Only the SHA-512 hash is verified. Installing .NET SDK 11.0.100-preview.5.26227.104: 0% Installing .NET SDK 11.0.100-preview.5.26227.104: 48% Installing .NET SDK 11.0.100-preview.5.26227.104: 83% Installed at D:\a\_work\1\s\.dotnet: .NET SDK 11.0.100-preview.5.26227.104 Downloading package Microsoft.DotNet.Arcade.Sdk, version 11.0.0-beta.26277.111. Package 'Microsoft.DotNet.Arcade.Sdk' (11.0.0-beta.26277.111) successfully downloaded to 'D:\a\_work\1\s\.packages\'. Downloading vswhere 3.1.7 INFO: Tests will run against full MSBuild in C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Current\Bin\msbuild.exe dotnetup binary is less than 24 hours old; skipping re-download. Installing dotnet (runtime) 6.0.36, dotnet (runtime) 7.0.20, dotnet (runtime) 8.0.28, dotnet (runtime) 9.0.17, dotnet (runtime) 10.0.9, dotnet (runtime) 11.0.0-preview.6.26277.111, aspnet (runtime) 11.0.0-preview.6.26277.111 to D:\a\_work\1\s\.dotnet... ⚠ Daily builds are not code-signed. Only the SHA-512 hash is verified. Installing dotnet (runtime) 10.0.9: 59% Installing dotnet (runtime) 11.0.0-preview.6.26277.111: 0% Installing dotnet (runtime) 9.0.17: 58% Installed at D:\a\_work\1\s\.dotnet: dotnet (runtime) 6.0.36 dotnet (runtime) 10.0.9 dotnet (runtime) 8.0.28 dotnet (runtime) 7.0.20 dotnet (runtime) 9.0.17 aspnet (runtime) 11.0.0-preview.6.26277.111 The following installs failed: dotnet (runtime) 11.0.0-preview.6.26277.111: Failed to extract .NET archive for version 11.0.0-preview.6.26277.111: Could not find a part of the path 'C:\Users\cloudtest\AppData\Local\Temp\ecl2kgt3q\dotnet-62f51424-63c9-4e6a-b09b-380303c69000.tar'. Error: Failed to extract .NET archive for version 11.0.0-preview.6.26277.111: Could not find a part of the path 'C:\Users\cloudtest\AppData\Local\Temp\ecl2kgt3q\dotnet-62f51424-63c9-4e6a-b09b-380303c69000.tar'. Failed to install shared frameworks (6.0, 7.0, 8.0, 9.0, 10.0, 11.0.0-preview.6.26277.111, aspnetcore@11.0.0-preview.6.26277.111) to 'D:\a\_work\1\s\.dotnet' using dotnetup (exit code '1'); falling back to dotnet install script. GET [https://builds.dotnet.microsoft.com/dotnet/scripts/v1/dotnet-install.ps1](https://builds.dotnet.microsoft.com/dotnet/scripts/v1/dotnet-install.ps1) dotnet-install: Remote file [https://builds.dotnet.microsoft.com/dotnet/Runtime/6.0.0/dotnet-runtime-6.0.0-win-x64.zip](https://builds.dotnet.microsoft.com/dotnet/Runtime/6.0.0/dotnet-runtime-6.0.0-win-x64.zip) size is 32516454 bytes. dotnet-install: Downloaded file [https://builds.dotnet.microsoft.com/dotnet/Runtime/6.0.0/dotnet-runtime-6.0.0-win-x64.zip](https://builds.dotnet.microsoft.com/dotnet/Runtime/6.0.0/dotnet-runtime-6.0.0-win-x64.zip) size is 32516454 bytes. dotnet-install: The remote and local file sizes are equal. dotnet-install: Extracting the archive. dotnet-install: Adding to current process PATH: "D:\a\_work\1\s\.dotnet\". Note: This change will not be visible if PowerShell was run as a child process. dotnet-install: Note that the script does not ensure your Windows version is supported during the installation. dotnet-install: To check the list of supported versions, go to [https://learn.microsoft.com/dotnet/core/install/windows#supported-versions](https://learn.microsoft.com/dotnet/core/install/windows#supported-versions) dotnet-install: Installed version is 6.0.0 dotnet-install: Installation finished at InstallDotNetSharedFrameworksWithInstallScript, D:\a\_work\1\s\eng\restore-toolset.ps1: line 249 at InstallDotNetSharedFrameworks, D:\a\_work\1\s\eng\restore-toolset.ps1: line 226 at InitializeCustomSDKToolset, D:\a\_work\1\s\eng\restore-toolset.ps1: line 47 at , D:\a\_work\1\s\eng\restore-toolset.ps1: line 285 at InitializeCustomToolset, D:\a\_work\1\s\eng\common\build.ps1: line 99 at Build, D:\a\_work\1\s\eng\common\build.ps1: line 105 at , D:\a\_work\1\s\eng\common\build.ps1: line 190 at , D:\a\_work\_temp\3095a923-09e9-43ab-8e1e-bde4534a48b5.ps1: line 4 at , : line 1 ##[error](InitializeToolset) Failed to install shared framework 6.0 to 'D:\a\_work\1\s\.dotnet' using dotnet install script for architecture '' (exit code '1'). ##[error]PowerShell exited with code '1'.) exhibited with `dotnetup` in the full framework win leg: ``` Downloading package Microsoft.DotNet.Arcade.Sdk, version 11.0.0-beta.26277.111. Package 'Microsoft.DotNet.Arcade.Sdk' (11.0.0-beta.26277.111) successfully downloaded to 'D:\a_work\1\s.packages'. Downloading vswhere 3.1.7 INFO: Tests will run against full MSBuild in C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Current\Bin\msbuild.exe dotnetup binary is less than 24 hours old; skipping re-download. Installing dotnet (runtime) 6.0.36, dotnet (runtime) 7.0.20, dotnet (runtime) 8.0.28, dotnet (runtime) 9.0.17, dotnet (runtime) 10.0.9, dotnet (runtime) 11.0.0-preview.6.26277.111, aspnet (runtime) 11.0.0-preview.6.26277.111 to D:\a_work\1\s.dotnet... ⚠ Daily builds are not code-signed. Only the SHA-512 hash is verified. Installing dotnet (runtime) 10.0.9: 59% Installing dotnet (runtime) 11.0.0-preview.6.26277.111: 0% Installing dotnet (runtime) 9.0.17: 58% Installed at D:\a_work\1\s.dotnet: dotnet (runtime) 6.0.36 dotnet (runtime) 10.0.9 dotnet (runtime) 8.0.28 dotnet (runtime) 7.0.20 dotnet (runtime) 9.0.17 aspnet (runtime) 11.0.0-preview.6.26277.111 The following installs failed: dotnet (runtime) 11.0.0-preview.6.26277.111: Failed to extract .NET archive for version 11.0.0-preview.6.26277.111: Could not find a part of the path 'C:\Users\cloudtest\AppData\Local\Temp\ecl2kgt3q\dotnet-62f51424-63c9-4e6a-b09b-380303c69000.tar'. Error: Failed to extract .NET archive for version 11.0.0-preview.6.26277.111: Could not find a part of the path 'C:\Users\cloudtest\AppData\Local\Temp\ecl2kgt3q\dotnet-62f51424-63c9-4e6a-b09b-380303c69000.tar'. ``` Windows: FileShare.Read (no FileShare.Delete) means nothing can delete the archive while we hold it open, and a directory can't be removed while it still contains an undeletable file. So the scratch dir can't be reaped mid-extract. Unix: even if the path is unlinked, our open fd keeps the data readable for both passes. Each pass wraps the shared stream in a non-owning GZipStream/TarReader (leaveOpen: true), and only the gzip wrapper is disposed — the shared handle stays open the entire time. This closes the create/close/reopen-by-path window that produced the original DirectoryNotFoundException. The test-only ExtractTarContents(string tarPath, ...) overload is preserved (it opens its own FileShare.Read handle and delegates), so existing tests are unchanged. Tests: 12 total, 0 failed, 2 skipped; the two tar.gz tests pass individually. --- .../Internal/DotnetArchiveExtractor.cs | 117 ++++++++++-------- .../DotnetArchiveExtractorTests.cs | 39 ++++++ 2 files changed, 102 insertions(+), 54 deletions(-) diff --git a/src/Installer/Microsoft.Dotnet.Installation/Internal/DotnetArchiveExtractor.cs b/src/Installer/Microsoft.Dotnet.Installation/Internal/DotnetArchiveExtractor.cs index 684194e2ab73..3a398721e516 100644 --- a/src/Installer/Microsoft.Dotnet.Installation/Internal/DotnetArchiveExtractor.cs +++ b/src/Installer/Microsoft.Dotnet.Installation/Internal/DotnetArchiveExtractor.cs @@ -303,57 +303,51 @@ private static string ResolveEntryDestPath(string entryName, string targetDir, M /// private static void ExtractTarArchive(string archivePath, string targetDir, IProgressTask? installTask, MuxerHandler? muxerHandler = null, Action? onEntryExtracted = null, Func? shouldSkipEntry = null) { - string decompressedPath = DecompressTarGzIfNeeded(archivePath, out bool needsDecompression); + bool isGzip = archivePath.EndsWith(".gz", StringComparison.OrdinalIgnoreCase); - try - { - long totalEntries = CountTarEntries(decompressedPath); - installTask?.MaxValue = totalEntries > 0 ? totalEntries : 1; + // Hold a single read handle on the archive for the entire extraction. FileShare.Read + // keeps the file locked against deletion on Windows, and on Unix an open handle keeps the + // underlying data readable even if the path is unlinked. Either way the scratch directory + // cannot be reaped out from under us (e.g. by CI temp cleanup) in the window between the + // entry-count pass and the extraction pass. Both passes seek this shared stream back to the + // start and wrap it in a fresh, non-owning GZipStream/TarReader so the handle stays open. + using var archiveStream = new FileStream(archivePath, FileMode.Open, FileAccess.Read, FileShare.Read); - ExtractTarContents(decompressedPath, targetDir, installTask, muxerHandler, onEntryExtracted, shouldSkipEntry); - } - finally - { - // Clean up temporary decompressed file - if (needsDecompression && File.Exists(decompressedPath)) - { - File.Delete(decompressedPath); - } - } + long totalEntries = CountTarEntries(archiveStream, isGzip); + installTask?.MaxValue = totalEntries > 0 ? totalEntries : 1; + + archiveStream.Seek(0, SeekOrigin.Begin); + ExtractTarContents(archiveStream, isGzip, targetDir, installTask, muxerHandler, onEntryExtracted, shouldSkipEntry); } /// - /// Decompresses a .tar.gz file if needed, returning the path to the tar file. + /// Wraps an already-open tar archive stream for reading, layering in gzip decompression when + /// needed. The returned reader leaves open so a single file + /// handle can be reused across multiple passes; callers dispose the returned stream only when + /// it is a gzip wrapper (never the shared archive stream). /// - private static string DecompressTarGzIfNeeded(string archivePath, out bool needsDecompression) - { - needsDecompression = archivePath.EndsWith(".gz", StringComparison.OrdinalIgnoreCase); - if (!needsDecompression) - { - return archivePath; - } - - string decompressedPath = archivePath.Replace(".gz", "", StringComparison.Ordinal); - - using FileStream originalFileStream = File.OpenRead(archivePath); - using FileStream decompressedFileStream = File.Create(decompressedPath); - using GZipStream decompressionStream = new GZipStream(originalFileStream, CompressionMode.Decompress); - decompressionStream.CopyTo(decompressedFileStream); - - return decompressedPath; - } + private static Stream OpenTarReadStream(Stream archiveStream, bool isGzip) + => isGzip ? new GZipStream(archiveStream, CompressionMode.Decompress, leaveOpen: true) : archiveStream; /// - /// Counts the number of entries in a tar file for progress reporting. + /// Counts the number of entries in a tar archive for progress reporting. /// - private static long CountTarEntries(string tarPath) + private static long CountTarEntries(Stream archiveStream, bool isGzip) { long totalFiles = 0; - using var tarStream = File.OpenRead(tarPath); - var tarReader = new TarReader(tarStream); - while (tarReader.GetNextEntry() is not null) + Stream tarStream = OpenTarReadStream(archiveStream, isGzip); + try { - totalFiles++; + using var tarReader = new TarReader(tarStream, leaveOpen: true); + while (tarReader.GetNextEntry() is not null) + { + totalFiles++; + } + } + finally + { + // Only dispose the gzip wrapper; the shared archive stream stays open for the next pass. + if (isGzip) { tarStream.Dispose(); } } return totalFiles; } @@ -364,27 +358,42 @@ private static long CountTarEntries(string tarPath) /// internal static void ExtractTarContents(string tarPath, string targetDir, IProgressTask? installTask, MuxerHandler? muxerHandler = null, Action? onEntryExtracted = null, Func? shouldSkipEntry = null) { - using var tarStream = File.OpenRead(tarPath); - var tarReader = new TarReader(tarStream); - TarEntry? entry; - - // Defer hard link creation until after all regular files are extracted, - // since the target file may not exist yet when the hard link entry is encountered. - var deferredHardLinks = new List<(string DestPath, string TargetPath)>(); + bool isGzip = tarPath.EndsWith(".gz", StringComparison.OrdinalIgnoreCase); + using var archiveStream = new FileStream(tarPath, FileMode.Open, FileAccess.Read, FileShare.Read); + ExtractTarContents(archiveStream, isGzip, targetDir, installTask, muxerHandler, onEntryExtracted, shouldSkipEntry); + } - while ((entry = tarReader.GetNextEntry()) is not null) + private static void ExtractTarContents(Stream archiveStream, bool isGzip, string targetDir, IProgressTask? installTask, MuxerHandler? muxerHandler, Action? onEntryExtracted, Func? shouldSkipEntry) + { + Stream tarStream = OpenTarReadStream(archiveStream, isGzip); + try { - bool skip = shouldSkipEntry?.Invoke(entry.Name) ?? false; - if (!skip) + using var tarReader = new TarReader(tarStream, leaveOpen: true); + TarEntry? entry; + + // Defer hard link creation until after all regular files are extracted, + // since the target file may not exist yet when the hard link entry is encountered. + var deferredHardLinks = new List<(string DestPath, string TargetPath)>(); + + while ((entry = tarReader.GetNextEntry()) is not null) { - ProcessTarEntry(entry, targetDir, muxerHandler, deferredHardLinks); + bool skip = shouldSkipEntry?.Invoke(entry.Name) ?? false; + if (!skip) + { + ProcessTarEntry(entry, targetDir, muxerHandler, deferredHardLinks); + } + + onEntryExtracted?.Invoke(entry.Name); + installTask?.Value += 1; } - onEntryExtracted?.Invoke(entry.Name); - installTask?.Value += 1; + CreateDeferredHardLinks(deferredHardLinks); + } + finally + { + // Only dispose the gzip wrapper; the shared archive stream is owned by the caller. + if (isGzip) { tarStream.Dispose(); } } - - CreateDeferredHardLinks(deferredHardLinks); } private static void ProcessTarEntry(TarEntry entry, string targetDir, MuxerHandler? muxerHandler, List<(string DestPath, string TargetPath)> deferredHardLinks) diff --git a/test/dotnetup.Tests/DotnetArchiveExtractorTests.cs b/test/dotnetup.Tests/DotnetArchiveExtractorTests.cs index 44b7a2884195..2f0ab5e5adc6 100644 --- a/test/dotnetup.Tests/DotnetArchiveExtractorTests.cs +++ b/test/dotnetup.Tests/DotnetArchiveExtractorTests.cs @@ -461,6 +461,45 @@ public void Commit_ExtractsTarGzArchive_Correctly() File.ReadAllText(runtimeFile).Should().Be("runtime-content"); } + [Fact] + public void Commit_ExtractsTarGzArchive_WhenDecompressedTarPathAlreadyExists() + { + using var testEnv = DotnetupTestUtilities.CreateTestEnvironment(); + + var installRoot = new DotnetInstallRoot(testEnv.InstallPath, InstallerUtilities.GetDefaultInstallArchitecture()); + var version = new ReleaseVersion(9, 0, 0); + + var request = new DotnetInstallRequest( + installRoot, + new UpdateChannel("9.0"), + InstallComponent.Runtime, + new InstallRequestOptions()); + + byte[] tarGzContent = CreateTarGzWithFiles( + ("shared/Microsoft.NETCore.App/9.0.0/System.Runtime.dll", "runtime-content")); + + var mockDownloader = new MockArchiveDownloader + { + CreateFakeArchive = true, + FakeArchiveContent = tarGzContent, + ArchiveFileExtension = ".tar.gz" + }; + + var releaseManifest = new ReleaseManifest(); + var progressTarget = new NullProgressTarget(); + + using var extractor = new DotnetArchiveExtractor(request, version, releaseManifest, progressTarget, mockDownloader); + + extractor.Prepare(); + Directory.CreateDirectory(mockDownloader.DownloadCalls[0].DestinationPath.Replace(".gz", "", StringComparison.Ordinal)); + + extractor.Commit(); + + var runtimeFile = Path.Combine(testEnv.InstallPath, "shared", "Microsoft.NETCore.App", "9.0.0", "System.Runtime.dll"); + File.Exists(runtimeFile).Should().BeTrue("tar.gz extraction should not depend on a sibling decompressed .tar staging file"); + File.ReadAllText(runtimeFile).Should().Be("runtime-content"); + } + [PlatformSpecificFact(TestPlatforms.Windows)] public void Commit_ExtractsZipArchive_Correctly() {