Skip to content

Reduce sdk-build job timeout from 150 to 50 minutes#54752

Open
MichaelSimons wants to merge 1 commit into
mainfrom
michaelsimons/reduce-build-timeout-to-45min
Open

Reduce sdk-build job timeout from 150 to 50 minutes#54752
MichaelSimons wants to merge 1 commit into
mainfrom
michaelsimons/reduce-build-timeout-to-45min

Conversation

@MichaelSimons

@MichaelSimons MichaelSimons commented Jun 12, 2026

Copy link
Copy Markdown
Member

Now that tests are queued to Helix rather than run inline, the build legs complete much faster. The previous 150-minute timeout wastes significant compute when a job hangs (e.g. build 1462067 where a macOS job hung for the full 150 minutes on SDK install).

Analysis

Sampled 7 recent successful PR builds (definition 101) and measured build leg durations (excluding Dotnet-Format Integration Tests):

Job Min Avg Max
TestBuild: macOS (arm64) 24.1 34.8 39.9
AoT: macOS (arm64) 24.3 26.7 29.0
TestBuild: windows (x64) 21.7 23.2 27.8
FullFramework: windows (x64) 22.0 23.3 25.6
TestBuild: linux (x64) 16.8 17.3 18.1
AoT: windows (x64) 15.3 16.3 18.5
TestBuild: linux (arm64) 10.1 10.4 10.6

The worst-case observed is ~40 minutes (macOS arm64 TestBuild). A 50-minute timeout provides ~25% headroom above the observed maximum while cutting wasted compute by ~67% when hangs occur.

Copilot AI review requested due to automatic review settings June 12, 2026 21:04
@MichaelSimons MichaelSimons requested a review from MiYanni as a code owner June 12, 2026 21:04

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR reduces the default Azure Pipelines job timeout for the sdk-build job template from 150 minutes to 45 minutes, aligning timeouts with the faster “tests queued to Helix” workflow and limiting wasted compute when jobs hang.

Changes:

  • Lower timeoutInMinutes default in eng/pipelines/templates/jobs/sdk-build.yml from 150 to 45 minutes.

Now that tests are queued to Helix rather than run inline, the build
legs complete much faster (worst case ~40 min on macOS). The previous
150-minute timeout wastes significant compute when a job hangs.

50 minutes provides ~25% headroom above the observed worst case.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@MichaelSimons MichaelSimons force-pushed the michaelsimons/reduce-build-timeout-to-45min branch from 7367ad1 to 29dac69 Compare June 12, 2026 21:09
@MichaelSimons MichaelSimons changed the title Reduce sdk-build job timeout from 150 to 45 minutes Reduce sdk-build job timeout from 150 to 50 minutes Jun 12, 2026
@MichaelSimons MichaelSimons requested a review from a team June 12, 2026 21:10

@nagilson nagilson left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was initially hesitant to lower this because I've seen it go over 150 in the past and succeed, but the data you demonstrated here was helpful in building confidence.

In theory having links might be helpful to backup the data, but I also recognize the CI runs will be deleted shortly.

Now, why dotnetup got stuck at :
Downloading .NET SDK 11.0.100-preview.5.26227.104 ( 29.0 MB / 183.6 MB): 15%
is of interest to me although the most concerning is perhaps that it it's not respecting the 10 minute timeout and ran for 2 hours. I'll investigate or file an issue.

@MichaelSimons

Copy link
Copy Markdown
Member Author

I was initially hesitant to lower this because I've seen it go over 150 in the past and succeed, but the data you demonstrated here was helpful in building confidence.

Keep in mind the Helix Monitor Job is what allows this change. The "build" jobs no longer wait on helix, they simply queue the workitems.

@MichaelSimons

Copy link
Copy Markdown
Member Author

@nagilson - FYI dotnetup failure just happened in the fullFX leg.

dotnetup binary is less than 24 hours old; skipping re-download.
Installing dotnet (runtime) 6.0.36, dotnet (runtime) 7.0.20, dotnet (runtime) 8.0.28, dotnet (runtime) 9.0.17, dotnet (runtime) 10.0.9, dotnet (runtime) 11.0.0-preview.6.26277.111, aspnet (runtime) 11.0.0-preview.6.26277.111 to D:\a\_work\1\s\.dotnet...
⚠  Daily builds are not code-signed. Only the SHA-512 hash is verified.
Installing  dotnet (runtime)                             10.0.9: 59%
Installing  dotnet (runtime)         11.0.0-preview.6.26277.111: 0%
Installing  dotnet (runtime)                             9.0.17: 58%
Installed at D:\a\_work\1\s\.dotnet:
  dotnet (runtime) 6.0.36
  dotnet (runtime) 10.0.9
  dotnet (runtime) 8.0.28
  dotnet (runtime) 7.0.20
  dotnet (runtime) 9.0.17
  aspnet (runtime) 11.0.0-preview.6.26277.111
The following installs failed:
  dotnet (runtime) 11.0.0-preview.6.26277.111: Failed to extract .NET archive for version 11.0.0-preview.6.26277.111: Could not find a part of the path 'C:\Users\cloudtest\AppData\Local\Temp\ecl2kgt3q\dotnet-62f51424-63c9-4e6a-b09b-380303c69000.tar'.
Error: Failed to extract .NET archive for version 11.0.0-preview.6.26277.111: Could not find a part of the path 'C:\Users\cloudtest\AppData\Local\Temp\ecl2kgt3q\dotnet-62f51424-63c9-4e6a-b09b-380303c69000.tar'.
Failed to install shared frameworks (6.0, 7.0, 8.0, 9.0, 10.0, 11.0.0-preview.6.26277.111, aspnetcore@11.0.0-preview.6.26277.111) to 'D:\a\_work\1\s\.dotnet' using dotnetup (exit code '1'); falling back to dotnet install script.
GET https://builds.dotnet.microsoft.com/dotnet/scripts/v1/dotnet-install.ps1
dotnet-install: Remote file https://builds.dotnet.microsoft.com/dotnet/Runtime/6.0.0/dotnet-runtime-6.0.0-win-x64.zip size is 32516454 bytes.
dotnet-install: Downloaded file https://builds.dotnet.microsoft.com/dotnet/Runtime/6.0.0/dotnet-runtime-6.0.0-win-x64.zip size is 32516454 bytes.
dotnet-install: The remote and local file sizes are equal.
dotnet-install: Extracting the archive.
dotnet-install: Adding to current process PATH: "D:\a\_work\1\s\.dotnet\". Note: This change will not be visible if PowerShell was run as a child process.
dotnet-install: Note that the script does not ensure your Windows version is supported during the installation.
dotnet-install: To check the list of supported versions, go to https://learn.microsoft.com/dotnet/core/install/windows#supported-versions
dotnet-install: Installed version is 6.0.0
dotnet-install: Installation finished
at InstallDotNetSharedFrameworksWithInstallScript, D:\a\_work\1\s\eng\restore-toolset.ps1: line 249
at InstallDotNetSharedFrameworks, D:\a\_work\1\s\eng\restore-toolset.ps1: line 226
at InitializeCustomSDKToolset, D:\a\_work\1\s\eng\restore-toolset.ps1: line 47
at <ScriptBlock>, D:\a\_work\1\s\eng\restore-toolset.ps1: line 285
at InitializeCustomToolset, D:\a\_work\1\s\eng\common\build.ps1: line 99
at Build, D:\a\_work\1\s\eng\common\build.ps1: line 105
at <ScriptBlock>, D:\a\_work\1\s\eng\common\build.ps1: line 190
at <ScriptBlock>, D:\a\_work\_temp\3095a923-09e9-43ab-8e1e-bde4534a48b5.ps1: line 4
at <ScriptBlock>, <No file>: line 1

@nagilson

Copy link
Copy Markdown
Member

@nagilson - FYI dotnetup failure just happened in the fullFX leg.

dotnetup binary is less than 24 hours old; skipping re-download.
Installing dotnet (runtime) 6.0.36, dotnet (runtime) 7.0.20, dotnet (runtime) 8.0.28, dotnet (runtime) 9.0.17, dotnet (runtime) 10.0.9, dotnet (runtime) 11.0.0-preview.6.26277.111, aspnet (runtime) 11.0.0-preview.6.26277.111 to D:\a\_work\1\s\.dotnet...
⚠  Daily builds are not code-signed. Only the SHA-512 hash is verified.
Installing  dotnet (runtime)                             10.0.9: 59%
Installing  dotnet (runtime)         11.0.0-preview.6.26277.111: 0%
Installing  dotnet (runtime)                             9.0.17: 58%
Installed at D:\a\_work\1\s\.dotnet:
  dotnet (runtime) 6.0.36
  dotnet (runtime) 10.0.9
  dotnet (runtime) 8.0.28
  dotnet (runtime) 7.0.20
  dotnet (runtime) 9.0.17
  aspnet (runtime) 11.0.0-preview.6.26277.111
The following installs failed:
  dotnet (runtime) 11.0.0-preview.6.26277.111: Failed to extract .NET archive for version 11.0.0-preview.6.26277.111: Could not find a part of the path 'C:\Users\cloudtest\AppData\Local\Temp\ecl2kgt3q\dotnet-62f51424-63c9-4e6a-b09b-380303c69000.tar'.
Error: Failed to extract .NET archive for version 11.0.0-preview.6.26277.111: Could not find a part of the path 'C:\Users\cloudtest\AppData\Local\Temp\ecl2kgt3q\dotnet-62f51424-63c9-4e6a-b09b-380303c69000.tar'.
Failed to install shared frameworks (6.0, 7.0, 8.0, 9.0, 10.0, 11.0.0-preview.6.26277.111, aspnetcore@11.0.0-preview.6.26277.111) to 'D:\a\_work\1\s\.dotnet' using dotnetup (exit code '1'); falling back to dotnet install script.
GET https://builds.dotnet.microsoft.com/dotnet/scripts/v1/dotnet-install.ps1
dotnet-install: Remote file https://builds.dotnet.microsoft.com/dotnet/Runtime/6.0.0/dotnet-runtime-6.0.0-win-x64.zip size is 32516454 bytes.
dotnet-install: Downloaded file https://builds.dotnet.microsoft.com/dotnet/Runtime/6.0.0/dotnet-runtime-6.0.0-win-x64.zip size is 32516454 bytes.
dotnet-install: The remote and local file sizes are equal.
dotnet-install: Extracting the archive.
dotnet-install: Adding to current process PATH: "D:\a\_work\1\s\.dotnet\". Note: This change will not be visible if PowerShell was run as a child process.
dotnet-install: Note that the script does not ensure your Windows version is supported during the installation.
dotnet-install: To check the list of supported versions, go to https://learn.microsoft.com/dotnet/core/install/windows#supported-versions
dotnet-install: Installed version is 6.0.0
dotnet-install: Installation finished
at InstallDotNetSharedFrameworksWithInstallScript, D:\a\_work\1\s\eng\restore-toolset.ps1: line 249
at InstallDotNetSharedFrameworks, D:\a\_work\1\s\eng\restore-toolset.ps1: line 226
at InitializeCustomSDKToolset, D:\a\_work\1\s\eng\restore-toolset.ps1: line 47
at <ScriptBlock>, D:\a\_work\1\s\eng\restore-toolset.ps1: line 285
at InitializeCustomToolset, D:\a\_work\1\s\eng\common\build.ps1: line 99
at Build, D:\a\_work\1\s\eng\common\build.ps1: line 105
at <ScriptBlock>, D:\a\_work\1\s\eng\common\build.ps1: line 190
at <ScriptBlock>, D:\a\_work\_temp\3095a923-09e9-43ab-8e1e-bde4534a48b5.ps1: line 4
at <ScriptBlock>, <No file>: line 1

Got it, thanks - Im treating this as 2 issues, the install script fallback logic using '' and then the transient issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants