slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."

[jeffschwMSFT]

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
1378245	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#66262
1377963	dotnet/maui	Controls SearchBar,Shape,Slider	Log
1378171	dotnet/maui	Controls CollectionView	Log
1378124	dotnet/aspnetcore	Tests: macOS	Log
1378024	dotnet/maui	Controls Layout	Log
1378083	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#66314
1377967	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#126872
1377944	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#125491
1377906	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#115996
1377873	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#126566
1377804	dotnet/runtime	osx-arm64 Release Libraries_CheckedCoreCLR	Log
1377729	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#66303
1377623	dotnet/maui	Controls Shell	Log
1377497	dotnet/maui	Controls CollectionView	Log
1377665	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#126864
1377472	dotnet/maui	Controls SearchBar,Shape,Slider	Log
2950411	dotnet-sdk	TestBuild: macOS (x64)	Log	#59580
1377624	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log
1377337	dotnet/maui	Controls CollectionView	Log	dotnet/maui#34932
2950448	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
1377095	dotnet/maui	Controls CollectionView	Log
2950376	dotnet-maui	RunOniOS_MauiDebug ARM64	Log
2950412	dotnet-sdk	AoT: macOS (x64)	Log	#59942
1377088	dotnet/maui	Controls Shell	Log
2950363	dotnet-maui	RunOniOS_BlazorDebug ARM64	Log
2950331	dotnet-runtime	coreclr Pri1 Runtime Tests Run osx arm64 checked	Log
2950335	dotnet-sdk	Darwin	Log	#59726
2950334	dotnet-sdk	Darwin	Log	#59725
1376955	dotnet/maui	Controls CollectionView	Log
2950275	dotnet-maui	RunOniOS_MauiNativeAOT ARM64	Log
2949761	dotnet-maui	Controls Image,ImageButton,IndicatorView,InputTransparent,IsEnabled,IsVisible	Log
1376779	dotnet/maui	Controls Shell	Log	dotnet/maui#34573
1377204	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log	dotnet/runtime#125439
1377205	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#126173
1377207	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#126173
1377218	dotnet/runtime	osx-arm64 Release CoreCLR_Libraries	Log
1376773	dotnet/maui	Controls SearchBar,Shape,Slider	Log
1377127	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#126427
1377119	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#125298
1377116	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#126309
1377098	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#66251
1376682	dotnet/maui	Controls Shell	Log
2950152	dotnet-efcore	macOS	Log
1377000	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#126840
1376974	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#126562
1376549	dotnet/maui	Controls Shell	Log
2950081	dotnet-maui	Build macOS (Debug)	Log
2950001	dotnet-maui	RunOniOS_MauiNativeAOT ARM64	Log
1376923	dotnet/aspnetcore	Tests: macOS	Log
2950048	dotnet-maui	Controls Shell	Log
1376899	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#126836
1376876	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#115996
2949998	dotnet-maui	RunOniOS_BlazorRelease x64	Log
2949744	dotnet-maui	Controls Layout	Log
2949934	dotnet-runtime	Libraries SuperPMI collection libraries_tests_no_tiered_compilation Checked coreclr osx arm64 Release	Log
2949865	dotnet-maui	RunOniOS_MauiNativeAOT x64	Log
2949920	dotnet-maui	Controls Layout	Log
1376800	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#125180
1376719	dotnet/maui	Controls Shell	Log	dotnet/maui#34932
2949884	dotnet-maui	RunOniOS_MauiRelease x64	Log
1376514	dotnet/maui	Controls Shell	Log
1376414	dotnet/maui	Controls Shell	Log	dotnet/maui#34600
2949866	dotnet-maui	Controls ScrollView	Log
2949738	dotnet-maui	Controls Shell	Log
1376744	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#126834
1376724	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#65460
1376736	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#126833
2949760	dotnet-maui	RunOniOS_BlazorRelease_CoreCLR ARM64	Log
1376651	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#117261
1376577	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#100729
2949737	dotnet-maui	RunOniOS_MauiNativeAOT x64
1376568	dotnet/runtime	ios-arm64 Release AllSubsets_Mono_Smoke	Log	dotnet/runtime#126701
1376256	dotnet/maui	Controls Shell	Log	dotnet/maui#34758
1376527	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#66294
1376279	dotnet/maui	Controls CollectionView	Log	dotnet/maui#34532
1376496	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#66291
1376490	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#66290
1376400	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log	dotnet/runtime#126693
1376110	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log
1376328	dotnet/runtime	osx-arm64 Release Libraries_CheckedCoreCLR	Log
1376255	dotnet/runtime	osx-arm64 Release NativeAOT_Libs	Log	dotnet/runtime#126824
2949448	dotnet-maui	Controls CollectionView	Log
1376032	dotnet/maui	Controls Layout	Log
1376028	dotnet/maui	Controls CarouselView	Log	dotnet/maui#30369
2949314	dotnet-maui	Controls CollectionView	Log
1375879	dotnet/aspnetcore	Tests: macOS	Log
2949330	dotnet-runtime	coreclr Pri1 Runtime Tests Run osx arm64 checked	Log
1375765	dotnet/maui	Controls Layout	Log
1375853	dotnet/maui	Controls CollectionView	Log
2949244	dotnet-maui	Controls CollectionView	Log
2949279	dotnet-maui	RunOniOS_MauiDebug ARM64	Log
1375718	dotnet/maui	Controls CollectionView	Log
2949243	dotnet-maui	RunOniOS_BlazorDebug ARM64	Log
1375562	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log
1375542	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#53800
1375525	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_Smoke	Log	dotnet/runtime#126803
1375208	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#64807
1373372	dotnet/maui	Controls CollectionView	Log	dotnet/maui#34673
1375004	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#126478
1374830	dotnet/maui	Controls Layout	Log
Displaying 100 of 1275 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
54	322	1275

[lewing]

🤔

[nagilson]

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

[ivanpovazan]

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

• On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

• While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

[garath]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

[garath]

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.

[ivanpovazan]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

We are still experiencing the problem.

Should we try to change the queue to osx.13.arm64?

@dotnet/dnceng

[ilyas1974]

Looks at the failing builds, I'm noticing the jobs that are timing out in the hosted pool (Azure Pipelines) are still using the older hardware. I would recommend moving the workloads to the mac-latest-internal or mac-14-arm64 agent specifications. These have the latest mac hardware associated with them.