Skip to content

Turn on Helix Job Monitor #38751

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wtgodbe wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

Turn on Helix Job Monitor #38751

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
db524e4
Enable Helix Job Monitor
Aug 5, 2026
34bf525
Validate Helix monitor result
Aug 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions .config/dotnet-tools.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{
"version": 1,
"isRoot": true,
"tools": {
"microsoft.dotnet.helix.jobmonitor": {
"version": "11.0.0-beta.26401.101",
"commands": [
"dotnet-helix-job-monitor"
]
}
}
}
13 changes: 13 additions & 0 deletions azure-pipelines-internal-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -645,6 +645,9 @@ extends:
HelixAccessToken: $(_HelixAccessToken)
SYSTEM_ACCESSTOKEN: $(System.AccessToken)
DotNetBuildsInternalReadSasToken: $(dotnetbuilds-internal-container-read-token)
- template: /eng/common/core-templates/job/helix-job-monitor.yml@self
parameters:
helixAccessToken: $(HelixApiAccessToken)
- stage: validate

@AndriySvyryd AndriySvyryd Aug 5, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need to change the validate logic to also check whether the corresponding Helix monitor job succeeded

@wtgodbe wtgodbe Aug 5, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

34bf525

@AndriySvyryd AndriySvyryd Aug 5, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally, we should check the result of each individual Helix job to preserve the validation logic, though I am not sure whether this is currently possible to do here (feature request?)
Otherwise, add $helixJobMonitorResult to each item in $groupResults instead of failing outright

@wtgodbe wtgodbe Aug 5, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally, we should check the result of each individual Helix job to preserve the validation logic

What would be the benefit of this? The helix jobs no longer depend on the test results, they just send the tests off and then report green. If one fails, the monitor will fail too

@AndriySvyryd AndriySvyryd Aug 5, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What would be the benefit of this? The helix jobs no longer depend on the test results, they just send the tests off and then report green. If one fails, the monitor will fail too

Right, the benefit would be from checking the helix monitoring jobs for test failures specific to that leg

@wtgodbe wtgodbe Aug 5, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the benefit would be from checking the helix monitoring jobs for test failures specific to that leg

But the individual legs don't fail when there are test failures - as soon as the tests are sent to helix, they complete w/ success. Only the helix monitor job will ever fail for test failures.

@AndriySvyryd AndriySvyryd Aug 5, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only the helix monitor job will ever fail for test failures.

Yes and we need to make that failure more granular, so that we can continue to check only the relevant failures in the validation groups.

displayName: Validate
dependsOn: build
Expand All @@ -670,8 +673,18 @@ extends:
HelixMacOsArm64Result: $[ stageDependencies.build.Helix_macOS_ARM64.result ]
HelixWindowsArm64Result: $[ stageDependencies.build.Helix_Windows_Arm64.result ]
HelixUbuntuCosmosResult: $[ stageDependencies.build.Helix_Ubuntu_Cosmos.result ]
HelixJobMonitorResult: $[ stageDependencies.build.HelixJobMonitor.result ]
steps:
- pwsh: |
$helixJobMonitorResult = "$(HelixJobMonitorResult)"
Write-Host "Helix Job Monitor result: $helixJobMonitorResult"

if ($helixJobMonitorResult -ne 'Succeeded')
{
Write-Error "Helix Job Monitor did not succeed: $helixJobMonitorResult."
exit 1
}

$groupResults = @{
Windows = @("$(WindowsResult)", "$(HelixWindowsResult)")
Linux = @("$(LinuxResult)", "$(HelixUbuntuResult)")
Expand Down
11 changes: 11 additions & 0 deletions azure-pipelines-public.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -509,6 +509,7 @@ stages:
env:
HelixAccessToken: $(_HelixAccessToken)
SYSTEM_ACCESSTOKEN: $(System.AccessToken)
- template: /eng/common/core-templates/job/helix-job-monitor.yml
- stage: validate
displayName: Validate
dependsOn: build
Expand All @@ -533,8 +534,18 @@ stages:
HelixMacOsArm64Result: $[ stageDependencies.build.Helix_macOS_ARM64.result ]
HelixWindowsArm64Result: $[ stageDependencies.build.Helix_Windows_Arm64.result ]
HelixUbuntuCosmosResult: $[ stageDependencies.build.Helix_Ubuntu_Cosmos.result ]
HelixJobMonitorResult: $[ stageDependencies.build.HelixJobMonitor.result ]
steps:
- pwsh: |
$helixJobMonitorResult = "$(HelixJobMonitorResult)"
Write-Host "Helix Job Monitor result: $helixJobMonitorResult"

if ($helixJobMonitorResult -ne 'Succeeded')
{
Write-Error "Helix Job Monitor did not succeed: $helixJobMonitorResult."
exit 1
}

$groupResults = @{
Windows = @("$(WindowsResult)", "$(HelixWindowsResult)")
Linux = @("$(LinuxResult)", "$(HelixUbuntuResult)")
Expand Down
2 changes: 2 additions & 0 deletions eng/Version.Details.props
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ This file should be imported by eng/Versions.props
<!-- dotnet-dotnet dependencies -->
<MicrosoftDotNetArcadeSdkPackageVersion>11.0.0-beta.26401.101</MicrosoftDotNetArcadeSdkPackageVersion>
<MicrosoftDotNetBuildTasksTemplatingPackageVersion>11.0.0-beta.26401.101</MicrosoftDotNetBuildTasksTemplatingPackageVersion>
<MicrosoftDotNetHelixJobMonitorPackageVersion>11.0.0-beta.26401.101</MicrosoftDotNetHelixJobMonitorPackageVersion>
<MicrosoftDotNetHelixSdkPackageVersion>11.0.0-beta.26401.101</MicrosoftDotNetHelixSdkPackageVersion>
<MicrosoftExtensionsCachingMemoryPackageVersion>11.0.0-rc.1.26401.101</MicrosoftExtensionsCachingMemoryPackageVersion>
<MicrosoftExtensionsConfigurationPackageVersion>11.0.0-rc.1.26401.101</MicrosoftExtensionsConfigurationPackageVersion>
Expand All @@ -29,6 +30,7 @@ This file should be imported by eng/Versions.props
<!-- dotnet-dotnet dependencies -->
<MicrosoftDotNetArcadeSdkVersion>$(MicrosoftDotNetArcadeSdkPackageVersion)</MicrosoftDotNetArcadeSdkVersion>
<MicrosoftDotNetBuildTasksTemplatingVersion>$(MicrosoftDotNetBuildTasksTemplatingPackageVersion)</MicrosoftDotNetBuildTasksTemplatingVersion>
<MicrosoftDotNetHelixJobMonitorVersion>$(MicrosoftDotNetHelixJobMonitorPackageVersion)</MicrosoftDotNetHelixJobMonitorVersion>
<MicrosoftDotNetHelixSdkVersion>$(MicrosoftDotNetHelixSdkPackageVersion)</MicrosoftDotNetHelixSdkVersion>
<MicrosoftExtensionsCachingMemoryVersion>$(MicrosoftExtensionsCachingMemoryPackageVersion)</MicrosoftExtensionsCachingMemoryVersion>
<MicrosoftExtensionsConfigurationVersion>$(MicrosoftExtensionsConfigurationPackageVersion)</MicrosoftExtensionsConfigurationVersion>
Expand Down
4 changes: 4 additions & 0 deletions eng/Version.Details.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,10 @@
<Uri>https://github.com/dotnet/dotnet</Uri>
<Sha>5da2de22f7c6e2ebe7ed4c16c614b00183145d3c</Sha>
</Dependency>
<Dependency Name="Microsoft.DotNet.Helix.JobMonitor" Version="11.0.0-beta.26401.101">
<Uri>https://github.com/dotnet/dotnet</Uri>
<Sha>5da2de22f7c6e2ebe7ed4c16c614b00183145d3c</Sha>
</Dependency>
<Dependency Name="Microsoft.DotNet.Helix.Sdk" Version="11.0.0-beta.26401.101">
<Uri>https://github.com/dotnet/dotnet</Uri>
<Sha>5da2de22f7c6e2ebe7ed4c16c614b00183145d3c</Sha>
Expand Down
1 change: 1 addition & 0 deletions eng/helix.proj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
<DotNetCliVersion>$(NETCoreSdkVersion)</DotNetCliVersion>

<EnableAzurePipelinesReporter>true</EnableAzurePipelinesReporter>
<EnableHelixJobMonitor Condition = "'$(SYSTEM_ACCESSTOKEN)' != ''">true</EnableHelixJobMonitor>
<FailOnTestFailure>true</FailOnTestFailure>
<SqlServerTests>$(RepoRoot)/test/EFCore.SqlServer.FunctionalTests/*.csproj;$(RepoRoot)/test/EFCore.SqlServer.HierarchyId.Tests/*.csproj;$(RepoRoot)/test/EFCore.CrossStore.FunctionalTests/*.csproj;$(RepoRoot)/test/EFCore.OData.FunctionalTests/*.csproj;$(RepoRoot)/test/EFCore.AspNet.SqlServer.FunctionalTests/*.csproj;$(RepoRoot)/test/EFCore.VisualBasic.FunctionalTests/*.vbproj;$(RepoRoot)/test/EFCore.FSharp.FunctionalTests/*.fsproj</SqlServerTests>
<CosmosTests>$(RepoRoot)/test/EFCore.Cosmos.FunctionalTests/*.csproj</CosmosTests>
Expand Down
Loading