Known Test Repository Issue Sometimes Misses Test Failures

[nagilson]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=1196220

Build leg reported

Microsoft.DotNet.Cli.New.IntegrationTests.DotnetNewSearchTests.BasicTest

Pull Request

dotnet/sdk#51591

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "The SSL connection",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

So, I believe there is an issue with the known issue flagger, where it is failing to mark certain failures as known, even if they are a known issue. Please let me know if I am mistaken.

Here's the issue:
Looking at this image, this test was not considered a known issue:
https://github.com/dotnet/sdk/pull/51591/checks?check_run_id=54740778719

I believe that, because it's showing as a test failure, and other similar test failures are counted in the above section:

Furthermore, it appeared to be blocking build-analysis from being green.

Here's such an example of 1 of several 'known issues' that flag this issue:
#5015
dotnet/sdk#51574
#6410

As you can see in, #51574 looks for the string "The SSL connection could not be established" as a ErrorMessage which is supposed to use String.Contains. The test, however, that I think was not flagged as a 'known issue', did indeed have that error message string. It looks like some were flagged, while others were not. Here's the output of that 'unknown' test failure:

Expected command to exit with 0 but it did not.\nFile Name: /Users/runner/work/1/s/artifacts/bin/redist/Release/dotnet/dotnet\nArguments: new console --search --debug:custom-hive /Users/runner/work/1/s/artifacts/tmp/Release/dotnet-new.IntegrationTests/SharedHomeDirectory/20251106211951401\nExit Code: 103\nStdOut:\nWarning: use of 'dotnet new --search' is deprecated. Use 'dotnet new search' instead.\nFor more information, run: \n   dotnet new search -h\n\nSearching for the templates...\nStdErr:\nMatches from template source: NuGet.org\nSearch failed: Failed to update search cache. (The SSL connection could not be established, see inner exception.) (The SSL connection could not be established, see inner exception.)\n\nFor details on the exit code, refer to https://aka.ms/templating-exit-codes#103\n [[Full log]](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1196220&view=ms.vss-test-web.build-test-results-tab&runId=33106418&resultId=100166&paneView=debug)

While there may be a confounding variable in that there are now several known issues which are tracking substrings of each other, or may overlap, the whole reason there are multiple in the first place is because the first existing known test repository issue did not flag all of the tests which failed with the known issue.

Could this alternatively be a limitation of:

1. A limit of how many tests can be flagged as a known issue for the same known issue
2. A different 'failing configuration' which causes issues to be missed?
3. A misunderstanding of which strings get searched? Note that I believe the error is also in the 'helix logs'.
4. A failure of my own accord to file the known issue? @dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=1196220
Error message validated: [The SSL Connection C]
Result validation: ❌ Known issue did not match with the provided build.
Validation performed at: 11/6/2025 10:39:11 PM UTC

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=1196220
Error message validated: [The SSL connection]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 11/6/2025 10:42:09 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
1196423	dotnet/machinelearning	Build	Log	dotnet/machinelearning#7543
1196398	dotnet/aspnetcore	Run build.sh	Log	dotnet/aspnetcore#64262
1196235	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#121356
1196186	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#121417
1196168	dotnet/runtime	Send tests to Helix (Unix)	Log

Build	Definition	Test	Pull Request
1196358	dotnet/sdk	Microsoft.DotNet.Cli.New.IntegrationTests.DotnetNewSearchTests.CanShowDeprecationMessage_WhenLegacyCommandIsUsed	dotnet/sdk#51558
1196220	dotnet/sdk	Microsoft.DotNet.Cli.New.IntegrationTests.DotnetNewSearchTests.CanShowDeprecationMessage_WhenLegacyCommandIsUsed	dotnet/sdk#51591

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
7	7	7

Report

Build	Definition	Step Name	Console log	Pull Request
2959523	dotnet-dotnet	Build	Log
2958411	dotnet-maui	Build the samples	Log
1388539	dotnet/sdk	🟣 Build	Log	dotnet/sdk#53447
1387575	dotnet/sdk	🟣 Build	Log	dotnet/sdk#53714
2953633	dotnet-maui	Controls (v18.5) TabbedPage,TableView,TimePicker,TitleView,ToolbarItem,Triggers	Log
1383021	dotnet/sdk	Build	Log
2953212	dotnet-sdk	Build	Log	#59726
2951958	dotnet-maui	Controls (v18.5) Shell	Log
1381780	dotnet/maui	Controls (vlatest) Entry	Log
2952778	dotnet-aspnetcore	Run build.sh (azureLinux30Net8BuildAmd64)	Log	#60175
1380895	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#53836
1380629	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#53557
1380238	dotnet/maui	Publish the android_ui_tests_controls_30 test results	Log
1379907	dotnet/maui	Publish the mac_ui_tests_controls test results	Log
2951358	dotnet-maui	Publish the android_ui_tests_controls_30 test results	Log
2951333	dotnet-maui	Publish the android_ui_tests_controls_30 test results	Log
1380249	dotnet/maui	Publish the android_ui_tests_controls_30 test results	Log	dotnet/maui#34959
2951170	dotnet-maui	Publish the ios_ui_tests_mono_controls_latest test results	Log
2950638	dotnet-maui	Publish the android_ui_tests_controls_30 test results	Log
1378733	dotnet/maui	Controls TabbedPage,TableView,TimePicker,TitleView,ToolbarItem,Triggers	Log
1378662	dotnet/sdk	Build	Log	dotnet/sdk#53860
2950364	dotnet-maui	Controls (vlatest) SafeAreaEdges,Shadow	Log
2949761	dotnet-maui	Controls (vlatest) Entry	Log
2949744	dotnet-maui	Controls (vlatest) Editor,Effects,Essentials,FlyoutPage,Focus,Fonts,Frame,Gestures,GraphicsView	Log
1376779	dotnet/maui	Controls CollectionView	Log	dotnet/maui#34573
2949849	dotnet-performance	Run performance job script	Log
2949738	dotnet-maui	Controls (vlatest) CarouselView	Log
1376199	dotnet/maui	Install dotnet preview without workloads	Log	dotnet/maui#34923
2948055	dotnet-runtime	Build product	Log	#59967
2947064	dotnet-maui	Controls (v18.5) CollectionView	Log
2947209	dotnet-maui	Controls (vlatest) CollectionView	Log
1370505	dotnet/aspnetcore	Run build.sh	Log
1370346	dotnet/aspnetcore	Run build.sh	Log	dotnet/aspnetcore#65958
1370266	dotnet/runtime	Build product	Log	dotnet/runtime#126647
2945576	dotnet-performance	Run performance job script	Log
1365431	dotnet/maui	Controls Lifecycle,ManualReview,Maps	Log	dotnet/maui#34796
2942589	dotnet-maui	Controls (vlatest) Cells,CheckBox,ContextActions,CustomRenderers,DatePicker,Dispatcher,DisplayAlert,DisplayPrompt,DragAndDrop	Log
1362028	dotnet/maui	Controls WebView	Log	dotnet/maui#34573
2941150	dotnet-maui	Controls (v18.5) ViewBaseTests,VisualStateManager,Window	Log
1361948	dotnet/aspnetcore	Run build.sh	Log	dotnet/aspnetcore#66003
1359985	dotnet/maui	Controls CollectionView	Log	dotnet/maui#34664
2940237	dotnet-maui	Controls (v18.5) Editor,Effects,Essentials,FlyoutPage,Focus,Fonts,Frame,Gestures,GraphicsView	Log
1359507	dotnet/maui	Controls (vlatest) WebView	Log
1357884	dotnet/maui	Controls Page,Performance,Picker,ProgressBar	Log
1358474	dotnet/aspnetcore	Run build.sh	Log	dotnet/aspnetcore#64562
1357796	dotnet/maui	Controls ListView	Log
2939384	dotnet-maui	Controls (vlatest) Border,BoxView,Brush,Button	Log
2939246	dotnet-maui	Controls (vlatest) Editor,Effects,FlyoutPage,Focus,Fonts,Frame,Gestures,GraphicsView	Log

Build	Definition	Test	Pull Request
1394388	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1390570	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_SslProtocols_Test.GetAsync_AllowedSSLVersion_Succeeds
1390469	dotnet/sdk	Microsoft.DotNet.Cli.New.IntegrationTests.DotnetNewSearchTests.BasicTest	dotnet/sdk#54034
1390399	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1389302	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1385254	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_SslProtocols_Test.GetAsync_AllowedSSLVersion_Succeeds
1384035	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1383491	dotnet/sdk	Microsoft.NET.Build.Containers.UnitTests.dll.2.WorkItemExecution	dotnet/sdk#53941
1382941	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1381576	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_SslProtocols_Test.GetAsync_AllowedSSLVersion_Succeeds
1381361	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1380981	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1380407	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_SslProtocols_Test.GetAsync_AllowedSSLVersion_Succeeds
1380051	dotnet/sdk	Microsoft.NET.Build.Containers.UnitTests.dll.2.WorkItemExecution	dotnet/sdk#53864
2951443	dotnet-sdk	Microsoft.NET.Build.Containers.UnitTests.dll.2.WorkItemExecution	#59942
1379288	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1371557	dotnet/sdk	Microsoft.NET.Build.Containers.UnitTests.dll.2.WorkItemExecution	dotnet/sdk#53745
1369661	dotnet/runtime	Workloads-NoWebcil-Wasm.Build.Tests.MainWithArgsTests.WorkItemExecution	dotnet/runtime#126322
1365329	dotnet/runtime	System.Net.Http.Functional.Tests.TelemetryTest_Http30.EventSource_SuccessfulRequest_LogsStartStop
1363759	dotnet/runtime	System.Net.Http.Functional.Tests.HttpClientHandlerTest_Http3.Public_Interop_ExactVersion_BufferContent_Success
1362750	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_SslProtocols_Test.GetAsync_AllowedSSLVersion_Succeeds
1361814	dotnet/runtime	System.Net.Http.Functional.Tests.TelemetryTest_Http30.EventSource_UnsuccessfulRequest_LogsStartFailedStop
1361815	dotnet/runtime	System.Net.Http.Functional.Tests.HttpClientHandlerTest_Http3.Public_Interop_ExactVersion_BufferContent_Success
1360430	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	9	72

[nagilson]

I'm still seeing this issue where the string contains match is identical, yet the known issue labeler doesn't think it's a known issue.