ObjectDisposedException fail-fast in DownloadAndRegisterEpsAsync progress callback (Progress<T> on ThreadPool touches disposed CTS) — WinML 1.2.3

[andrewtheart]

Summary

An intermittent, unhandled System.ObjectDisposedException: The CancellationTokenSource has been disposed. is thrown on a ThreadPool thread from inside the SDK's own progress-callback marshaling during FoundryLocalManager.DownloadAndRegisterEpsAsync(...). Because it is unhandled on a ThreadPool thread, it fail-fasts the entire host process (0xc0000409). The calling application has no way to catch it.

Environment

• Package: Microsoft.AI.Foundry.Local.WinML 1.2.3 — faulting module Microsoft.AI.Foundry.Local.Core.dll, assembly/file version 1.2.0.0
• Host: WinUI 3 / Windows App SDK desktop app targeting net10.0-windows (the .NET Runtime crash event reported CoreCLR Version: 9.0.17)
• OS: Windows 11 Pro, build 26200 (x64)
• GPU / EP: NVIDIA GeForce RTX 5090 (CUDA execution provider)

What the app was doing

One-time initialization. After FoundryLocalManager.CreateAsync(...) we register the hardware execution providers with a progress callback, then fetch the catalog:

await manager.DownloadAndRegisterEpsAsync(
    (epName, pct) => { /* report progress to the UI */ },
    cancellationToken);

The crash occurs during this call. It is intermittent — in our logs, EP registration completed successfully 23 times and fail-fasted once. It appears most likely when the EPs are already cached, i.e. the download_and_register_eps command returns very quickly.

Crash details

Windows Error Reporting — Application Error (Event ID 1000):

Faulting application name: <host>.exe
Faulting module name:      Microsoft.AI.Foundry.Local.Core.dll, version: 1.2.0.0
Exception code:            0xc0000409

.NET Runtime (Event ID 1026):

Description: The process was terminated due to an unhandled exception.
Exception Info: System.ObjectDisposedException: The CancellationTokenSource has been disposed.
   at Microsoft.AI.Foundry.Local.NativeInterop.<>c__DisplayClass13_0.<ExecuteCommandWithCallbackManaged>b__9(String, Double)
   at System.Progress`1.InvokeHandlers(Object)
   at System.Threading.QueueUserWorkItemCallbackDefaultContext`1.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.WindowsThreadPool.DispatchCallback(IntPtr, IntPtr, IntPtr)

Root cause (analysis)

ExecuteCommandWithCallbackManaged marshals native progress updates through a System.Progress<T>, whose handler invocations are posted to the ThreadPool (the Progress<T> is constructed on a background thread with no SynchronizationContext). The command also uses a per-command CancellationTokenSource.

When the download_and_register_eps command finishes — quickly, when EPs are already cached — the SDK disposes that CTS, but a progress callback (b__9) is still queued on the ThreadPool. When that queued callback runs, it touches the now-disposed CTS, throwing ObjectDisposedException. Since it runs as a ThreadPool work item with no handler, the runtime terminates the process.

The throw happens inside the SDK's own wrapper (b__9), before control reaches the app-supplied (epName, pct) callback, so the calling application cannot observe or handle it (it is not surfaced through AppDomain.UnhandledException in a way that can prevent termination).

Note: appears already addressed on main

This managed-interop code path does not appear to exist in the current open-source tree — ExecuteCommandWithCallbackManaged, ExecuteWithTracker, ExecuteCommandManaged, and a Progress<T>-based NativeInterop all return no hits — and the current EP-download paths use disposal-safe patterns:

• sdk/cs routes EP progress through a plain string ICoreInterop.CallbackFn (no Progress<T>).
• sdk_v2/cs checks ct?.IsCancellationRequested (safe after dispose) inside the native callback.

So this looks specific to the compiled 1.2.x WinML assembly and likely resolved by the SDK v2 / unified-package refactor.

Ask

1. Can you confirm this Progress<T> EP-progress / disposed-CTS race is gone in the unified/v2 package?
2. Would you consider a 1.2.x servicing backport — e.g. guard the progress handler so it cannot touch the CTS after disposal, or defer disposing the per-command CTS until all queued Progress<T> callbacks have drained?

Thanks!

[hovrawl]

While this reads like and AI Filed report, I am actually getting this issue myself when following the latest example on the Foundry Docs, and it happens MOST times for me with this snippet:

var eps = manager.DiscoverEps();
int maxNameLen = 30;
Console.WriteLine("Available execution providers:");
Console.WriteLine($"  {"Name".PadRight(maxNameLen)}  Registered");
Console.WriteLine($"  {new string('─', maxNameLen)}  {"──────────"}");
foreach (var ep in eps)
{
    Console.WriteLine($"  {ep.Name.PadRight(maxNameLen)}  {ep.IsRegistered}");
}

// Download and register all execution providers with per-EP progress.
// EP packages include dependencies and may be large.
// Download is only required again if a new version of the EP is released.
// For cross platform builds there is no dynamic EP download and this will return immediately.
Console.WriteLine("\nDownloading execution providers:");
if (eps.Length > 0)
{
    string currentEp = "";
    await manager.DownloadAndRegisterEpsAsync((epName, percent) =>
    {
        if (epName != currentEp)
        {
            if (currentEp != "")
            {
                Console.WriteLine();
            }
            currentEp = epName;
        }
        Console.Write($"\r  {epName.PadRight(maxNameLen)}  {percent,6:F1}%");
        
    });
    Console.WriteLine();
}
else
{
    Console.WriteLine("No execution providers to download.");
}
Console.WriteLine("Foundry started");
Console.WriteLine("Loading catalog...");
var modelLoadCt = CancellationToken.None;
var catalog = await manager.GetCatalogAsync(modelLoadCt);

Starting Foundry
Available execution providers:
  Name                            Registered
  ──────────────────────────────  ──────────
  WebGpuExecutionProvider         False
  OpenVINOExecutionProvider       False

Downloading execution providers:
Unhandled exception. System.ObjectDisposedException: The CancellationTokenSource has been disposed.
   at Microsoft.AI.Foundry.Local.NativeInterop.<>c__DisplayClass13_0.<ExecuteCommandWithCallbackManaged>b__9(String, Double) + 0xc5
   at System.Progress`1.InvokeHandlers(Object) + 0x46
   at System.Threading.QueueUserWorkItemCallbackDefaultContext`1.Execute() + 0x2f
   at System.Threading.ThreadPoolWorkQueue.Dispatch() + 0x1a3
   at System.Threading.WindowsThreadPool.DispatchCallback(IntPtr, IntPtr, IntPtr) + 0x67

Worth noting that even when it works, the providers never save as registered even when using the model correctly.

[andrewtheart]

It is AI but thats irrelevant I suppose. It's a legitimate issue.