SourceFileUpdateTests.RenameSourceFile: flaky failure — FileNotFoundException on Foo.cs / missing '> Renamed.cs' output

[MichaelSimons]

Build Information

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1490592
Leg Name: TestBuild: macOS (arm64) - osx.15.arm64.open — Helix work item dotnet-watch.Tests.dll.2 (Finished, exit code 2)
Pull request: #55109

Error Details

Microsoft.DotNet.Watch.UnitTests.SourceFileUpdateTests.RenameSourceFile intermittently fails on the macOS arm64 Helix leg. The useMove: false variant hits a FileNotFoundException on the renamed dependency file (Foo.cs) and never observes the expected > Renamed.cs hot-reload output, so the assertion fails. It passed on retry (build 1491201), confirming it is flaky rather than a product regression. (The useMove: true variant is skipped due to the known limitation tracked in #43320.)

failed RenameSourceFile (False) (5s 768ms)
  Assertion failed.
  Failed to find expected prefix: '> Renamed.cs'
   at Microsoft.DotNet.Watch.UnitTests.SourceFileUpdateTests.RenameSourceFile(Boolean useMove) in /_/test/dotnet-watch.Tests/HotReload/SourceFileUpdateTests.cs:233
...
dotnet watch ❌ An unexpected error occurred: System.IO.FileNotFoundException: Could not find file '/private/tmp/helix/working/AB0409A7/t/dotnetSdkTests.xBTNK8Z7/RenameSourceF---E7D36796/Dependency/Foo.cs'.
   at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirError)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(...)

Note: the existing KBE #40006 does not cover this — its pattern is hard-coded to dotnet-watch.Tests.dll.1 and the old ... has failed Helix.Sdk message format, whereas this failure is in the dll.2 partition and surfaces via the newer Monitor Helix Jobs failed (Finished, exit code 2) format. The pattern below matches the test failure directly in the Helix console log, so ExcludeConsoleLog is false.

Error Message

{
    "ErrorMessage": ["failed RenameSourceFile", "SourceFileUpdateTests.RenameSourceFile"],
    "BuildRetry": true,
    "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=1490592
Error message validated: [failed RenameSourceFile SourceFileUpdateTests.RenameSourceFile]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 7/5/2026 7:15:15 AM UTC

Report

Build	Repository	Test	Pull Request
1521428	dotnet/sdk	RenameSourceFile
1511599	dotnet/sdk	RenameSourceFile	#55144
1490592	dotnet/sdk	RenameSourceFile	#55109

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	1	3

[MichaelSimons]

Root cause analysis

Investigation of the intermittent macOS arm64 failure of RenameSourceFile(useMove: false). This is an assessment plus a suggested product change for the dotnet-watch owners to evaluate ? no code change is being proposed via PR at this time.

What the test does

test/dotnet-watch.Tests/HotReload/SourceFileUpdateTests.cs ? RenameSourceFile(false) renames a dependency source file with a non-atomic delete+create:

File.Delete(oldFilePath);            // Dependency/Foo.cs
File.WriteAllText(newFilePath, ...); // Dependency/Renamed.cs

and then asserts hot reload prints > Renamed.cs.

Observed failure

dotnet watch An unexpected error occurred: System.IO.FileNotFoundException: Could not find file '.../Dependency/Foo.cs'.

The expected > Renamed.cs output is therefore never produced. It passed on retry, i.e. flaky, not a product regression. (useMove: true remains skipped per #43320.)

Why it happens

1. macOS FileSystemWatcher is backed by FSEvents, which coalesces/reorders events. The deletion of Foo.cs can surface as an Update, and a delete-?-add sequence for a path is collapsed to Update by HotReloadDotNetWatcher.NormalizePathChanges.
2. In HotReloadDotNetWatcher.CaptureChangedFilesSnapshot, Foo.cs is present in the evaluation result, so its change is kept as Update:

   if (evaluationResult.Files.TryGetValue(changedPath.Path, out var existingFileItem))
   {
       var changeKind = changedPath.Kind == ChangeKind.Add ? ChangeKind.Update : changedPath.Kind;
       return new ChangedFile(existingFileItem, changeKind);
   }

3. When no change in the batch is classified as Add, AnalyzeFileChanges sets evaluationRequired == false, so the code calls compilationHandler.UpdateFileContentAsync(...) instead of doing a full re-evaluation.
4. HotReloadMSBuildWorkspace.UpdateFileContentAsync reads file content for Add/Update changes. Reading the already-deleted Foo.cs throws FileNotFoundException, which escapes to Program.cs ("An unexpected error occurred") and aborts the iteration ? so hot reload never runs.

Why it is intermittent

It depends entirely on how/when the OS coalesces and orders the delete of Foo.cs vs. the create of Renamed.cs. When the create of Renamed.cs is seen as an Add, dotnet-watch does a full re-evaluation (reloading the graph from disk) and never reads the missing Foo.cs ? that ordering passes. The bad ordering (delete surfaced as Update, no Add in the batch) is what triggers the crash, and it only reproduced on macOS.

Assessment

This looks like a genuine product robustness gap rather than a pure test-timing issue: renaming a watched source file via delete+create is a legitimate scenario (several editors/tools rename files that way), and dotnet-watch crashing with an unhandled FileNotFoundException when a watched file briefly/permanently disappears is undesirable. A test-only workaround (e.g. reordering to create-before-delete) would mask the fragility and reduce coverage of the delete-then-add path.

Suggested product change (for the owning team to consider)

In CaptureChangedFilesSnapshot, when a change is classified as Add/Update but the target file no longer exists on disk, downgrade it to Delete before it reaches the workspace ? a file that is not present cannot be added or updated (it was removed/renamed away), so the workspace should drop the document rather than attempt (and fail) to read it:

var changeKind = changedPath.Kind;

// A change reported as Add or Update may target a file that has already been
// deleted or renamed away by the time we process the change (delete+create rename,
// or coalesced/reordered OS watcher events). Reading such a file would throw
// FileNotFoundException, so treat a missing file as a deletion instead.
if (changeKind is ChangeKind.Add or ChangeKind.Update && !File.Exists(changedPath.Path))
{
    changeKind = ChangeKind.Delete;
}

Notes / trade-offs to weigh:

• The existence check runs at snapshot-capture time (after debounce/normalization), so in the rename race Foo.cs is reliably reclassified as Delete.
• For a transient temp-write/rename-over save where a file is momentarily missing, we'd emit Delete now and the subsequent Add/Update event re-adds it, so content still converges ? but this is worth validating against the editors you care about.
• Alternatively, the read in HotReloadMSBuildWorkspace.UpdateFileContentAsync could tolerate a missing file for Update/Add, but that lives outside this repo (Microsoft.CodeAnalysis.ExternalAccess.HotReload), so handling it in dotnet-watch is the more self-contained option.

The suggested change compiles (Microsoft.DotNet.HotReload.Watch.csproj, 0 warnings / 0 errors), but the repro is macOS-only so it could not be exercised end-to-end locally.

[MichaelSimons]

@tmat - can you take a look at this issue? I included a copilot accessment of the RCA. Given the recommended product change, I will hand it over to you. TIA.