Fix CI flakiness: MSB4216 task host failures, dotnet-watch hangs, NuGet errors by mmitche · Pull Request #53424 · dotnet/sdk

mmitche · 2026-03-12T20:04:26Z

Summary

This PR addresses three categories of intermittent CI failures identified through systematic analysis of 30 recent builds (57% failure rate baseline, ~30% after excluding clearly-broken PRs).

Root Cause 1 - MSB4216 task host failures on macOS Helix

Impact: Tests using NuGet package tasks with TaskHostFactory (ComputeWasmBuildAssets from WebAssembly SDK, ComputeManagedAssemblies from ILLink) fail intermittently on macOS Helix machines.

Fix: Export DOTNET_HOST_PATH in both RunTestsOnHelix.sh and RunTestsOnHelix.cmd so MSBuild can locate the dotnet executable when spawning out-of-process task hosts.

Root Cause 2 - dotnet-watch Aspire_BuildError_ManualRestart test hang

Impact: Test hangs for 60+ minutes on Helix, consuming the entire work item timeout.

Fix:

Close stdin pipe before killing process tree in AwaitableProcess.DisposeAsync()
Add 30-second timeout for process exit wait
Reduce DCP operation timeouts from 100,000s (~27 hours) to 300s (5 minutes)

Root Cause 3 - Noisy NuGet source removal errors

Impact: Helix test setup produces confusing error messages when trying to remove internal NuGet sources that don't exist in public CI builds.

Fix: Suppress errors from dotnet nuget remove source commands using || true (bash) and 2>nul (cmd).

Validation

Building on the feature branch to establish baseline pass rate. Target: 25+ consecutive passing builds.

Copilot

Pull request overview

This PR hardens Helix/CI test execution for Hot Reload / dotnet-watch scenarios by reducing hang risk and improving the reliability of Helix environment setup.

Changes:

Replace “effectively infinite” DCP/Aspire timeouts used by watch-based tests with bounded (5-minute) values to prevent multi-hour Helix hangs.
Improve test process cleanup by closing stdin prior to termination and adding a bounded wait for process exit.
Update Helix test entrypoint scripts to set DOTNET_HOST_PATH and make NuGet source removal resilient when sources are absent.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
test/Microsoft.DotNet.HotReload.Test.Utilities/WatchableApp.cs	Switches DCP/Aspire-related environment timeouts to bounded values for CI.
test/Microsoft.DotNet.HotReload.Test.Utilities/AwaitableProcess.cs	Adjusts disposal/termination behavior to reduce hangs during cleanup.
build/RunTestsOnHelix.sh	Sets `DOTNET_HOST_PATH` and makes `dotnet nuget remove source` tolerant of missing sources.
build/RunTestsOnHelix.cmd	Sets `DOTNET_HOST_PATH` and suppresses errors when removing non-existent NuGet sources.

You can also share your feedback on Copilot code review. Take the survey.

Copilot · 2026-03-12T20:08:07Z

test/Microsoft.DotNet.HotReload.Test.Utilities/AwaitableProcess.cs

+            catch (OperationCanceledException)
+            {
+                Logger.Log($"Process {Id} did not exit within 30 seconds after Kill()");


If the process still hasn’t exited after the 30s timeout, DisposeAsync only logs and continues. That can leave a runaway dotnet-watch process (or child processes) running in the Helix work item, causing resource leaks and cross-test interference while also hiding the failure signal. Consider failing the test/cleanup in this case (or at least making the cleanup path retry/force termination and surface the problem), rather than just logging and proceeding to dispose the Process handle.

Suggested change

catch (OperationCanceledException)

{

Logger.Log($"Process {Id} did not exit within 30 seconds after Kill()");

catch (OperationCanceledException ex)

{

Logger.Log($"Process {Id} did not exit within 30 seconds after Kill()");

throw new TimeoutException($"Process {Id} did not exit within 30 seconds after Kill().", ex);

Copilot · 2026-03-12T20:08:08Z

test/Microsoft.DotNet.HotReload.Test.Utilities/AwaitableProcess.cs

+            try
+            {
+                Process.StandardInput.Close();
+            }
+            catch


The broad catch around closing StandardInput/Kill() swallows all exceptions without logging. Since this code was added to address a platform-specific hang, swallowing the exception makes it hard to diagnose when stdin can’t be closed (or why). Consider at least logging the exception details (or narrowing the caught exception types) so cleanup failures are actionable in CI logs.

tmat · 2026-03-13T02:29:33Z

src/Dotnet.Watch/Watch/Aspire/AspireServiceFactory.cs

            _isDisposed = true;

            // wait for all in-flight process initialization to complete:
+            // If no session initialization is in-flight (_pendingSessionInitializationCount == 0),


Not entirely correct either. Ok to merge, I'll follow up with better fix.

@tmat This is an automatic fix by the AI for flakiness. Don't merge this...when it's gotten 25 passing runs we'll take a second pass over this to smooth it out.

Yeah, I figured. It looks very AI like. It's great it found the issue. I'll work on a better fix over the weekend.

tmat · 2026-03-13T02:34:13Z

watch changes lgtm

mmitche · 2026-03-13T12:44:20Z

🎯 Milestone: 5 Consecutive Passes

Validation Results

Build	Jobs	Result
1333280	18/18 ✅	Passed
1333378	18/18 ✅	Passed
1333489	18/18 ✅	Passed
1333567	18/18 ✅	Passed
1333595	18/18 ✅	Passed

Root Causes Fixed

MSB4216 TaskHostFactory — DOTNET_HOST_PATH not set in Helix scripts
dotnet-watch Aspire hang — Per-operation timeout was inheriting the 2-hour Helix work-item timeout instead of being capped at 5 minutes. Also fixed semaphore deadlock in AspireServiceFactory.DisposeAsync, added stdin close before process kill, and set DCP timeout environment variables.
GZipCompress file lock — Parallel.For races with antivirus/file indexer; added retry with exponential backoff
NuGet source removal noise — Suppressed stderr from removing non-existent NuGet sources
Missing runtimeconfig.json — Added MSBuild target to include runtimeconfig.json in NuGet packages for test tool projects

Baseline vs Current

Before: ~57% failure rate (17/30 builds failed on main)
After: 5/5 consecutive perfect builds (100% pass rate so far)

Continuing validation toward 25 consecutive passes target.

akoeplinger · 2026-03-13T13:04:38Z

build/RunTestsOnHelix.cmd


+REM Set DOTNET_HOST_PATH so MSBuild task hosts can locate the dotnet executable.
+REM Without this, tasks from NuGet packages that use TaskHostFactory fail with MSB4216.
+set DOTNET_HOST_PATH=%DOTNET_ROOT%\dotnet.exe


@ViktorHofer does this make sense?

akoeplinger · 2026-03-13T13:06:39Z

build/RunTestsOnHelix.cmd

-dotnet nuget remove source dotnet-tools-transport --configfile %TestExecutionDirectory%\nuget.config
-dotnet nuget remove source dotnet-libraries --configfile %TestExecutionDirectory%\nuget.config
-dotnet nuget remove source dotnet-eng --configfile %TestExecutionDirectory%\nuget.config
+REM Remove feeds not needed for tests. Errors from non-existent sources


any idea which feeds remain? maybe we should have a separate nuget.config just for the tests

mmitche · 2026-03-13T13:44:05Z

@akoeplinger This is an automated run...let it go for a while..don't merge or approve. I'll have it go through another pass and also reivew at the end of when it thinks it gets a stable run.

Root causes fixed: 1. MSB4216 task host failures - DOTNET_HOST_PATH not set in Helix 2. dotnet-watch test hangs - AwaitableProcess timeout cap, disposal race, semaphore deadlock 3. GZipCompress file lock in BlazorWasm parallel compression 4. Missing runtimeconfig.json in test tool NuGet packages 5. CompilationHandler IOException during hot reload file updates 6. BrowserTests.LaunchesBrowserOnStart timing race - sync to async wait 7. NuGet source removal noise in Helix scripts

tmat · 2026-03-16T18:47:33Z

#53271 has a better fix for the dotnet-watch race condition.

Copilot AI review requested due to automatic review settings March 12, 2026 20:04

github-actions bot added the Area-Infrastructure label Mar 12, 2026

Copilot started reviewing on behalf of mmitche March 12, 2026 20:05 View session

Copilot AI reviewed Mar 12, 2026

View reviewed changes

mmitche force-pushed the fix/ci-flakiness/main branch from 83d044f to ae5112f Compare March 12, 2026 20:21

mmitche requested a review from a team as a code owner March 12, 2026 20:21

mmitche force-pushed the fix/ci-flakiness/main branch 2 times, most recently from a35fa75 to d9f4358 Compare March 12, 2026 23:54

mmitche requested review from a team and tmat as code owners March 12, 2026 23:54

build-analysis bot mentioned this pull request Mar 13, 2026

dotnet-watch tests failing #40006

Open

mmitche force-pushed the fix/ci-flakiness/main branch from d9f4358 to 9949b40 Compare March 13, 2026 01:26

mmitche requested a review from a team as a code owner March 13, 2026 01:26

mmitche force-pushed the fix/ci-flakiness/main branch from 9949b40 to ed04d76 Compare March 13, 2026 02:28

tmat reviewed Mar 13, 2026

View reviewed changes

mmitche force-pushed the fix/ci-flakiness/main branch 3 times, most recently from 2eb12cb to bcae687 Compare March 13, 2026 12:43

mmitche force-pushed the fix/ci-flakiness/main branch from bcae687 to 1644c42 Compare March 13, 2026 12:45

akoeplinger reviewed Mar 13, 2026

View reviewed changes

mmitche force-pushed the fix/ci-flakiness/main branch 7 times, most recently from 4cb1ca9 to ea8fad3 Compare March 14, 2026 00:12

mmitche force-pushed the fix/ci-flakiness/main branch 12 times, most recently from aa94871 to fac0ce7 Compare March 15, 2026 00:52

mmitche force-pushed the fix/ci-flakiness/main branch from fac0ce7 to 189c634 Compare March 15, 2026 15:40

tmat mentioned this pull request Mar 16, 2026

Improve shutdown sequence of Aspire service and fix race condition #53271

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix CI flakiness: MSB4216 task host failures, dotnet-watch hangs, NuGet errors#53424

Fix CI flakiness: MSB4216 task host failures, dotnet-watch hangs, NuGet errors#53424
mmitche wants to merge 1 commit intomainfrom
fix/ci-flakiness/main

mmitche commented Mar 12, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Mar 12, 2026

Uh oh!

Copilot AI Mar 12, 2026

Uh oh!

tmat Mar 13, 2026

Uh oh!

mmitche Mar 13, 2026

Uh oh!

tmat Mar 13, 2026

Uh oh!

tmat commented Mar 13, 2026

Uh oh!

mmitche commented Mar 13, 2026

Uh oh!

akoeplinger Mar 13, 2026

Uh oh!

akoeplinger Mar 13, 2026

Uh oh!

mmitche commented Mar 13, 2026

Uh oh!

tmat commented Mar 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

mmitche commented Mar 12, 2026

Summary

Root Cause 1 - MSB4216 task host failures on macOS Helix

Root Cause 2 - dotnet-watch Aspire_BuildError_ManualRestart test hang

Root Cause 3 - Noisy NuGet source removal errors

Validation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

tmat Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

mmitche Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

tmat Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

tmat commented Mar 13, 2026

Uh oh!

mmitche commented Mar 13, 2026

🎯 Milestone: 5 Consecutive Passes

Validation Results

Root Causes Fixed

Baseline vs Current

Uh oh!

akoeplinger Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

akoeplinger Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

mmitche commented Mar 13, 2026

Uh oh!

tmat commented Mar 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants