Add progress reporting during heap enumeration

[danmoseley]

Add progress reporting during heap enumeration

Commands like verifyheap and dumpheap -stat walk the entire managed heap but produce zero output during the walk. On large dumps (e.g., 19 GB / 248M objects), this means minutes of complete silence on both stdout and stderr, indistinguishable from a hang.

This PR adds periodic progress output (every 10 seconds) during heap enumeration for the two worst offenders:

• verifyheap -- 7+ minutes of silence before printing a single line
• dumpheap -stat -- ~80 seconds of silence before printing the statistics table

Example output

> dumpheap -stat
Scanning heap: 3,200 MB / 16,300 MB (20%)...
Scanning heap: 6,500 MB / 16,300 MB (40%)...
Scanning heap: 9,800 MB / 16,300 MB (60%)...
Scanning heap: 13,100 MB / 16,300 MB (80%)...
Scanning heap: 16,100 MB / 16,300 MB (99%)...
Statistics:
          MT      Count     TotalSize Class Name
...
Total 248,737,850 objects, 17,528,701,450 bytes

Without the fix, there would be no output at all before the Statistics: line.

Design choices

Output format: Progress lines are written via Console.WriteLine(), producing one line every ~10 seconds that scrolls up. Two other formats were considered:

• Dots (. . . . .) -- minimal but gives no indication of how far along or how much is left
• In-place \r overwrite -- cleanest UX in interactive terminals, but produces a single growing line when stdout is redirected to a file, and depends on terminal \r support

WriteLine is the most portable choice; for verifyheap on this dump it produces ~43 progress lines over 7 minutes, which seems an acceptable trade-off for the information provided.

Scope: Only verifyheap (always) and dumpheap -stat (when -stat is used) enable progress. Commands that already stream output continuously (dumpheap without -stat, finalizequeue, gchandles) don't need it. gcheapstat and dumpasync also have silent walks but use different enumeration paths; they can be addressed in a follow-up.

Performance

The progress mechanism adds negligible overhead:

• A single Stopwatch.ElapsedMilliseconds read per valid object (~5ns), compared to the per-object work of reading method tables, sizes, and verifying references (orders of magnitude more expensive)
• The actual Console.WriteLine call happens only once every 10 seconds
• Segment metadata is materialized into a List<ClrSegment> upfront (typically a handful of segments) to avoid double-enumerating the filtered segment list

Changes

• HeapWithFilters.cs -- Added ProgressCallback property and a ProgressIntervalMs constant. When set, EnumerateFilteredObjects() computes total heap size from segment metadata and periodically invokes the callback.
• ProgressReporter.cs -- New helper class encapsulating the time-gated progress logic, with a static FormatProgressMessage method.
• VerifyHeapCommand.cs -- Sets progress callback before heap walk.
• DumpHeapCommand.cs -- Sets progress callback when -stat is used.
• Unit tests -- ProgressReporterTests covering callback timing, byte tracking, and message formatting.

Test data (19.3 GB dump, 248M objects, warm cache)

Command	Before (silent)	After (first progress at)
verifyheap	7:16 silence	~10s
dumpheap -stat	1:30 silence	~10s

Fixes #5760

[danmoseley]

One thing not mentioned above -- while a user may know to wait without feedback, the Copilot CLI (at least for me) did not. So this may be more intersting to fix than before.

I'm assuming that changing the console output here is not considered a breaking change...

[Copilot]

Pull request overview

Adds periodic progress reporting during managed heap enumeration to avoid long “silent” periods in verifyheap and dumpheap -stat, using a time-gated reporter and a callback hook in the heap enumeration pipeline.

Changes:

• Added ProgressReporter helper and integrated it into HeapWithFilters.EnumerateFilteredObjects() via an optional ProgressCallback.
• Enabled progress reporting for verifyheap (always) and dumpheap -stat (when -stat is used).
• Added a new unit test project with ProgressReporterTests.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
src/Microsoft.Diagnostics.ExtensionCommands/HeapWithFilters.cs	Adds progress callback plumbing and reporting during object enumeration.
src/Microsoft.Diagnostics.ExtensionCommands/ProgressReporter.cs	Introduces time-gated progress reporting + message formatting.
src/Microsoft.Diagnostics.ExtensionCommands/VerifyHeapCommand.cs	Enables heap-scan progress output for verifyheap.
src/Microsoft.Diagnostics.ExtensionCommands/DumpHeapCommand.cs	Enables heap-scan progress output for dumpheap -stat.
src/Microsoft.Diagnostics.ExtensionCommands/Microsoft.Diagnostics.ExtensionCommands.csproj	Grants unit tests access to internals via InternalsVisibleTo.
src/tests/Microsoft.Diagnostics.ExtensionCommands.UnitTests/Microsoft.Diagnostics.ExtensionCommands.UnitTests.csproj	Adds new unit test project for progress reporting.
src/tests/Microsoft.Diagnostics.ExtensionCommands.UnitTests/ProgressReporterTests.cs	Adds unit tests for reporting behavior and formatting.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[danmoseley]

flaky tests-

1. SOS.UnitTests (~14 tests per Windows leg) — Known issue. Caused by CDB SecureLoadDotNetExtensions rejecting
   unsigned DAC DLLs. PR Disable desktop CLR and single-file DAC tests blocked by CDB SecureLoadDotNetExtensions #5759 specifically fixes this. Identical failure counts on PR Update SDK to 10.0.105 and bump test runtime versions #5762 (unrelated SDK update).
   Your code doesn't touch SOS tests or CDB infrastructure.
2. EventPipe.UnitTests (1 test) — Flaky test. Failed on your Win x64 Debug, but failed on PR Update SDK to 10.0.105 and bump test runtime versions #5762's macOS Debug.
   Different platform each time = classic flaky behavior. Your changes don't touch EventPipe.
3. NETCore.Client.UnitTests (1 test on Alpine + Ubuntu) — Flaky test. Your changes only modify heap enumeration
   progress reporting (HeapWithFilters.cs, ProgressReporter.cs, DumpHeapCommand.cs, VerifyHeapCommand.cs) — completely
   unrelated to the NETCore.Client IPC library. This test exercises EventPipe sessions and process discovery.