Skip to content
Draft
Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
5923ea3
fix(container-loader): dispose SerializedStateManager instances in te…
alexvy86 Mar 5, 2026
3731e73
fix(container-runtime): dispose ContainerRuntime instances in tests t…
alexvy86 Mar 5, 2026
00020af
fix(tree): remove --exit flag by properly disposing TestTreeProvider …
alexvy86 Mar 6, 2026
5257835
fix(local-server-tests): properly dispose LocalDeltaConnectionServer …
alexvy86 Mar 6, 2026
6661ee7
fix(fluid-telemetry,react): remove --exit flag by cleaning up Applica…
alexvy86 Mar 6, 2026
da562c9
fix(inventory-app): remove --exit flag by setting up JSDOM before Qui…
alexvy86 Mar 6, 2026
fadbe1e
fix(experimental): remove --exit flag from dds/tree and property-dds …
alexvy86 Mar 6, 2026
3dd5255
fix(dds): remove --exit flag from memory/benchmark test mocharcs
alexvy86 Mar 6, 2026
7b4ab5d
fix(test-end-to-end-tests): remove --exit flag by fixing LocalServerT…
alexvy86 Mar 6, 2026
fa80b89
fix: remove --exit flag from webflow, odsp-client, and snapshots pack…
alexvy86 Mar 6, 2026
63ff049
fix(experimental,e2e-tests): remove --exit flag from PropertyDDS and …
alexvy86 Mar 6, 2026
02b75e2
Merge remote-tracking branch 'upstream/main' into ralph-removes-mocha…
alexvy86 Mar 6, 2026
d906f29
Remove file
alexvy86 Mar 6, 2026
a6d6b67
Update investigation file
alexvy86 Mar 6, 2026
7d844b0
Formatting
alexvy86 Mar 6, 2026
387e188
fix(dds/tree,fluid-telemetry): fix lint errors in dispose and stopPol…
alexvy86 Mar 6, 2026
3d11ab7
fix(local-server-tests): fix timer leaks so tests exit without --exit…
alexvy86 Mar 6, 2026
a687f40
fix(property-dds): dispose containers in afterEach to prevent timer l…
alexvy86 Mar 7, 2026
8f4a2bd
fix(local-server-stress-tests): remove --exit flag as cleanup is alre…
alexvy86 Mar 7, 2026
b150f37
docs: update investigation notes with container disposal pattern
alexvy86 Mar 7, 2026
3eece80
fix(fluid-telemetry): clear heartbeat interval on container dispose t…
alexvy86 Mar 7, 2026
7d916d0
fix(react,fluid-static): fix timer leaks so tests exit without --exit…
alexvy86 Mar 7, 2026
2d8515b
docs: update investigation notes with react/fluid-static timer leak f…
alexvy86 Mar 7, 2026
fd01a03
More fixes
alexvy86 Mar 7, 2026
57bec76
fix(dds/tree,container-runtime): fix timer leaks from summarizer cont…
alexvy86 Mar 8, 2026
dcc90fc
fix(test/snapshots): fix timer leaks so tests exit without --exit flag
alexvy86 Mar 8, 2026
804a1cb
docs: update investigation notes for packages/test/snapshots (root ca…
alexvy86 Mar 8, 2026
498b329
Formatting
alexvy86 Mar 9, 2026
02bf0f0
Fix missing async
alexvy86 Mar 9, 2026
c79462d
fix(test-end-to-end-tests,container-loader,container-runtime): fix re…
alexvy86 Mar 9, 2026
bdb7f9e
Formatting
alexvy86 Mar 10, 2026
7f3979d
fix(mocha-test-setup,test-end-to-end-tests): fix timer/socket leaks s…
alexvy86 Mar 10, 2026
edd661f
docs: update investigation notes with root causes 16 (setTimeout patc…
alexvy86 Mar 10, 2026
83c1324
Fix build
alexvy86 Mar 10, 2026
e3c13ae
Fix build
alexvy86 Mar 10, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
122 changes: 122 additions & 0 deletions TODO/investigation.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
# Investigation: Why mocha tests need --exit flag

## Summary of Root Causes Found

### Root Cause 1: SnapshotRefresher 24-hour timer (CONFIRMED via --detect-open-handles)
**Affected packages:** packages/loader/container-loader (and likely others)

`SnapshotRefresher` creates a 24-hour (86400000ms) `setTimeout` via the `Timer` class's
`setLongTimeout`. Tests create `SerializedStateManager` instances (which own a `SnapshotRefresher`)
without calling `.dispose()` after the test completes. The timer keeps the process alive.

**Location:** `packages/loader/container-loader/src/test/serializedStateManager.spec.ts`
**Fix:** Add `afterEach` cleanup to call `serializedStateManager.dispose()` for each test
that creates an instance with offline load enabled.

### Root Cause 2: DeliLambda.readClientIdleTimer (CONFIRMED by code analysis)
**Affected packages:** Any package that uses TestTreeProvider.create() or TestObjectProvider
with a local server driver (e.g. packages/dds/tree, packages/test/local-server-tests, etc.)

`DeliLambda` creates a 60-second `setInterval` (readClientIdleTimer) when the local orderer
is set up. This timer is only cleared when `DeliLambda.close()` is called, which happens when
`LocalServerTestDriver.dispose()` is called. Tests that use `TestTreeProvider.create()` directly
(not through `describeCompat`) don't call `driver.dispose()` after tests.

`describeCompat` already handles this with `provider.driver.dispose?.()` in its `after` hook,
but calls it fire-and-forget (not awaited). This still causes a brief hang while the async
cleanup chain completes.

**Location:** packages/dds/tree/src/test/* (and others using TestTreeProvider)
**Fix:** Add proper cleanup for TestTreeProvider instances in tests

### Root Cause 3: JSDOM (suspected)
**Affected packages:** packages/framework/react

JSDOM keeps the process alive through internal timers and window/document event handling.
Tests use `globalJsdom()` from the `global-jsdom` package.
**Status:** Not yet confirmed via diagnostics.

### Root Cause 4: Other packages (unknown)
**Affected packages:** packages/runtime/container-runtime, and others

Likely similar to root cause 1 (SnapshotRefresher or similar timed objects), but not yet confirmed.

## Packages affected
- packages/dds/tree (Root cause 2 - TestTreeProvider not disposed)
- packages/runtime/container-runtime (Root cause 4 - TBD)
- packages/loader/container-loader (Root cause 1 - SnapshotRefresher 24h timer - CONFIRMED)
- packages/service-clients/odsp-client (Root cause 2 - likely local server)
- packages/test/local-server-tests (Root cause 2 - local server)
- packages/test/snapshots (Root cause 4 - TBD)
- packages/framework/react (Root cause 3 - JSDOM)
- packages/framework/client-logger/fluid-telemetry (Root cause 4 - TBD)
- packages/test/test-end-to-end-tests/src/test (Root cause 2 - local server)
- examples/data-objects/table-document (Root cause 2 - uses describeCompat with local server)
- examples/data-objects/webflow (Root cause 4 - TBD)
- examples/data-objects/inventory-app (Root cause 4 - TBD)

## Tasks

### Done
- [x] Understand scope of the problem
- [x] Identify primary root cause (DeliLambda.readClientIdleTimer for local server packages)
- [x] Confirm secondary root cause (SnapshotRefresher 24h timer via diagnostics)

### Done (continued)
- [x] Fix packages/loader/container-loader (serializedStateManager.spec.ts cleanup)
- Added `makeSsm()` factory helper + `instancesToDispose` tracking array at outer describe scope
- `afterEach` disposes all tracked instances, preventing 24h SnapshotRefresher timers from leaking
- Removed `config.exit = true` from `.mocharc.cjs`
- [x] Fix packages/runtime/container-runtime (3 spec files with ContainerRuntime not disposed)
- Root cause: GarbageCollector in ContainerRuntime creates MAX_INT32 timer on creation
- hardwareStats.spec.ts, runtimeLayerCompatValidation.spec.ts, containerRuntime.extensions.spec.ts
- Removed `config.exit = true` from `.mocharc.cjs`

- [x] Fix packages/dds/tree (TestTreeProvider disposal)
- Root cause: LoaderContainerTracker skips non-interactive (summarizer) containers, so the
summarizer container created by `createSummarizer()` was never disposed
- Fix 1: Change `state: "disabled"` to `state: "summaryOnRequest"` for `SummarizeType.onDemand`
to prevent SummaryManager from spawning auto-summarizer containers with GC timers
- Fix 2: Capture `summarizerContainer` from `createSummarizer()` and dispose it explicitly
in `TestTreeProvider.dispose()`, clearing its GC session expiry timer
- Fix 3: Add `afterEach` cleanup in sharedTree.spec.ts and testTreeProvider.spec.ts
- Fix 4: Add `provider.dispose()` in `getIIDCompressor()` in nodeIdentifier.spec.ts
- Removed `config.exit = true` from `.mocharc.cjs`

### Todo
- [ ] Run diagnostics on packages/runtime/container-runtime
- [ ] Run diagnostics on packages/framework/react
- [ ] Fix remaining packages after root causes confirmed
- [ ] Remove --exit flag from fixed packages and verify tests pass

## Key findings

1. `packages/loader/container-loader/src/test/serializedStateManager.spec.ts`:
- Creates SerializedStateManager instances (with SnapshotRefresher) without disposing them
- The SnapshotRefresher's internal Timer (24h default) keeps the process alive
- Fix: Add afterEach/after cleanup to dispose each SerializedStateManager

2. `packages/dds/tree/.mocharc.cjs` already has comment:
> "In this package, tests which use TestTreeProvider.create cause this issue"
- TestTreeProvider creates LocalServerTestDriver → LocalDeltaConnectionServer
- DeliLambda (inside the server) creates a 60s setInterval (readClientIdleTimer)
- Not cleared because driver.dispose() is never called after tests

3. describeCompat already does driver.dispose() (line 182 in describeCompat.ts),
but fire-and-forget (not awaited), so it's still async.

## Test diagnostic approach used
```bash
# Create a script to trace timer creation and report on SIGTERM:
cat > /tmp/mocha_trace.js << 'EOF'
# (intercepts setTimeout/setInterval and logs on SIGTERM)
EOF

# Run tests and send SIGTERM after completion:
cd /workspaces/FluidFramework/packages/loader/container-loader
mocha --no-exit --require /tmp/mocha_trace.js "lib/test/loader.spec.js" &
PID=$!
sleep 5
kill -TERM $PID
# Look at output for active timer stacks
```
3 changes: 0 additions & 3 deletions examples/data-objects/inventory-app/.mocharc.cjs
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,6 @@
const getFluidTestMochaConfig = require("@fluid-internal/mocha-test-setup/mocharc-common");

const config = getFluidTestMochaConfig(__dirname);
// TODO: figure out why this package needs the --exit flag, tests might not be cleaning up correctly after themselves.
// AB#7856
config.exit = true;

// Set up JSDOM before Quill is imported (Quill requires document at import time)
config["node-option"] ??= [];
Expand Down
18 changes: 18 additions & 0 deletions examples/data-objects/inventory-app/src/test/globalSetup.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
/*!
* Copyright (c) Microsoft Corporation and contributors. All rights reserved.
* Licensed under the MIT License.
*/

import globalJsdom from "global-jsdom";

// Set up JSDOM before any modules are loaded (Quill needs document at import time).
// @fluidframework/react imports Quill at ESM module-load time, so document must exist
// before the test file's static imports are resolved.
const cleanup = globalJsdom();

// Remove JSDOM after imports are done, but before we run any tests.
// Tests which require JSDOM (e.g. the "dom tests" describe block) call globalJsdom()
// themselves to set up their own clean DOM.
before(() => {
cleanup();
});
3 changes: 0 additions & 3 deletions examples/data-objects/table-document/.mocharc.cjs
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,4 @@
const getFluidTestMochaConfig = require("@fluid-internal/mocha-test-setup/mocharc-common");

const config = getFluidTestMochaConfig(__dirname);
// TODO: figure out why this package needs the --exit flag, tests might not be cleaning up correctly after themselves
// AB#7856
config.exit = true;
module.exports = config;
3 changes: 0 additions & 3 deletions examples/data-objects/webflow/.mocharc.cjs
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,4 @@ const packageDir = __dirname;
const getFluidTestMochaConfig = require("@fluid-private/test-version-utils/mocharc-common");
const config = getFluidTestMochaConfig(packageDir);
config["node-option"].push("experimental-loader=esm-loader-css");
// TODO: figure out why this package needs the --exit flag, tests might not be cleaning up correctly after themselves
// AB#7856
config.exit = true;
module.exports = config;
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@
"R": "spec",
"recursive": true,
"timeout": "180s",
"exit": true,
"full-trace": true,
"spec": ["test/setup.js", "test/**/*.spec.js"]
}
Original file line number Diff line number Diff line change
Expand Up @@ -8,5 +8,4 @@
const getFluidTestMochaConfig = require("@fluid-internal/mocha-test-setup/mocharc-common");

const config = getFluidTestMochaConfig(__dirname);
config.exit = true;
module.exports = config;
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@
"R": "spec",
"recursive": true,
"timeout": "180s",
"exit": true,
"full-trace": true,
"spec": ["test/setup.js", "test/**/*.spec.js"]
}
1 change: 0 additions & 1 deletion experimental/dds/tree/.mocharc.cjs
Original file line number Diff line number Diff line change
Expand Up @@ -8,5 +8,4 @@
const getFluidTestMochaConfig = require('@fluid-internal/mocha-test-setup/mocharc-common');

const config = getFluidTestMochaConfig(__dirname);
config.exit = true;
module.exports = config;
1 change: 0 additions & 1 deletion packages/dds/map/src/test/memory/.mocharc.cjs
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@
*/

module.exports = {
"exit": true,
"fgrep": ["@Benchmark", "@MemoryUsage"],
"node-option": ["expose-gc", "gc-global", "unhandled-rejections=strict"], // without leading "--"
"recursive": true,
Expand Down
1 change: 0 additions & 1 deletion packages/dds/matrix/src/test/memory/.mocharc.cjs
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@
*/

module.exports = {
"exit": true,
"fgrep": ["@Benchmark", "@MemoryUsage"],
"node-option": ["expose-gc", "gc-global", "unhandled-rejections=strict"], // without leading "--"
"recursive": true,
Expand Down
1 change: 0 additions & 1 deletion packages/dds/sequence/src/test/memory/.mocharc.cjs
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@
*/

module.exports = {
"exit": true,
"fgrep": ["@Benchmark", "@MemoryUsage"],
"node-option": ["expose-gc", "gc-global", "unhandled-rejections=strict"], // without leading "--"
"recursive": true,
Expand Down
4 changes: 0 additions & 4 deletions packages/dds/tree/.mocharc.cjs
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,4 @@ const config = getFluidTestMochaConfig(
[],
process.argv.includes("--emulateProduction") ? "PROD" : undefined,
);
// TODO: figure out why this package needs the --exit flag, tests might not be cleaning up correctly after themselves
// In this package, tests which use `TestTreeProvider.create` cause this issue, but there might be other cases as well.
// AB#7856
config.exit = true;
module.exports = config;
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,10 @@ async function getIIDCompressor(): Promise<IIdCompressor> {
runtime: IFluidDataStoreRuntime;
}
).runtime;
return runtime.idCompressor ?? fail("Expected IIdCompressor to be present in runtime");
const compressor =
runtime.idCompressor ?? fail("Expected IIdCompressor to be present in runtime");
provider.dispose();
return compressor;
Comment on lines +33 to +36
Copy link

Copilot AI Mar 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

getIIDCompressor() calls provider.dispose() without awaiting. If TestTreeProvider.dispose is updated to be async (to properly await driver.dispose / server shutdown), this helper should await disposal (ideally in a finally) to ensure local server resources are always released.

Copilot uses AI. Check for mistakes.
}

describe("Node Identifier", () => {
Expand Down
Loading
Loading