Enabled running both Vitest and Karma tests for packages.

[martnpaneq]

🚀 Summary

• Extended runAutomatedTests to support both Karma and Vitest test frameworks simultaneously during the Karma-to-Vitest migration
• The runner automatically detects which framework each package uses by checking scripts.test in its package.json — if it contains vitest, the package is routed to Vitest; otherwise Karma
• Vitest packages are run in a single pnpm vitest process using the --project flag, leveraging the existing workspace vitest.config.mjs
• Vitest packages are run, each in a separate process, to keep backwards compatibility for -f argument
• Karma packages continue to work exactly as before
• Mixed runs (some packages on Karma, some on Vitest) work transparently — the runner executes Karma first, then Vitest, and aggregates errors from both
• Coverage reports from both runners are merged into a unified coverage/lcov.info with consistent relative paths
• Watch mode is blocked for mixed Karma + Vitest selections (must be run separately)

How it works

Two packages: utils and upload are "mock" migrated to Vitest in ckeditor5 on branch i/4307-test-wrapper-v2.

Given pnpm test -f upload -f utils -f emoji -b ChromeHeadless --coverage:

1. All test files are collected via glob patterns (unchanged)
2. partitionByRunner() reads each package's package.json and splits files:
   - emoji → Karma (no vitest in test script)
   - upload, utils → Vitest ("test": "vitest run")
3. Karma runs first with only its files in the entry bundle and coverage scope
4. A single Vitest process runs: pnpm vitest --run --project upload --project utils
5. Coverage from both runners is merged with consistent relative SF: paths

---

📌 Related issues

• Closes ckeditor/ckeditor5-internal#4307

---

Test plan

• Checkout i/4307-test-wrapper-v2 branch in ckeditor5 repository.
• pnpm test -f upload -f utils -f emoji -b ChromeHeadless --coverage from the repository root
• Vitest-only run: pnpm test -f utils -f upload -b ChromeHeadless (no Karma started)
• Karma-only run: pnpm test -f emoji -b ChromeHeadless (no Vitest started)
• Coverage reports: Karma coverage/ contains only Karma packages, Vitest coverage-vitest/ contains only Vitest packages

Warning

The integration with coverage report sent to lcov was not tested, because it is run only on master branch. Here is the reasoning behind why it should work correctly, without running it.

See screenshot

---

Note

Medium Risk
Moderate risk because it changes the core automated test runner flow (file selection, coverage scoping, process spawning) and could alter CI/local test behavior across packages.

Overview
runAutomatedTests() can now execute Karma and Vitest tests in one invocation. It collects all matching test files, detects each package’s runner via its package.json scripts.test, partitions the selection, then runs Karma first and Vitest second.

Adds a Vitest execution path that spawns pnpm vitest per detected project (passing --run/--watch, optional coverage to coverage-vitest, and explicit file paths), restricts watch mode for mixed-runner selections, and aggregates errors when both runners fail. Karma entry generation is adjusted to include only Karma-selected files to avoid instrumenting Vitest packages.

Updates/extends runautomatedtests unit tests to cover Vitest-only, mixed-runner, multi-project Vitest runs, watch-mode rejection, and failure aggregation, and adds a changelog entry documenting the new behavior.

^{Written by Cursor Bugbot for commit e7e72c1. This will update automatically on new commits. Configure here.}

[martnpaneq]

I prepared a corresponding branch in ckeditor5 for testing purposes i/4307-test-wrapper-v2. Two packages: utils and upload are "mock" migrated to Vitest. You can see that the way of displaying tests is different for them than for other packages.

https://app.circleci.com/pipelines/github/ckeditor/ckeditor5/16639/workflows/ff76ca2a-c325-42a5-b752-e1fe4557924c/jobs/108501

[Copilot]

Pull request overview

This PR updates the ckeditor5-dev-tests automated test runner to support running Karma and Vitest in the same invocation, which is needed to enable a gradual Karma→Vitest migration across packages.

Changes:

• Refactors runAutomatedTests() to collect matched test files, detect each package’s runner via package.json scripts.test, and partition execution between Karma and Vitest.
• Adds a Vitest runner that spawns a single pnpm vitest process with --project flags and blocks --watch for mixed Karma+Vitest selections.
• Adds coverage merging to produce a unified coverage/lcov.info, plus extensive new unit tests for Vitest-only, mixed runs, and coverage merging.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File	Description
packages/ckeditor5-dev-tests/lib/tasks/runautomatedtests.js	Implements mixed Karma+Vitest orchestration, Vitest spawning, runner partitioning, and coverage merging.
packages/ckeditor5-dev-tests/tests/tasks/runautomatedtests.js	Extends unit tests to cover Vitest-only runs, mixed routing, watch restrictions, error aggregation, and coverage merging.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Coverage merging unreachable when any test runner fails
  • Moved coverage merging to run before aggregated runner errors are thrown so unified coverage is produced even when tests fail.

Or push these changes by commenting:

@cursor push 02ce42a643

Preview (02ce42a643)

diff --git a/packages/ckeditor5-dev-tests/lib/tasks/runautomatedtests.js b/packages/ckeditor5-dev-tests/lib/tasks/runautomatedtests.js
--- a/packages/ckeditor5-dev-tests/lib/tasks/runautomatedtests.js
+++ b/packages/ckeditor5-dev-tests/lib/tasks/runautomatedtests.js
@@ -69,13 +69,13 @@
 		}
 	}
 
+	if ( options.coverage ) {
+		mergeCoverageReports( karmaFiles.length > 0, vitestProjects.length > 0 );
+	}
+
 	if ( errors.length ) {
 		throw aggregateErrors( errors );
 	}
-
-	if ( options.coverage ) {
-		mergeCoverageReports( karmaFiles.length > 0, vitestProjects.length > 0 );
-	}
 }
 
 // -- Glob resolution & file collection -----------------------------------------------------------

_{This Bugbot Autofix run was free. To enable autofix for future PRs, go to the Cursor dashboard.}

[martnpaneq]

After merging this PR, some changes in ckedtor5 and ckeditor5-commercial have to be made:

1. Enabled running mixed Vitest and Karma tests with merged coverage report ckeditor5#19968
2. https://github.com/ckeditor/ckeditor5-commercial/pull/9528

In order to rewrite a single package to Vitest:

1. Introduce "test": "vitest run" in package.json scripts.
2. Add a new vitest.config.mjs file in the root of the package:

import { defineProject } from 'vitest/config';

export default defineProject( {
	test: {
		name: 'upload', // TODO: Replace with the actual package name.
		include: [ 'tests/*' ],
		exclude: [ 'tests/manual/*', 'tests/_utils/*' ]
	}
} );

[filipsobol]

Add a changelog entry, please.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}