SEC-10094 feat(workflow): PR test analysis

[saturninoabril]

Summary

Per rollout plan - https://hub.mattermost.com/private-core/pl/gt5hkzyrfiduze6rq3bta91www

• Add PR Test Analysis workflow that uses AI to evaluate test coverage adequacy on pull requests targeting main and release-* branches
• Add PR Test Analysis Override workflow that allows authorized users to override a failing test analysis status via /test-analysis-override <reason> comment
• Both workflows delegate to reusable workflows in mattermost-test-automation-toolkit, following the same pattern as mattermost/mattermost

Ticket Link

https://mattermost.atlassian.net/browse/SEC-10094

Release Note

NONE

[coderabbitai]

📝 Walkthrough

Walkthrough

Two new GitHub Actions workflows are added to enable PR test analysis and override functionality. The PR Test Analysis workflow triggers on PR lifecycle events and manual dispatch, delegating analysis logic to an external reusable workflow with configurable Claude model selection. The PR Test Analysis Override workflow allows manual override of test analysis via issue comments.

Changes

Cohort / File(s)

Summary

PR Test Analysis Workflows .github/workflows/pr-test-analysis.yml, .github/workflows/pr-test-analysis-override.yml

Two new GitHub Actions workflows added for PR test analysis and override functionality. PR Test Analysis triggers on pull request events and manual dispatch, passing PR number, target repository, and Claude model configuration to a reusable workflow. PR Test Analysis Override triggers on issue comments with /test-analysis-override command and forwards comment metadata to a reusable workflow. Both workflows delegate execution to external reusable workflows in mattermost/mattermost-test-automation-toolkit and require appropriate GitHub tokens and API secrets.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The pull request title accurately describes the main change: adding PR test analysis workflows. It is specific, concise, and clearly conveys the primary purpose of the changeset.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description check	✅ Passed	The pull request description clearly relates to the changeset, outlining the addition of two new GitHub Actions workflows with specific details about their functionality and integration with reusable workflows.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch SEC-10094_pr-test-analysis

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In @.github/workflows/pr-test-analysis-override.yml:
- Around line 22-23: The current guard uses
startsWith(github.event.comment.body, '/test-analysis-override') which allows
empty or whitespace-only reasons; update the condition to require a non-empty
reason by replacing the startsWith check with a regex match that enforces a
space and at least one non-whitespace character (e.g.
matches(github.event.comment.body, '^/test-analysis-override\\s+\\S+')) while
keeping the contains(...) author_association check unchanged so only
OWNER/MEMBER/COLLABORATOR comments are allowed.
- Around line 9-10: Change the concurrency group to be workflow-specific so runs
from pr-test-analysis.yml and pr-test-analysis-override.yml don’t share a
namespace: update the existing "group: test-analyzer-${{
github.event.issue.number }}" to include a workflow identifier (for example
append or prefix with the workflow name or use ${{ github.workflow }}) so the
key becomes unique to this override workflow, and keep "cancel-in-progress:
false" as-is; modify the "group" entry referenced in this file to reflect that
change.

In @.github/workflows/pr-test-analysis.yml:
- Line 40: Replace the mutable `@main` reference with a pinned commit SHA for the
reusable workflow; locate the uses line currently written as "uses:
mattermost/mattermost-test-automation-toolkit/.github/workflows/pr-test-analysis.yml@main"
(and the analogous entry in pr-test-analysis-override.yml) and change the `@main`
suffix to the specific commit SHA of the external repo you want to lock to,
update both occurrences to the same verified SHA, and commit the change so the
workflow uses an immutable reference.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 2690c160-ba77-4a5a-86b8-8ba84d913af7

📥 Commits

Reviewing files that changed from the base of the PR and between c4e6ebc and 657285c.

📒 Files selected for processing (2)

• .github/workflows/pr-test-analysis-override.yml
• .github/workflows/pr-test-analysis.yml

[github-actions]

Coverage Comparison Report

Generated on April 15, 2026 at 05:02:31 UTC

+-----------------+------------+------------+-----------+
| Metric          | Main       | This PR    | Diff      |
+-----------------+------------+------------+-----------+
| Lines           |     85.17% |     85.17% |     0.00% |
| Statements      |     85.03% |     85.03% |     0.00% |
| Branches        |     72.22% |     72.22% |     0.00% |
| Functions       |     84.03% |     84.03% |     0.00% |
+-----------------+------------+------------+-----------+
| Total           |     81.61% |     81.61% |     0.00% |
+-----------------+------------+------------+-----------+

[saturninoabril]

Merging this now