Skip to content

ci(conformance): integrate MCP server conformance test pipeline#3106

Open
anubhav756 wants to merge 6 commits intomainfrom
anubhav-conformance
Open

ci(conformance): integrate MCP server conformance test pipeline#3106
anubhav756 wants to merge 6 commits intomainfrom
anubhav-conformance

Conversation

@anubhav756
Copy link
Copy Markdown
Contributor

@anubhav756 anubhav756 commented Apr 20, 2026

Description

This PR integrates the official MCP conformance testing framework to automatically validate the strict protocol compliance of MCP Toolbox server.

Changes

  • Adds a GitHub Action that builds and spins up the server binary, then targets it with the MCP conformance runner.
    • It uses a conformance-baseline.yml file to track known gaps without blocking contributors.
  • Adds a 6:00 AM cron job that runs the tier-check audit tool.
    • It automatically creates and updates a self-maintaining dashboard issue (labeled periodic-failure) so maintainers can easily track operational SLA and label requirements needed for Tier 1 status.

Note

We have opened an upstream PR to fix the Repository Health table, in the generated GitHub issue, in the official conformance tool: modelcontextprotocol/conformance#239
image

@anubhav756 anubhav756 self-assigned this Apr 20, 2026
@anubhav756 anubhav756 force-pushed the anubhav-conformance branch from e500d8e to 3ca25fe Compare April 20, 2026 22:09
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a conformance-baseline.yml file to the root directory. The reviewer suggests moving this file to a more appropriate location, such as the tests/ or ci/ directory, to maintain a clean project structure and align with existing organizational patterns.

Comment thread conformance-baseline.yml
@@ -0,0 +1,15 @@
# Copyright 2026 Google LLC
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Consider moving the conformance-baseline.yml file to a more appropriate directory, such as tests/ or a dedicated ci/ folder, to maintain a clean root directory. This aligns with the project's structure where integration tests are grouped in the tests/ directory.

@anubhav756 anubhav756 force-pushed the anubhav-conformance branch from 3ca25fe to 82dbf77 Compare April 20, 2026 22:13
@anubhav756 anubhav756 force-pushed the anubhav-conformance branch from bc51272 to abf9cd1 Compare April 20, 2026 22:32
@anubhav756 anubhav756 force-pushed the anubhav-conformance branch from abf9cd1 to ce3aad2 Compare April 20, 2026 22:45
@anubhav756 anubhav756 force-pushed the anubhav-conformance branch from ce3aad2 to 83ed31f Compare April 20, 2026 22:50
@anubhav756 anubhav756 force-pushed the anubhav-conformance branch 2 times, most recently from 76ed811 to af4e745 Compare April 20, 2026 23:00
@anubhav756 anubhav756 force-pushed the anubhav-conformance branch 2 times, most recently from ce64dad to b24e7b5 Compare April 20, 2026 23:11
@anubhav756 anubhav756 marked this pull request as ready for review April 20, 2026 23:14
@anubhav756 anubhav756 requested a review from a team as a code owner April 20, 2026 23:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant