Follow-up: a6 test / skills / docs review findings

[guoqqqi]

Background

Before putting a6 into real use, we completed a full review and local rerun of the current test system, skills, and documentation.

Detailed Feishu report:

• https://www.feishu.cn/docx/AR0Qd3d1xo4I0Bx2PGQccwCBnXe

Current Conclusions

• The baseline CI test suite passes.
• With the current CI E2E workflow deployment model, the a6 E2E suite can also be reproduced successfully in a local environment.
• skills metadata validation passes.
• After running the skills E2E suite locally against a real environment, all cases pass except skywalking, which is skipped because the required conditional plugin is missing.
• The current overall code coverage is approximately 52.4%.

This indicates that a6 already has a relatively complete automation baseline, but there are still gaps before it can be considered ready for “100% coverage / full pre-release validation”.

Follow-up Items

P0

• Clarify the conditional capability boundary for the skywalking skill
  • TestSkillPluginSkywalking is currently skipped with the current image
  • We need to make it explicit that this is a conditional skip caused by a missing plugin, not a fully covered and passing case

P1

• Add compatibility tests for debug logs container naming

  • The current setup passes with the CI container name apisix
  • However, the test currently assumes a hard-coded container name, which can easily lead to mismatches in local reproduction

• Add capability-boundary coverage for Global Rule labels

  • The related case is currently skipped because APISIX does not support it

• Add tests for shared modules and low-coverage modules

  • Such as selector, tableprinter, root, configutil, and similar modules

• Add more combined scenario coverage

  • service + route + plugin-config
  • Real traffic verification for multi-node upstreams
  • More non-happy-path and invalid-parameter branches

P2

• Continue raising low-coverage packages one by one based on the coverage report by adding more unit tests

Suggested Outputs

• Track conditional-capability cases separately from real passing cases
• Add another round of test supplementation using semantic scenarios as the unit of coverage
• Keep tests and documentation aligned in follow-up PRs

[guoqqqi]

The notes below break down the current a6 status so the issue can continue to be analyzed and advanced directly.

Current Status

• Baseline CI tests: successfully rerun locally
• CI E2E: successfully rerun locally using the current workflow assumptions, including host-network mode and container names apisix / httpbin
• Skills validation: passed
• Skills E2E: executed against a real environment; all cases passed except skywalking, which was skipped because the required plugin condition was not present
• Overall coverage: 52.4%

Semantic Cases That Have Already Passed in a Real Environment

• Basic CLI execution, version, and completion
• Context create, switch, delete, and error handling
• Config dump / validate / diff / sync
• Route CRUD, list, filtering, YAML/JSON output, and real traffic forwarding
• Service CRUD, Upstream binding, and Route binding via Service ID
• Upstream CRUD, multi-node behavior, and health checks
• Consumer / Credential / Consumer Group CRUD
• Real key-auth traffic flow for Consumers
• Plugin / Plugin Metadata / Plugin Config CRUD
• Global Rule CRUD
• Secret CRUD
• SSL CRUD / create-from-file / status querying
• Stream Route CRUD
• Bulk delete / export / label operations
• Basic debug trace capability

Current Skills Status

• ./scripts/validate-skills.sh: passed
• test/e2e/skills: executed against a real environment
• Skills currently verified in a real environment include:
  • auth: key-auth, basic-auth, jwt-auth, hmac-auth
  • traffic: proxy-rewrite, traffic-split, limit-count, limit-req, redirect, response-rewrite
  • observability: http-logger, datadog, zipkin, prometheus
  • recipes: canary, blue-green, api-versioning, health-check, circuit-breaker, graphql-proxy, mtls, multi-tenant
  • persona/shared: developer, operator, shared
• The only skill case currently not passing is:
  • TestSkillPluginSkywalking
  • Reason: the current APISIX image does not include the skywalking plugin, so this is a conditional skip rather than a functional failure

Current Skips / Conditional Capabilities

• TestCoverageFlags_DebugLogsTypeFlag
• TestCoverageFlags_DebugLogsOutputJSONFlag
• TestExport_GlobalRule
• TestSkillPluginSkywalking

These should not be counted directly as “fully covered”.

Follow-up TODOs

P0

• Clarify the conditional capability boundary for skywalking
  • There should be a passing validation path when the plugin exists
  • There should be a stable skip semantic when the plugin does not exist

P1

• Add compatibility tests for debug logs container naming
  • The current setup passes with the CI container name
  • But the test still assumes a hard-coded container name

- Add capability-boundary coverage for Global Rule labels

• Add more combined scenario coverage

  • service + route + plugin-config
  • Real traffic verification for multi-node upstreams
  • More non-happy-path and invalid-parameter branches

• Add tests for low-coverage shared modules

  • selector
  • tableprinter
  • root
  • configutil

P2

• Continue adding unit tests package by package based on the coverage report

Documentation Follow-ups

• Add clarification for conditional plugins such as skywalking
• Add more direct entry points in the user guide for context, config, update, and version
• The documentation should clearly distinguish between “verified in a real environment” and “conditional capability / skip”

[guoqqqi]

The next-stage testing architecture work has now been split into dedicated tracking issues so that the follow-up can be executed in phases:

• test: refactor automation strategy around real APISIX E2E and Ginkgo #19 umbrella: refactor automation strategy around real APISIX E2E and Ginkgo
• docs(test): redefine unit-vs-e2e policy for a6 #20 policy/docs update
• test(e2e): bootstrap Ginkgo/Gomega framework for real APISIX scenarios #23 Ginkgo/Gomega E2E bootstrap
• test: migrate core command coverage from mocked Admin API tests to real E2E #22 first migration batch for core resources
• test: migrate remaining resource commands and remove legacy mock-command coverage #21 remaining migration and legacy cleanup

This keeps #14 as the original findings issue while the actual strategy migration and execution are tracked separately. The current execution order is: #20 -> #23 -> #22 -> #21.