Skip to content

project tests: capture output for SKIP and UNEXRUN cases into meson-test-run.xml#15590

Open
jon-turney wants to merge 2 commits intomesonbuild:masterfrom
jon-turney:testlog-xml-improvements
Open

project tests: capture output for SKIP and UNEXRUN cases into meson-test-run.xml#15590
jon-turney wants to merge 2 commits intomesonbuild:masterfrom
jon-turney:testlog-xml-improvements

Conversation

@jon-turney
Copy link
Member

@jon-turney jon-turney commented Feb 28, 2026

As observed in #15534 (comment) investigating CI result changes involving skipped tests is harder than it should be, because we don't record any information about those tests into the testlog.

Future work?

  • The whole 'result is None' (sometimes) thing makes this code more complex than it needs to be. Maybe create an appropriate TestResult object instead in those cases?
  • The XML log contains text which has GitHub ::group:: folding directives inside. This kind of indicates a confusion somewhere?
  • Maybe the plaintext log is actually pretty useless now we're more intelligent about what we write to stdout? Or should just be a capture of that?
  • I think there's now a formal schema for JUnit XML, so perhaps we should verify our output against that? Certainly, this would help check I've added properties in the right way.

@jon-turney jon-turney force-pushed the testlog-xml-improvements branch 2 times, most recently from c2ebd83 to 0d344de Compare March 11, 2026 20:50
@dcbaker
Copy link
Member

dcbaker commented Mar 11, 2026

The JUnit was (when I originally wrote it) validated against the JUnit 4 schema. I think there's a newer one now, and there have been changes since the code was written.

@jon-turney jon-turney changed the title Improvements to XML testlog Improvements to project tests meson-test-run.xml Mar 12, 2026
@jon-turney
Copy link
Member Author

The JUnit was (when I originally wrote it) validated against the JUnit 4 schema. I think there's a newer one now, and there have been changes since the code was written.

Uh, this is my fault for giving this PR a bad title: this is about the XML log written by run_project_tests, not the the XML log written by 'meson test'.

There is indeed a schema conformance test for the JUnit XML output from 'meson test'.

@jon-turney jon-turney force-pushed the testlog-xml-improvements branch from 0d344de to 09d7475 Compare March 16, 2026 21:36
@jon-turney
Copy link
Member Author

  • The whole 'result is None' (sometimes) thing makes this code more complex than it needs to be. Maybe create an appropriate TestResult object instead in those cases?

This seems more difficult than I guessed, so maybe it's not worth it...

  • I think there's now a formal schema for JUnit XML, so perhaps we should verify our output against that? Certainly, this would help check I've added properties in the right way.

Unfortunately, it seems that xml.etree doesn't have schema validation. lxml can do that, so maybe changing to use that would be ok (since I don't think the "only python stdlib" applies to the testsuite)?

It also seems that I'd misunderstood the JUnit schema, and while that has arbitrary properties you can attach to a testsuite, there's nothing similar for individual testcases. So drop the bit that uses that to attach the meson.log file.

@jon-turney jon-turney changed the title Improvements to project tests meson-test-run.xml project tests: capture output for SKIP and UNEXRUN cases into meson-test-run.xml Mar 16, 2026
@jon-turney jon-turney marked this pull request as ready for review March 16, 2026 21:48
@jon-turney jon-turney requested a review from jpakkane as a code owner March 16, 2026 21:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants