Backward compatibility testing infrastructure#2031
Backward compatibility testing infrastructure#2031darioAnongba wants to merge 6 commits intomainfrom
Conversation
Add two methods to IntegratedNetworkHarness for backward compatibility testing. NewNodeWithBinary starts a node with a specific binary path (e.g. an older release), while UpgradeNode stops a node and restarts it with the harness's default (current) binary, preserving data directories.
Add a script that builds the tapd-integrated binary from a specific git tag using a temporary worktree. Built binaries are cached locally so subsequent invocations are instant. This supports backward compatibility testing without requiring release artifact publishing.
Add wire format fixtures for v0.8 RFQ messages (request, accept, reject, HTLC) and a backward compatibility test that decodes each fixture and verifies round-trip encoding. Fixtures are generated deterministically via the gen_test_vectors build tag and tested on every PR.
Add v0.8 wire format fixtures for channel messages (funding blob, commitment blob, HTLC blob) organized under testdata/compat/v0.8/. These are captured from real protocol interactions. A backward compatibility test verifies that each fixture can be decoded by the current code on every PR.
Add TestCustomChannelsCompat which runs a subset of custom channel tests (core, force close, v1 upgrade) with one node running an older binary for each configured historical version. The test builds old binaries from source via build-compat-binary.sh and caches them locally. The compatVersions list is initially empty with a TODO to populate once v0.8.0 is released.
Add Makefile targets: - itest-cc-compat: runs backward compat integration tests - build-compat-binary: builds an old tapd-integrated from a git tag - gen-compat-fixtures: regenerates wire format fixture files Add a GitHub Actions workflow that triggers on release/* pushes and manual dispatch to run the compat test suite with cached old binaries.
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request introduces a comprehensive backward compatibility testing framework to prevent regressions in wire format encoding/decoding and protocol interactions. It establishes two layers of testing: per-PR fixture decode tests for immediate feedback on wire format changes, and mixed-version integration tests for more thorough runtime and interaction protocol validation. This new infrastructure allows for testing against historical binary versions, simulating real-world upgrade scenarios, and ensuring the long-term stability and interoperability of the system. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Changelog
Ignored Files
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a comprehensive backward compatibility testing framework, which is a great addition for ensuring protocol stability across versions. The implementation includes two layers of testing: wire format fixture tests and mixed-version integration tests. The new Makefile targets, test harness extensions for running nodes with different binaries, and the build script for old binaries are well-designed. I have one suggestion to improve the maintainability of the test code by avoiding the use of environment variables to pass state within tests.
| // Store the old binary path in the harness so test cases can | ||
| // retrieve it. We use an environment variable as a simple | ||
| // side channel. | ||
| t.Setenv("COMPAT_OLD_BINARY", oldBinary) |
There was a problem hiding this comment.
Using environment variables to pass data between test setup and test cases can be brittle. A more robust approach would be to extend the ccHarnessTest struct to include the oldBinary path. This would make the data flow explicit and avoid potential issues with parallel test execution if not handled carefully, even though t.Setenv provides some protection.
For example:
// in this file
ht := &ccHarnessTest{
t: t1,
testCase: tc,
lndHarness: net,
oldBinary: oldBinary, // new field
}
// in test files that need it
oldBinaryPath := ht.oldBinary
if oldBinaryPath != "" {
// use NewNodeWithBinary
} else {
// use NewNode
}This would require modifying the ccHarnessTest struct and how it's used in the test cases, but it leads to cleaner and more maintainable test code.
Pull Request Test Coverage Report for Build 23209855627Details
💛 - Coveralls |
darioAnongba
changed the title
2 participants
rfqmsg(request, accept, reject, HTLC) andtapchannelmsg(funding blob, commitment blob, HTLC blob). Backward compat tests decode each fixture with current code and verify round-trip encoding on every PR.NewNodeWithBinaryandUpgradeNodetoIntegratedNetworkHarnessso tests can start nodes on older binaries and simulate upgrades mid-operation.scripts/build-compat-binary.shbuildstapd-integratedfrom any git tag using a temporary worktree and caches the result in~/.tapd-compat-bins/.TestCustomChannelsCompatruns core, force close, and v1 upgrade tests against each configured historical version. ThecompatVersionslist is empty until v0.8.0 is tagged.make itest-cc-compat,make build-compat-binary,make gen-compat-fixtures. GitHub Actions workflow triggers onrelease/*pushes andworkflow_dispatch.Design
Two layers catch different classes of regressions:
Bitcoin Core takes a similar approach (
feature_backwards_compatibility.pyon demand, not every PR) but their protocol evolves much more slowly.Closes #2027