fix(kona): unbreak kona-host-client-offline-cannon (target spec + fresh witness)#20167
Merged
fix(kona): unbreak kona-host-client-offline-cannon (target spec + fresh witness)#20167
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #20167 +/- ##
==========================================
- Coverage 11.4% 11.4% -0.1%
==========================================
Files 669 669
Lines 72658 72658
==========================================
- Hits 8332 8315 -17
- Misses 64182 64199 +17
Partials 144 144
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
ajsutton
force-pushed
the
aj/fix/kona-offline-cannon-target-spec
branch
from
a72e856 to
b4db834
Compare
2 tasks
wwared
approved these changes
Apr 21, 2026
| AGREED_L2_OUTPUT_ROOT=$(cast rpc --rpc-url $OP_NODE_ADDRESS "optimism_outputAtBlock" $(cast 2h $((CLAIMED_L2_BLOCK_NUMBER - 1))) | jq -r .outputRoot) | ||
| AGREED_L2_HEAD_HASH=$(cast block --rpc-url $L2_NODE_ADDRESS $((CLAIMED_L2_BLOCK_NUMBER - 1)) --json | jq -r .hash) | ||
| L1_ORIGIN_NUM=$(cast rpc --rpc-url $OP_NODE_ADDRESS "optimism_outputAtBlock" $(cast 2h $((CLAIMED_L2_BLOCK_NUMBER - 1))) | jq -r .blockRef.l1origin.number) | ||
| L1_HEAD=$(cast block --rpc-url $L1_NODE_ADDRESS $((L1_ORIGIN_NUM + 30)) --json | jq -r .hash) |
Contributor
There was a problem hiding this comment.
Question: Is there a particular reason why the "+ 30" offset is used here for L1_HEAD? I know the other kona run-client justfile jobs use the same value
Contributor
Author
There was a problem hiding this comment.
honestly, no idea. It does seem very arbitrary and probably not entirely the right approach, but is just following what was previously done (as far as I can tell anyway). It probably should be finding the required L1 head from the safe head db but kona-node didn't support it (maybe still doesn't - not sure if it landed). We may need to circle back around and improve this later but it seems low priority since it has managed to find a working setup.
Adds a just recipe that runs kona-host in native mode against real RPCs to collect preimages and packages them into a zstd-compressed tar in the layout expected by the kona-host-client-offline CI job. Arguments mirror run-client-native for consistency. Prints the computed boundary hashes so operators can sync .circleci/continue/rust-ci.yml when regenerating for a new block. Requires RPC endpoints with archive state available for the target block; pruned EL nodes will reject optimism_outputAtBlock for old blocks.
The committed witness tar at holocene-op-sepolia-26215604-witness.tar.zst is ~1 year old and tracks a post-Holocene test block chosen when the offline host+client test was introduced. kona-host-client-offline-cannon has been failing because the rebuilt kona-client's preimage access pattern no longer matches what was recorded back then. Regenerates against a recent op-sepolia finalized block so the test exercises current hardfork code paths (through Jovian). Uses the new generate-witness-tar just recipe. Drops the hardfork prefix from the tar filename since the block is past multiple hardforks. Boundaries (see .circleci/continue/rust-ci.yml): BLOCK_NUMBER=42427365 L2_CLAIM=0xaaf7cb1725cb9766899cbd78866270d8561e79b91c00be5c53adb7699e0e7715 L2_OUTPUT_ROOT=0xac90a2e8cfee33ecb47b73f3e83f9aa18ff7ecf28bc8d34b2de857919e50b6c6 L2_HEAD=0xc5fa07d52e83f7a2e68de356e7a7bfc9306a9b28666ec6c9cc6e4f8e1b268278 L1_HEAD=0xab33322985684f2c6fbcf375bdcb69cc0f10547e58cb5a7b889172ee9ea77aef L2_CHAIN_ID=11155420
… path The cannon-builder:v1.0.0 Docker image ships with a MIPS64 target spec that has target-c-int-width=64. The correct n64 ABI value is 32, and the wrong value caused LLVM to miscompile str equality comparisons — the branch on the memcmp return was eliminated, making str == str always return false. Fixed upstream in commit 04ab310 for the reproducible prestate path, but kona-host-client-offline-cannon goes through rust/kona/justfile's build-cannon-client and lint-cannon recipes which invoke the baked image directly and never picked up the corrected spec. Mount the already-corrected spec from rust/kona/docker/cannon/ over /mips64-unknown-none.json inside the container so cargo resolves it instead of the broken baked-in file. Same mechanism cannon-repro.dockerfile uses at its COPY step, adapted for docker run. Reproducer: https://github.com/ajsutton/rust-mips64-str-miscompile
The offline-cannon witness is ~7MB and changes every time the test block number is refreshed, which bloats git history with binary churn. Mirror op-program's compat-data approach: host the tar as a GitHub release asset on ethereum-optimism/chain-test-data and download it on demand with etag caching. - Add scripts/fetch-witness-tar.sh — curl with --etag-save/--etag-compare so repeated local runs skip re-download when the release asset is unchanged (same pattern as op-program/scripts/run-compat.sh). - Tag convention is kona-YYYY-MM-DD, namespaced to avoid colliding with the op-program compat data tags already hosted in the same repo. - CI gets a Fetch witness data step and a WITNESS_TAR_URL env var, inserted before the existing Decompress witness data step. - generate-witness-tar now prints the exact gh release create/upload commands so regenerators follow the same ritual. - Drop the committed tar and gitignore *.tar.zst in testdata/ so regenerated artefacts cannot sneak back into the tree.
…ffline The CI Fetch + Decompress steps and WITNESS_TAR_NAME/URL env vars were really an implementation detail of running the offline cannon test. Moving them into a `prepare-witness-data` recipe wired as a dependency of `run-client-cannon-offline` means: - CI shrinks to just the boundary env vars + one just invocation. - Local runs of `just run-client-cannon-offline ...` automatically fetch and decompress (no separate command to remember). - The tar name and release URL live next to the block/claim/etc. boundary metadata in a single recipe, so refreshing for a new block only touches the justfile (plus a new chain-test-data release). Also drops the fetch script's hardcoded URL default — the just recipe is now the single source of truth for the canonical release URL.
wwared
force-pushed
the
aj/fix/kona-offline-cannon-target-spec
branch
from
c225233 to
8641beb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The
kona-host-client-offline-cannonCI job has been silently broken: thecannon-builder:v1.0.0Docker image ships with a MIPS64 target spec that hastarget-c-int-width: 64, but the n64 ABI specifies 32. The wrong value causes LLVM to miscompilestrequality comparisons —str == stralways returns false — which breaks kona-client under cannon.Upstream fix #19730 (commit 04ab310) corrected the JSON in the source tree and plumbed it through
cannon-repro.dockerfile'sCOPYstep, but did not reachrust/kona/justfile'sbuild-cannon-client/lint-cannonrecipes — those still invoke the baked image directly. Those recipes feed the offline-cannon CI job.This PR:
build-cannon-client/lint-cannon: bind-mount the correctedrust/kona/docker/cannon/mips64-unknown-none.jsonover/mips64-unknown-none.jsoninside the container. Same mechanismcannon-repro.dockerfileuses, adapted fordocker run.holocene-op-sepolia-26215604-witness.tar.zstwas recorded ~1 year ago against a pre-fix kona-client. Regenerated against op-sepolia finalized block #42427365 using a correctly-built kona-host. Updates.circleci/continue/rust-ci.yml(BLOCK_NUMBER,L2_CLAIM,L2_OUTPUT_ROOT,L2_HEAD,L1_HEAD,WITNESS_TAR_NAME). Drops the hardfork prefix from the filename.generate-witness-tarjust recipe: encapsulates the regeneration procedure (native kona-host run + tar zst package) with arguments mirroringrun-client-native. Prints the boundary hashes so operators can sync the CI config when regenerating for a new block.Reproducer for the str miscompile: https://github.com/ajsutton/rust-mips64-str-miscompile
Test plan
kona-host-client-offline-cannonpasses on this PR (was hanging/timing out on develop)kona-lint-cannonstill passes (same mount path)required-rust-cigate passes🤖 Generated with Claude Code