discuss: trie mask issue

[0xaatif]

Relates #583

I'm a bit confused by mpt_trie::trie_subsets::create_trie_subset.
AIUI any subset should NOT change the state root of a trie - it just hides subtries behind a hash indirection.

Note that #583 calls hiding subtries "masking", mpt_trie calls it "subsetting". The StateMpt::mask function is a thin (and I think low risk) wrapper around the mpt_trie functionality:

zk_evm/trace_decoder/src/typed_mpt.rs

Lines 341 to 351 in cfe6d42

	fn mask(&mut self, addresses: impl IntoIterator<Item = TrieKey>) -> anyhow::Result<()> {
	let inner = mpt_trie::trie_subsets::create_trie_subset(
	self.typed.as_hashed_partial_trie(),
	addresses.into_iter().map(TrieKey::into_nibbles),
	)?;
	self.typed = TypedMpt {
	inner,
	_ty: PhantomData,
	};
	Ok(())
	}

Here we branch on masking a particular subtrie:

zk_evm/trace_decoder/src/core.rs

Lines 466 to 473 in cfe6d42

	if !precompiled_addresses.contains(&addr.compat()) {
	// TODO(0xaatif): this looks like an optimization,
	// but if it's omitted, the tests fail...
	if std::env::var_os("SKIP_MASK_PRECOMPILED").is_some_and(|it| it == "true") {
	} else {
	state_mask.insert(TrieKey::from_address(addr));
	}
	}

I've added an SKIP_MASK_PRECOMPILED for the sake of this discussion.

If we DO mask those addresses:

$ git rev-parse --short HEAD
cfe6d42d
$ cargo test --quiet --package trace_decoder --test consistent-with-header -- --quiet
loading test vectors...done

running 14 tests
..............
test result: ok. 14 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 13.20s

Everything is ok.
If we DO NOT mask those addresses:

$ SKIP_MASK_PRECOMPILED=true cargo test --quiet --package trace_decoder --test consistent-with-header -- --quiet
loading test vectors...done
...
test result: FAILED. 4 passed; 10 failed; 0 ignored; 0 measured; 0 filtered out; finished in 12.46s

Eek! Let's see more detail about one of the failures

$ SKIP_MASK_PRECOMPILED=true cargo test --quiet --package trace_decoder --test consistent-with-header -- --exact b4_dev@1
loading test vectors...done
...
Assertion failed at trace_decoder/tests/consistent-with-header.rs:34:17:
  check!( pairs().position(|(before, after)| {
                        before.trie_roots_after.state_root != after.tries.state_trie.hash()
                    }) == None )
with expansion:
  Some(1) == None

Assertion failed at trace_decoder/tests/consistent-with-header.rs:50:17:
  check!( gen_inputs
                        .last()
                        .map(|it| it.trie_roots_after.state_root.compat()) == Some(header.state_root) )
with diff:
  Some(
<     0xb2415bb7cb3c0dea9520a546500dc54cbe444afadd85e68bd688e3a96bf69196,
>     0x038e7e3c67c9166523c4b2bae746f5d7cf72480af4294bb900f252faf05d04e9,
  )
...

I'll try and work on a smaller test case :)

[Nashtare]

I'm not sure I understand the replication to begin with..

zk_evm/trace_decoder/src/core.rs

Lines 466 to 473 in cfe6d42

	if !precompiled_addresses.contains(&addr.compat()) {
	// TODO(0xaatif): this looks like an optimization,
	// but if it's omitted, the tests fail...
	if std::env::var_os("SKIP_MASK_PRECOMPILED").is_some_and(|it| it == "true") {
	} else {
	state_mask.insert(TrieKey::from_address(addr));
	}
	}

We enter the if condition if we do NOT have a precompile address at hand, so your SKIP_MASK_PRECOMPILED flag is irrelevant here. What #488 aimed at doing was, in the case we do have a precompile, to check if it effectively is present in the initial trie witness (i.e. not hashed out), and mark it (or mask with your terminology? I'm not sure) as touched if there is a path down the trie for it.

[BGluth]

Yeah creating a trie subset should always produce a trie with an identical hash, so that's a bug.

Maybe subset isn't the best name here. It's true that the trie has only subset of the original trie's node and it's a different trie, but it's a bit confusing as well since the trie is the same except for hash nodes.

I would just run this through trie diff and see where they diverge.