You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This is very similar to #20779. When a generated code has split classes and they try to access protected fields in BufferedRowIterator, an IllegalAccessError will happen.
I am not making partitionIndex public too because I cannot find a real example that can trigger an IllegalAccessError on it too. In the code base, partitionIndex is mostly accessed via addPartitionInitializationStatement. Since it is used in the partition initialization code, not in split classes, there should be no issue with it.
Why are the changes needed?
Fix runtime error in large queries.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Unit test.
Was this patch authored or co-authored using generative AI tooling?
pan3793
changed the title
[SPARK-56134] Make BufferedRowIterator.unsafeRow public to avoid IllegalAccessError.
[SPARK-56134][SQL] Make BufferedRowIterator.unsafeRow public to avoid IllegalAccessError.
Mar 23, 2026
The PR description mentioned that the partitionIndex is not included in this PR. Practically that's okay, because currently the partition initialization code is not yet subject to code splitting.
As can be seen from the init mutable state's case, when these init code grows big, there may be a future point in time that someone decides that partition init code also need to support splitting. But that might still be okay.
If I were to do it, I'd probably do specialized splitting so that the split functions specific to partition init would still take partitionIndex as the argument, then we still don't need to make it "public" since it's not a field to begin with.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This is very similar to #20779. When a generated code has split classes and they try to access protected fields in
BufferedRowIterator, anIllegalAccessErrorwill happen.I am not making
partitionIndexpublic too because I cannot find a real example that can trigger anIllegalAccessErroron it too. In the code base,partitionIndexis mostly accessed viaaddPartitionInitializationStatement. Since it is used in the partition initialization code, not in split classes, there should be no issue with it.Why are the changes needed?
Fix runtime error in large queries.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Unit test.
Was this patch authored or co-authored using generative AI tooling?
No.