Skip to content

[SPARK-56441][BUILD] Upgrade lz4-java to 1.11.0 #55295

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
dongjoon-hyun wants to merge 4 commits into from
Closed

[SPARK-56441][BUILD] Upgrade lz4-java to 1.11.0 #55295

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
f0b243b
[SPARK-56441] Upgrade `lz4-java` to 1.11.0
dongjoon-hyun Apr 10, 2026
44ca725
Benchmark results for *LZ4TPCDSDataBenchmark (JDK 17, Scala 2.13, spl…
dongjoon-hyun Apr 10, 2026
d337d09
Benchmark results for *LZ4TPCDSDataBenchmark (JDK 21, Scala 2.13, spl…
dongjoon-hyun Apr 10, 2026
1ac85e0
Benchmark results for *LZ4TPCDSDataBenchmark (JDK 25, Scala 2.13, spl…
dongjoon-hyun Apr 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions core/benchmarks/LZ4TPCDSDataBenchmark-jdk21-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,16 @@
Benchmark LZ4CompressionCodec
================================================================================================

OpenJDK 64-Bit Server VM 21.0.10+7-LTS on Linux 6.14.0-1017-azure
OpenJDK 64-Bit Server VM 21.0.10+7-LTS on Linux 6.17.0-1010-azure
AMD EPYC 7763 64-Core Processor
Compression: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Compression 4 times 2376 2382 9 0.0 593993111.8 1.0X
Compression 4 times 2342 2349 9 0.0 585571861.0 1.0X

OpenJDK 64-Bit Server VM 21.0.10+7-LTS on Linux 6.14.0-1017-azure
OpenJDK 64-Bit Server VM 21.0.10+7-LTS on Linux 6.17.0-1010-azure
AMD EPYC 7763 64-Core Processor
Decompression: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Decompression 4 times 731 757 29 0.0 182787977.2 1.0X
Decompression 4 times 748 776 24 0.0 187029239.5 1.0X


8 changes: 4 additions & 4 deletions core/benchmarks/LZ4TPCDSDataBenchmark-jdk25-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,16 @@
Benchmark LZ4CompressionCodec
================================================================================================

OpenJDK 64-Bit Server VM 25.0.2+10-LTS on Linux 6.14.0-1017-azure
OpenJDK 64-Bit Server VM 25.0.2+10-LTS on Linux 6.17.0-1010-azure
AMD EPYC 7763 64-Core Processor
Compression: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Compression 4 times 2373 2380 10 0.0 593284150.8 1.0X
Compression 4 times 2343 2353 15 0.0 585680834.0 1.0X

OpenJDK 64-Bit Server VM 25.0.2+10-LTS on Linux 6.14.0-1017-azure
OpenJDK 64-Bit Server VM 25.0.2+10-LTS on Linux 6.17.0-1010-azure
AMD EPYC 7763 64-Core Processor
Decompression: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Decompression 4 times 694 710 23 0.0 173453930.5 1.0X
Decompression 4 times 721 729 9 0.0 180327142.3 1.0X


12 changes: 6 additions & 6 deletions core/benchmarks/LZ4TPCDSDataBenchmark-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,16 @@
Benchmark LZ4CompressionCodec
================================================================================================

OpenJDK 64-Bit Server VM 17.0.18+8-LTS on Linux 6.14.0-1017-azure
AMD EPYC 7763 64-Core Processor
OpenJDK 64-Bit Server VM 17.0.18+8-LTS on Linux 6.17.0-1010-azure
AMD EPYC 9V74 80-Core Processor
Compression: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Compression 4 times 2376 2383 10 0.0 593917596.8 1.0X
Compression 4 times 2543 2547 6 0.0 635681307.2 1.0X

OpenJDK 64-Bit Server VM 17.0.18+8-LTS on Linux 6.14.0-1017-azure
AMD EPYC 7763 64-Core Processor
OpenJDK 64-Bit Server VM 17.0.18+8-LTS on Linux 6.17.0-1010-azure
AMD EPYC 9V74 80-Core Processor
Decompression: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Decompression 4 times 759 761 2 0.0 189753707.8 1.0X
Decompression 4 times 767 768 2 0.0 191709504.8 1.0X


2 changes: 1 addition & 1 deletion dev/deps/spark-deps-hadoop-3-hive-2.3
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -189,7 +189,7 @@ log4j-api/2.25.4//log4j-api-2.25.4.jar
log4j-core/2.25.4//log4j-core-2.25.4.jar
log4j-layout-template-json/2.25.4//log4j-layout-template-json-2.25.4.jar
log4j-slf4j2-impl/2.25.4//log4j-slf4j2-impl-2.25.4.jar
lz4-java/1.10.4//lz4-java-1.10.4.jar
lz4-java/1.11.0//lz4-java-1.11.0.jar
metrics-core/4.2.37//metrics-core-4.2.37.jar
metrics-graphite/4.2.37//metrics-graphite-4.2.37.jar
metrics-jmx/4.2.37//metrics-jmx-4.2.37.jar
Expand Down
2 changes: 1 addition & 1 deletion pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -889,7 +889,7 @@
<dependency>
<groupId>at.yawk.lz4</groupId>
<artifactId>lz4-java</artifactId>
<version>1.10.4</version>
<version>1.11.0</version>
</dependency>
<dependency>
<groupId>com.github.luben</groupId>
Expand Down