Migrate benchmarks to kotlinx-benchmark for JVM+Native#512
Draft
andrewparmet wants to merge 25 commits intoopen-toast:mainfrom
Draft
Migrate benchmarks to kotlinx-benchmark for JVM+Native#512andrewparmet wants to merge 25 commits intoopen-toast:mainfrom
andrewparmet wants to merge 25 commits intoopen-toast:mainfrom
Conversation
andrewparmet
force-pushed
the
native-benchmarks
branch
3 times, most recently
from
ad76523 to
48c0e25
Compare
New commonMain benchmarks run on both JVM (backed by JMH) and Native, covering all the same scenarios as the existing JMH benchmarks: deserialize, serialize, passthrough, mutate-and-serialize, copy-append, and string-heavy workloads. Streaming benchmarks use kotlinx.io.Buffer directly (works on all platforms) instead of ByteArrayOutputStream. Existing JMH benchmarks preserved in jvmMain for the legacy runner and protobuf-java/Wire comparison modules. Run: ./gradlew :benchmarks:protokt-benchmarks:macosArm64Benchmark ./gradlew :benchmarks:protokt-benchmarks:jvmBenchmark
- benchmarks-util: multiplatform module with ProtobufBenchmarkSet interface (kotlinx-benchmark Blackhole), randomUtf8String (kotlin Random), readDatasetBytes expect/actual - JvmProtobufBenchmarkSet: extends base with InputStream-streaming methods for protobuf-java and Wire - Remove duplicate ProtoktBenchmarks.kt from jvmMain (commonMain version covers all benchmarks) - protobuf-java/Wire benchmarks: switch to kotlin.random.Random, implement JvmProtobufBenchmarkSet
…Poet version alignment
…terface
- ProtobufBenchmarkSet<T>: generic over blackhole type, avoids
kotlinx-benchmark runtime resolution issues in the util module
- All annotations switched from org.openjdk.jmh to kotlinx.benchmark
- Shared benchmark config via configureBenchmarks() with Gradle
property overrides (-PbenchmarkInclude, -PbenchmarkExclude, etc.)
- Removed legacy JMH runner, main() functions, kapt, application plugin
- protobuf-java and Wire: register("main") benchmark target for JVM
- README updated with unified API
- Add allopen plugin for @State-annotated classes (JMH requires open
classes and methods); remove manual `open` keyword from all benchmark
classes
- benchmarks-conventions: allopen + kotlinx-benchmark-runtime, drop kapt
(kotlinx-benchmark handles JMH annotation processing)
- protobuf-java/Wire: register("main") target, kotlinx.benchmark
annotations, remove legacy JMH main() functions
- protokt-benchmarks: add allopen for JVM target, remove kapt
- BenchmarkConfiguration: add jmhIgnoreLock to prevent stale lock issues
- Restore 20K string comment, clean up buildSrc deps, remove stray files
- Merge DatasetReader + RandomUtf8 into BenchmarkUtils.kt (commonMain) - Keep ProtobufBenchmarkSet and JvmProtobufBenchmarkSet as own files - Add ProtoktJvmBenchmarks with InputStream-based streaming deserialization benchmarks (deserializeLargeStreaming, etc.) - Fix kotlin.random.Random import
- @param for collectionFactory in commonMain benchmarks - expect/actual applyBenchmarkConfig: JVM uses System.setProperty, native sets collectionFactoryOverride directly via friend paths - runtimeFriendPaths() on benchmark module for internal access - persistent-collections dependency moved to commonMain
…tory parameterization
…le native collection factory parameterization
ftell returns Int on mingwX64 making .toInt() redundant, but it's needed on other native targets where ftell returns Long.
andrewparmet
force-pushed
the
native-benchmarks
branch
from
48c0e25 to
b3824a3
Compare
andrewparmet
commented
Apr 14, 2026
| } | ||
| } | ||
|
|
||
| fun Project.friendPaths(vararg friendProjectPaths: String) { |
Collaborator
Author
There was a problem hiding this comment.
Since native can't use system properties to configure codec/collection factory, we have to use friend paths now. This structure decouples friend path declaration from the runtime module so we can expose the internal members necessary to set codec/coll. fac. from the benchmark projects.
… targets Add @param for codec to ProtoktMultiplatformBenchmarks so JVM results break down by codec (matching RESULTS.md). On native the codec param iterates but has no effect (single built-in codec). Also add enableNativeTargets() to benchmark modules for macosArm64 execution.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PersistentCollectionFactoryparameterization on both JVM and NativeHighlights from results
PersistentCollectionFactoryshows no measurable overhead vsDefaultCollectionFactoryon any benchmarkTest plan