Microbenchmarks with JMH / Java Microbenchmark Harness

Writing microbenchmarks for parts of our applications is not always easy – especially when the internals of the virtual machine, the just-in-time-compiler and such things are coming into effect.

Java Microbenchmark Harness is a tool that takes care of creating JVM warmup-cycles, handling benchmark-input-parameters and running benchmarks as isolated processes etc.

Now following a few short examples for writing microbenchmarks with JMH.

Figure 1. Java JMH Microbenchmarks running in IntelliJ

Project Setup

Using Maven we simply need to add the following two dependencies to our project’s pom.xml:

<dependencies>
    <dependency>
        <groupId>org.openjdk.jmh</groupId>
        <artifactId>jmh-core</artifactId>
        <version>1.19</version>
    </dependency>
    <dependency>
        <groupId>org.openjdk.jmh</groupId>
        <artifactId>jmh-generator-annprocess</artifactId>
        <version>1.19</version>
    </dependency>
</dependencies>

In addition, we’re adding the Maven Shade Plugin to assemble a fat-jar for running the benchmarks:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>2.2</version>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>shade</goal>
            </goals>
            <configuration>
                <finalName>benchmark</finalName>
                <transformers>
                    <transformer
                    implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                    <mainClass>org.openjdk.jmh.Main</mainClass>
                    </transformer>
                </transformers>
                <filters>
                    <filter>
                        <artifact>*:*</artifact>
                        <excludes>
                            <exclude>META-INF/*.SF</exclude>
                            <exclude>META-INF/*.DSA</exclude>
                            <exclude>META-INF/*.RSA</exclude>
                        </excludes>
                    </filter>
                </filters>
            </configuration>
        </execution>
    </executions>
</plugin>

Setup using Maven Archetype

More easier is to use the provided Maven archetype org.openjdk.jmh:jmh-java-benchmark-archetype to generate a new benchmark project e.g.:

mvn archetype:generate \
          -DinteractiveMode=false \
          -DarchetypeGroupId=org.openjdk.jmh \
          -DarchetypeArtifactId=jmh-java-benchmark-archetype \
          -DgroupId=com.hascode.tutorial \
          -DartifactId=jmh-benchmark-sample \
          -Dversion=1.0.0

Simple Benchmark

We’re now ready to implement our first, simple benchmark using org.openjdk.jmh.Main as starting point and runner and by annotating the method to benchmark with @Benchmark – that’s all!

package com.hascode.tutorial;

import java.io.IOException;
import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.runner.RunnerException;

public class DefaultsBenchmarkExample {

  public static void main(String[] args) throws IOException, RunnerException {
    org.openjdk.jmh.Main.main(args);
  }

  @Benchmark
  public void sampleMethod(){

  }
}

We may run our benchmarks now using a fat-jar e.g. like this:

mvn clean package && java -cp target/benchmark.jar com.hascode.tutorial.DefaultsBenchmarkExample

# JMH version: 1.19
# VM version: JDK 1.8.0_131, VM 25.131-b11
# VM invoker: /usr/lib/jvm/jdk1.8.0_131/jre/bin/java
# VM options: <none>
# Warmup: 20 iterations, 1 s each
# Measurement: 20 iterations, 1 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: com.hascode.tutorial.DefaultsBenchmarkExample.sampleMethod

# Run progress: 0.00% complete, ETA 00:13:45
# Fork: 1 of 10
# Warmup Iteration   1: 3605181491.535 ops/s
# Warmup Iteration   2: 3600786119.281 ops/s
# Warmup Iteration   3: 3614718349.890 ops/s
# Warmup Iteration   4: 3123247490.136 ops/s
[..]
Iteration   1: 3556337136.253 ops/s
Iteration   2: 3280579831.750 ops/s
Iteration   3: 3528262154.060 ops/s
Iteration   4: 3492611702.649 ops/s
Iteration   5: 3527094300.955 ops/s
[..]

# Run progress: 4.85% complete, ETA 00:13:12
# Fork: 2 of 10
# Warmup Iteration   1: 3549568604.608 ops/s
# Warmup Iteration   2: 3589118701.881 ops/s
# Warmup Iteration   3: 3553118270.819 ops/s
[..]

Result "com.hascode.tutorial.DefaultsBenchmarkExample.sampleMethod":
  3545540088.367 ±(99.9%) 17575281.787 ops/s [Average]
  (min, avg, max) = (3051953885.239, 3545540088.367, 3652281748.873), stdev = 74414843.592
  CI (99.9%): [3527964806.580, 3563115370.154] (assumes normal distribution)

or using an IDE:

Figure 2. Java JMH Microbenchmarks running in IntelliJ

Tuning the Warmup Configuration

In the following example, we’ll be adding some configuration for the vm warmup using the @Warmup annotation.

package com.hascode.tutorial;

import java.io.IOException;
import java.util.concurrent.TimeUnit;
import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.annotations.Fork;
import org.openjdk.jmh.annotations.Warmup;
import org.openjdk.jmh.runner.RunnerException;

public class WarmupConfigExample {

  public static void main(String[] args) throws IOException, RunnerException {
    org.openjdk.jmh.Main.main(args);
  }

  @Benchmark
  @Fork(0)
  @Warmup(iterations = 10, time = 500, timeUnit = TimeUnit.MILLISECONDS)
  public void sampleMethod() {

  }
}

# JMH version: 1.19
# VM version: JDK 1.8.0_131, VM 25.131-b11
# VM invoker: /usr/lib/jvm/jdk1.8.0_131/jre/bin/java
# VM options:
# Warmup: 10 iterations, 500 ms each
# Measurement: 20 iterations, 1 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: com.hascode.tutorial.WarmupConfigExample.sampleMethod

# Run progress: 0.00% complete, ETA 00:00:25
# Fork: N/A, test runs in the host VM
# *** WARNING: Non-forked runs may silently omit JVM options, mess up profilers, disable compiler hints, etc. ***
# *** WARNING: Use non-forked runs only for debugging purposes, not for actual performance runs. ***
# Warmup Iteration   1: 3533060659.401 ops/s
# Warmup Iteration   2: 3513785656.121 ops/s
# Warmup Iteration   3: 3595708958.880 ops/s
# Warmup Iteration   4: 3576232220.453 ops/s
# Warmup Iteration   5: 3482621862.885 ops/s
# Warmup Iteration   6: 3521885445.695 ops/s
# Warmup Iteration   7: 3545500280.775 ops/s
# Warmup Iteration   8: 3523578528.180 ops/s
# Warmup Iteration   9: 3547621249.120 ops/s
# Warmup Iteration  10: 3487009994.200 ops/s
Iteration   1: 3571613038.862 ops/s
Iteration   2: 3594909751.769 ops/s
Iteration   3: 3551277284.450 ops/s
Iteration   4: 3538796644.638 ops/s
Iteration   5: 3530181250.100 ops/s
Iteration   6: 3500270929.480 ops/s
Iteration   7: 3539906819.276 ops/s
Iteration   8: 3489324994.493 ops/s
Iteration   9: 3510483375.086 ops/s
Iteration  10: 3533637346.608 ops/s
Iteration  11: 3542051580.501 ops/s
Iteration  12: 3535201317.693 ops/s
Iteration  13: 3543712258.627 ops/s
Iteration  14: 3489945232.067 ops/s
Iteration  15: 3484418015.987 ops/s
Iteration  16: 3502514821.205 ops/s
Iteration  17: 3535290894.166 ops/s
Iteration  18: 3476540124.706 ops/s
Iteration  19: 3489130100.256 ops/s
Iteration  20: 3496757588.873 ops/s

Result "com.hascode.tutorial.WarmupConfigExample.sampleMethod":
  3522798168.442 ±(99.9%) 27407380.668 ops/s [Average]
  (min, avg, max) = (3476540124.706, 3522798168.442, 3594909751.769), stdev = 31562380.336
  CI (99.9%): [3495390787.774, 3550205549.110] (assumes normal distribution)

# Run complete. Total time: 00:00:25

Benchmark                          Mode  Cnt           Score          Error  Units
WarmupConfigExample.sampleMethod  thrpt   20  3522798168.442 ± 27407380.668  ops/s

Parameterized Benchmarks

In the next example, we’ll be implementing a parameterized benchmark:

package com.hascode.tutorial;

import java.io.IOException;
import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.annotations.BenchmarkMode;
import org.openjdk.jmh.annotations.Fork;
import org.openjdk.jmh.annotations.Level;
import org.openjdk.jmh.annotations.Mode;
import org.openjdk.jmh.annotations.Param;
import org.openjdk.jmh.annotations.Scope;
import org.openjdk.jmh.annotations.Setup;
import org.openjdk.jmh.annotations.State;
import org.openjdk.jmh.runner.RunnerException;

public class ParametersExample {

  public static void main(String[] args) throws IOException, RunnerException {
    org.openjdk.jmh.Main.main(args);
  }

  @Fork(value = 1, warmups = 1)
  @Benchmark
  @BenchmarkMode(Mode.Throughput)
  public void runSample(BenchmarkParams params) {
    for (int i = 0; i <= params.loops; i++) {
      params.sb.append("num ").append(i).append("\n");
    }
    System.out.println(params.sb.toString());
  }

  @State(Scope.Benchmark)
  public static class BenchmarkParams {

    @Param({"1", "20", "40", "100", "1000"})
    public int loops;

    public StringBuffer sb;

    @Setup(Level.Invocation)
    public void setup() {
      sb = new StringBuffer();
    }
  }
}

Output:

# JMH version: 1.19
# VM version: JDK 1.8.0_131, VM 25.131-b11
# VM invoker: /usr/lib/jvm/jdk1.8.0_131/jre/bin/java
# VM options: <none>
# Warmup: 20 iterations, 1 s each
# Measurement: 20 iterations, 1 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: com.hascode.tutorial.ParametersExample.runSample
# Parameters: (loops = 1)

# Run progress: 0.00% complete, ETA 00:06:40
# Warmup Fork: 1 of 1
# Warmup Iteration   1: num 0
num 1

num 0
num 1

num 0
num 1

num 0
num 1

num 0
num 1

num 0
num 1

num 0
num 1

num 0
num 1

[..]

num 994
num 995
num 996
num 997
num 998
num 999
num 1000

400.640 ops/s

Result "com.hascode.tutorial.ParametersExample.runSample":
  376.221 ±(99.9%) 29.341 ops/s [Average]
  (min, avg, max) = (248.222, 376.221, 400.640), stdev = 33.790
  CI (99.9%): [346.880, 405.563] (assumes normal distribution)

# Run complete. Total time: 00:06:51

Benchmark                    (loops)   Mode  Cnt       Score      Error  Units
ParametersExample.runSample        1  thrpt   20  150800.311 ± 6103.086  ops/s
ParametersExample.runSample       20  thrpt   20   18131.695 ±  307.077  ops/s
ParametersExample.runSample       40  thrpt   20    9302.495 ±  472.481  ops/s
ParametersExample.runSample      100  thrpt   20    3837.838 ±  233.328  ops/s
ParametersExample.runSample     1000  thrpt   20     376.221 ±   29.341  ops/s

Programmatic Configuration

In addition we may skip using annotations and use the integrated OptionsBuilder to configure our benchmarks:

package com.hascode.tutorial;

import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.RunnerException;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;

public class ProgrammaticalConfigExample {

  public static void main(String[] args) throws RunnerException {
    Options opts = new OptionsBuilder()
        .include(".*")
        .warmupIterations(10)
        .measurementIterations(20)
        .jvmArgs("-Xms4g", "-Xmx4g")
        .shouldDoGC(true)
        .forks(1)
        .build();

    new Runner(opts).run();
  }
}

Output:

# JMH version: 1.19
# VM version: JDK 1.8.0_131, VM 25.131-b11
# VM invoker: /usr/lib/jvm/jdk1.8.0_131/jre/bin/java
# VM options: -Xms4g -Xmx4g
# Warmup: 10 iterations, 1 s each
# Measurement: 20 iterations, 1 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Average time, time/op
# Benchmark: com.hascode.tutorial.ProgrammaticalConfigExample.sample

# Run progress: 0.00% complete, ETA 00:00:30
# Fork: 1 of 1
# Warmup Iteration   1: ≈ 10⁻¹⁰ s/op
# Warmup Iteration   2: ≈ 10⁻¹⁰ s/op
# Warmup Iteration   3: ≈ 10⁻¹⁰ s/op
# Warmup Iteration   4: ≈ 10⁻¹⁰ s/op
# Warmup Iteration   5: ≈ 10⁻¹⁰ s/op
# Warmup Iteration   6: ≈ 10⁻¹⁰ s/op
# Warmup Iteration   7: ≈ 10⁻¹⁰ s/op
# Warmup Iteration   8: ≈ 10⁻¹⁰ s/op
# Warmup Iteration   9: ≈ 10⁻¹⁰ s/op
# Warmup Iteration  10: ≈ 10⁻¹⁰ s/op
Iteration   1: ≈ 10⁻¹⁰ s/op
Iteration   2: ≈ 10⁻¹⁰ s/op
Iteration   3: ≈ 10⁻¹⁰ s/op
Iteration   4: ≈ 10⁻¹⁰ s/op
Iteration   5: ≈ 10⁻¹⁰ s/op
Iteration   6: ≈ 10⁻¹⁰ s/op
Iteration   7: ≈ 10⁻¹⁰ s/op
Iteration   8: ≈ 10⁻¹⁰ s/op
Iteration   9: ≈ 10⁻¹⁰ s/op
Iteration  10: ≈ 10⁻¹⁰ s/op
Iteration  11: ≈ 10⁻¹⁰ s/op
Iteration  12: ≈ 10⁻¹⁰ s/op
Iteration  13: ≈ 10⁻¹⁰ s/op
Iteration  14: ≈ 10⁻¹⁰ s/op
Iteration  15: ≈ 10⁻¹⁰ s/op
Iteration  16: ≈ 10⁻¹⁰ s/op
Iteration  17: ≈ 10⁻¹⁰ s/op
Iteration  18: ≈ 10⁻¹⁰ s/op
Iteration  19: ≈ 10⁻¹⁰ s/op
Iteration  20: ≈ 10⁻¹⁰ s/op

Result "com.hascode.tutorial.ProgrammaticalConfigExample.sample":
  ≈ 10⁻¹⁰ s/op

# Run complete. Total time: 00:00:49

Benchmark                           Mode  Cnt    Score     Error  Units
ProgrammaticalConfigExample.sample  avgt   20  ≈ 10⁻¹⁰             s/op

OpenJDK JMH Examples

There are plenty of good examples to be found in the OpenJDK Mercurial repository here: http://hg.openjdk.java.net/code-tools/jmh/file/tip/jmh-samples/src/main/java/org/openjdk/jmh/samples/

JMH IntelliJ Plugin

There is a plugin for IntelliJ that helps generating benchmarks (assuming that both dependencies for jmh-core and annotations are on the classpath:

Figure 3. IntelliJ JMH Plugin Installation

Figure 4. IntelliJ JMH Plugin - Actions

Tutorial Sources

Please feel free to download the tutorial sources from my GitHub repository, fork it there or clone it using Git:

git clone https://github.com/hascode/jmh-benchmark-sample.git

Resources

OpenJDK: JMH

Troubleshooting

"Error: A JNI error has occurred, please check your installation and try again Exception in thread “main” java.lang.NoClassDefFoundError: org/openjdk/jmh/runner/RunnerException at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Class.java:2701) at java.lang.Class.privateGetMethodRecursive(Class.java:3048) at java.lang.Class.getMethod0(Class.java:3018) at java.lang.Class.getMethod(Class.java:1784) at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544) at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526) Caused by: java.lang.ClassNotFoundException: org.openjdk.jmh.runner.RunnerException at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) … 7 more" Use the Maven Shade Plugin to assemble a fat Jar, that then may be run using java -jar fatjar.jar package.MainClass

Project Setup#

Setup using Maven Archetype#

Simple Benchmark#

Tuning the Warmup Configuration#

Parameterized Benchmarks#

Programmatic Configuration#

OpenJDK JMH Examples#

JMH IntelliJ Plugin#

Tutorial Sources#

Resources#

Troubleshooting#