I recently stumbled upon a nice framework that allows to convert simple JUnit tests into micro benchmarks named JUnitBenchmarks.

It allows to set basic benchmark options and and to generate charts by adding some simple annotations and a test rule to your tests.

One might argue if it is wise to mix the aspects, testing and benchmarking and I’d agree for sure – nevertheless I think this framework can be handy sometimes so let’s create some benchmarks using JUnit and JUnitBenchmarks..

Dependencies / Project Setup

Of course we need to add some dependencies to our project .. first of all – of course junit and junit-benchmarks.

For the examples where we’re collecting test data for multiple runs to visualize the results in some nice charts we need an embedded database, that’s why we’re adding the h2 database.

Finally I can’t and I won’t live with all my beloved hamcrest matchers (not only the few ones bundled with junit) and therefore hamcrest-all is the last dependency needed to run the following examples.

Maven

If you’re the Maven guy, please feel free to add the following dependencies to your pom.xml:

<dependencies>
	<dependency>
		<groupId>junit</groupId>
		<artifactId>junit</artifactId>
		<version>4.11</version>
		<scope>test</scope>
	</dependency>
	<dependency>
		<groupId>com.carrotsearch</groupId>
		<artifactId>junit-benchmarks</artifactId>
		<version>0.6.0-SNAPSHOT</version>
		<scope>test</scope>
	</dependency>
	<dependency>
		<groupId>com.h2database</groupId>
		<artifactId>h2</artifactId>
		<version>1.3.160</version>
		<scope>test</scope>
	</dependency>
	<dependency>
		<groupId>org.hamcrest</groupId>
		<artifactId>hamcrest-all</artifactId>
		<version>1.3</version>
		<scope>test</scope>
	</dependency>
</dependencies>

SBT

On the other side if you’re using SBT, please feel free to add the following dependencies to your build.sbt:

libraryDependencies ++= Seq(
 "junit" % "junit" % "4.11" % "test",
 "com.carrotsearch" % "junit-benchmarks" % "0.6.0-SNAPSHOT" % "test",
 "com.h2database" % "h2" % "1.3.160" % "test",
 "org.hamcrest" % "hamcrest-all" % "1.3" % "test"
)

Now let’s add some classes that we’re going to reuse in the following tests and benchmarks.

The first one is ClassUnderTest.java – it does nothing special just some slow down action, useless loop processing and fattening the string constant pool ..

package com.hascode.tutorial;

public class ClassUnderTest {
	public boolean doTest() throws InterruptedException {
		for (int i = 0; i < 3000000; i++)
			new String("foo_" + i);

		Thread.sleep(500);
		return true;
	}
}

The second one is used for our benchmarks on different map implementations – that’s why we implement a correct hashCode/equals behaviour here – and because we’re using a TreeMap, the class implements the Comparable interface.

package com.hascode.tutorial;

public class Foo implements Comparable<Foo> {
	private final Integer x;

	public Foo(final int x) {
		this.x = x;
	}

	@Override
	public int hashCode() {
		final int prime = 31;
		int result = 1;
		result = prime * result + x;
		return result;
	}

	@Override
	public boolean equals(final Object obj) {
		if (this == obj)
			return true;
		if (obj == null)
			return false;
		if (getClass() != obj.getClass())
			return false;
		Foo other = (Foo) obj;
		if (x != other.x)
			return false;
		return true;
	}

	@Override
	public int compareTo(final Foo o) {
		return x.compareTo(o.x);
	}
}

Now that we’ve got the boring stuff covered, lets write some tests and benchmark them …

Benchmarking using JUnit Test Rules

To benchmark a test is quite easy – we just need to add a test rule to the test class and we get some simple benchmarking for free.

In addition we may fine-tune the benchmarks for each test method using @BenchmarkOptions. In the following example the test method testSomethingWithWarmups has 4 warmup rounds and 20 benchmark rounds.

package com.hascode.tutorial;

import static org.junit.Assert.assertTrue;

import org.junit.Rule;
import org.junit.Test;

import com.carrotsearch.junitbenchmarks.BenchmarkRule;

public class SimpleTestBenchmark {
	ClassUnderTest classUnderTest = new ClassUnderTest();

	@Rule
	public BenchmarkRule benchmarkRun = new BenchmarkRule();

	@Test
	public void testSomething() throws Exception {
		assertTrue(classUnderTest.doTest());
	}
}

Running the benchmarks produces the following output:

SimpleTestBenchmark.testSomething: [measured 10 out of 15 rounds, threads: 1 (sequential)]
round: 0.64 [+- 0.00], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 12, GC.time: 0.01, time.total: 9.78, time.warmup: 3.38, time.bench: 6.40

Configuring Benchmark Options

We’re able to configure the benchmarks at class or method level using @BenchmarkOptions to set option like the benchmark rounds, warmup rounds, disable calls to the garbage collector, set the time of clock for measuring and the number of threads that should execute the benchmarked method in parallel.

package com.hascode.tutorial;

import static org.junit.Assert.assertTrue;

import org.junit.Rule;
import org.junit.Test;

import com.carrotsearch.junitbenchmarks.BenchmarkOptions;
import com.carrotsearch.junitbenchmarks.BenchmarkRule;
import com.carrotsearch.junitbenchmarks.Clock;

@BenchmarkOptions(benchmarkRounds = 10, warmupRounds = 5, callgc = false, clock = Clock.REAL_TIME, concurrency = 4)
public class BenchmarkOptionsExample {
	ClassUnderTest classUnderTest = new ClassUnderTest();

	@Rule
	public BenchmarkRule benchmarkRun = new BenchmarkRule();

	@Test
	public void testSomething() throws Exception {
		assertTrue(classUnderTest.doTest());
	}

	@BenchmarkOptions(benchmarkRounds = 20, warmupRounds = 4)
	@Test
	public void testSomethingWithAnotherSetup() throws Exception {
		assertTrue(classUnderTest.doTest());
	}
}

Running the tests produces the following output:

BenchmarkOptionsExample.testSomethingWithAnotherSetup: [measured 20 out of 24 rounds, threads: 1 (sequential)]
round: 0.65 [+- 0.01], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 37, GC.time: 0.02, time.total: 15.65, time.warmup: 2.73, time.bench: 12.91
BenchmarkOptionsExample.testSomething: [measured 10 out of 15 rounds, threads: 1 (sequential)]
round: 0.65 [+- 0.01], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 37, GC.time: 0.02, time.total: 9.80, time.warmup: 3.26, time.bench: 6.54

Visualizing Benchmarks

With JUnitBenchmarks it is easy to visualize the benchmark data. For the following benchmarks we’re comparing three different implementations of java.util.Map:

HashMap, LinkedHashMap and TreeMap.

Of course the benchmark results are somehow obvious looking at the way that we’re adding data to the maps and the fact that TreeMap orders incoming entries but for this tutorial it should make a sufficient example.

To specify where to look for benchmark results for further processing please add the following parameters to your run configuration

-Djub.consumers=CONSOLE,H2 -Djub.db.file=.benchmarks

Bar Chart

In our first example, we’re creating a bar chart from our test results. We’re configuring the chart using the AxisRange and BenchmarkMethodChart annotations.

The first one allows us to specify the axis range for our chart, the second one triggers the chart creation.

package com.hascode.tutorial;

import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.is;
import static org.hamcrest.Matchers.notNullValue;

import java.util.HashMap;
import java.util.LinkedHashMap;
import java.util.Map;
import java.util.TreeMap;

import org.junit.Rule;
import org.junit.Test;

import com.carrotsearch.junitbenchmarks.BenchmarkRule;
import com.carrotsearch.junitbenchmarks.annotation.AxisRange;
import com.carrotsearch.junitbenchmarks.annotation.BenchmarkMethodChart;

@AxisRange(min = 0, max = 1)
@BenchmarkMethodChart(filePrefix = "map-types-benchmark-barchart")
public class MapTypesBenchmarkWithBarChartExample {
	@Rule
	public BenchmarkRule benchmarkRun = new BenchmarkRule();

	static final int MAX_ENTRIES = 1500000;

	@Test
	public void hashMap() throws Exception {
		testMap(new HashMap<Integer, Foo>());
	}

	@Test
	public void linkedHashMap() throws Exception {
		testMap(new LinkedHashMap<Integer, Foo>());
	}

	@Test
	public void treeMap() throws Exception {
		testMap(new TreeMap<Integer, Foo>());
	}

	private void testMap(final Map<Integer, Foo> map) {
		for (int i = 0; i < MAX_ENTRIES; i++) {
			map.put(i, new Foo(i));
		}
		assertThat(map.size(), is(MAX_ENTRIES));
		for (int i = 0; i < MAX_ENTRIES; i++) {
			assertThat(map.get(i), is(notNullValue()));
		}
	}
}

Running the example should produce the following output:

MapTypesBenchmarkWithBarChartExample.treeMap: [measured 10 out of 15 rounds, threads: 1 (sequential)]
round: 0.52 [+- 0.02], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 11, GC.time: 0.55, time.total: 8.86, time.warmup: 3.65, time.bench: 5.21
MapTypesBenchmarkWithBarChartExample.hashMap: [measured 10 out of 15 rounds, threads: 1 (sequential)]
round: 0.10 [+- 0.03], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 6, GC.time: 0.29, time.total: 1.60, time.warmup: 0.55, time.bench: 1.05
MapTypesBenchmarkWithBarChartExample.linkedHashMap: [measured 10 out of 15 rounds, threads: 1 (sequential)]
round: 0.13 [+- 0.03], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 6, GC.time: 0.21, time.total: 1.93, time.warmup: 0.68, time.bench: 1.25

If you take a look into the project directory you should see the following web site:

map types bar chart
Figure 1. Map insertion benchmark results as a bar chart

History Chart

In the second example, we’re creating a history chart. It is important to know, that you need to run a few rounds to collect data (that is stored in the h2 database).

package com.hascode.tutorial;

import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.is;
import static org.hamcrest.Matchers.notNullValue;

import java.util.HashMap;
import java.util.LinkedHashMap;
import java.util.Map;
import java.util.TreeMap;

import org.junit.Rule;
import org.junit.Test;

import com.carrotsearch.junitbenchmarks.BenchmarkRule;
import com.carrotsearch.junitbenchmarks.annotation.BenchmarkHistoryChart;
import com.carrotsearch.junitbenchmarks.annotation.LabelType;

@BenchmarkHistoryChart(filePrefix = "map-types-benchmark-history-chart", labelWith = LabelType.CUSTOM_KEY, maxRuns = 20)
public class HistoryChartOutputTests {
	@Rule
	public BenchmarkRule benchmarkRun = new BenchmarkRule();

	static final int MAX_ENTRIES = 1500000;

	@Test
	public void hashMap() throws Exception {
		testMap(new HashMap<Integer, Foo>());
	}

	@Test
	public void linkedHashMap() throws Exception {
		testMap(new LinkedHashMap<Integer, Foo>());
	}

	@Test
	public void treeMap() throws Exception {
		testMap(new TreeMap<Integer, Foo>());
	}

	private void testMap(final Map<Integer, Foo> map) {
		for (int i = 0; i < MAX_ENTRIES; i++) {
			map.put(i, new Foo(i));
		}
		assertThat(map.size(), is(MAX_ENTRIES));
		for (int i = 0; i < MAX_ENTRIES; i++) {
			assertThat(map.get(i), is(notNullValue()));
		}
	}
}

Running the test should produce some output like this:

HistoryChartOutputTests.treeMap: [measured 10 out of 15 rounds, threads: 1 (sequential)]
round: 0.62 [+- 0.14], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 12, GC.time: 1.24, time.total: 10.00, time.warmup: 3.81, time.bench: 6.19
HistoryChartOutputTests.hashMap: [measured 10 out of 15 rounds, threads: 1 (sequential)]
round: 0.13 [+- 0.04], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 8, GC.time: 0.57, time.total: 2.23, time.warmup: 0.91, time.bench: 1.32
HistoryChartOutputTests.linkedHashMap: [measured 10 out of 15 rounds, threads: 1 (sequential)]
round: 0.16 [+- 0.04], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 10, GC.time: 0.67, time.total: 2.42, time.warmup: 0.83, time.bench: 1.59
map types history chart
Figure 2. Map insertion benchmark results in a history chart

Tutorial Sources

Please feel free to download the tutorial sources from my GitHub repository, fork it there or clone it using Git:

git clone https://github.com/hascode/junit-benchmarking-tutorial.git

Resources