Mutation testing makes an interesting addition to the classical test coverage metrics.

They seed mutations (errors) into the code, run the project’s tests afterwards and if the tests fail, the mutation is killed – otherwise it lived and we have a possible indication of an issue with our tests.

In the following short tutorial. I’d like to demonstrate how to setup mutation tests with the PIT/Pitest library and Maven and generate reports.

book bean detailed mutation report 1024x952
Figure 1. Detailed mutation report for BookBean

Adding the Maven Plugin

To integrate Pitest, we’re simply adding the following plug-in to our project’s pom.xml.

<plugin>
  <groupId>org.pitest</groupId>
  <artifactId>pitest-maven</artifactId>
  <version>1.1.5</version>
</plugin>

In addition to write some tests, we’re adding junit and hamcrest-all to our project.

Target Classes and Tests Configuration

We may specify a pattern for classes to be mutated and tests to be used by adding the following lines to the plug-in definition in the pom.xml.

If nothing is specified, the project’s groupId is taken as a default. More detailed information about the other configuration flags can be found in the documentation here.

<configuration>
	<targetClasses>
		<param>com.your.package.root.for.mutation*</param>
	</targetClasses>
	<targetTests>
		<param>com.your.package.root.for.tests*</param>
	</targetTests>
</configuration>

Our Example Project

This is our small project that’s going to be the target for our mutation tests:

Book Entity

Just a simple POJO, immutable, with a toString method implemented

package com.hascode.tutorial.entity;

public class Book {
	private final String title;
	private final String id;

	public Book(final String title, final String id) {
		this.title = title;
		this.id = id;
	}

	// getter ommitted,

	@Override
	public String toString() {
		StringBuilder builder = new StringBuilder();
		builder.append("Book [title=").append(title).append(", id=").append(id).append("]");
		return builder.toString();
	}
}

Book Bean

Another POJO, creates new Books, adding an unique identifier and checking some constraints.

package com.hascode.tutorial.service;

import java.util.UUID;

import com.hascode.tutorial.entity.Book;

public class BookBean {
	private static final int MIN_LENGTH = 3;

	public Book create(final String title) {
		if (title == null) {
			throw new IllegalArgumentException("title must be set");
		}
		if (title.length() <= MIN_LENGTH) {
			throw new IllegalArgumentException("title must have a minimal length of " + MIN_LENGTH);
		}
		return new Book(title, UUID.randomUUID().toString().toUpperCase());
	}
}

Book Bean Test

A simple test, verifying the book bean’s behaviour for the success path and two possible error paths.

package com.hascode.tutorial.service;

import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.isEmptyOrNullString;
import static org.hamcrest.Matchers.not;
import static org.hamcrest.Matchers.notNullValue;

import org.junit.Test;

import com.hascode.tutorial.entity.Book;

public class BookBeanTest {
	private static final String TITLE = "Some book";

	BookBean bookBean = new BookBean();

	@Test
	public void shouldCreateBook() throws Exception {
		Book book = bookBean.create(TITLE);
		assertThat(book, notNullValue());
		assertThat(book.getTitle(), equalTo(TITLE));
		assertThat(book.getId(), not(isEmptyOrNullString()));
	}

	@Test(expected = IllegalArgumentException.class)
	public void shouldFailCreateBookWithNoTitleGiven() throws Exception {
		bookBean.create(null);
	}

	@Test(expected = IllegalArgumentException.class)
	public void shouldFailCreateBookWithTooShortTitleGiven() throws Exception {
		bookBean.create("ab");
	}
}

Running Mutations

We’re now ready to run mutation tests. This is done by simply executing the following Maven goal:

$ mvn org.pitest:pitest-maven:mutationCoverage
[..]
[INFO] Found plugin : Default csv report plugin
[INFO] Found plugin : Default xml report plugin
[INFO] Found plugin : Default html report plugin
[INFO] Found plugin : Default limit mutations plugin
[INFO] Found shared classpath plugin : Default mutation engine
[INFO] Adding org.pitest:pitest to SUT classpath
[INFO] Mutating from /data/project/pitest-tutorial/target/classes
[INFO] Defaulting to group id (com.hascode.tutorial*)
3:48:50 PM PIT >> INFO : Verbose logging is disabled. If you encounter an problem please enable it before reporting an issue.
3:48:50 PM PIT >> INFO : Sending 1 test classes to slave
3:48:50 PM PIT >> INFO : Sent tests to slave
3:48:50 PM PIT >> INFO : SLAVE : 3:48:50 PM PIT >> INFO : Checking environment
3:48:50 PM PIT >> INFO : SLAVE : 3:48:50 PM PIT >> INFO : Found  3 tests
3:48:50 PM PIT >> INFO : SLAVE : 3:48:50 PM PIT >> INFO : Dependency analysis reduced number of potential tests by 0
3:48:50 PM PIT >> INFO : SLAVE : 3:48:50 PM PIT >> INFO : 3 tests received
\3:48:50 PM PIT >> INFO : Calculated coverage in 0 seconds.
3:48:50 PM PIT >> INFO : Created  2 mutation test units
-3:48:51 PM PIT >> INFO : Completed in 1 seconds
================================================================================
- Timings
================================================================================
> scan classpath : < 1 second
> coverage and dependency analysis : < 1 second
> build mutation tests : < 1 second
> run mutation analysis : < 1 second
--------------------------------------------------------------------------------
> Total  : 1 seconds
--------------------------------------------------------------------------------
================================================================================
- Statistics
================================================================================
>> Generated 7 mutations Killed 5 (71%)
>> Ran 7 tests (1 tests per mutation)
================================================================================
- Mutators
================================================================================
> org.pitest.mutationtest.engine.gregor.mutators.ConditionalsBoundaryMutator
>> Generated 1 Killed 0 (0%)
> KILLED 0 SURVIVED 1 TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 0
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.ReturnValsMutator
>> Generated 4 Killed 3 (75%)
> KILLED 3 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 1
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.NegateConditionalsMutator
>> Generated 2 Killed 2 (100%)
> KILLED 2 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 0
--------------------------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.044s
[INFO] Finished at: Sun May 10 15:48:51 CEST 2015
[INFO] Final Memory: 12M/205M
[INFO] ------------------------------------------------------------------------

Mutation Test Report

Now we’re ready to have a look at the generated HTML report of the mutation test in target/pit-reports/yyyyMMddhhmm.

pitest mutation report overview
Figure 2. Pitest Mutation Report Overview
book bean mutation overview
Figure 3. BookBean Mutation Report Overview
book bean detailed mutation report 1024x952
Figure 4. Detailed mutation report for BookBean

Tutorial Sources

Please feel free to download the tutorial sources from my GitHub repository, fork it there or clone it using Git:

git clone https://github.com/hascode/pitest-tutorial.git