Writing Java Integration Tests for MongoDB

MongoDB is matured, document-oriented, cross-platform NoSQL database system with drivers available for a bunch of different programming languages.

In the following short examples I’m going to write some integration tests for MongoDB using the MongoDB Java driver and the Flapdoodle library to create an embedded MongoDB instance for testing.

We’re going to write tests for a simple persist-and-query scenarion and for a map-reduce function and in addition I’m going to show how to bind the start and stop of a MongoDB instance to a Maven goal using the embedmongo-maven-plugin.

Figure 1. MongoDB Console

Dependencies

We need to add the following two dependencies to our mavenized project’s pom.xml to be able to run the following examples – one of course – is the mongo-java-driver the other one is flapdoodle-mongo-embed that makes our life so easy here when writing our tests.

The flapdoodle project can be found at the following GitHub project: https://github.com/flapdoodle-oss/de.flapdoodle.embed.mongo

<dependencies>
	<dependency>
		<groupId>org.mongodb</groupId>
		<artifactId>mongo-java-driver</artifactId>
		<version>2.11.3</version>
	</dependency>
	<dependency>
		<groupId>de.flapdoodle.embed</groupId>
		<artifactId>de.flapdoodle.embed.mongo</artifactId>
		<version>1.36</version>
	</dependency>
</dependencies>

In addition, I’ve added some common dependencies like junit, hamcrest and guava..

Integration Test Examples

I have added two example tests here:

The first tests adds some data (books) to the mongo database and queries afterwards.

The second test uses a map-reduce algorithm to count the occurrence of each book category in the given datasets.

The MongodForTestsFactory allows us to set up the environment that we need to run our tests here.

package it;

import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.hasItem;

import java.util.Collections;
import java.util.Comparator;
import java.util.List;
import java.util.Set;
import java.util.UUID;

import org.junit.After;
import org.junit.Before;
import org.junit.Test;

import com.google.common.collect.Lists;
import com.mongodb.BasicDBObject;
import com.mongodb.DB;
import com.mongodb.DBCollection;
import com.mongodb.DBCursor;
import com.mongodb.DBObject;
import com.mongodb.MapReduceCommand;
import com.mongodb.MapReduceOutput;
import com.mongodb.MongoClient;

import de.flapdoodle.embed.mongo.distribution.Version;
import de.flapdoodle.embed.mongo.tests.MongodForTestsFactory;

public class MongoDbIntegrationTest {
	private static final String BOOK_COLLECTION_NAME = "bookCollection";
	private static final String AUTHOR_1 = "Tim Tester";
	private static final String AUTHOR_2 = "Vaughn Vernon";
	private static final String BOOK1_TITLE = "Some book";
	private static final String BOOK2_TITLE = "Implementing Domain Driven Design";
	private static final String BOOK3_TITLE = "Anemic Domain Model and why I love it";

	MongodForTestsFactory factory;
	MongoClient mongo;

	@Before
	public void setup() throws Exception {
		factory = MongodForTestsFactory.with(Version.Main.PRODUCTION);
		mongo = factory.newMongo();
	}

	@After
	public void teardown() throws Exception {
		if (factory != null)
			factory.shutdown();
	}

	@Test
	public void shouldPersistAndFindBooks() {
		DB db = mongo.getDB("test-" + UUID.randomUUID());
		DBCollection collection = db.createCollection(BOOK_COLLECTION_NAME,
				new BasicDBObject());

		Set<String> collectionNames = db.getCollectionNames();
		assertThat(collectionNames, hasItem(BOOK_COLLECTION_NAME));

		BasicDBObject book1 = new BasicDBObject();
		book1.put("title", BOOK1_TITLE);
		book1.put("author", AUTHOR_1);

		BasicDBObject book2 = new BasicDBObject();
		book2.put("title", BOOK2_TITLE);
		book2.put("author", AUTHOR_2);

		BasicDBObject book3 = new BasicDBObject();
		book3.put("title", BOOK3_TITLE);
		book3.put("author", AUTHOR_1);

		collection.insert(book1);
		collection.insert(book2);
		collection.insert(book3);

		assertThat(collection.find().size(), equalTo(3));

		BasicDBObject query1 = new BasicDBObject("title", BOOK1_TITLE);
		BasicDBObject bookFromDb1 = (BasicDBObject) collection.findOne(query1);
		assertThat((String) bookFromDb1.get("title"), equalTo(BOOK1_TITLE));
		assertThat((String) bookFromDb1.get("author"), equalTo(AUTHOR_1));

		BasicDBObject query2 = new BasicDBObject("author", AUTHOR_1);
		DBCursor docCursor = collection.find(query2);
		assertThat(docCursor.size(), equalTo(2));

		BasicDBObject bookByAuthor1 = (BasicDBObject) docCursor.next();
		assertThat((String) bookByAuthor1.get("title"), equalTo(BOOK1_TITLE));
		assertThat((String) bookByAuthor1.get("author"), equalTo(AUTHOR_1));

		BasicDBObject bookByAuthor2 = (BasicDBObject) docCursor.next();
		assertThat((String) bookByAuthor2.get("title"), equalTo(BOOK3_TITLE));
		assertThat((String) bookByAuthor2.get("author"), equalTo(AUTHOR_1));
	}

	@Test
	public void shouldCountCategoriesUsingMapReduce() throws Exception {
		DB db = mongo.getDB("test-" + UUID.randomUUID());
		DBCollection collection = db.createCollection(BOOK_COLLECTION_NAME,
				new BasicDBObject());
		BasicDBObject book1 = new BasicDBObject()
				.append("title", BOOK1_TITLE)
				.append("author", AUTHOR_1)
				.append("categories",
						new String[] { "crime", "horror", "mystery" });

		BasicDBObject book2 = new BasicDBObject()
				.append("title", BOOK2_TITLE)
				.append("author", AUTHOR_2)
				.append("categories",
						new String[] { "science", "mystery", "sports" });

		BasicDBObject book3 = new BasicDBObject()
				.append("title", BOOK3_TITLE)
				.append("author", AUTHOR_1)
				.append("categories",
						new String[] { "horror", "science", "romance" });

		collection.insert(book1);
		collection.insert(book2);
		collection.insert(book3);

		// java please buy some groovy-style multiline support ''' :\
		String map = "function(){" + "this.categories.forEach("
				+ "function(category){emit(category, {count:1});}" + ");"
				+ "};";
		String reduce = "function(key, values){" + "var sum = 0;"
				+ "for(var i=0;i<values.length;i++)"
				+ "sum += values[i].count;" + "return {count: sum};" + "};";
		MapReduceOutput output = collection.mapReduce(map, reduce, null,
				MapReduceCommand.OutputType.INLINE, null);
		List<DBObject> result = Lists.newArrayList(output.results());
		Collections.sort(result, bookComparator);

		assertThat((String) result.get(0).get("_id"), equalTo("crime"));
		DBObject count1 = (DBObject) result.get(0).get("value");
		assertThat((Double) count1.get("count"), equalTo(1.0D));

		assertThat((String) result.get(1).get("_id"), equalTo("horror"));
		DBObject count2 = (DBObject) result.get(1).get("value");
		assertThat((Double) count2.get("count"), equalTo(2.0D));

		assertThat((String) result.get(2).get("_id"), equalTo("mystery"));
		DBObject count3 = (DBObject) result.get(2).get("value");
		assertThat((Double) count3.get("count"), equalTo(2.0D));

		// [..]
	}

	static Comparator<DBObject> bookComparator = new Comparator<DBObject>() {
		public int compare(final DBObject o1, final DBObject o2) {
			return ((String) o1.get("_id")).compareTo((String) o2.get("_id"));
		}
	};
}

Map-Reduce Human Readable

One simple feature on my personal wishlist for a future version of the Java language is the possibility to declare multiline strings in an easy an elegant way – perhaps the way it is possible in Groovy using ”’.

Because we do not have this feature yet and the mapper and reducer in MongoDB are written in JavaScript – and in our case – stored in a Java string it is – let’s say the optimum to read.

Therefore I’ve added a readable version of the functions here:

Map

function () {
    this.categories.forEach(function (category) {
        emit(category, {
            count: 1
        });
    })
};

Reduce

function (key, values) {
    var sum = 0;
    for (var i = 0; i < values.length; i++)
        sum += values[i].count;

    return {
        count: sum
    };
}

BSON Output

{ "_id" : "crime" , "value" : { "count" : 1.0}}
{ "_id" : "horror" , "value" : { "count" : 2.0}}
{ "_id" : "mystery" , "value" : { "count" : 2.0}}
{ "_id" : "romance" , "value" : { "count" : 1.0}}
{ "_id" : "science" , "value" : { "count" : 2.0}}
{ "_id" : "sports" , "value" : { "count" : 1.0}}

Integration in the Maven Build Lifecycle

The embedmongo-maven-plugin allows us to start or stop an instance of MongoDB during a Maven build.

The plugin is highly configurable as shown in the example that I’ve take from the project’s documentation on GitHub.

<plugin>
    <groupId>com.github.joelittlejohn.embedmongo</groupId>
    <artifactId>embedmongo-maven-plugin</artifactId>
    <version>0.1.9</version>
    <executions>
        <execution>
            <id>start</id>
            <goals>
                <goal>start</goal>
            </goals>
            <configuration>
                <port>37017</port> <!-- optional, default 27017 -->
                <randomPort>true</randomPort> <!-- optional, default is false, if true allocates a random port and overrides embedmongo.port -->
                <version>2.0.4</version>  <!-- optional, default 2.2.1 -->
                <databaseDirectory>/tmp/mongotest</databaseDirectory>  <!-- optional, default is a new dir in java.io.tmpdir -->
                <logging>file</logging> <!-- optional (file|console|none), default console -->
                <logFile>${project.build.directory}/myfile.log</logFile> <!-- optional, can be used when logging=file, default is ./embedmongo.log -->
                <logFileEncoding>utf-8</logFileEncoding> <!-- optional, can be used when logging=file, default is utf-8 -->
                <proxyHost>myproxy.company.com</proxyHost>  <!-- optional, default is none -->
                <proxyPort>8080</proxyPort>  <!-- optional, default 80 -->
                <proxyUser>joebloggs</proxyUser>  <!-- optional, default is none -->
                <proxyPassword>pa55w0rd</proxyPassword>  <!-- optional, default is none -->
                <bindIp>127.0.0.1</bindIp> <!-- optional, default is to listen on all interfaces -->
            </configuration>
        </execution>
        <execution>
            <id>stop</id>
            <goals>
                <goal>stop</goal>
            </goals>
        </execution>
    </executions>
</plugin>

Now when running a connected Maven goal, an embedded mongodb process is spawned and destroyed afterwards .. could look similar to this Maven output:

[..]
[mongod output] Wed Oct 16 20:31:23 versionCmpTest passed
[mongod output] Wed Oct 16 20:31:23 BackgroundJob starting: DataFileSync
[mongod output] Wed Oct 16 20:31:23 versionArrayTest passed
[mongod output] Wed Oct 16 20:31:23 shardKeyTest passed
[mongod output] Wed Oct 16 20:31:23 isInRangeTest passed
[mongod output] Wed Oct 16 20:31:23 shardObjTest passed
[mongod output] Wed Oct 16 20:31:23 [initandlisten] MongoDB starting : pid=5110 port=27017 dbpath=/tmp/embedmongo-db-b28a30d3-ed1a-405e-8bde-d3fdda957d66 64-bit host=styx
[mongod output] Wed Oct 16 20:31:23 [initandlisten] db version v2.2.1, pdfile version 4.5
[mongod output] Wed Oct 16 20:31:23 [initandlisten] git version: d6764bf8dfe0685521b8bc7b98fd1fab8cfeb5ae
[mongod output] Wed Oct 16 20:31:23 [initandlisten] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49
[mongod output] Wed Oct 16 20:31:23 [initandlisten] options: { dbpath: "/tmp/embedmongo-db-b28a30d3-ed1a-405e-8bde-d3fdda957d66", noauth: true, nohttpinterface: true, nojournal: true, noprealloc: true, port: 27017, smallfiles: true, verbose: true }
[mongod output] Wed Oct 16 20:31:23 [initandlisten] Unable to check for journal files due to: boost::filesystem::basic_directory_iterator constructor: No such file or directory: "/tmp/embedmongo-db-b28a30d3-ed1a-405e-8bde-d3fdda957d66/journal"
[mongod output] Wed Oct 16 20:31:23 [initandlisten] flushing directory /tmp/embedmongo-db-b28a30d3-ed1a-405e-8bde-d3fdda957d66
[mongod output] Wed Oct 16 20:31:23 [initandlisten] opening db:  local
[mongod output] Wed Oct 16 20:31:23 [initandlisten] enter repairDatabases (to check pdfile version #)
[mongod output] Wed Oct 16 20:31:23 [initandlisten] done repairDatabases
[mongod output] Wed Oct 16 20:31:23 BackgroundJob starting: snapshot
[mongod output] Wed Oct 16 20:31:23 BackgroundJob starting: ClientCursorMonitor
[mongod output] Wed Oct 16 20:31:23 BackgroundJob starting: PeriodicTask::Runner
[mongod output] Wed Oct 16 20:31:23 [initandlisten] fd limit hard:4096 soft:4096 max conn: 3276
[mongod output] Wed Oct 16 20:31:23 BackgroundJob starting: TTLMonitor
[mongod output] Wed Oct 16 20:31:23 [initandlisten] waiting for connections on port 27017
[mongod output] [INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 5.691s
[INFO] Finished at: Wed Oct 16 20:31:23 CEST 2013
[INFO] Final Memory: 8M/204M
[INFO] ------------------------------------------------------------------------
Oct 16, 2013 8:31:23 PM de.flapdoodle.embed.mongo.AbstractMongoProcess stop
INFO: try to stop mongod
Wed Oct 16 20:31:23 [initandlisten] connection accepted from 127.0.0.1:58700 #1 (1 connection now open)
[mongod output] Wed Oct 16 20:31:23 [conn1] run command admin.$cmd { shutdown: 1, force: true }
[mongod output] Wed Oct 16 20:31:23 [conn1] terminating, shutdown command received
[mongod output] Wed Oct 16 20:31:23 dbexit: shutdown called
[mongod output] Wed Oct 16 20:31:23 [conn1] shutdown: going to close listening sockets...
[mongod output] Wed Oct 16 20:31:23 [conn1] closing listening socket: 5
[mongod output] Wed Oct 16 20:31:23 [conn1] closing listening socket: 6
[mongod output] Wed Oct 16 20:31:23 [conn1] removing socket file: /tmp/mongodb-27017.sock
[mongod output] Wed Oct 16 20:31:23 [conn1] shutdown: going to flush diaglog...
[mongod output] Wed Oct 16 20:31:23 [conn1] shutdown: going to close sockets...
[mongod output] Wed Oct 16 20:31:23 [conn1] shutdown: waiting for fs preallocator...
[mongod output] Wed Oct 16 20:31:23 [conn1] shutdown: closing all files...
[mongod output] Wed Oct 16 20:31:23 [conn1] closeAllFiles() finished
[mongod output] Wed Oct 16 20:31:23 [conn1] shutdown: removing fs lock...
[mongod output] Wed Oct 16 20:31:23 dbexit: really exiting now
[mongod output]

Tutorial Sources

Please feel free to download the tutorial sources from my GitHub repository, fork it there or clone it using Git:

git clone https://bitbucket.org/hascode/mongodb-java-testing-tutorial.git

Dependencies#

Integration Test Examples#

Map-Reduce Human Readable#

Map#

Reduce#

BSON Output#

Integration in the Maven Build Lifecycle#

Tutorial Sources#

Resources#