A lot of boilerplate code is written when developers need to test their applications with different connected systems like databases, stream platforms and other collaborators.

Docker allows to handle those dependencies but there is still some glue code required to bind the container’s lifecycle and the configuration to the concrete integration test.

Testcontainers is a testing library that offers lightweight throwaway instances of anything able to run in a Docker container, with bindings to configure the specific containers and also provides wrappers to manage our own custom containers.

In the following short tutorial I am going to demonstrate how to start Apache Kafka as well as a classical Postgresql database from a JUnit 5 integration test.

Testcontainers Integration Testing
Figure 1. Testcontainers Integration Testing

Prerequisites

We need two things for the following tutorial: Docker installed (see the exact requirements) and Maven.

Since we’re using JUnit5, we need to add the JUnit dependencies (I’ve used the JUnit BOM for these dependencies here) and also the testcontainers-junit adapter library.

Using Maven our project’s pom.xml should include the following basic dependencies:

pom.xml
<dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>org.junit</groupId>
        <artifactId>junit-bom</artifactId>
        <version>5.3.2</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
    </dependencies>
</dependencyManagement>

<dependencies>
    <!-- JUnit 5 -->
    <dependency>
      <groupId>org.junit.jupiter</groupId>
      <artifactId>junit-jupiter-api</artifactId>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.junit.jupiter</groupId>
      <artifactId>junit-jupiter-engine</artifactId>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.junit.platform</groupId>
      <artifactId>junit-platform-launcher</artifactId>
      <scope>test</scope>
    </dependency>
    <!-- Testcontainers -->
    <dependency>
      <groupId>org.testcontainers</groupId>
      <artifactId>junit-jupiter</artifactId>
      <version>${testcontainers.version}</version>
      <scope>test</scope>
    </dependency>
</dependencies>

Examples

Kafka

Our first example targets my favorite stream processing platform, Apache Kafka….

Dependencies

We need to add two dependencies .. one for the testcontainers-kafka support and the kafka client library:

<!-- Kafka Container -->
<dependency>
    <groupId>org.testcontainers</groupId>
    <artifactId>kafka</artifactId>
    <version>${testcontainers.version}</version>
    <scope>test</scope>
</dependency>

<!-- Kafka Client/Producer -->
<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
    <version>0.9.0.1</version>
</dependency>

Integration Test

This is our Kafka integration test. We’re producing and consuming some records send over the containerized Kafka instance.

This is not a real valid test but demonstrates that our container is running and available.

ContainerizedKafkaIT.java
package it;

import static org.junit.jupiter.api.Assertions.assertTrue;
import java.util.Arrays;
import java.util.Properties;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.stream.IntStream;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.Producer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.junit.Assert;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import org.testcontainers.containers.KafkaContainer;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;

@Testcontainers
public class ContainerizedKafkaIT {

  public static final String MY_TOPIC = "my-topic";
  public static final int NUMBER_OF_MESSAGES = 100;

  @Container
  public KafkaContainer kafkaContainer = new KafkaContainer();

  @Test
  @DisplayName("kafka server should be running")
  void shouldBeRunningKafka() throws Exception {
    assertTrue(kafkaContainer.isRunning());
  }

  @Test
  @DisplayName("should send and receive records over kafka")
  void shouldSendAndReceiveMessages() throws Exception {
    var servers = kafkaContainer.getBootstrapServers();
    System.out.printf("servers: %s%n", servers);

    var props = new Properties();
    props.put("bootstrap.servers", servers);
    props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
    props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
    props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
    props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
    props.put("group.id", "group-1");
    props.put("auto.offset.reset", "earliest");

    var counter = new AtomicInteger(0);
    CountDownLatch waitABit = new CountDownLatch(1);

    new Thread(() -> {
      KafkaConsumer<String, String> kafkaConsumer = new KafkaConsumer<>(props);
      kafkaConsumer.subscribe(Arrays.asList(MY_TOPIC));
      while (counter.get()<NUMBER_OF_MESSAGES) {
        ConsumerRecords<String, String> records = kafkaConsumer.poll(200);
        records.forEach(record -> {
          System.out
                  .printf("%d # offset: %d, value = %s%n", counter.incrementAndGet(), record.offset(),
                          record.value());
        });
      }
      waitABit.countDown();

    }).start();

    try (
            Producer<String, String> producer = new KafkaProducer<>(props)) {
      IntStream.range(0, NUMBER_OF_MESSAGES).forEach(i -> {
        var msg = String.format("my-message-%d", i);
        producer.send(new ProducerRecord<>(MY_TOPIC, msg));
        System.out.println("Sent:" + msg);
      });
    }

    waitABit.await();
    Assert.assertEquals(NUMBER_OF_MESSAGES, counter.get());
  }
}

The test should produce an output similar to this one but the tests needs some serious improvements, so results may differ…

Testing Kafka with Testcontainers
Figure 2. Testing Kafka with Testcontainers
 ℹ︎ Checking the system...
 ✔ Docker version should be at least 1.6.0
 ✔ Docker environment should have more than 2GB free disk space
servers: PLAINTEXT://localhost:32902
Sent:my-message-0
Sent:my-message-1
Sent:my-message-2
Sent:my-message-3
Sent:my-message-4
Sent:my-message-5
Sent:my-message-6
Sent:my-message-7
Sent:my-message-8
Sent:my-message-9
Sent:my-message-10
Sent:my-message-11
Sent:my-message-12
Sent:my-message-13
Sent:my-message-14
Sent:my-message-15
Sent:my-message-16
Sent:my-message-17
Sent:my-message-18
Sent:my-message-19
Sent:my-message-20
Sent:my-message-21
Sent:my-message-22
Sent:my-message-23
Sent:my-message-24
Sent:my-message-25
Sent:my-message-26
Sent:my-message-27
[..]
Sent:my-message-98
Sent:my-message-99
1# offset: 20, value = my-message-20
2# offset: 21, value = my-message-21
3# offset: 22, value = my-message-22
4# offset: 23, value = my-message-23
5# offset: 24, value = my-message-24
6# offset: 25, value = my-message-25
7# offset: 26, value = my-message-26
8# offset: 27, value = my-message-27
9# offset: 28, value = my-message-28
[..]
79# offset: 98, value = my-message-98
80# offset: 99, value = my-message-99

Postgresql

To provide another example, we will now write an integration test for a classical RDBMS, Postgresql...

Dependencies

We need to add the dependencies for the testcontainers-postgresql adapter and the JDBC driver needed:

<!-- Postgres Container -->
<dependency>
    <groupId>org.testcontainers</groupId>
    <artifactId>postgresql</artifactId>
    <version>${testcontainers.version}</version>
    <scope>test</scope>
</dependency>

<!-- Postgres Driver for JDBC Connection -->
<dependency>
    <groupId>org.postgresql</groupId>
    <artifactId>postgresql</artifactId>
    <version>42.2.5</version>
</dependency>

Integration Test

Again we’re writing two fast integration tests to verify that the database container is started and another one writing and reading from the started Postgresql database.

ContainerizedPostgresIT.java
package it;

import static org.junit.jupiter.api.Assertions.assertTrue;

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import org.testcontainers.containers.PostgreSQLContainer;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;

@Testcontainers
public class ContainerizedPostgresIT {

  @Container
  private PostgreSQLContainer postgresqlContainer = new PostgreSQLContainer()
      .withDatabaseName("mydb")
      .withUsername("user")
      .withPassword("secret");

  @Test
  @DisplayName("postgres should be running")
  void shouldBeRunningPostgres() throws Exception {
    System.out.printf("postgres db running, db-name: '%s', user: '%s', jdbc-url: '%s'%n ",
        postgresqlContainer.getDatabaseName(),
        postgresqlContainer.getUsername(),
        postgresqlContainer.getJdbcUrl());
    assertTrue(postgresqlContainer.isRunning());
  }

  @Test
  @DisplayName("should write and read from database")
  void shouldReadFromDatabase() throws Exception {
    Class.forName("org.postgresql.Driver");
    var connection = DriverManager
        .getConnection(postgresqlContainer.getJdbcUrl(), postgresqlContainer.getUsername(),
            postgresqlContainer.getPassword());
    connection.prepareStatement("CREATE DATABASE article_db").execute();
    connection.prepareStatement("CREATE table articles (title VARCHAR , url VARCHAR)").execute();
    connection.prepareStatement("INSERT INTO articles VALUES('Implementing Reactive Client-Server Communication over TCP or Websockets with RSocket and Java','https://www.hascode.com/2018/11/implementing-reactive-client-server-communication-over-tcp-or-websockets-with-rsocket-and-java/')").execute();
    connection.prepareStatement("INSERT INTO articles VALUES('Setting up Kafka Brokers for Testing with Kafka-Unit','https://www.hascode.com/setting-up-kafka-brokers-for-testing-with-kafka-unit/')").execute();
    connection.prepareStatement("INSERT INTO articles VALUES('Managing Architecture Decision Records with ADR-Tools','https://www.hascode.com/managing-architecture-decision-records-with-adr-tools/')").execute();

    var result = connection
        .prepareStatement("SELECT title,url FROM articles ORDER BY title ASC").executeQuery();
    result.next();
    assertTrue(result.getString("title").equals( "Implementing Reactive Client-Server Communication over TCP or Websockets with RSocket and Java"));
    result.next();
    assertTrue(result.getString("title").equals( "Managing Architecture Decision Records with ADR-Tools"));
    result.next();
    assertTrue(result.isLast() && result.getRow() == 3);
  }
}

The test should produce an output similar to this one:

Postgresql testing with Testcontainers
Figure 3. Postgresql testing with Testcontainers
ℹ︎ Checking the system...
✔ Docker version should be at least 1.6.0
✔ Docker environment should have more than 2GB free disk space
postgres db running, db-name: 'mydb', user: 'user', jdbc-url: 'jdbc:postgresql://localhost:32915/mydb'

Tutorial Sources

Please feel free to download the tutorial sources from my GitHub repository, fork it there or clone it using Git:

git clone https://github.com/hascode/testcontainers-tutorial.git

Resources

Article Updates

  • 2020-02-08: Updated the Kafka example to start reading data from the beginning (thanks @Jan Vermeir for the hint)

Other Testing Tutorials of Mine

Please feel free to have a look at other testing tutorial of mine (an excerpt):

And more…