When implementing distributed systems, client-server architectures and simple applications with network related functionalities, everything is fine when we’re in the development or in the testing stage because the network is reliable and the communicating systems are not as stressed as they are in production.

But to sleep well we want to validate how resilient we have implemented our systems, how they behave when the network fails, the latency rises, the bandwidth is limited, connections time out and so on.

In the following tutorial I will demonstrate how to set up a testing environment to simulate different classical network problems with a tool named Toxiproxy and I will show how to integrate it with the well known Java testing stack with Maven and JUnit.

JUnit Setup using Toxiproxy
Figure 1. JUnit Setup using Toxiproxy, Docker, Maven, Wiremock

About Toxiproxy

We want to simulate a test environment where we may control different possible network problems from within the scope of our (JUnit) test.

Toxiproxy allows us to create different proxy instances and add so called toxics to them to simulate typical network related problems.

Supported toxics are:

  • latency

  • down

  • bandwidth

  • slow_close

  • timeout

  • slicer

  • limit_data

A complete list with more detailed information can be found on the project’s documentation here.

The Toxiproxy Java Client allows us to control a running Toxiproxy server and to create new proxies and add toxics using a Java API.

So the communication flow when using Toxiproxy looks similar to this simplified example:

Toxiproxy communication flow
Figure 2. Toxiproxy communication flow

Application Under Test

Our application under tests consists of a single client class that communicates with a configurable remote service via HTTP protocol.

When not interested in the setup of our application under test, we may skip directly to the test setup.

Maven

We’re just using the new HTTP client therefore we need to use at least Java version 9 so we’re adding the following properties to our project’s pom.xml:

<properties>
	[..]
	<java.version>9</java.version>
	<maven.compiler.source>${java.version}</maven.compiler.source>
	<maven.compiler.target>${java.version}</maven.compiler.target>
</properties>

Client Application

This is our sample REST client. It exposes two methods that we’ll be testing later, one that simply calls a remote service and prints the duration of the operation.

The other method sends a specified amount of data to a remote service and again prints the duration.

We’re using the new HTTP Client added in Java 9 here.

package com.hascode.tutorial;

import java.net.URI;
import java.time.Duration;
import java.time.Instant;
import jdk.incubator.http.HttpClient;
import jdk.incubator.http.HttpClient.Version;
import jdk.incubator.http.HttpRequest;
import jdk.incubator.http.HttpRequest.BodyProcessor;
import jdk.incubator.http.HttpResponse;

public class UpstreamService {

  public void callRestEndpoint(String url) {
    HttpRequest request = HttpRequest.newBuilder()
        .uri(URI.create(url))
        .header("Accept", "application/json")
        .GET().build();
    Instant start = Instant.now();
    HttpClient
        .newBuilder()
        .version(Version.HTTP_1_1)
        .build()
        .sendAsync(request, HttpResponse.BodyHandler.asString()).thenApply(HttpResponse::body)
        .thenAccept(System.out::println).join();
    long durationInSeconds = Duration.between(start, Instant.now()).getSeconds();
    System.out.printf("the request took %d seconds%n", durationInSeconds);
  }

  public void sendToRestEndpoint(String url, int size) {
    byte[] body = new byte[size];

    HttpRequest request = HttpRequest.newBuilder()
        .uri(URI.create(url))
        .header("Content-type", " application/octet-stream")
        .POST(BodyProcessor.fromByteArray(body)).build();
    Instant start = Instant.now();
    HttpClient
        .newBuilder()
        .version(Version.HTTP_1_1)
        .build()
        .sendAsync(request, HttpResponse.BodyHandler.asString()).thenApply(HttpResponse::body)
        .thenAccept(System.out::println).join();
    long durationInSeconds = Duration.between(start, Instant.now()).getSeconds();
    System.out.printf("uploading %d kb took %d seconds%n", (size/1024), durationInSeconds);
  }
}

Java 9 Module Declaration

To use the HTTP client, we need to add the following module-info.java to our project:

module-info.java
module toxiproxy.tutorial {
  requires jdk.incubator.httpclient;
}

Test Setup

Now that we have a sample application we’re ready to create a setup to test the behavior of the application when dealing with network related problems.

Our setup is this:

  • The fabric8 Docker-Maven-Plugin starts and stops Docker instances with specified images and is bound to the Maven phase named integration-test

  • The Docker-Maven-Plugin uses the Toxiproxy Docker Image and runs a full Toxiproxy server in a container listening for control connections on port 8474

  • The Docker-Maven-Plugin is configured to wait until port 8474 is listening

  • Wiremock is used to simulate a remote REST (like) service, the life-cycle of this service is controlled using JUnit Test Rules

  • The Toxiproxy Java Client is used to create a control connection to the Toxiproxy server and create new proxies and add toxics (network problems) to them

  • The application or component under test uses one of the Toxiproxy proxies so that the traffic flows through Toxiproxy and the specified problems are added to the upstream or downstream (or both) that is delegated by the server

JUnit Setup using Toxiproxy
Figure 3. JUnit Setup using Toxiproxy, Docker, Maven, Wiremock

Dependencies

For our setup we need to add these additional dependencies to our project’s pom.xml:

  • JUnit

  • Wiremock

  • Toxiproxy Java Client

pom.xml
<dependencies>
	<dependency>
	  <groupId>eu.rekawek.toxiproxy</groupId>
	  <artifactId>toxiproxy-java</artifactId>
	  <version>2.1.3</version>
	</dependency>
	<dependency>
	  <groupId>com.github.tomakehurst</groupId>
	  <artifactId>wiremock</artifactId>
	  <version>2.18.0</version>
	  <scope>test</scope>
	</dependency>
	<dependency>
	  <groupId>junit</groupId>
	  <artifactId>junit</artifactId>
	  <version>4.12</version>
	  <scope>test</scope>
	</dependency>
</dependencies>

To control the startup and shutdown of the Toxiproxy server and to bind it to the testing lifecycle, we need to add the following plugin references and configurations to our pom.xml to achieve that..

  • … before integration tests are run, the Docker image shopify/toxiproxy is started and we’re waiting until port 8474 is listening

  • … after our integration tests have run, the Docker instance is stopped and volumes are removed

<build>
	<plugins>
	  <!-- start toxiproxy via docker and docker-maven-plugin -->
	  <plugin>
		<groupId>io.fabric8</groupId>
		<artifactId>docker-maven-plugin</artifactId>
		<version>0.26.0</version>
		<executions>
		  <execution>
			<id>prepare-it-toxiproxy</id>
			<phase>pre-integration-test</phase>
			<goals>
			  <goal>start</goal>
			</goals>
			<configuration>
			  <images>
				<image>
				  <name>shopify/toxiproxy</name>
				  <alias>it-toxiproxy</alias>
				  <run>
					<network>
					  <mode>host</mode>
					</network>
					<wait>
					  <tcp>
						<host>localhost</host>
						<ports>
						  <port>8474</port>
						</ports>
					  </tcp>
					  <kill>2000</kill>
					</wait>
				  </run>
				</image>
			  </images>
			</configuration>
		  </execution>
		  <execution>
			<id>remove-it-toxiproxy</id>
			<phase>post-integration-test</phase>
			<goals>
			  <goal>stop</goal>
			</goals>
			<configuration>
			  <removeVolumes>true</removeVolumes>
			</configuration>
		  </execution>
		</executions>
	  </plugin>
	</plugins>
</build>

Integration Tests

Now we may finally write our integration tests..

JUnit Test Setup / Teardown

Before our tests run, we’re starting a new Wiremock instance, this is done using a simple TestRule.

In addition we’re creating the client connection to the ToxiProxy server.

When tearing down our application, we’re removing the proxy we created:

@Rule
public WireMockRule wireMockRule = new WireMockRule(9999);

ToxiproxyClient client;
Proxy httpProxy;

@Before
public void setup() throws Exception {
	client = new ToxiproxyClient("127.0.0.1", 8474);
	httpProxy = client.createProxy("http-tproxy", "127.0.0.1:8888", "127.0.0.1:9999");
}

@After
	public void teardown() throws Exception {
	httpProxy.delete();
}

Latency Test

In our first test, we want to test the behavior of our REST client when there is a latency of 12 seconds added to each connection (downstream) :

@Test
public void latencyTest() throws Exception {
	// create toxic
	httpProxy.toxics().latency("latency-toxic", ToxicDirection.DOWNSTREAM, 12_000).setJitter(15);

	// create fake rest endpoint
	stubFor(get(urlEqualTo("/rs/date"))
		.withHeader("Accept", equalTo("application/json"))
		.willReturn(aResponse()
			.withStatus(200)
			.withHeader("Content-Type", "application/json")
			.withBody(String.format("{\"now\":\"%s\"}", LocalDateTime.now()))));

	// call rest service over toxiproxy
	UpstreamService upstreamService = new UpstreamService();
	upstreamService.callRestEndpoint("http://localhost:8888/rs/date");

	// verify something happened
	verify(getRequestedFor(urlMatching("/rs/date"))
		.withHeader("Accept", matching("application/json")));
}

Limited Bandwidth Test

In our second test setup we want to test the behavior of our application when it is forced to deal with limited bandwidth .. e.g. 1.5Mbit upstream.

@Test
public void bandWidthTest() throws Exception {
	// create toxic with 1.5Mbit bandwidth limit
	httpProxy.toxics().bandwidth("bandwidth-toxic", ToxicDirection.UPSTREAM, 150);

	// create fake rest endpoint
	stubFor(post(urlEqualTo("/rs/data/upload"))
		.withHeader("Content-type", equalTo("application/octet-stream"))
		.willReturn(aResponse()
			.withStatus(200)
			.withHeader("Content-Type", "text-plain")
			.withBody("data received")));

	// call rest service over toxiproxy
	UpstreamService upstreamService = new UpstreamService();
	upstreamService.sendToRestEndpoint("http://localhost:8888/rs/data/upload", 1_048_576);

	// verify something happened
	verify(postRequestedFor(urlMatching("/rs/data/upload"))
		.withHeader("Content-type", matching("application/octet-stream")));
}

Running the Tests

At last, we may run the tests in our IDE or using Maven and our beloved command-line like this:

$ mvn integration-test
[..]
[INFO] --- docker-maven-plugin:0.26.0:start (prepare-it-toxiproxy) @ toxiproxy-tutorial ---
[INFO] DOCKER> [shopify/toxiproxy:latest] "it-toxiproxy": Start container cf2f0b20f6c1
[INFO] DOCKER> [shopify/toxiproxy:latest] "it-toxiproxy": Waiting for ports [8474] directly on container with IP ().
[INFO] DOCKER> [shopify/toxiproxy:latest] "it-toxiproxy": Waited on tcp port '[localhost/127.0.0.1:8474]' 521 ms
[INFO]
[INFO] --- maven-failsafe-plugin:2.22.0:integration-test (default) @ toxiproxy-tutorial ---
[INFO]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running it.RestConnectionIT
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
{"now":"2018-07-29T16:10:38.042490"}
the request took 12 seconds
data received
uploading 1024 kb took 7 seconds
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.272 s - in it.RestConnectionIT
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------

We may see, that …

  • … a Docker image indeed has been started

  • … that we were waiting until the designated TCP port was available

  • … that in our first test, some latency was added to the request

  • … that in the second test the limited bandwidth caused some delay when uploading data

Toxiproxy and Docker

Of course we may control Toxiproxy and inspect its state directly using Docker, so the following snippets might be helpful:

First of all we should pull the Docker image:

docker pull shopify/toxiproxy

I’m often writing the instance ID to an environment variable for a quicker access … e.g.:

exportTPROXY_ID=`docker run --rm-td--net=host shopify/toxiproxy`

Now I may use the toxiproxy-cli binaries that reside in /go/bin.

List Existing Proxies

This command list all proxy instances that we have created with their state and their listening interface and their upstream destination:

docker exec -it $TPROXY_ID /bin/sh -c '/go/bin/toxiproxy-cli ls'
Name                    Listen          Upstream                Enabled         Toxics
======================================================================================
http-tproxy             127.0.0.1:8888  app.hascode.com:80      enabled         1

Inspect Toxics for a Proxy

If a proxy exists, we may inspect it to see which toxics are assigned:

docker exec -it $TPROXY_ID /bin/sh -c '/go/bin/toxiproxy-cli inspect http-tproxy'
Name: http-tproxy       Listen: 127.0.0.1:8888  Upstream: app.hascode.com:80
======================================================================
Upstream toxics:
Proxy has no Upstream toxics enabled.

Downstream toxics:
latency-toxic:  type=latency    stream=downstream       toxicity=1.00   attributes=[    jitter=15       latency=20      ]

Hint: add a toxic with `toxiproxy-cli toxic add`

For more detailed information, please feel free to consult the Toxiproxy documentation.

Tutorial Sources

Please feel free to download the tutorial sources from my GitHub repository, fork it there or clone it using Git:

git clone https://github.com/hascode/toxiproxy-tutorial.git

Troubleshooting

  • “[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile (default-testCompile) on project: Fatal error compiling: invalid target release: 1.9 → [Help 1]“ With Java 9 the new version-string-scheme has changed and therefore 1.9 is not valid. For more details, please consult the following announcement from Oracle.

  • “java.lang.NoClassDefFoundError: jdk/incubator/http/HttpRequest
    at it.RestConnectionIT.latencyTest(RestConnectionIT.java:60)
    Caused by: java.lang.ClassNotFoundException: jdk.incubator.http.HttpRequest
    at it.RestConnectionIT.latencyTest(RestConnectionIT.java:60)”
    We need to add the module to our test invocation. One way is to add the following configuration for the Failsafe Plugin to our pom.xml:

    <plugin>
    	<groupId>org.apache.maven.plugins</groupId>
    	<artifactId>maven-failsafe-plugin</artifactId>
    	<version>2.22.0</version>
    	<configuration>
    	  <argLine>--add-modules jdk.incubator.httpclient</argLine>
    	</configuration>
    	[..]
    </plugin>
    1. Adam Bien has written a nice article about using the Java 9 HTTP client with JUnit and Maven here.

  • “[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project toxiproxy-tutorial: Compilation failure: Compilation failure:
    [ERROR] /data/project/toxiproxy-tutorial/src/main/java/com/hascode/tutorial/UpstreamService.java:[6,21] package jdk.incubator.http is not visible
    [ERROR] (package jdk.incubator.http is declared in module jdk.incubator.httpclient, which is not in the module graph)”
    We need to add our module descriptor (module-info.java) with the requirement for jdk.incubator.httpclient like this one:

    module toxiproxy.tutorial {
      requires jdk.incubator.httpclient;
    }