When implementing distributed systems, client-server architectures and simple applications with network related functionalities, everything is fine when we’re in the development or in the testing stage because the network is reliable and the communicating systems are not as stressed as they are in production.
But to sleep well we want to validate how resilient we have implemented our systems, how they behave when the network fails, the latency rises, the bandwidth is limited, connections time out and so on.
In the following tutorial I will demonstrate how to set up a testing environment to simulate different classical network problems with a tool named Toxiproxy and I will show how to integrate it with the well known Java testing stack with Maven and JUnit.
About Toxiproxy
We want to simulate a test environment where we may control different possible network problems from within the scope of our (JUnit) test.
Toxiproxy allows us to create different proxy instances and add so called toxics to them to simulate typical network related problems.
Supported toxics are:
-
latency
-
down
-
bandwidth
-
slow_close
-
timeout
-
slicer
-
limit_data
A complete list with more detailed information can be found on the project’s documentation here.
The Toxiproxy Java Client allows us to control a running Toxiproxy server and to create new proxies and add toxics using a Java API.
So the communication flow when using Toxiproxy looks similar to this simplified example:
Application Under Test
Our application under tests consists of a single client class that communicates with a configurable remote service via HTTP protocol.
When not interested in the setup of our application under test, we may skip directly to the test setup.
Maven
We’re just using the new HTTP client therefore we need to use at least Java version 9 so we’re adding the following properties to our project’s pom.xml:
<properties>
[..]
<java.version>9</java.version>
<maven.compiler.source>${java.version}</maven.compiler.source>
<maven.compiler.target>${java.version}</maven.compiler.target>
</properties>
Client Application
This is our sample REST client. It exposes two methods that we’ll be testing later, one that simply calls a remote service and prints the duration of the operation.
The other method sends a specified amount of data to a remote service and again prints the duration.
We’re using the new HTTP Client added in Java 9 here.
package com.hascode.tutorial;
import java.net.URI;
import java.time.Duration;
import java.time.Instant;
import jdk.incubator.http.HttpClient;
import jdk.incubator.http.HttpClient.Version;
import jdk.incubator.http.HttpRequest;
import jdk.incubator.http.HttpRequest.BodyProcessor;
import jdk.incubator.http.HttpResponse;
public class UpstreamService {
public void callRestEndpoint(String url) {
HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create(url))
.header("Accept", "application/json")
.GET().build();
Instant start = Instant.now();
HttpClient
.newBuilder()
.version(Version.HTTP_1_1)
.build()
.sendAsync(request, HttpResponse.BodyHandler.asString()).thenApply(HttpResponse::body)
.thenAccept(System.out::println).join();
long durationInSeconds = Duration.between(start, Instant.now()).getSeconds();
System.out.printf("the request took %d seconds%n", durationInSeconds);
}
public void sendToRestEndpoint(String url, int size) {
byte[] body = new byte[size];
HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create(url))
.header("Content-type", " application/octet-stream")
.POST(BodyProcessor.fromByteArray(body)).build();
Instant start = Instant.now();
HttpClient
.newBuilder()
.version(Version.HTTP_1_1)
.build()
.sendAsync(request, HttpResponse.BodyHandler.asString()).thenApply(HttpResponse::body)
.thenAccept(System.out::println).join();
long durationInSeconds = Duration.between(start, Instant.now()).getSeconds();
System.out.printf("uploading %d kb took %d seconds%n", (size/1024), durationInSeconds);
}
}
Java 9 Module Declaration
To use the HTTP client, we need to add the following module-info.java to our project:
module toxiproxy.tutorial {
requires jdk.incubator.httpclient;
}
Test Setup
Now that we have a sample application we’re ready to create a setup to test the behavior of the application when dealing with network related problems.
Our setup is this:
-
The fabric8 Docker-Maven-Plugin starts and stops Docker instances with specified images and is bound to the Maven phase named integration-test
-
The Docker-Maven-Plugin uses the Toxiproxy Docker Image and runs a full Toxiproxy server in a container listening for control connections on port 8474
-
The Docker-Maven-Plugin is configured to wait until port 8474 is listening
-
Wiremock is used to simulate a remote REST (like) service, the life-cycle of this service is controlled using JUnit Test Rules
-
The Toxiproxy Java Client is used to create a control connection to the Toxiproxy server and create new proxies and add toxics (network problems) to them
-
The application or component under test uses one of the Toxiproxy proxies so that the traffic flows through Toxiproxy and the specified problems are added to the upstream or downstream (or both) that is delegated by the server
Dependencies
For our setup we need to add these additional dependencies to our project’s pom.xml:
-
JUnit
-
Wiremock
-
Toxiproxy Java Client
<dependencies>
<dependency>
<groupId>eu.rekawek.toxiproxy</groupId>
<artifactId>toxiproxy-java</artifactId>
<version>2.1.3</version>
</dependency>
<dependency>
<groupId>com.github.tomakehurst</groupId>
<artifactId>wiremock</artifactId>
<version>2.18.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
<scope>test</scope>
</dependency>
</dependencies>
To control the startup and shutdown of the Toxiproxy server and to bind it to the testing lifecycle, we need to add the following plugin references and configurations to our pom.xml to achieve that..
-
… before integration tests are run, the Docker image shopify/toxiproxy is started and we’re waiting until port 8474 is listening
-
… after our integration tests have run, the Docker instance is stopped and volumes are removed
<build>
<plugins>
<!-- start toxiproxy via docker and docker-maven-plugin -->
<plugin>
<groupId>io.fabric8</groupId>
<artifactId>docker-maven-plugin</artifactId>
<version>0.26.0</version>
<executions>
<execution>
<id>prepare-it-toxiproxy</id>
<phase>pre-integration-test</phase>
<goals>
<goal>start</goal>
</goals>
<configuration>
<images>
<image>
<name>shopify/toxiproxy</name>
<alias>it-toxiproxy</alias>
<run>
<network>
<mode>host</mode>
</network>
<wait>
<tcp>
<host>localhost</host>
<ports>
<port>8474</port>
</ports>
</tcp>
<kill>2000</kill>
</wait>
</run>
</image>
</images>
</configuration>
</execution>
<execution>
<id>remove-it-toxiproxy</id>
<phase>post-integration-test</phase>
<goals>
<goal>stop</goal>
</goals>
<configuration>
<removeVolumes>true</removeVolumes>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
Integration Tests
Now we may finally write our integration tests..
JUnit Test Setup / Teardown
Before our tests run, we’re starting a new Wiremock instance, this is done using a simple TestRule.
In addition we’re creating the client connection to the ToxiProxy server.
When tearing down our application, we’re removing the proxy we created:
@Rule
public WireMockRule wireMockRule = new WireMockRule(9999);
ToxiproxyClient client;
Proxy httpProxy;
@Before
public void setup() throws Exception {
client = new ToxiproxyClient("127.0.0.1", 8474);
httpProxy = client.createProxy("http-tproxy", "127.0.0.1:8888", "127.0.0.1:9999");
}
@After
public void teardown() throws Exception {
httpProxy.delete();
}
Latency Test
In our first test, we want to test the behavior of our REST client when there is a latency of 12 seconds added to each connection (downstream) :
@Test
public void latencyTest() throws Exception {
// create toxic
httpProxy.toxics().latency("latency-toxic", ToxicDirection.DOWNSTREAM, 12_000).setJitter(15);
// create fake rest endpoint
stubFor(get(urlEqualTo("/rs/date"))
.withHeader("Accept", equalTo("application/json"))
.willReturn(aResponse()
.withStatus(200)
.withHeader("Content-Type", "application/json")
.withBody(String.format("{\"now\":\"%s\"}", LocalDateTime.now()))));
// call rest service over toxiproxy
UpstreamService upstreamService = new UpstreamService();
upstreamService.callRestEndpoint("http://localhost:8888/rs/date");
// verify something happened
verify(getRequestedFor(urlMatching("/rs/date"))
.withHeader("Accept", matching("application/json")));
}
Limited Bandwidth Test
In our second test setup we want to test the behavior of our application when it is forced to deal with limited bandwidth .. e.g. 1.5Mbit upstream.
@Test
public void bandWidthTest() throws Exception {
// create toxic with 1.5Mbit bandwidth limit
httpProxy.toxics().bandwidth("bandwidth-toxic", ToxicDirection.UPSTREAM, 150);
// create fake rest endpoint
stubFor(post(urlEqualTo("/rs/data/upload"))
.withHeader("Content-type", equalTo("application/octet-stream"))
.willReturn(aResponse()
.withStatus(200)
.withHeader("Content-Type", "text-plain")
.withBody("data received")));
// call rest service over toxiproxy
UpstreamService upstreamService = new UpstreamService();
upstreamService.sendToRestEndpoint("http://localhost:8888/rs/data/upload", 1_048_576);
// verify something happened
verify(postRequestedFor(urlMatching("/rs/data/upload"))
.withHeader("Content-type", matching("application/octet-stream")));
}
Running the Tests
At last, we may run the tests in our IDE or using Maven and our beloved command-line like this:
$ mvn integration-test
[..]
[INFO] --- docker-maven-plugin:0.26.0:start (prepare-it-toxiproxy) @ toxiproxy-tutorial ---
[INFO] DOCKER> [shopify/toxiproxy:latest] "it-toxiproxy": Start container cf2f0b20f6c1
[INFO] DOCKER> [shopify/toxiproxy:latest] "it-toxiproxy": Waiting for ports [8474] directly on container with IP ().
[INFO] DOCKER> [shopify/toxiproxy:latest] "it-toxiproxy": Waited on tcp port '[localhost/127.0.0.1:8474]' 521 ms
[INFO]
[INFO] --- maven-failsafe-plugin:2.22.0:integration-test (default) @ toxiproxy-tutorial ---
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running it.RestConnectionIT
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
{"now":"2018-07-29T16:10:38.042490"}
the request took 12 seconds
data received
uploading 1024 kb took 7 seconds
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.272 s - in it.RestConnectionIT
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
We may see, that …
-
… a Docker image indeed has been started
-
… that we were waiting until the designated TCP port was available
-
… that in our first test, some latency was added to the request
-
… that in the second test the limited bandwidth caused some delay when uploading data
Toxiproxy and Docker
Of course we may control Toxiproxy and inspect its state directly using Docker, so the following snippets might be helpful:
First of all we should pull the Docker image:
docker pull shopify/toxiproxy
I’m often writing the instance ID to an environment variable for a quicker access … e.g.:
exportTPROXY_ID=`docker run --rm-td--net=host shopify/toxiproxy`
Now I may use the toxiproxy-cli binaries that reside in /go/bin.
List Existing Proxies
This command list all proxy instances that we have created with their state and their listening interface and their upstream destination:
docker exec -it $TPROXY_ID /bin/sh -c '/go/bin/toxiproxy-cli ls'
Name Listen Upstream Enabled Toxics
======================================================================================
http-tproxy 127.0.0.1:8888 app.hascode.com:80 enabled 1
Inspect Toxics for a Proxy
If a proxy exists, we may inspect it to see which toxics are assigned:
docker exec -it $TPROXY_ID /bin/sh -c '/go/bin/toxiproxy-cli inspect http-tproxy'
Name: http-tproxy Listen: 127.0.0.1:8888 Upstream: app.hascode.com:80
======================================================================
Upstream toxics:
Proxy has no Upstream toxics enabled.
Downstream toxics:
latency-toxic: type=latency stream=downstream toxicity=1.00 attributes=[ jitter=15 latency=20 ]
Hint: add a toxic with `toxiproxy-cli toxic add`
For more detailed information, please feel free to consult the Toxiproxy documentation.
Tutorial Sources
Please feel free to download the tutorial sources from my GitHub repository, fork it there or clone it using Git:
git clone https://github.com/hascode/toxiproxy-tutorial.git
Troubleshooting
-
“[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile (default-testCompile) on project: Fatal error compiling: invalid target release: 1.9 → [Help 1]“ With Java 9 the new version-string-scheme has changed and therefore 1.9 is not valid. For more details, please consult the following announcement from Oracle.
-
“java.lang.NoClassDefFoundError: jdk/incubator/http/HttpRequest
at it.RestConnectionIT.latencyTest(RestConnectionIT.java:60)
Caused by: java.lang.ClassNotFoundException: jdk.incubator.http.HttpRequest
at it.RestConnectionIT.latencyTest(RestConnectionIT.java:60)” We need to add the module to our test invocation. One way is to add the following configuration for the Failsafe Plugin to our pom.xml:<plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-failsafe-plugin</artifactId> <version>2.22.0</version> <configuration> <argLine>--add-modules jdk.incubator.httpclient</argLine> </configuration> [..] </plugin>
-
Adam Bien has written a nice article about using the Java 9 HTTP client with JUnit and Maven here.
-
-
“[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project toxiproxy-tutorial: Compilation failure: Compilation failure:
[ERROR] /data/project/toxiproxy-tutorial/src/main/java/com/hascode/tutorial/UpstreamService.java:[6,21] package jdk.incubator.http is not visible
[ERROR] (package jdk.incubator.http is declared in module jdk.incubator.httpclient, which is not in the module graph)” We need to add our module descriptor (module-info.java) with the requirement for jdk.incubator.httpclient like this one:module toxiproxy.tutorial { requires jdk.incubator.httpclient; }
Other Testing Tutorials of Mine
Please feel free to have a look at other testing tutorial of mine (an excerpt):
And more…