There is a new testing framework out there called Karate that is build on top of the popular Cucumber framework.

Karate makes it easy to script interactions with out web-services under test and verify the results. In addition it offers us a lot of useful features like parallelization, script/feature re-use, data-tables, JsonPath and XPath support, gherkin syntax, switchable staging-configurations and many others.

In the following tutorial we’ll be writing different scenarios and features for a real-world RESTful web-service to demonstrate some of its features.

karate bdd testing
Figure 1. Karate BDD Testing

Dependencies

Using Maven here, we just need to add one dependency for karate-junit4 when using JUnit or alternatively karate-testng when using TestNG to our project’s pom.xml:

<dependency>
    <groupId>com.intuit.karate</groupId>
    <artifactId>karate-junit4</artifactId>
    <version>0.2.9</version>
</dependency>

REST Resource under Test

Our sample REST resource allows us to ..

  • List all existing users

  • Create a new user

  • Update a user

  • Delete a user

  • Get a user (by id)

  • Authenticate a user (returning an auth-token)

  • Access a secured resource (having a valid auth-token)

More detailed information about the REST resource specification can be found in Appendix A: REST Resource Specification, all details about its implementation in Appendix B: REST Resource Implementation.

Setting up Karate with JUnit

We’re using JUnit as test framework here, so the only thing we need to setup to get our Karate tests running, is an entry class for JUnit, a configuration file for Karate in JavaScript format and our scenario file(s).

Test Structure

This is what our directory/file structure for the following testing examples looks like:

src/test
├── java
│   └── feature
│       └── user
│           └── UserManagementTest.java
└── resources
    ├── feature
    │   ├── security
    │   │   └── user-login.feature
    │   └── user
    │       └── user-management.feature
    └── karate-config.js

JUnit Runner

This simple class allows us to run our tests with the JUnit framework.

We may fine-tune which scenarios to include and where to find it using additional configuration but if we keep this test classes’ package and the scenario file’s path in sync, we don’t need to add anything further than the following two lines to our feature.user.UserManagementTest class.

More detailed information about naming conventions can be found in the project’s wiki here, detailed information about adding Cucumber options are available here.

package feature.user;

import org.junit.runner.RunWith;

import com.intuit.karate.junit4.Karate;

@RunWith(Karate.class)
public class UserManagementTest {}

Karate Configuration

This is our Karate configuration file named karate-config.js.

We’re specifying the base-url for our tests here (our REST service runs on localhost, port 9000).

For more detailed information about the Karate configuration file, please consult the dedicated section in the project’s wiki.

function() {
	return {
		baseUrl: 'http://localhost:9000'
	}
}

Scenarios

We’re now ready to write our scenarios and features down using Karate’s DSL.

As I am lazy I have written everything in one file in my project but separated into multiple parts in this tutorial:

Create and Fetch Users

In our first scenario, we’re going to do some CRUD operations so we’re going to ..

  • Verify that no user does exist in our application

  • Create a new user

  • Verify the created user

  • Verify we find the created user using his identifier

Feature: User management

Background:
* url baseUrl

Scenario: List existing users

Given path '/user'
When method GET
Then status 200
And match $ == []

Scenario: Create and receive new user

Given path '/user'
And request {id:'', name:'Fred', age:22, password:'abc123'}
When method POST
Then status 200
And match response == {id:'#uuid', name:'Fred', age:22, password:'abc123'}
And def user = response

Given path '/user/'+user.id
When method GET
Then status 200
And match $ == {id:'#(user.id)', name:'#(user.name)', age:#(user.age), password:'#(user.password)'}

Using Data Tables

Another nice features is to verify using data tables so this is our rewritten scenario executed for five users:

Feature: User management

Background:
* url baseUrl

Scenario Outline: Create multiple users and verify their id, name and age

Given path '/user'
And request {id:'', name:'<name>', age: <age>}
When method POST
Then status 200
And match $ == {id:'#uuid', name: '<name>', age: <age>}
And def user = response

Given path '/user/'+user.id
When method GET
And status 200
And match $ == {id:'#(user.id)', name:'#(user.name)', age:#(user.age)}

Examples:
| name  | age |
| Tim   | 42  |
| Liz   | 16  |
| Selma | 65  |
| Ted   | 12  |
| Luise | 19  |

Re-using Features

Another interesting feature is the possibility to re-use existing features by loading them into another feature.

In the following example there’s a secured resource that can only be accessed with a valid auth-token.

To obtain the auth-token, a user needs request a new auth-token using his credentials.

To re-use the login procedure, we’re storing it in a feature file named user-login.feature.

It’s important to know that we need to pass the user’s identifier and password to the feature and that the last line of the feature file specifies a variable named authToken that we may use then in our original feature file.

Feature: User management

Background:
* url baseUrl

Scenario: Login user

Given path '/user/login'
And request {id: '#(id)', password:'#(password)'}
When method POST
Then status 200
And match $ == {authToken:'#notnull'}
And def authToken = $.authToken

This is our scenario where a new created user tries to login, we’re verifying that the resource is not accessible without the security-token and then we’re calling the feature above to obtain the auth-token.

With this auth-token, we’re able to access the secured service.

Feature: User management

Background:
* url baseUrl

Scenario: Create and login user

Given path '/user'
And request {id:'', name:'Eleonore', age:31, password:'foobar'}
When method POST
Then status 200
And def user = response

Given path '/user/secured/date'
When method GET
Then status 403

Given path '/user/secured/date'
And def userLogin = call read('classpath:feature/security/user-login.feature') {id:'#(user.id)', password:'#(user.password)'}
And header Auth-Token = userLogin.authToken
When method GET
Then status 200
And match $ == {date:'#notnull'}

Using Assertions

We may use assertions to verify special conditions e.g. to assure non-functional requirements like the response-time:

Feature: User management

Background:
* url baseUrl

Scenario: Remove all users
Given path '/user'
When method DELETE
Then status 200
And assert responseTime < 1000

Given path '/user'
When method GET
Then status 200
And match $ == []
And assert responseTime < 1000

Running the Tests

We may run the tests now using our IDE of choice or using Maven in the command line.

I’m running the tests on Bitbucket Pipelines, please feel free to take a look at the full test execution there.

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running feature.user.UserManagementTest
13:02:44.279 [main] DEBUG com.intuit.karate.cucumber.CucumberRunner - init test class: class feature.user.UserManagementTest
13:02:44.390 [main] DEBUG com.intuit.karate.cucumber.CucumberRunner - loading feature: /data/project/karate-bdd-testing/target/test-classes/feature/user/user-management.feature
13:02:45.769 [main] DEBUG com.intuit.karate.LoggingFilter -
1 > GET http://localhost:9000/user

13:02:45.852 [qtp1279309678-23] INFO com.hascode.tutorial.UserResource - 0 users found
13:02:46.285 [main] DEBUG com.intuit.karate.LoggingFilter -
1 < 200
1 < Content-Length: 2
1 < Content-Type: application/json
1 < Date: Thu, 06 Apr 2017 11:02:45 GMT
1 < Server: Jetty(9.4.3.v20170317)
[]

[..]
1 > POST http://localhost:9000/user
1 > Content-Type: application/json
{"id":"","name":"Fred","age":22,"password":"abc123"}

13:02:46.481 [qtp1279309678-22] INFO com.hascode.tutorial.UserResource - new user User [name=Fred, id=5746BAF9-CAFE-4E37-8F94-145FBEB148B8, age=22, password=abc123] saved
13:02:46.484 [main] DEBUG com.intuit.karate.LoggingFilter -
1 < 200
1 < Content-Length: 88
1 < Content-Type: application/json
1 < Date: Thu, 06 Apr 2017 11:02:46 GMT
1 < Server: Jetty(9.4.3.v20170317)
{"age":22,"id":"5746BAF9-CAFE-4E37-8F94-145FBEB148B8","name":"Fred","password":"abc123"}
[..]
9 Scenarios (9 passed)
109 Steps (109 passed)
0m2.345s

Tests run: 118, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.36 sec

Results :

Tests run: 118, Failures: 0, Errors: 0, Skipped: 0

Running the tests in Eclipse IDE:

running karate tests junit in eclipse 1024x793
Figure 2. Running Karate Tests in Eclipse IDE

Tutorial Sources

Please feel free to download the tutorial sources from my GitHub repository, fork it there or clone it using Git:

git clone https://github.com/hascode/karate-bdd-testing.git

Is this really BDD?

No, the framework’s author, Peter Thomas has written a detailed article about this topic, please feel free to read: "Yes, Karate is not true BDD".

REST Testing Articles of mine

Troubleshooting

  • "java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NoClassDefFoundError: org/eclipse/jetty/util/Decorator at com.hascode.tutorial.RestServer.main(RestServer.java:14) … 6 more Caused by: java.lang.ClassNotFoundException: org.eclipse.jetty.util.Decorator at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) … 7 more" → The dependency jersey-container-jetty-http uses an older version of jetty-util, adding an exclusion to this dependency solves the problem:

    <dependency>
        <groupId>org.glassfish.jersey.containers</groupId>
        <artifactId>jersey-container-jetty-http</artifactId>
        <version>${jersey.version}</version>
        <exclusions>
            <exclusion>
                <groupId>org.eclipse.jetty</groupId>
                <artifactId>jetty-util</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
  • "[ERROR] Failed to execute goal com.sebastian-daschner:jaxrs-analyzer-maven-plugin:0.14:analyze-jaxrs (default) on project karate-bdd-testing: Execution default of goal com.sebastian-daschner:jaxrs-analyzer-maven-plugin:0.14:analyze-jaxrs failed: Unable to load the mojo ‘analyze-jaxrs’ (or one of its required components) from the plugin ‘com.sebastian-daschner:jaxrs-analyzer-maven-plugin:0.14′: com.google.inject.ProvisionException: Guice provision errors:
    [ERROR]
    [ERROR] 1) No implementation for org.eclipse.aether.RepositorySystem was bound.
    [ERROR] while locating com.sebastian_daschner.jaxrs_analyzer.maven.JAXRSAnalyzerMojo
    [ERROR] at ClassRealm[plugin>com.sebastian-daschner:jaxrs-analyzer-maven-plugin:0.14, parent: sun.misc.Launcher$AppClassLoader@4e25154f]
    [ERROR] while locating org.apache.maven.plugin.Mojo annotated with @com.google.inject.name.Named(value=com.sebastian-daschner:jaxrs-analyzer-maven-plugin:0.14:analyze-jaxrs)
    [ERROR]
    [ERROR] 1 error
    [ERROR] role: org.apache.maven.plugin.Mojo
    [ERROR] roleHint: com.sebastian-daschner:jaxrs-analyzer-maven-plugin:0.14:analyze-jaxrs
    [ERROR] -> [Help 1]
    "  – Please use Maven in version >= 3.2.1

Appendix A: REST Resource Specification

I have generated a plain-text description of the REST resource using the JAX-RS Analyzer Maven Plugin.

If you’re interested in other possibilities to derive documentation from existing JAX-RS web-services in AsciiDoc, Swagger or plain-text format, please feel free to read my tutorial: "Documenting RESTful Webservices in Swagger, AsciiDoc and Plain-Text with Maven and the JAX-RS Analyzer".

REST resources of karate-bdd-testing:
1.0.0

GET user:
 Request:
  No body

 Response:
  Content-Type: application/json
  Status Codes: 200
   Response Body: com.hascode.tutorial.UserResource$1
    {}

POST user:
 Request:
  Content-Type: application/json
  Request Body: com.hascode.tutorial.User
   {"age":0,"id":"string","name":"string","password":"string"}

 Response:
  Content-Type: application/json
  Status Codes: 200
   Response Body: com.hascode.tutorial.User
    {"age":0,"id":"string","name":"string","password":"string"}

DELETE user:
 Request:
  No body

 Response:
  Content-Type: */*
  Status Codes: 200

POST user/login:
 Request:
  Content-Type: */*
  Request Body: com.hascode.tutorial.UserResource$Credential
   {"id":"string","password":"string"}

 Response:
  Content-Type: */*
  Status Codes: 200
   Response Body: java.lang.String

  Status Codes: 401

GET user/secured/date:
 Request:
  No body
  Header Param: Auth-Token, java.lang.String

 Response:
  Content-Type: application/json
  Status Codes: 200
   Response Body: java.lang.String

  Status Codes: 403

GET user/{userId}:
 Request:
  No body
  Path Param: userId, java.lang.String

 Response:
  Content-Type: application/json
  Status Codes: 200
   Response Body:

Appendix B: REST Resource Implementation

This is the implementation of our RESTful web-service under test.

Using a tool like Wiremock would have been easier than implementing a real application but I’d like to simulate the usage of Karate with a real application.

Dependencies

We need to add multiple dependencies to our project’s pom.xml here. It is important to handle the possible version conflict of jetty-utils here by excluding it from jersey-container-jetty-http – for more information take a look at the troubleshooting section.

<dependency>
    <groupId>org.eclipse.jetty</groupId>
    <artifactId>jetty-server</artifactId>
    <version>${jetty.version}</version>
</dependency>
<dependency>
    <groupId>org.eclipse.jetty</groupId>
    <artifactId>jetty-servlet</artifactId>
    <version>${jetty.version}</version>
</dependency>
<dependency>
    <groupId>org.glassfish.jersey.core</groupId>
    <artifactId>jersey-server</artifactId>
    <version>${jersey.version}</version>
</dependency>
<dependency>
    <groupId>org.glassfish.jersey.containers</groupId>
    <artifactId>jersey-container-servlet-core</artifactId>
    <version>${jersey.version}</version>
</dependency>
<dependency>
    <groupId>org.glassfish.jersey.containers</groupId>
    <artifactId>jersey-container-jetty-http</artifactId>
    <version>${jersey.version}</version>
    <exclusions>
        <exclusion>
            <groupId>org.eclipse.jetty</groupId>
            <artifactId>jetty-util</artifactId>
        </exclusion>
    </exclusions>
</dependency>
<dependency>
    <groupId>org.glassfish.jersey.media</groupId>
    <artifactId>jersey-media-moxy</artifactId>
    <version>${jersey.version}</version>
</dependency>

User Entity

This is our user entity, serialized via JAX-B, setters, getters, toString etc. ommitted..

package com.hascode.tutorial;

import javax.xml.bind.annotation.XmlRootElement;

@XmlRootElement
public class User {
    private String name;
    private String id;
    private int age;
    private String password;
}

User Resource

This is our JAX-RS based user resource storing users and credentials in a simple hashmap – it’s just for demonstration ;)

For the path "secured" we’re delegating to an sub-resource responsible for secured resources. If you’re interested in such tricks, please feel free to have a look at my article: "JAX-RS Server API Snippets".

package com.hascode.tutorial;

import java.math.BigInteger;
import java.security.SecureRandom;
import java.util.Collection;
import java.util.Map;
import java.util.UUID;
import java.util.concurrent.ConcurrentHashMap;

import javax.ws.rs.Consumes;
import javax.ws.rs.DELETE;
import javax.ws.rs.GET;
import javax.ws.rs.POST;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
import javax.ws.rs.Produces;
import javax.ws.rs.core.GenericEntity;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import javax.xml.bind.annotation.XmlRootElement;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

@Path("/user")
public class UserResource {
    private static final Logger LOG = LoggerFactory.getLogger(UserResource.class);

    private static final Map<String, User> USER_STORE = new ConcurrentHashMap<>();
    private static final Map<String, String> TOKEN_TO_USERID = new ConcurrentHashMap<>();

    @XmlRootElement
    static class Credential {
        private String id;
        private String password;

        public String getId() {
            return id;
        }

        public void setId(String id) {
            this.id = id;
        }

        public String getPassword() {
            return password;
        }

        public void setPassword(String password) {
            this.password = password;
        }
    }

    @GET
    @Produces(MediaType.APPLICATION_JSON)
    public Response getAll() {
        Collection<User> users = USER_STORE.values();
        LOG.info("{} users found", users.size());

        return Response.ok(new GenericEntity<Collection<User>>(users) {
        }).build();
    }

    @Path("/{userId}")
    @GET
    @Produces(MediaType.APPLICATION_JSON)
    public Response getUserById(@PathParam("userId") String userId) {
        LOG.info("fetching user by user-id: {}", userId);
        return Response.ok(USER_STORE.get(userId)).build();
    }

    @POST
    @Consumes(MediaType.APPLICATION_JSON)
    @Produces(MediaType.APPLICATION_JSON)
    public Response save(User user) {
        user.setId(UUID.randomUUID().toString().toUpperCase());
        USER_STORE.put(user.getId(), user);

        LOG.info("new user {} saved", user);
        return Response.ok(user).build();
    }

    @DELETE
    public Response purgeUsers() {
        LOG.info("removing all {} users", USER_STORE.size());
        USER_STORE.clear();
        return Response.ok().build();
    }

    @POST
    @Path("/login")
    public Response login(Credential credential) {
        LOG.info("login user with id: '{}' and password '{}'", credential.getId(), credential.getPassword());
        if (validCredential(credential)) {
            LOG.info("login for user-id '{}' successful", credential.id);
            String authToken = authorize(credential);
            return Response.ok("{\"authToken\":\"" + authToken + "\"}").build();
        }
        return Response.status(401).build();
    }

    private String authorize(Credential credential) {
        String code = new BigInteger(130, new SecureRandom()).toString(20);
        TOKEN_TO_USERID.put(code, credential.getId());
        return code;
    }

    private boolean validCredential(Credential credential) {
        if (!USER_STORE.containsKey(credential.getId())) {
            LOG.warn("no user with id {} known", credential.getId());
            return false;
        }
        if (!USER_STORE.get(credential.id).getPassword().equals(credential.getPassword())) {
            LOG.warn("credentials are invalid");
            return false;
        }

        LOG.info("credentials are valid");
        return true;
    }

    @Path("/secured")
    public SecuredResource getSecuredResource() {
        return new SecuredResource(TOKEN_TO_USERID);
    }
}

Secured Resource

This resource allows to obtain the current date in a JSON format but an auth-token is required.

package com.hascode.tutorial;

import java.time.LocalDateTime;
import java.time.format.DateTimeFormatter;
import java.util.Map;

import javax.ws.rs.GET;
import javax.ws.rs.HeaderParam;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import javax.ws.rs.core.Response.Status;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class SecuredResource {
    private static final Logger LOG = LoggerFactory.getLogger(SecuredResource.class);

    private final Map<String, String> tokenToUserid;

    public SecuredResource(Map<String, String> tokenToUserid) {
        this.tokenToUserid = tokenToUserid;
    }

    @GET
    @Path("/date")
    @Produces(MediaType.APPLICATION_JSON)
    public Response getDate(@HeaderParam("Auth-Token") String authToken) {
        LOG.info("trying to access secured resource with authToken '{}'", authToken);
        if (!authorized(authToken)) {
            LOG.warn("not authorized");
            return Response.status(Status.FORBIDDEN).build();
        }
        return Response.ok(String.format("{\"date\":\"%s\"}",
                LocalDateTime.now().format(DateTimeFormatter.ofPattern("yyyy-MM-dd")))).build();
    }

    private boolean authorized(String authToken) {
        if (authToken == null || authToken.isEmpty()) {
            LOG.warn("no auth-token given");
            return false;
        }
        if (!tokenToUserid.containsKey(authToken)) {
            LOG.warn("invalid auth-token given");
            return false;
        }
        return true;
    }
}

Application Starter

The following code initializes and starts an embedded Jetty server, running on port 9000.

package com.hascode.tutorial;

import org.eclipse.jetty.server.Server;
import org.eclipse.jetty.servlet.ServletContextHandler;
import org.eclipse.jetty.servlet.ServletHolder;

public class App {
    private Server jettyServer;

    public App() {
        ServletContextHandler context = new ServletContextHandler(ServletContextHandler.SESSIONS);
        context.setContextPath("/");

        jettyServer = new Server(9000);
        jettyServer.setHandler(context);

        ServletHolder jerseyServlet = context.addServlet(org.glassfish.jersey.servlet.ServletContainer.class, "/*");
        jerseyServlet.setInitParameter("jersey.config.server.provider.classnames",
                UserResource.class.getCanonicalName());
        jerseyServlet.setInitOrder(0);
    }

    public void start() throws Exception {
        jettyServer.start();
    }

    public void stop() throws Exception {
        try {
            jettyServer.stop();
        } finally {
            jettyServer.destroy();
        }
    }

    public static void main(String[] args) throws Exception {
        new App().run();
    }

    private void run() throws Exception {
        try {
            jettyServer.start();
            jettyServer.join();
        } finally {
            jettyServer.destroy();
        }
    }

}

Jetty Logging Configuration

We’re adding the following jetty-logging.properties to our project:

org.eclipse.jetty.util.log.class=org.eclipse.jetty.util.log.StdErrLog
org.eclipse.jetty.LEVEL=OFF
com.hascode.LEVEL=INFO

Directory Structure

This is our final application’s directory structure:

src/main
├── java
│   └── com
│       └── hascode
│           └── tutorial
│               ├── App.java
│               ├── SecuredResource.java
│               ├── User.java
│               └── UserResource.java
└── resources
    └── jetty-logging.properties

Running the REST-Service

We may run our REST service now using our IDE or in the command-line like this:

mvn exec:java -Dexec.mainClass=com.hascode.tutorial.App

Afterwards, our RESTful web-service is available at http://localhost:9000/ and we may obtain its WADL at http://localhost:9000/application.wadl.

Article Updates

  • 2017-07-26: Link to SemaphoreCI article added.