BDD versus Karate

Now that we know about the core ideas and features of the Karate framework, let’s look at how it achieves writing meaningful test scenarios without extensive coding.

What is BDD?

BDD comes from the world of agile software development. It focuses on bringing together the different roles in a software project (for example developers, testers, and project owners) and talking about requirements. These requirements should then be put into clear scenarios so that, later, all roles can take these scenarios as a basis for their work.

A core part of BDD is a ubiquitous and simple DSL that can express desired behavior in natural language and is understood by every team member. Its purpose is to form a basis for the development of new features but also be a clear guideline for testing and acceptance. In the following sections, we will look at one of the most well-known and widely used DSLs – Gherkin.

The Gherkin language

If you have been in touch with the Cucumber Open BDD testing tool (https://cucumber.io/), you will have stumbled across the Gherkin language. Gherkin is a very concise and straightforward way to turn acceptance criteria into understandable and well-structured test scenarios. At its core, it is a plain-text format to describe how a system under test should behave. Cucumber can parse this format and link it to custom test code, so it provides a readable wrapper around the technical implementation of tests. This way, other development and QA team members, as well as people with non-technical roles, can read these specifications and know exactly what is expected from the system.

The following is an example of the contents of a typical Gherkin feature file (named webshop.feature):

Feature: Web shop tests
    @smoketest
    # This is an example comment
    Scenario: Product search works as expected
        Given I am on the Web shop homepage
        And I am logged in
        When I search for 'Packt'
        Then I get a list of items containing 'Packt'

Even though this feature file is taken from a website test (commonly called UI test), Gherkin can be used to describe any behavior of any system under test – potentially even cases that have nothing to do with software at all.

Let’s take a look at the different parts of the preceding file.

Features

Gherkin files have the .feature extension and always start with a Feature: line followed by a description. Each feature file collects test scenarios that belong to a defined generic business case indicated by the given description. In our example, the feature name Web shop tests shows that this is a collection of test cases for our imaginary online store.

Scenarios

A feature can include one or more scenarios that are associated with it. Each scenario is a test case related to a specific business case, user journey, or behavior.

Scenarios are made up of different keywords and parts that we will look at in the following sections.

Tags

Within feature files that contain multiple scenarios, you can use one or more custom tags to further group them into test suites (in our case, the tag @smoketest could mean that the following scenario belongs to a set of small, basic tests that are run after a deployment to quickly verify that our application is usable). In the course of this book, you will see how these can be used to pre-select the tests you want to run at a specific point in your software development lifecycle.

Steps

Scenarios are made up of different steps that are executed from top to bottom. Each one describes either the state of the system under test, certain actions, or assertions. Their generic nature makes them excellent for reuse across multiple test cases.

Keywords can be substituted

From the processing of the scenarios by a Gherkin-based framework, it does not matter at all which keywords are used in which order. They are used solely for comprehensibility and semantic correctness. In any case, you should get into the habit of following these conventions.

Next, let’s look at the different types of Gherkin keywords you can use.

The Given keyword

This keyword indicates the initial state and prerequisites of an application, its user, or data. In our example, Given I am on the Web shop homepage tells us where the users of the application start their journey in this test case. It is deliberately kept simple to increase comprehensibility and does not contain any technical details.

The When keyword

Any interactions with or changes to the system under test start with a When keyword. When I search for 'Packt' describes clearly what the users do and what their intention is but not how the webshop deals with this on a technical level. Later, we will see that Karate bends this core BDD rule and why it does it this way.

The Then keyword

The final part of most test scenarios is one or more assertions to verify a certain outcome or state of the system under test. This is expressed by Then as in Then I get a list of items containing 'Packt'. Again, this clearly states what outcome is expected from the system but not how exactly this is solved by the application.

The And keyword

The And keyword can connect multiple Given, When, or Then lines depending on if there is more than one prerequisite, action, or assertion. It is just “syntactic sugar” to make a scenario more readable.

In our example it connects two Given statements:

• The user is on the homepage and…
• …the user is logged in

The But keyword

Like And, the But keyword can be used to connect steps. It is not as common as its counterpart but can express certain conditions better. Take the following, for example:

Given I am on the Web shop homepage

And I am not logged in

That can also be formulated as follows:

Given I am on the Web shop homepage

But I am not logged in

This would emphasize even more that the focus of this test is on an unauthorized user.

Catch-all steps (*)

Gherkin has one more step keyword that all different step types (Given, When, Then, And, and But) can be substituted with. A * step is mostly used when it does not really fit the natural flow of what should be tested but instead contains utility functions or setup code, or helps with debugging test scenarios.

An example is Karate’s print function, which logs a string on the command line for simple tracing of values:

* print "The value of my variable is", myVariable

Here, it does not directly play a role in any part of the tested behavior. Instead, it can be considered as a utility step that still has to follow the chronological sequence of events but could easily be separated from them without affecting the overall test functionality. In Karate, an advantage of this is that these bullet points can be suppressed in the generated test reports, as we will later discover.

Comments

Comments can be used to make certain steps clearer, especially when they contain information that might be hard to understand for newer team members or colleagues that are not as familiar with the tested business domain.

Also, they can be used in case a test is temporarily deactivated to explain further why this was done or what needs to happen before it can be active again.

Comments can be placed anywhere in a Gherkin file. They start with a # symbol (like the # This is an example comment line in the example) and are ignored by Cucumber and Karate alike.

Additional Gherkin syntax

In this section, we only looked at the basic Gherkin keywords that will be necessary in the first steps toward writing test cases.

The Gherkin specification has some more advanced language constructs such as the following:

• Data tables: For specifying multiple sets of data for a step
• Example tables: For running scenarios with different sets of data
• Background scenarios: That define common steps in multiple scenarios
• Scenario outlines: For combining similar scenarios into one
• Docstrings: For more complex data definition in a step

All of these will play a role in later chapters when we look at code reuse, keeping tests free from code duplication, and the overall structuring and fragmentation of test suites

Writing good BDD scenarios

When doing BDD, test scenarios should typically have the following characteristics to keep them clear, understandable, and concise across your test suites:

• Scenarios should be able to run independently: This is important because depending on whether all or just a subset of scenarios is run or if they are run in multiple threads at the same time, we cannot always be sure of the order of execution. That means that if scenarios depend on each other (for example, one that creates a new user and one that deletes the new user), we can run into problems if both are executed at the same time or the wrong way around
• It makes sense to run scenarios in parallel early on to be sure that they can run without disturbing others.
• Scenarios should be concise and easy to follow: Your scenarios should not execute too many steps even though the Gherkin format is predestined for easy comprehension of the test cases. This is beneficial for a greater understandability and easier analysis of exactly what happened in case of an error.
• Scenarios should not test too many different things at the same time: This one is closely related to the second point – understandability. Commonly, a scenario fails as soon as a step fails, skipping all subsequent steps. Therefore, if a scenario tests too many things in succession, hints of faulty functions are lost.
• Scenarios should not duplicate other scenarios: In many cases, it might not make sense to do the same test repeatedly. For example, if you start multiple tests with the same request to the same API, you don’t necessarily need to check that this API delivers the expected data format in every individual scenario. This might not always apply, for example, when similar tests belong to different test suites that might be run at different points in time.
• Scenarios should not contain concrete technical information: When writing behavior-driven tests, the focus is on high-level behaviors and how a system is interacted with. There should not be any references to implementation details or certain elements a user interacts with. The basic idea is to describe what should be done, but not how.

Karate is not true BDD

Almost all these points are valid tips for writing good Karate tests. However, the last guideline (“Scenarios should not contain concrete technical information”) is one that we will revisit later in this chapter. Karate’s idea is radically different in this regard, and we will see both why this makes sense and why the framework has moved away from BDD altogether.

Now that we have an idea about BDD and the Gherkin language, let’s see in the next section how Gherkin statements can be linked to source code.

Glue code

So-called glue code builds the bridge between the Gherkin steps and the implementation of concrete test code (it glues these two together). This is how the test framework can execute the functions matching the respective step including the passed parameters.

Let’s look at an example implementation of this part of the scenario:

Given I am on the Web shop homepage
When I search for 'Packt'

First, we will look at how Cucumber deals with glue code so that we can better compare it to Karate’s approach afterward.

Cucumber glue code

The general mechanism of Cucumber-based frameworks is working by linking steps whose names fit predefined Cucumber expressions (https://github.com/cucumber/cucumber-expressions#readme) that are assigned to certain Java functions by annotations. In older versions of Cucumber, this matching of steps to glue code was done via regular expressions (https://en.wikipedia.org/wiki/Regular_expression), but Cucumber expressions turned out to be way easier to use when implementing glue code yourself:

Package blog.softwaretester.gherkin;
import io.cucumber.java.en.Given;
import io.cucumber.java.en.Then;
import io.cucumber.java.en.When;
public class StepDefinitions {
    @Given("I am on the Web shop homepage")
    public void goToHomepage() {
        System.out.println("Go to homepage");
    }
    @When("I search for {string}")
    public void search(final String searchTerm) {
        System.out.println("Search for " + searchTerm);
    }
}

The goToHomepage() function is invoked when Cucumber encounters a step definition that matches the @Given("I am on the Web shop homepage") annotation. It is good to know that, even though the @Given annotation is used here, this matches any other keyword, too, if the following text part is the same.

It is analogous to the step after that: search(final String searchTerm) is connected by the @When("I search for {string}") annotation. It is noteworthy that here, a dynamic parameter exists that can be any string (defined by the {string} placeholder) so that the following is true:

• This string parameter can be passed from the scenario step to the connected function (in our example, searchTerm)
• It can match any string so that here you avoid having to write a new function for each value – this ensures efficient code reusability

Demo steps

Please note that in the preceding method implementations, the steps do not have any concrete functionality except for printing out a message in order to show they are really being invoked.

When running the preceding example in Cucumber, we can see in the console log that this connection works as expected:

[INFO] Running blog.softwaretester.gherkin.RunCucumberTest
Go to homepage
Search for Packt

The complete example can be found at https://github.com/PacktPublishing/Writing-API-Tests-with-Karate/tree/main/chapter01/cucumber-glue-example.

Now that we know how Cucumber handles glue code, let’s check out why Karate separated from Cucumber and implemented its own approach.

The Karate way

Karate uses essentially the same mechanism and concepts as Cucumber but with some interesting differences.

Up until version 0.9.0 of Karate, it was based on Cucumber because of its rich feature set and very good support in common integrated development environments (IDEs). However, Peter Thomas decided in 2018 to separate from Cucumber while keeping Karate fully compatible.

The main reasons for this decision were that Karate should not be dependent on the timeline, priorities, and feature set of Cucumber but instead have the freedom to move ahead with development at a faster pace. Basically, development is more gradual with the Cucumber framework because it must support and coordinate many different language bindings. Since Karate supports only one programming language, there is of course a speed advantage.

Also, this move allowed Karate to control all phases of the test lifecycle, including reporting, and not be tied into the existing Cucumber infrastructure, specifications, and technologies.

Karate, as we have seen, borrows Gherkin but explicitly does not use BDD. The reasoning behind this is that Karate tests are not specifically aimed at non-technical roles but instead at testers and developers with a more technical background.

Peter Thomas explains it like this:

Karate makes sense especially for “platform” teams creating and maintaining web-services, and where product-owner involvement in acceptance-testing is not the highest priority.

Thomas, P. (2017). Karate is not true BDD, https://medium.com/hackernoon/yes-karate-is-not-true-bdd-698bf4a9be39.

Using Gherkin as a test DSL is intended to simplify testing and not just bluntly repeat what other frameworks already provide anyway.

Now, let’s look at an example of how Karate handles glue code internally.

Karate glue code

Karate provides common predefined steps plus the matching glue code directly inside the framework. That means that you do not have to write any code in many cases since the steps you need are already implemented.

Some steps go way beyond the simple conversion of string arguments into primitive method parameters but instead process those further. This makes it possible to use steps that contain JavaScript, JSONPath or XPath expressions, Java function calls, and more. We will see the details of this approach when we deal with this in Chapter 3, Writing Karate Tests.

We can check how this is done by looking at the internal implementation of a scenario step, in this case, Karate’s print statement:

* print "The value of my variable is", myVariable

In Karate’s ScenarioActions class (available in its GitHub repository https://github.com/karatelabs/karate), we find this piece of code that connects the preceding scenario step to the glue code:

@Override
@When("^print (.+)")
public void print(String exp) {
    engine.print(exp);
}

We can clearly see that this is done in analogy to Cucumber. The only differences are that this implementation is predefined and uses a regular expression instead of a Cucumber expression. In this example, the statement "^print (.+)" means find a line starting with print (^print), take everything that follows, and interpret it as a parameter ((.+)). Like in a Cucumber expression, the parameter is then passed on to a function that can process it.

Note

This was just an example to illustrate the inner workings of the framework. You will most likely never be in touch with this kind of code again when authoring software tests with Karate!

The downside of this approach of embedding glue code within the framework should not remain unmentioned. It basically means that this functionality is locked away and cannot be changed anymore by the framework users. Also, adding your own glue code is not possible like it would be in a Cucumber-based framework. Not only is this not available, but it is also not desired by the authors of the framework, as this could violate Karate’s conventions and lead to unforeseen errors. Since Karate’s step implementations are very flexible as they are, they can potentially cover just about any use case without ever needing to be extended with custom glue code anyway.

The overall goal is to keep the framework simple and consistent and limiting the possibilities can be seen as an advantage here.

In this chapter, we covered BDD, the Gherkin language elements, and its technical implementation in Cucumber and Karate. Also, we talked about why and how Karate deviated from Cucumber in some core ideas. Now, let’s look at the different data types that Karate can work with and how this makes it a big strong point for this framework.