Suppose a failing test blocks you to check-in a mission critical code, and you come to know that the owner of the code is on a vacation. What do you do? You try to fix the test or just comment out or delete the test to proceed with your check-in (committing files to a source control such as SVN), or you wait until the test is fixed.

Sometimes we comment out tests because the feature is not developed. JUnit came up with a solution for this. Instead of commenting a test, we can just ignore it by annotating the test method with @Ignore. Commenting out a test or code is bad as it does nothing but increases the code size and reduces its readability. Also, when you comment out a test, then the test report doesn't tell you anything about the commented-out test; however, if you ignore a test, then the test report will tell you that something needs to be fixed as some tests are ignored. So, you can keep track of the ignored test.

Use @Ignore("Reason: why do you want to ignore?"). Giving a proper description explains the intention behind ignoring the test. The following is an example of, where a test method is ignored because the holiday calculation is not working:

@Test
@Ignore("John's holiday stuff failing")
public void when_today_is_holiday_then_stop_alarm() {
}

The following is a screenshot from Eclipse:

You can place the @Ignore annotation on a test class, effectively ignoring all the contained tests.

JUnit was designed to allow execution in a random order, but typically they are executed in a linear fashion and the order is not guaranteed. The JUnit runner depends on reflection to execute the tests. Usually, the test execution order doesn't vary from run to run; actually, the randomness is environment-specific and varies from JVM to JVM. So, it's better that you never assume they'll be executed in the same order and depend on other tests, but sometimes we need to depend on the order.

For example, when you want to write slow tests to insert a row into a database, then first update the row and finally delete the row. Here, unless the insert function is executed, delete or update functions cannot run.

JUnit 4.11 provides us with an @FixMethodOrder annotation to specify the execution order. It takes enum MethodSorters.

To change the execution order, annotate your test class using @FixMethodOrder and specify one of the following available enum MethodSorters constant:

We will write a few tests to verify this behavior.

Add a TestExecutionOrder test and create tests, as shown in the following code snippet:

public class TestExecutionOrder {
  @Test   public void edit() throws Exception {
    System.out.println("edit executed");
  }
  @Test   public void create() throws Exception {
    System.out.println("create executed");
  }
  @Test   public void remove() throws Exception {
    System.out.println("remove executed");
  }  
}

Run the tests. The execution order may vary, but if we annotate the class with @FixMethodOrder(MethodSorters.NAME_ASCENDING), the tests will be executed in the ascending order as follows:

@FixMethodOrder(MethodSorters.NAME_ASCENDING)
public class TestExecutionOrder { … }

The following Eclipse screenshot displays the test execution in the ascending order:

In multisite projects, sporadically, a date or time zone tests fail in a local CI server but run fine in other servers in a different time zone. We can choose to not run those automatic tests in our local server.

Sometimes our tests fail due to a bug in a third-party code or external software, but we know that after some specific build or version, the bug will be fixed. Should we comment out the code and wait until the build is available?

In many projects, Jenkins (for test automation) and SONAR (for code-quality metrics) run in a server. It has been observed that due to low resources, the automatic tests run forever when SONAR is processing and the tests run simultaneously.

JUnit has the answer to all these issues. It recommends using an org.junit.Assume class.

Like Assert, Assume offers many static methods, such as assumeTrue(condition), assumeFalse(condition), assumeNotNull(condition), and assumeThat(condition). Before executing a test, we can check our assumption using the assumeXXX methods. If our assumption fails, then the assumeXXX methods throw AssumptionViolatedException, and the JUnit runner ignores the tests with failing assumptions.

So, basically, if our assumption is not true, the tests are just ignored. We can assume that the tests are run in the EST time zone; if the tests are run somewhere else, they will be ignored automatically. Similarly, we can assume that the third-party code version is higher than the build/version 123; if the build version is lower, the tests will be ignored.

Let's write the code to validate our assumption about Assume.

Here, we will try to solve the SONAR server issue. We will assume that SONAR is not running. If SONAR runs during the test execution, the assumption will fail and the tests will be ignored.

Create an Assumption test class. The following is the body of the class:

public class Assumption {
   
  boolean isSonarRunning = false;
  @Test 
  public void very_critical_test() throws Exception {
    isSonarRunning = true;
    Assume.assumeFalse(isSonarRunning);
    assertTrue(true);
  }
  
}

Here, for simplicity, we added a isSonarRunning variable to replicate a SONAR server facade. In the actual code, we can call an API to get the value. We will set the variable to false. Then, in the test, we will reset the value to true. This means SONAR is running. So, our assumption that SONAR is not running is false; hence, the test will be ignored.

The following screenshot shows that the test is ignored. We didn't annotate the test using @Ignore:

When we change the value of the isSonarRunning variable to false, as given in the following code snippet, the test will be executed:

public void very_critical_test() throws Exception {
    isSonarRunning = false;
    Assume.assumeFalse(isSonarRunning);
    assertTrue(true);
}

Continuous integration tools such as Jenkins can run multiple tools such as Sonar to acquire code-quality metrics. It's always a good practice to have a build pipeline where the code quality is only checked after the tests pass. This prevents the CPU-intensive tasks from occurring at the same time.

Assumption is also used in the @Before methods, but be careful not to overuse it. Assumption is good for use with TDD where one writes pretests ahead of time.

To run multiple test cases, JUnit 4 provides Suite.class and the @Suite.SuiteClasses annotation. This annotation takes an array (comma separated) of test classes.

Create a TestSuite class and annotate the class with @RunWith(Suite.class). This annotation will force Eclipse to use the suite runner.

Next, annotate the class with @Suite.SuiteClasses({ AssertTest.class, TestExecutionOrder.class, Assumption.class }) and pass comma-separated test class names.

The following is the code snippet:

import org.junit.runner.RunWith;
import org.junit.runners.Suite;

@RunWith(Suite.class)
@Suite.SuiteClasses({ AssertTest.class, TestExecutionOrder.class,Assumption.class })
public class TestSuite {

}

During execution, the suite will execute all the tests. The following is a screenshot of the suite run. Check whether it runs seven tests out of the three test fixtures: AssertTest, TestExecutionOrder, and Assumption.

A test suite is created for group-related tests such as a group of data access, API usage tests, or a group of input validation logic tests.

Joe Walnes created the assertThat(Object actual, Matcher matcher) method. General consensus is that assertThat is readable and more useful than assertEquals. The syntax of the assertThat method is as follows:

  public static void assertThat(Object actual, Matcher matcher

Here, Object is the actual value received and Matcher is an implementation of the org.hamcrest.Matcher interface. This interface comes from a separate library called hamcrest.jar.

A matcher enables a partial or an exact match for an expectation, whereas assertEquals uses an exact match. Matcher provides utility methods such as is, either, or, not , and hasItem. The Matcher methods use the builder pattern so that we can combine one or more matchers to build a composite matcher chain. Just like StringBuilder, it builds a string in multiple steps.

The following are a few examples of matchers and assertThat:

The preceding examples are more English than a JUnit test code. So, anyone can understand the intent of the code and test, and a matcher improves readability.

Hamcrest provides a utility matcher class called org.hamcrest.CoreMatchers.

A few utility methods of CoreMatchers are allOf, anyOf, both, either, describedAs, everyItem, is, isA, anything, hasItem, hasItems, equalTo, any, instanceOf, not, nullValue, notNullValue, sameInstance, theInstance ,startsWith, endsWith, and containsString. All these methods return a matcher.

We worked with assertEquals; so, let's start with equalTo. The equalTo method is equivalent to assertEquals.

Comparing matchers – equalTo, is, and not

Create a AssertThatTest.java JUnit test and static import org.hamcrest.CoreMatchers.*; as follows:

import static org.hamcrest.CoreMatchers.not;
import static org.hamcrest.CoreMatchers.is;
import static org.junit.Assert.assertThat;

import org.junit.Test;

public class AssertThatTest {

  @Test
  public void verify_Matcher() throws Exception {
    int age = 30;
    assertThat(age, equalTo(30));
    assertThat(age, is(30));
    
    assertThat(age, not(equalTo(33)));
    assertThat(age, is(not(33)));
  }
}

Set the age variable to 30 and then likewise for assertEquals and call equalTo, which here is Matcher. The equalTo method takes a value. If the Matcher value doesn't match the actual value, then assertThat throws an AssertionError exception.

Set the age variable value to 29 and rerun the test. The following error will occur:

The is(a) attribute takes a value and returns a Boolean and behaves similar to equalTo(a). The is(a) attribute is the same as is(equalTo(a)).

The not attribute takes a value or a matcher. In the preceding code, we used assertThat(age, is(not(33)));. This expression is nothing but age is not 33 and is more readable than the assert methods.

@Test
  public void verify_multiple_values() throws Exception {
        
    double marks = 100.00;
    assertThat(marks, either(is(100.00)).or(is(90.9)));
    
    assertThat(marks, both(not(99.99)).and(not(60.00)));
    
    assertThat(marks, anyOf(is(100.00),is(1.00),is(55.00),is(88.00),is(67.8)));
    
    assertThat(marks, not(anyOf(is(0.00),is(200.00))));
    
    assertThat(marks, not(allOf(is(1.00),is(100.00), is(30.00))));
  }

   @Test
  public void verify_collection_values() throws Exception {
        
    List<Double> salary =Arrays.asList(50.0, 200.0, 500.0);
    
    assertThat(salary, hasItem(50.00));
    assertThat(salary, hasItems(50.00, 200.00));
        assertThat(salary, not(hasItem(1.00)));
  }

@Test
  public void verify_Strings() throws Exception {
    String name = "John Jr Dale";
    assertThat(name, startsWith("John"));
    assertThat(name, endsWith("Dale"));
    assertThat(name, containsString("Jr"));
  }

Parameterized tests are used for multiple iterations over a single input to stress the object in test. The primary reason is to reduce the amount of test code.

In TDD, the code is written to satisfy a failing test. The production code logic is built from a set of test cases and different input values. For example, if we need to build a class that will return the factorial of a number, then we will pass different sets of data and verify that our implementation passes the validation.

We know that the factorial of 0 is 1, the factorial of 1 is 1, the factorial of 2 is 2, the factorial of 3 is 6, the factorial of 4 is 24, and so on.

So, if we write tests such as factorial_of_1_is_1 and factorial_of_4_is_24, then the test class will be polluted very easily. How many methods will we write?

We can create two arrays: one with the expected values and the other with the original numbers. Then, we can loop through the arrays and assert the result. We don't have to do this because the JUnit 4 framework provides us with a similar solution. It gives us a Parameterized runner.

We read about the @RunWith annotation in the preceding section. Parameterized is a special type of runner and can be used with the @RunWith annotation.

Parameterized comes with two flavors: constructor and method.

  @Parameters(name = "{index}: factorial({0})={1}")
    public static Collection<Object[]> factorialData() {
      return Arrays.asList(new Object[][] {
      
        { 0, 1 }, { 1, 1 }, { 2, 2 }, { 3, 6 }, { 4, 24 }, { 5, 120 },{ 6, 720 }  
      });
    }

JUnit tests are automated to get quick feedback after a change in the code. If a test runs for a long time, it violates the quick feedback principle. JUnit provides a timeout value (in milliseconds) in the @Test annotation to make sure that if a test runs longer than the specified value, the test fails.

The following is an example of a timeout:

  @Test(timeout=10)
  public void forEver() throws Exception {
    Thread.sleep(100000);
  }

Here, the test will fail automatically after 10 milliseconds. The following is an Eclipse screenshot that shows the error:

A theory is a kind of a JUnit test but different from the typical example-based JUnit tests, where we assert a specific data set and expect a specific outcome. JUnit theories are an alternative to JUnit's parameterized tests. A JUnit theory encapsulates the tester's understanding of an object's universal behavior. This means whatever a theory asserts is expected to be true for all data sets. Theories are useful for finding bugs in boundary-value cases.

Parameterized tests allow us to write flexible data-driven tests and separate data from the test methods. Theories are similar to parameterized tests—both allow us to specify the test data outside of the test case.

Parameterized tests are good but they have the following drawbacks:

Theory comes up with many annotations and a runner class. Let's examine the important annotations and classes in theory, as follows:

We will start with a simple theory and then explore more. Perform the following steps:

Rules allow very flexible addition or redefinition of the behavior of each test method in a test class. Rules are like Aspect Oriented Programming (AOP); we can do useful things before and/or after the actual test execution. You can find more information about AOP at http://en.wikipedia.org/wiki/Aspect-oriented_programming.

We can use the inbuilt rules or define our custom rule.

In this section, we will look at the inbuilt rules and create our custom Verifier and WatchMan rule.

@Test(timeout=10)

import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.Timeout;

public class TimeoutTest {
    
    @Rule
    public Timeout globalTimeout =  new Timeout(20);
    
    @Test
    public void testInfiniteLoop1() throws InterruptedException{
      Thread.sleep(30);
    }
    
    @Test
    public void testInfiniteLoop2() throws InterruptedException{
      Thread.sleep(30);
    }
    
}

public class ExpectedExceptionRuleTest {

   @Rule
    public ExpectedException thrown= ExpectedException.none();

    @Test
    public void throwsNothing() {

    }

    @Test
    public void throwsNullPointerException() {
      thrown.expect(NullPointerException.class);
      throw new NullPointerException();
    }

    @Test
    public void throwsIllegalStateExceptionWithMessage() {
      thrown.expect(IllegalStateException.class);
      thrown.expectMessage("Is this a legal state?");
     
      throw new IllegalStateException("Is this a legal state?");
    }
}

@Rule
  public TemporaryFolder folder = new TemporaryFolder();

  @Test
  public void testUsingTempFolder() throws IOException {
    File createdFile = folder.newFile("myfile.txt");
    File createdFolder = folder.newFolder("mysubfolder");
    
  }

Exploring the ErrorCollector rule

The ErrorCollector rule allows the execution of a test to continue after the first problem is found (for example, to collect all the incorrect rows in a table and report them all at once) as follows:

import org.junit.rules.ErrorCollector;
import static org.hamcrest.CoreMatchers.equalTo;

public class ErrorCollectorTest {

   @Rule
   public ErrorCollector collector = new ErrorCollector();
    
   @Test
   public void fails_after_execution() {
   collector.checkThat("a", equalTo("b"));
   collector.checkThat(1, equalTo(2));
   collector.checkThat("ae", equalTo("g"));
   }
}

In this example, none of the verification passes but the test still finishes its execution, and at the end, notifies all errors.

The following is the log—the arrows indicate the errors—and also note that only one test method is being executed but Eclipse indicates three failures:

public class VerifierRuleTest {
  private String errorMsg = null;

  @Rule
  public TestRule rule = new Verifier() {
    protected void verify() {
      assertNull("ErrorMsg should be null after each test execution",errorMsg);
    }
  };
  
  
  @Test
  public void testName() throws Exception {
    errorMsg = "Giving a value";
  }
}

@FixMethodOrder(MethodSorters.NAME_ASCENDING)
public class TestWatcherTest {

  private static String dog = "";

    @Rule
    public TestWatcher watchman = new TestWatcher() {
      @Override
      public Statement apply(Statement base, Description description) {
        return super.apply(base, description);
      }

      @Override
      protected void succeeded(Description description) {
        dog += description.getDisplayName() + " " + "success!\n";
      }

      @Override
      protected void failed(Throwable e, Description description) {
        dog += description.getDisplayName() + " " + e.getClass().getSimpleName() + "\n";
      }

      @Override
      protected void starting(Description description) {
        super.starting(description);
      }

      @Override
      protected void finished(Description description) {
        super.finished(description);
      }
    };

    @Test
    public void red_test() {
      fail();
    }

    @Test
    public void green() {
    }
  
    @AfterClass
    public static void afterClass() {
      System.out.println(dog);
    }
}

green(com.packtpub.junit.recap.rule.TestWatcherTest) success!
red_test(com.packtpub.junit.recap.rule.TestWatcherTest) AssertionError

public class TestNameRuleTest {

  @Rule
    public TestName name = new TestName();
    
    @Test
    public void testA() {
      assertEquals("testA", name.getMethodName());
    }
    
    @Test
    public void testB() {
      assertEquals("testB", name.getMethodName());
    }
}

class Resource{
  public void open() {
    System.out.println("Opened");
  }
  
  public void close() {
    System.out.println("Closed");
  }
  
  public double get() {
    return Math.random();
  }
}

public class ExternalResourceTest {
  Resource resource;
  public @Rule TestName name = new TestName();

  public @Rule ExternalResource rule = new ExternalResource() {
    @Override protected void before() throws Throwable {
      resource = new Resource();
      resource.open();
      System.out.println(name.getMethodName());
    }
    
    @Override protected void after()  {
      resource.close();
      System.out.println("\n");
    }
  };
  
  @Test
  public void someTest() throws Exception {
    System.out.println(resource.get());
  }
  
  @Test
  public void someTest2() throws Exception {
    System.out.println(resource.get());
  }
}

Opened
someTest2
0.5872875884671511
Closed

Opened
someTest
0.395586457988541
Closed

The Categories runner runs only the classes and methods that are annotated with either the category given with the @IncludeCategory annotation or a subtype of that category. Either classes or interfaces can be used as categories. Subtyping works, so if you use @IncludeCategory(SuperClass.class), a test marked @Category({SubClass.class}) will be run.

We can exclude categories by using the @ExcludeCategory annotation.

We can define two interfaces using the following code:

public interface SmartTests { /* category marker */ }
public interface CrazyTests { /* category marker */ }

public class SomeTest {
  @Test
  public void a() {
    fail();
  }

  @Category(CrazyTests.class)
  @Test
  public void b() {
  }
}

@Category({CrazyTests.class, SmartTests.class})
public class OtherTest {
  @Test
  public void c() {

  }
}

@RunWith(Categories.class)
@IncludeCategory(CrazyTests.class)
@SuiteClasses( { SomeTest.class, OtherTest.class }) // Note that Categories is a kind of Suite
public class CrazyTestSuite {
  // Will run SomeTest.b and OtherTest.c, but not SomeTest.a
}

@RunWith(Categories.class)
@IncludeCategory(CrazyTests.class)
@ExcludeCategory(SmartTests.class)
@SuiteClasses( { SomeTest.class, OtherTest.class }) 
public class CrazyTestSuite {
  // Will run SomeTest.b, but not SomeTest.a or OtherTest.c
}