What’s New in JUnit 5.4

It’s a new year and with that comes another release of the JUnit 5 framework! In this article we will look at some of the big new features released in JUnit 5.4.

Ordering Test Case Execution

I have been personally looking forward to this feature for sometime now. While unit tests by definition should be isolated from one another, JUnit covers a space larger than “just” unit testing. In my case, I have been wanting to be able to explicitly define test execution order to resolve an issue around an integration test scenario in a project demonstrating JUnit 5.

The goal of the integration test is to validate that the application can communicate with a Postgres database. In the test class, which is making use of TestContainers, three behaviors are being verified, reading, mapping, and writing to a database. For reading from the database, a simple count of the number of records is being used, which would obviously be impacted by writing a new record to the database. While tests in JUnit 5 are executed in a consistent order, it is “intentionally nonobvious” how that order is determined. With JUnit 5.4, we can finally define an explicit test execution order.

Let’s take a look at how to order test cases in a class (full class here):

@ContextConfiguration(classes = { HotelApplication.class }, initializers = ITCustomerJUnit5Repo.Initializer.class)
@DirtiesContext(classMode = ClassMode.BEFORE_EACH_TEST_METHOD)
@TestMethodOrder(OrderAnnotation.class)
public class ITCustomerJUnit5Repo {
//Removed code, for clarity
@Test
@Order(1)
public void testCountNumberOfCustomersInDB() {
assertEquals(2, repo.count());
}
@Test
public void testRetrieveCustomerFromDatabase() {
Customer customer = repo.findAll().iterator().next();
assertEquals("John", customer.getFirstName());
assertEquals("Doe", customer.getLastName());
assertEquals("Middle", customer.getMiddleName());
assertEquals("", customer.getSuffix());
}
@Test
public void testAddCustomerToDB() throws ParseException {
Customer customer = new Customer.CustomerBuilder().firstName("BoJack").middleName("Horse").lastName("Horseman")
.suffix("Sr.").build();
repo.save(customer);
assertEquals(3, repo.count());
}
}

To enable ordering tests cases in a class, the class must be annotated with the @TestMethodOrder extension and an ordering type of either AlphanumericOrderAnnotation, or Random must be provided.

  • Alphanumeric orders test execution based on the method name* of the test case.
  • OrderAnnotation allows for a custom defined execution order using @Order like shown above.
  • Random orders test cases pseudo-randomly, the random seed can be defined by setting the property junit.jupiter.execution.order.random.seed in your build file.
  • You can also create your own custom method orderer by implementing the interface org.junit.jupiter.api.MethodOrderer

*A test case’s @DisplayName, if defined, will not be used to determine ordering.

Order Only the Tests that Matter

When using OrderAnnotation you should note, and this can be seen in the code example above, you don’t have to define an execution order for every test case in a class. In the example above only one test has an explicit execution order, testCountNumberOfCustomersInDB, as that is the only test case that will be impacted by a change in state. By default JUnit will execute any tests without a defined execution order after all tests that do have a defined execution order. If you have multiple unordered tests, as is the case above, they will be executed in the default deterministic, but “nonobvious” execution order that JUnit 5 typically uses.

This design decision is not only helpful for the obvious reason of requiring less work, but it also helps prevent polluting tests with superfluous information. Adding an execution order to a test that does not need it, it could lead to confusion. If a test begins to fail, a developer or test automation specialist might spend time fiddling with execution order when the cause of the failure is unrelated to execution order. By leaving a test without a defined execution order it is stating this test is not impacted by state change. In short, it should be actively encouraged to omit @Order on test cases that do not require it.

Extension Ordering

The new ordering functionality isn’t limited to just ordering the execution of test cases. You can also order how programmatically registered extensions, i.e. extensions registered with @RegisterExentsion, are executed. This can be useful when a test(s) has complex setup/teardown behavior and that setup/teardown has separate domains. For example testing the behavior of how a cache and database are used.

While extensions by default execute in a consistent order, like test cases, that order is “intentionally nonobvious”. With @Order an explicit and consistent extension execution order can be defined. In the below example a simple extension is defined which prints out the value passed into its constructor:

public class TestExtensionOrdering {
@RegisterExtension
@Order(3)
static ExampleJUnit5Extension extensionA = new ExampleJUnit5Extension("A");
@RegisterExtension
@Order(2)
static ExampleJUnit5Extension extensionB = new ExampleJUnit5Extension("B");
@RegisterExtension
@Order(1)
static ExampleJUnit5Extension extensionC = new ExampleJUnit5Extension("C");
@Test
public void testCaseA() {
// Do nothing
}
@Test
public void testCaseB() {
// Do nothing
}
public static class ExampleJUnit5Extension
implements BeforeAllCallback, AfterAllCallback, BeforeEachCallback, AfterEachCallback {
private String value;
public ExampleJUnit5Extension(String value) {
this.value = value;
}
@Override
public void beforeAll(ExtensionContext context) throws Exception {
System.out.println("Executing beforeAll with value:" + value);
}
@Override
public void afterAll(ExtensionContext context) throws Exception {
System.out.println("Executing afterAll with value:" + value);
}
@Override
public void afterEach(ExtensionContext context) throws Exception {
System.out.println("Executing afterEach with value:" + value);
}
@Override
public void beforeEach(ExtensionContext context) throws Exception {
System.out.println("Executing beforeEach with value:" + value);
}
}
}

Here is the console output from executing the above test class:

Executing beforeAll with value:C
Executing beforeAll with value:B
Executing beforeAll with value:A
Executing beforeEach with value:C
Executing beforeEach with value:B
Executing beforeEach with value:A
Executing afterEach with value:A
Executing afterEach with value:B
Executing afterEach with value:C
Executing beforeEach with value:C
Executing beforeEach with value:B
Executing beforeEach with value:A
Executing afterEach with value:A
Executing afterEach with value:B
Executing afterEach with value:C
Executing afterAll with value:A
Executing afterAll with value:B
Executing afterAll with value:C

Aggregate Artifact

A frequent question/concern I have heard when presenting on JUnit 5 has been the large number of dependencies that are required when using JUnit 5. With the 5.4 release the JUnit team will now start providing the junit-jupiteraggregate artifact. JUnit-Jupiter bundles junit-jupiter-api, junit-jupiter-params, so collectively this artifact should cover most of the needs when using JUnit 5. This change should help slim down the maven and gradle files of projects using JUnit 5, as well as make JUnit 5 easier to use in general. Below shows the “slimming” effect of the new aggregate artifact:

<!– New aggregate dependency –>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter</artifactId>
<scope>test</scope>
</dependency>
<!– Old dependencies –>
<!– <dependency> –>
<!– <groupId>org.junit.jupiter</groupId> –>
<!– <artifactId>junit-jupiter-api</artifactId> –>
<!– <scope>test</scope> –>
<!– </dependency> –>
<!– <dependency> –>
<!– <groupId>org.junit.jupiter</groupId> –>
<!– <artifactId>junit-jupiter-engine</artifactId> –>
<!– <scope>test</scope> –>
<!– </dependency> –>
<!– <dependency> –>
<!– <groupId>org.junit.jupiter</groupId> –>
<!– <artifactId>junit-jupiter-params</artifactId> –>
<!– <scope>test</scope> –>
<!– </dependency> –>

TempDir

@TempDir began its life originally as part of the JUnit-Pioneer third-party library. With the release of 5.4, @TempDir has been added as a native feature of the JUnit framework. @TempDir makes the process of validating some file I/O behavior easier by handling the setup and teardown of a temporary directory within the lifecycle of a test class. @TempDir can be injected in two ways, as a method argument or as a class field and must be used with either a Path or File type. @TempDir cannot be injected as a constructor argument. Let’s take a look at @TempDir in action:

@TestMethodOrder(OrderAnnotation.class)
public class TestTempDir {
@TempDir
static Path classTempDir;
@TempDir
static File classTempDirAsFile;
@Test
@Order(1)
public void useAsClassValue() throws IOException {
File file = classTempDir.resolve("temp.txt").toFile();
FileUtils.write(file, "A", StandardCharsets.ISO_8859_1, true);
assertEquals("A", FileUtils.readFileToString(file, StandardCharsets.ISO_8859_1));
}
@Test
@Order(2)
public void useAsClassValuePart2() throws IOException {
File file = classTempDir.resolve("temp.txt").toFile();
FileUtils.write(file, "B", StandardCharsets.ISO_8859_1, true);
assertEquals("AB", FileUtils.readFileToString(file, StandardCharsets.ISO_8859_1));
}
@Test
@Order(3)
public void injectAsMethodValue(@TempDir Path argumentTempDir) throws IOException {
File file = argumentTempDir.resolve("temp.txt").toFile();
FileUtils.write(file, "C", StandardCharsets.ISO_8859_1, true);
assertEquals("ABC", FileUtils.readFileToString(file, StandardCharsets.ISO_8859_1));
}
@Test
@Order(4)
public void injectAsMethodValuePart2(@TempDir Path argumentTempDir) throws IOException {
File file = argumentTempDir.resolve("temp.txt").toFile();
FileUtils.write(file, "D", StandardCharsets.ISO_8859_1, true);
assertEquals("ABCD", FileUtils.readFileToString(file, StandardCharsets.ISO_8859_1));
}
@Test
@Order(5)
public void useAsClassFileValue() throws IOException {
File file = new File(classTempDirAsFile, "temp.txt");
FileUtils.write(file, "E", StandardCharsets.ISO_8859_1, true);
assertEquals("ABCDE", FileUtils.readFileToString(file, StandardCharsets.ISO_8859_1));
}
@Test
@Order(6)
public void injectAsMethodFileValue(@TempDir File tempFile) throws IOException {
File file = new File(classTempDirAsFile, "temp.txt");
FileUtils.write(file, "F", StandardCharsets.ISO_8859_1, true);
assertEquals("ABCDEF", FileUtils.readFileToString(file, StandardCharsets.ISO_8859_1));
}
}

view raw
TempDirExample.java
hosted with ❤ by GitHub

Note: The same directory is shared across a test class even if you inject a@TempDir in multiple locations.

TestKit

TestKit was added in 5.4 as a way to perform meta-analysis on a test suite. TestKit can be used to check the number of; executed tests, passed tests, failed tests, skipped tests, as well as a few other behaviors. Let’s take a look at how you can check for tests being skipped when executing a test suite.

public class TestKitExample {
@Test
void failIfTestsAreSkipped() {
Events testEvents = EngineTestKit
.engine("junit-jupiter")
.selectors(selectClass(TestKitSubject.class))
.execute()
.tests();
testEvents.assertStatistics(stats > stats.skipped(1));
}
}

view raw
TestKitExample.java
hosted with ❤ by GitHub

public class TestKitSubject {
@Test
public void fakeRunningTest() {
}
@Test
@Disabled
public void fakeDisabledTest() {
}
}

view raw
TestKitSubject.java
hosted with ❤ by GitHub

To use TestKit you will need to add the junit-platform-testkit dependency to your build file.

But That’s not All…

Another new feature added with 5.4 is the new Display Name Generator. Lee Turner already wrote a great article on the new display name generator, so rather than re-explaining it this article, check his: https://leeturner.me/blog/2019/02/building-a-camel-case-junit5-displaynamegenerator.html

This is only a highlight of some of the new features in JUnit 5.4, to view all the new features, changes, and bug fixes, checkout the release notes the JUnit team maintains: https://junit.org/junit5/docs/current/release-notes/

Also be sure to check out the JUnit 5 user guides for examples on how to use all the features in JUnit 5: https://junit.org/junit5/docs/current/user-guide/index.html

Conclusion

I have been continually impressed by the JUnit team’s steady work improving the JUnit 5 framework. In a little under a year and a half we have now seen four minor releases. As someone who has come to deeply appreciate and advocate for automated testing over the past couple of years, I am happy to see the JUnit team aggressively adding new features to JUnit 5 and taking in feedback from the community and other testing frameworks like Spock, TestNG, and others.

To view the code used in this article check out my project github page here: https://github.com/wkorando/WelcomeToJunit5

One thought on “What’s New in JUnit 5.4

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s