What’s New in JUnit 5.5

A new minor version of JUnit 5, 5.5, was released on June 30th. That means it’s time for yet another article looking at what’s new in this release of JUnit 5! The JUnit team keeps up their blistering pace of a new minor release roughly every four months, and while this release might not quite have a “killer” new feature like previous releases have had, the aggregate of all the tweaks and enhancements have actually made this a pretty big release, far larger than I will be able to cover here. Let’s dive in and take a look at a some of the key new features and changes introduced in JUnit 5.5.

Declarative Timeouts

Ideally automated tests should execute quickly, however some tests; integration, feature, end-to-end tests, etc., might require complicated setup and/or interact with remote resources. This behavior might occasionally lead to situations where a test can run excessively long. To address this concern @Timeout was introduced to provide declarative timeout support for test cases and lifecycle methods. Here is a code example showing some of the ways @Timeout can be used. I’ll go into more detail on its usage below:

@Timeout isn’t limited to being placed on test cases themselves. Along with the aforementioned test cases, @Timeout can also be placed at the type level, where it will provide a default timeout for all test cases declared in the class, this can be overridden by adding a @Timeout to a test case. @Timeout can also be added to lifecycle methods, @BeforeAll, @BeforeEach​, @AfterEach, @AfterAll​.

@Timeout also provides flexibility in setting the unit for timeout length. By default @Timeout uses seconds for its unit of measurement, however @Timeout also has the unit field which can take a value of TimeUnit. Hopefully you’ll never need to declare a TimeUnit value larger than seconds, but having the flexibility to use milliseconds, microseconds, or even nanoseconds, can be helpful when testing race conditions and following the principle of “fail fast” even in the automated testing world (even a unit as small as a second can start to add up in a large enough test suite).

Timeouts can also be declared as system properties. Like with @Timeout the default unit is seconds, but can be changed by adding the following labels:

ns|μs|ms|s|m|h|d

More specific properties override less specific ones (i.e. setting timeout.beforeall, would override the value of timeout.lifecycle). This provides a way of setting sensible defaults for how long tests should run for that can be easily overridden if needed. Below is the full list of timeout system properties:

junit.jupiter.execution.timeout.default
junit.jupiter.execution.timeout.testable.method.default
junit.jupiter.execution.timeout.test.method.default
junit.jupiter.execution.timeout.testtemplate.method.default
junit.jupiter.execution.timeout.testfactory.method.default
junit.jupiter.execution.timeout.lifecycle.method.default
junit.jupiter.execution.timeout.beforeall.method.default
junit.jupiter.execution.timeout.beforeeach.method.default
junit.jupiter.execution.timeout.aftereach.method.default
junit.jupiter.execution.timeout.afterall.method.default

@RegisterExtension Updates

@RegisterExtension was added in JUnit 5.1 to support registering extensions programmatically. Since 5.1 @RegisterExtension has mostly remained unchanged, save for the addition of being able to declare their execution order which was added in 5.4. With JUnit 5.5 @RegisterExtension has a number of small but helpful changes made to it, let’s take a look at them.

@RegisterExtension Gets Loud 📣

There are several constraints around how @RegisterExtension can be used. Two of them are the field annotated with @RegisterExtension must not be private and the assigned value must implement an Extension interface. With JUnit 5.5 violating these constraints will cause an exception to be thrown, whereas previously fields that violated these constraints would be silently ignored. The below gif demonstrates the change in behavior.

@RegisterExtension Gets Secretive

In previous releases of JUnit, the declared field of @RegisterExtension must also implement an Extension interface. The JUnit team relaxed this constraint that only the assigned value of the field must implement Extension. This is helpful for Extension designers to hide fields that users of an extension should not be changing and, generally make an API that is cleaner and easier to use.

TestExecutionListener Fails More Quietly

While JUnit has made misusage of @RegisterExtension “noisier”, with JUnit 5.5 exceptions thrown by TestExecutionListeners have gotten quieter. In JUnit 5.4 and before when an exception was thrown by a TestExecutionListener this caused the entire test run to terminate. This behavior isn’t really desirable. Along with listeners existing within the world of reporting, so not necessarily “critical”, even a test throwing an exception would terminate a test run.

With JUnit 5.5 when a TestExecutionListener throws an exception, the stacktrace is instead printed to the console at the WARN level. This allows the test suite execution to continue, but for the information about an error occurring not being lost, and also being clearly indicated.

MethodOrder Random Seed Print Out

In JUnit 5.4 method ordering was added. One of the default implementation of method ordering is random method ordering. Random method ordering could be useful for checking that tests don’t have any unnecessary relationships between them. By running tests in a random order, this can validate that testCaseD doesn’t depend upon/is impacted by behavior occurring in testCaseC.

Note: This is generally only referring to unit tests. It can be appropriate for other types of automated tests to have dependencies upon one another.

If a seed isn’t supplied via the junit.jupiter.execution.order.random.seed property, JUnit will generate a seed for you. With JUnit 5.5, when this happens JUnit will also print log statement displaying the seed value used. This allows a test run to be recreated if a problem was discovered. The below gif demonstrates the behavior:

Execute Classes in Parallel

The JUnit team has made further improvements to their support of parallel test execution. In JUnit 5.5 the configuration parameter: ​junit.jupiter.execution.parallel.mode.classes.default has been added, that allows support for defining how parallel test execution should interact at the class level. Like with ​​junit.jupiter.execution.parallel.mode.default, there are two default values accepted SAME_THREAD and CONCURRENT.  Additionally junit.jupiter.execution.parallel.mode.classes.default will default to the value that junit.jupiter.execution.parallel.mode.default is set to, which is SAME_THREAD by default.

The goal of this change was to make it easier to configure parallel test execution in your test suite. This behavior previously could had been implemented by manually adding @Execution(MODE) to the declaration of every class. The problem with that though is in the description, it required manual work. The addition junit.jupiter.execution.parallel.mode.classes.default allows a default behavior that can be applied to an entire test suite from a single location that can be overridden if needed. Below is a gif showing how a test suite would execute using the different settings:

Default Test Discovery Implementations

A core characteristic of JUnit 5 is its extensibility. Between the needs of library developers and the needs of individual organizations, the JUnit framework cannot reasonably natively support every use case. To address this concern, the JUnit team has worked to make JUnit 5 a foundation that is exceptionally easy to build off of. This characteristic continues with the addition of default implementations for test discovery for both Jupiter and Vintage JUnit tests.

The default test discovery implementations will make it easier to write custom TestEngine implementations. Generally this isn’t something most developers will need to do, but if you have an unusual use case or a need not quite being met by existing TestEngines, then have default discovery implementations will make writing your own TestEngine a little easier.

I would recommend checking out the associated GitHub issues behind this change for more in-depth information: 1739 and 1798.

Deprecation of EnableIf/DisabledIf

Introduced with the release of JUnit 5 was the ability to conditionally disable tests. This feature has gradually been enhanced over time with the JUnit team providing several sensible defaults like disabling (or enabling) a test based upon OS, Java version, and by system property.

However with JUnit 5.5 the JUnit team will be deprecating @EnabledIf and @DisabledIf for eventual removal (currently slated to be removed in 5.6). @EnabledIf/@DisabliedIf provided a mechanism for writing out a script (JavaScript, Groovy, et al.) to evaluate if a test should be executed. Ultimately though this method provided little benefit over writing an implementation of DisabledCondition, while having the drawbacks of creating a maintenance headache as the script would have to be copy and pasted if it was to be reused and also being slower to execute.

Even setting aside the above concerns, the expiration date of using script based conditions was nigh. Nashhorn, Java’s JavaScript engine, was deprecated with Java 11 deprecated and schedule for removal, and script based conditions don’t play nicely when using the module path. If you are using @EnabledIfor @DisabliedIf in your projects, it would be a good idea to start migrating away from them now.

Conclusion

Because there were so many changes in this release, I was only able to cover a small portion of them (smaller than even normal) to see all the changes included in this release be sure to check out the release notes. And always be sure to check out the JUnit user guides for more in-depth information on how to write automated tests with JUnit 5.

To view the code used in this article, check out my github repo on JUnit5.