A new minor version of JUnit 5, 5.5, was released on June 30th. That means it’s time for yet another article looking at what’s new in this release of JUnit 5! The JUnit team keeps up their blistering pace of a new minor release roughly every four months, and while this release might not quite have a “killer” new feature like previous releases have had, the aggregate of all the tweaks and enhancements have actually made this a pretty big release, far larger than I will be able to cover here. Let’s dive in and take a look at a some of the key new features and changes introduced in JUnit 5.5.

Declarative Timeouts

Ideally automated tests should execute quickly, however some tests; integration, feature, end-to-end tests, etc., might require complicated setup and/or interact with remote resources. This behavior might occasionally lead to situations where a test can run excessively long. To address this concern @Timeout was introduced to provide declarative timeout support for test cases and lifecycle methods. Here is a code example showing some of the ways @Timeout can be used. I’ll go into more detail on its usage below:

@Timeout isn’t limited to being placed on test cases themselves. Along with the aforementioned test cases, @Timeout can also be placed at the type level, where it will provide a default timeout for all test cases declared in the class, this can be overridden by adding a @Timeout to a test case. @Timeout can also be added to lifecycle methods, @BeforeAll, @BeforeEach​, @AfterEach, @AfterAll​.

@Timeout also provides flexibility in setting the unit for timeout length. By default @Timeout uses seconds for its unit of measurement, however @Timeout also has the unit field which can take a value of TimeUnit. Hopefully you’ll never need to declare a TimeUnit value larger than seconds, but having the flexibility to use milliseconds, microseconds, or even nanoseconds, can be helpful when testing race conditions and following the principle of “fail fast” even in the automated testing world (even a unit as small as a second can start to add up in a large enough test suite).

Timeouts can also be declared as system properties. Like with @Timeout the default unit is seconds, but can be changed by adding the following labels:

ns|μs|ms|s|m|h|d

More specific properties override less specific ones (i.e. setting timeout.beforeall, would override the value of timeout.lifecycle). This provides a way of setting sensible defaults for how long tests should run for that can be easily overridden if needed. Below is the full list of timeout system properties:

junit.jupiter.execution.timeout.default
junit.jupiter.execution.timeout.testable.method.default
junit.jupiter.execution.timeout.test.method.default
junit.jupiter.execution.timeout.testtemplate.method.default
junit.jupiter.execution.timeout.testfactory.method.default
junit.jupiter.execution.timeout.lifecycle.method.default
junit.jupiter.execution.timeout.beforeall.method.default
junit.jupiter.execution.timeout.beforeeach.method.default
junit.jupiter.execution.timeout.aftereach.method.default
junit.jupiter.execution.timeout.afterall.method.default

@RegisterExtension Updates

@RegisterExtension was added in JUnit 5.1 to support registering extensions programmatically. Since 5.1 @RegisterExtension has mostly remained unchanged, save for the addition of being able to declare their execution order which was added in 5.4. With JUnit 5.5 @RegisterExtension has a number of small but helpful changes made to it, let’s take a look at them.

@RegisterExtension Gets Loud 📣

There are several constraints around how @RegisterExtension can be used. Two of them are the field annotated with @RegisterExtension must not be private and the assigned value must implement an Extension interface. With JUnit 5.5 violating these constraints will cause an exception to be thrown, whereas previously fields that violated these constraints would be silently ignored. The below gif demonstrates the change in behavior.

@RegisterExtension Gets Secretive

In previous releases of JUnit, the declared field of @RegisterExtension must also implement an Extension interface. The JUnit team relaxed this constraint that only the assigned value of the field must implement Extension. This is helpful for Extension designers to hide fields that users of an extension should not be changing and, generally make an API that is cleaner and easier to use.

TestExecutionListener Fails More Quietly

While JUnit has made misusage of @RegisterExtension “noisier”, with JUnit 5.5 exceptions thrown by TestExecutionListeners have gotten quieter. In JUnit 5.4 and before when an exception was thrown by a TestExecutionListener this caused the entire test run to terminate. This behavior isn’t really desirable. Along with listeners existing within the world of reporting, so not necessarily “critical”, even a test throwing an exception would terminate a test run.

With JUnit 5.5 when a TestExecutionListener throws an exception, the stacktrace is instead printed to the console at the WARN level. This allows the test suite execution to continue, but for the information about an error occurring not being lost, and also being clearly indicated.

MethodOrder Random Seed Print Out

In JUnit 5.4 method ordering was added. One of the default implementation of method ordering is random method ordering. Random method ordering could be useful for checking that tests don’t have any unnecessary relationships between them. By running tests in a random order, this can validate that testCaseD doesn’t depend upon/is impacted by behavior occurring in testCaseC.

Note: This is generally only referring to unit tests. It can be appropriate for other types of automated tests to have dependencies upon one another.

If a seed isn’t supplied via the junit.jupiter.execution.order.random.seed property, JUnit will generate a seed for you. With JUnit 5.5, when this happens JUnit will also print log statement displaying the seed value used. This allows a test run to be recreated if a problem was discovered. The below gif demonstrates the behavior:

Execute Classes in Parallel

The JUnit team has made further improvements to their support of parallel test execution. In JUnit 5.5 the configuration parameter: ​junit.jupiter.execution.parallel.mode.classes.default has been added, that allows support for defining how parallel test execution should interact at the class level. Like with ​​junit.jupiter.execution.parallel.mode.default, there are two default values accepted SAME_THREAD and CONCURRENT.  Additionally junit.jupiter.execution.parallel.mode.classes.default will default to the value that junit.jupiter.execution.parallel.mode.default is set to, which is SAME_THREAD by default.

The goal of this change was to make it easier to configure parallel test execution in your test suite. This behavior previously could had been implemented by manually adding @Execution(MODE) to the declaration of every class. The problem with that though is in the description, it required manual work. The addition junit.jupiter.execution.parallel.mode.classes.default allows a default behavior that can be applied to an entire test suite from a single location that can be overridden if needed. Below is a gif showing how a test suite would execute using the different settings:

Default Test Discovery Implementations

A core characteristic of JUnit 5 is its extensibility. Between the needs of library developers and the needs of individual organizations, the JUnit framework cannot reasonably natively support every use case. To address this concern, the JUnit team has worked to make JUnit 5 a foundation that is exceptionally easy to build off of. This characteristic continues with the addition of default implementations for test discovery for both Jupiter and Vintage JUnit tests.

The default test discovery implementations will make it easier to write custom TestEngine implementations. Generally this isn’t something most developers will need to do, but if you have an unusual use case or a need not quite being met by existing TestEngines, then have default discovery implementations will make writing your own TestEngine a little easier.

I would recommend checking out the associated GitHub issues behind this change for more in-depth information: 1739 and 1798.

Deprecation of EnableIf/DisabledIf

Introduced with the release of JUnit 5 was the ability to conditionally disable tests. This feature has gradually been enhanced over time with the JUnit team providing several sensible defaults like disabling (or enabling) a test based upon OS, Java version, and by system property.

However with JUnit 5.5 the JUnit team will be deprecating @EnabledIf and @DisabledIf for eventual removal (currently slated to be removed in 5.6). @EnabledIf/@DisabliedIf provided a mechanism for writing out a script (JavaScript, Groovy, et al.) to evaluate if a test should be executed. Ultimately though this method provided little benefit over writing an implementation of DisabledCondition, while having the drawbacks of creating a maintenance headache as the script would have to be copy and pasted if it was to be reused and also being slower to execute.

Even setting aside the above concerns, the expiration date of using script based conditions was nigh. Nashhorn, Java’s JavaScript engine, was deprecated with Java 11 deprecated and schedule for removal, and script based conditions don’t play nicely when using the module path. If you are using @EnabledIfor @DisabliedIf in your projects, it would be a good idea to start migrating away from them now.

Conclusion

Because there were so many changes in this release, I was only able to cover a small portion of them (smaller than even normal) to see all the changes included in this release be sure to check out the release notes. And always be sure to check out the JUnit user guides for more in-depth information on how to write automated tests with JUnit 5.

To view the code used in this article, check out my github repo on JUnit5.

One of the big improvements that came in JUnit 5 was support for dependency injection via constructors and methods. Since the release of JUnit 5 in September 2017, third-party libraries, like mockito, have started providing native support for constructor and method injection. In this article we will take a quick look at how to use constructor and method injection with mockito and then look at why you should start injecting mocks as method arguments in your automated tests.

How to Inject a Mock as an Argument

Starting with 2.21.0, current version 2.25.0, mockito has provided support for injecting mocks as both constructor and method arguments. Let’s looks at how you can start using dependency injection with mockito.

In your test class you will need to annotate it with @ExtendWith(MockitoExtension.class). Then for any arguments you would like mockito to provide a mock for, you simply annotate the argument with @Mock. Here is an example of using mockito dependency injection in action:

Pretty simple and straight forward. Let’s now look at why you should start using method injection of mocks.

The Case for Injecting Mocks as Method Arguments

There are three major benefits that come from automated testing: speed, repeatability, and auditability. The first two are pretty well understood benefits of automated testing, auditability however is if not less well understood, definitely less often discussed. Auditability, within the context of automated testing, refers to the quality of being able to see what code has been tested and the intent of the test.

Code coverage can be achieved without spending much time thinking about how other people, developers, test engineers, business analyst, etc, might use automated tests to understand (i.e. audit) the system the tests are covering. Tests with names like testSuccess, testFail, testFail2, can be executed just fine, but do little to communicate their intent. For an automated test suite to be properly auditable, tests names need to clearly convey the intent of what behavior is being tested. While a test with a name of testRollbackAddUserAddressValidationError​ is a bit of a mouth full, it pretty clearly describes what scenario the test is covering.

While testRollbackAddUserAddressValidationError()​ provides intent, to understand the scope of the test, what dependencies the code under test interacts with, it would require inspecting the code within the test case itself. However we can begin to communicate scope by injecting mocks as method arguments. If we were to do that with the above test we will would have testRollbackAddUserAddressValidationError(@Mock UserDao userDao). Now just from reading the signature of the test case we can determine that the scope of the test also includes interacting with the UserDao class.

When executing tests as a group, we can better see the benefits of injecting mocks as method arguments. Below is an example of running two test classes performing the same set of tests, but one is using mocks at the class level, while the other is using method injection. From the JUnit report alone, we can understand that UserService depends upon the UserDao and AddressDao classes.

Note: Another new feature in JUnit 5 are nested tests, which is being used here.

Conclusion

Injecting mocks as method arguments isn’t game changing, but it can help make tests easier to read, and thus audit, thru being able to communicate in the signature the scope of the test. While there will be instance where passing a mock in as a method argument isn’t practical, generally that should be rare*, and so hopefully this article encourages you to generally use method injection when you are working with mocks.

The code used in this article can be found in this gist, and also this repo.

* Complex mock setup should be seen as a smell that either (or all) the mock, the test, or the code under test has too many responsibilities

JUnit 5 offers a number of improvements over JUnit 4. In this article we will take a quick look at how exceptions are handled and verified in JUnit 4, and then see how the new assertThrows() in JUnit 5 improves the usability and readability when catching and verifying exceptions.

Handling and Verifying Exceptions in JUnit 4

In JUnit 4 there are two primary ways of handling exceptions. The most commonly used method is with the expected field in @Test. An alternative way of handling exceptions is by using a @Rule and ExpectedException. Below are examples of both: 

While both methods are capable of catching and verifying exceptions, each have issues that impact their usability and readability. Let’s step through some of these issues with expected and ExpectedException.

When using expected,  not only are you putting some of the assertion behavior into the definition of the test case, verifying fields within the thrown exception is a bit clunky. To verify the fields of an exception you’d have to add a try/catch within the test case, and within the catch block perform the additional assertions and then throw the caught exception.

When using ExpectedException you have to initially declare it with ​none(), no exception expected, which is a bit confusing. Within a test case you define the expected behavior before the method under test. This would be similar to if you were using a mock, but it’s not intuitive as a thrown exception is a “returned” value, not a dependency nor internal to the code under test.

These oddities significantly impacted the usability and readability of test cases in JUnit 4 that verified exception behavior. The latter is by no means a trivial problem as “easy to read” is probably one of, if not the, most import characteristics of test code. So it is not surprising then that exception handling behavior was heavily rewritten in JUnit 5.

Introducing assertThrows()

In JUnit 5, the above two methods of handling and verifying exceptions have been rolled into the much more straightforward and easier to use assertThrows(). assertThrows() requires two arguments, Class <T> and Executable, assertThrows() can also take an optional third argument of either String or Supplier<String> which can be used for providing a custom error message if the assertion fails. assertThrows() returns the thrown exception, which allows for further inspection and verification of the fields within the thrown exception.

Below is an example of assertThrows() in action:

As can be seen in the above, assertThrows()  is much cleaner and easier to use than either method in JUnit 4. Let’s take a bit closer look at assertThrows() and some of its  more subtle improvements as well.

The second argument, the Executable is where the requirement of Java 8 in JUnit 5 starts to show its benefits. Executable is a functional interface, which allows for, with the use of a lambda, directly executing the code under test within the declaration of assertThrows(). This makes it not only easier to check for if an exception thrown, but also allows assertThrows() to return the thrown exception so additional verification can be done.

Conclusion

assertThrows() offers significant improvements to usability and readability when verifying exception behavior for code under test. This is consistent with many of the changes made in JUnit 5, which have made the writing and reading of tests easier. If you haven’t yet made the switch to JUnit 5, I hope this seeing the improvements in exception handling and verification helps to build the case for making the switch.

The code used in this article can be found here: https://github.com/wkorando/junit-5-simple-demonstrator.

EDIT: An earlier version of this blog said that assertThrows()​ doesn’t support exception subtypes, that is incorrect.

It’s a new year and with that comes another release of the JUnit 5 framework! In this article we will look at some of the big new features released in JUnit 5.4.

Ordering Test Case Execution

I have been personally looking forward to this feature for sometime now. While unit tests by definition should be isolated from one another, JUnit covers a space larger than “just” unit testing. In my case, I have been wanting to be able to explicitly define test execution order to resolve an issue around an integration test scenario in a project demonstrating JUnit 5.

The goal of the integration test is to validate that the application can communicate with a Postgres database. In the test class, which is making use of TestContainers, three behaviors are being verified, reading, mapping, and writing to a database. For reading from the database, a simple count of the number of records is being used, which would obviously be impacted by writing a new record to the database. While tests in JUnit 5 are executed in a consistent order, it is “intentionally nonobvious” how that order is determined. With JUnit 5.4, we can finally define an explicit test execution order.

Let’s take a look at how to order test cases in a class (full class here):

To enable ordering tests cases in a class, the class must be annotated with the @TestMethodOrder extension and an ordering type of either Alphanumeric, OrderAnnotation, or Random must be provided.

• Alphanumeric orders test execution based on the method name* of the test case.
• OrderAnnotation allows for a custom defined execution order using @Order like shown above.
• Random orders test cases pseudo-randomly, the random seed can be defined by setting the property junit.jupiter.execution.order.random.seed in your build file.
• You can also create your own custom method orderer by implementing the interface org.junit.jupiter.api.MethodOrderer

*A test case’s @DisplayName, if defined, will not be used to determine ordering.

Order Only the Tests that Matter

When using OrderAnnotation you should note, and this can be seen in the code example above, you don’t have to define an execution order for every test case in a class. In the example above only one test has an explicit execution order, testCountNumberOfCustomersInDB, as that is the only test case that will be impacted by a change in state. By default JUnit will execute any tests without a defined execution order after all tests that do have a defined execution order. If you have multiple unordered tests, as is the case above, they will be executed in the default deterministic, but “nonobvious” execution order that JUnit 5 typically uses.

This design decision is not only helpful for the obvious reason of requiring less work, but it also helps prevent polluting tests with superfluous information. Adding an execution order to a test that does not need it, it could lead to confusion. If a test begins to fail, a developer or test automation specialist might spend time fiddling with execution order when the cause of the failure is unrelated to execution order. By leaving a test without a defined execution order it is stating this test is not impacted by state change. In short, it should be actively encouraged to omit @Order on test cases that do not require it.

Extension Ordering

The new ordering functionality isn’t limited to just ordering the execution of test cases. You can also order how programmatically registered extensions, i.e. extensions registered with @RegisterExentsion, are executed. This can be useful when a test(s) has complex setup/teardown behavior and that setup/teardown has separate domains. For example testing the behavior of how a cache and database are used.

While extensions by default execute in a consistent order, like test cases, that order is “intentionally nonobvious”. With @Order an explicit and consistent extension execution order can be defined. In the below example a simple extension is defined which prints out the value passed into its constructor:

Here is the console output from executing the above test class:

Executing beforeAll with value:C
Executing beforeAll with value:B
Executing beforeAll with value:A
Executing beforeEach with value:C
Executing beforeEach with value:B
Executing beforeEach with value:A
Executing afterEach with value:A
Executing afterEach with value:B
Executing afterEach with value:C
Executing beforeEach with value:C
Executing beforeEach with value:B
Executing beforeEach with value:A
Executing afterEach with value:A
Executing afterEach with value:B
Executing afterEach with value:C
Executing afterAll with value:A
Executing afterAll with value:B
Executing afterAll with value:C

Aggregate Artifact

A frequent question/concern I have heard when presenting on JUnit 5 has been the large number of dependencies that are required when using JUnit 5. With the 5.4 release the JUnit team will now start providing the junit-jupiteraggregate artifact. JUnit-Jupiter bundles junit-jupiter-api, junit-jupiter-params, so collectively this artifact should cover most of the needs when using JUnit 5. This change should help slim down the maven and gradle files of projects using JUnit 5, as well as make JUnit 5 easier to use in general. Below shows the “slimming” effect of the new aggregate artifact:

TempDir

@TempDir began its life originally as part of the JUnit-Pioneer third-party library. With the release of 5.4, @TempDir has been added as a native feature of the JUnit framework. @TempDir makes the process of validating some file I/O behavior easier by handling the setup and teardown of a temporary directory within the lifecycle of a test class. @TempDir can be injected in two ways, as a method argument or as a class field and must be used with either a Path or File type. @TempDir cannot be injected as a constructor argument. Let’s take a look at @TempDir in action:

Note: The same directory is shared across a test class even if you inject a@TempDir in multiple locations.

TestKit

TestKit was added in 5.4 as a way to perform meta-analysis on a test suite. TestKit can be used to check the number of; executed tests, passed tests, failed tests, skipped tests, as well as a few other behaviors. Let’s take a look at how you can check for tests being skipped when executing a test suite.

To use TestKit you will need to add the junit-platform-testkit dependency to your build file.

But That’s not All…

Another new feature added with 5.4 is the new Display Name Generator. Lee Turner already wrote a great article on the new display name generator, so rather than re-explaining it this article, check his: https://leeturner.me/blog/2019/02/building-a-camel-case-junit5-displaynamegenerator.html

This is only a highlight of some of the new features in JUnit 5.4, to view all the new features, changes, and bug fixes, checkout the release notes the JUnit team maintains: https://junit.org/junit5/docs/current/release-notes/

Also be sure to check out the JUnit 5 user guides for examples on how to use all the features in JUnit 5: https://junit.org/junit5/docs/current/user-guide/index.html

Conclusion

I have been continually impressed by the JUnit team’s steady work improving the JUnit 5 framework. In a little under a year and a half we have now seen four minor releases. As someone who has come to deeply appreciate and advocate for automated testing over the past couple of years, I am happy to see the JUnit team aggressively adding new features to JUnit 5 and taking in feedback from the community and other testing frameworks like Spock, TestNG, and others.

To view the code used in this article check out my project github page here: https://github.com/wkorando/WelcomeToJunit5