An Extended Discussion on Customization in JUnit 5

Inspiration often comes in twos for me. While reviewing a recent blog article, What’s New in JUnit 5.4, a colleague suggested I go into more depth on the usage of extensions in JUnit 5. Then in my twitter timeline I saw this from one of the core committers of the JUnit 5 framework:Screen Shot 2019-03-27 at 5.34.07 PM

Source: Twitter.

Later on in the thread, what was trying to be done with the “hack” could had been accomplished by creating a custom extension that’s available to the public.

The above tells me two things; there is a need for a deep dive on the JUnit 5 extension model and a need to explain the extendability aspect of the JUnit 5 framework. When I talk extendability, I’m specifically referring to the quality of being able to build on top of the existing framework that the JUnit team has provided in JUnit 5. Whereas hacks have often been the modus operandi for getting around the limits of frameworks (rather those limits were intentional or not!), the JUnit team went to great strides to make JUnit 5 extendable, and we’ll see in this series how to take advantage of that quality.

JUnit 5’s extension model and extensibility are by no means trivial subjects, so to make it more digestible, this will be a three part blog series. The subject of each blog article will look like this:

  1. Introduction to and using the JUnit 5 extension model
  2. The JUnit 5 extension lifecycle and building a custom extension
  3. Understanding and using extensibility in JUnit 5

In this article, we will take a high level overview of the extensions model from the perspective of a user of extensions; well learn why the extension model was introduced, how this improves upon what was in JUnit 4, the different ways to register an extension, and how to define the order of extension execution. 

The JUnit 5 Extension Model

JUnit 5 was a fundamental re-write and re-design of the JUnit framework. Some areas largely remained the same, though with a few enhancements, like assertions. Other areas were completely overhauled, which includes runner (@RunWith), MethodRule (@Rule), and TestRule (@ClassRule), being rolled into the new extension model.

The benefits of this overhaul can be experienced in a number of ways. A pretty obvious one is you can now declare multiple extensions at the class level whereas before you could only declare a single @RunWith:

@ExtendWith(SpringExtension.class)
@ExtendWith(MockitoExtension.class)
public class TestSomeStuff{...
}

A bit more subtle, parameterized tests and normal tests can now co-exist in the same class:

public class ParameterizedAndNormalTestsLivingTogether{

   @Test
   pubic void aNormalTest(){
      ...
   }

   @ParameterizedTest
   @ValueSource(strings = { "val1", "val2" })
   public void aParameterizedTest(String val) {
   ...
   }
}

Note: @ParameterizedTest is built using the extension model

If you haven’t run into the constraints imposed by the previous Runner and Rule architecture, I can assure you it’s quite the painful experience when you do! So being able to register multiple extensions in the same test class or locate a parameterized test and normal test in the same test class are reasons to celebrate. But this is only just scratching the surface of the extension model, so let’s start going deeper.

Registering Extensions

There are three different ways to register an extension in JUnit 5: declaratively, programmatically, and automatically. Each way of registering an extension comes with specific rules, constraints, and benefits. Let’s step through the different types of ways to register extensions and understand when and why you might prefer using one method of the other.

Declaratively Registering Extensions

Extensions can be registered declaratively with an annotation at the class, method, or test interface level, and even with a composed annotation (which will be covered in-depth in the article on extendability). The code samples above are examples of registering extensions declaratively.

Declarative registering of extensions is probably the easiest way of registering an extension, which can be made even easier with a composed annotation. For example it is easier to remember how to write @ParameterizedTest when you want to declare a parameterized test than @ExtendWith(ParameterizedTestExtension.class).

As you are using an annotation to register an extension all the constraints with using annotations are there, such as only being able to pass static values to the extension. Also a test class cannot easily reference extensions that have been registered declaratively.

Programmatically Registering Extensions

Extensions can be registered programmatically with @RegisterExtension. There are a few rules regarding programmatically registered extensions. First, an extension field cannot be private. Second, the extension field cannot be null at time of evaluation. Finally an extension can either a static or instance field. A static extension has access to the BeforeAll, AfterAll, and TestIntancePostProcessor steps of the extension life cycle.

Registering a programmatic extension would look like this:

@RegisterExtension
SomeExtension extension = new SomeExtension();

Test classes have a much greater degree of freedom when interacting with a programmatically registered extension as they are just another field within the test class. This can be great for retrieving values out of an extension to verify expected behavior, passing values into the extension to manipulate its state at runtime, as well as other uses.

Automatically Registering Extensions

The final way to register an extension is with the Java Service Loader. The Java Service Loader can best be described as arcane, at least I generally get blank stares or looks of confusion when I bring it up. Though like many arcane things, it can be very powerful for both good and ill!

The Java Service Loader can be used to automatically register extensions within a test suite. This can be helpful as it allows certain behaviors to happen automatically when executing tests. The flip side to this, depending on the type of work that is occurring within the extension, this could have a non-trivial impact on your test suite runtime, it could also interfere in a non-obvious way with how a test behaves (the person executing the test might not realize the extension is being executed because it wasn’t registered locally). So to quote Uncle Ben:

remember-with-great-power-comes-great-responsibility

Registering an Automatic Extension

Registering an automatic extension is a more involved process than the other two ways, let’s quickly walk through the steps:

  1. Create a folder named META-INF on the base of your classpath
  2. Create a folder named services under META-INF
  3. Create a file named org.junit.jupiter.api.extension.Extension under services
  4. In org.junit.jupiter.api.extension.Extension add the fully qualified name of the extension you want registered, for example: my.really.cool.Extension
  5. Pass in -Djunit.jupiter.extensions.autodetection.enabled=true as a JVM argument (how to do this will vary based on your IDE)
    1. Configure your build file to automatically pass in the above argument. Here is an example using Surefire in maven:
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <configuration>
        <properties>
            <configurationParameters>
               junit.jupiter.extensions.autodetection.enabled=true
            </configurationParameters>
        </properties>
    </configuration>
</plugin>

You can see a full example of the above here. Note META-INF is located under /src/test/resources.

Doing these steps every time you would want to use an automatic extension in a project is a bit involved, in the article on extendability we’ll take a look at how to make automatic extensions more practical to work with.

Ordering Extension Execution

As of JUnit 5.4 there are two ways to order how extensions are instantiated during the test cycle. Ordering extension execution could be useful in the realm of higher level tests; that is test above the unit test level. Integration tests, functional tests, feature tests, and other such similar tests might require complex setup and tear down steps.

Even for test code, it is still important to follow principles like single responsibility. If for example you have a feature test that verifies the behavior for when your application interacts with a database and cache, it would be better to locate the logic for setting up and tearing down the database in one extension and similar behavior for the cache in a separate extension, instead of putting all that behavior in a single extension. This allows for great reusability of each extension as well as making them easier to comprehend.

For all ways of ordering extension execution, the order of execution is inverted for “after steps”. So if you have three extensions named A, B, and C, each implementing the BeforeEach and AfterEach behavior, then going into a test method the execution order would be A -> B -> C, while the execution order leaving the test method would be C -> B -> A.

Order of Declaration

When registering an extension declaratively, the order of declaration within the test class is the order in which the extensions are registered and executed. Take the below example:

@ExtendWith(FirstExtension.class)
@ExtendWith(SecondExtension.class)
public class TestExtensionExecutionOrdering(){

   @ExtendWith(ThirdExtension.class)
   public void testExtensions(){
   }
}

When executing the test method testExtensions() the execution order going in would be FirstExtension -> SecondExtension -> ThirdExtension and going out of testExtensions() it would be ThirdExtension -> SecondExtension -> FirstExtension.

I haven’t personally used this feature a whole lot. I have a lot of confidence that, from a framework perspective, this feature behaves as designed. What I worry about however is the extent this feature would be understood by most developers and test engineers. In my experience, the order in which annotations are declared in a class or on a method is not something that developers and test engineers often think about or interact with. If this concern is ever surfaced, it’s often for stylistic reasons, for example; the shortest annotation should be declared first.

The good news is, is through the enhancements to extendibility that I mentioned in the introduction to this article, a custom annotation could be created and shared that includes the declaration of multiple extensions in their proper order. We will take a deeper look at custom annotations, and other examples of extensibility later in this series.

Ordering Programmatically Registered Extensions

Ordering extension registration and execution by order of declaration has been a feature of JUnit 5 since its initial release. With JUnit 5.4 programmatically registered extensions can also be executed in a manually defined order (programmatically registered extensions have always been executed in a consistent order, but it is “intentionally non-obvious”).

To define the execution order of a programmatically registered extension the annotation @Order(n) needs to be added to the declaration of the extension field. You do not need to add an annotation at class level like you would for ordering test methods to enable this behavior. However like when ordering test method execution, you do not need to order every extension. Extensions that do not have a defined execution order are executed after all extensions that do, following the “consistent, but intentionally non-obvious” process mentioned above. So in the below example:

public class TestClass{
   @RegisterExtension
   @Order(1)
   BaseExtension baseExtension = new BaseExtension();

   @RegisterExtension
   @Order(2)
   SecondaryExtension secondaryExtension = new SecondaryExtension();

   @RegisterExtension
   AuxillaryExtension secondaryExtension = new AuxillaryExtension();

   public void testExtensions(){
   }
}

BaseExtension is executed first, SecondaryExtension second, and AuxillaryExtension, and any other extension, executed after.

Also note that programmatically registered extensions will be executed after all extensions that have been registered declaratively and automatically. So aa programmatically registered extension with an annotated with @Order(1) may not be the first extension to be executed when running the test. So keep that in mind!

Conclusion

The new extensions model added a lot of much needed (and appreciated!) flexibility when it replaced the runner and rules architecture from JUnit 4. In the next article in the series we will take an in-depth look at the lifecycle of an extension and build our own custom extension!

The code used in this article, and series, can be found here.

Why You Should Start Injecting Mocks as Method Arguments

One of the big improvements that came in JUnit 5 was support for dependency injection via constructors and methods. Since the release of JUnit 5 in September 2017, third-party libraries, like mockito, have started providing native support for constructor and method injection. In this article we will take a quick look at how to use constructor and method injection with mockito and then look at why you should start injecting mocks as method arguments in your automated tests.

How to Inject a Mock as an Argument

Starting with 2.21.0, current version 2.25.0, mockito has provided support for injecting mocks as both constructor and method arguments. Let’s looks at how you can start using dependency injection with mockito.

In your test class you will need to annotate it with @ExtendWith(MockitoExtension.class). Then for any arguments you would like mockito to provide a mock for, you simply annotate the argument with @Mock. Here is an example of using mockito dependency injection in action:

Pretty simple and straight forward. Let’s now look at why you should start using method injection of mocks.

The Case for Injecting Mocks as Method Arguments

There are three major benefits that come from automated testing: speed, repeatability, and auditability. The first two are pretty well understood benefits of automated testing, auditability however is if not less well understood, definitely less often discussed. Auditability, within the context of automated testing, refers to the quality of being able to see what code has been tested and the intent of the test.

Code coverage can be achieved without spending much time thinking about how other people, developers, test engineers, business analyst, etc, might use automated tests to understand (i.e. audit) the system the tests are covering. Tests with names like testSuccess, testFail, testFail2, can be executed just fine, but do little to communicate their intent. For an automated test suite to be properly auditable, tests names need to clearly convey the intent of what behavior is being tested. While a test with a name of testRollbackAddUserAddressValidationError​ is a bit of a mouth full, it pretty clearly describes what scenario the test is covering.

While testRollbackAddUserAddressValidationError()​ provides intent, to understand the scope of the test, what dependencies the code under test interacts with, it would require inspecting the code within the test case itself. However we can begin to communicate scope by injecting mocks as method arguments. If we were to do that with the above test we will would have testRollbackAddUserAddressValidationError(@Mock UserDao userDao). Now just from reading the signature of the test case we can determine that the scope of the test also includes interacting with the UserDao class.

When executing tests as a group, we can better see the benefits of injecting mocks as method arguments. Below is an example of running two test classes performing the same set of tests, but one is using mocks at the class level, while the other is using method injection. From the JUnit report alone, we can understand that UserService depends upon the UserDao and AddressDao classes.

Screen Shot 2019-03-12 at 11.20.15 AM.png

Note: Another new feature in JUnit 5 are nested tests, which is being used here.

Conclusion

Injecting mocks as method arguments isn’t game changing, but it can help make tests easier to read, and thus audit, thru being able to communicate in the signature the scope of the test. While there will be instance where passing a mock in as a method argument isn’t practical, generally that should be rare*, and so hopefully this article encourages you to generally use method injection when you are working with mocks.

The code used in this article can be found in this gist, and also this repo.

* Complex mock setup should be seen as a smell that either (or all) the mock, the test, or the code under test has too many responsibilities

Handling and Verifying Exceptions in JUnit 5

JUnit 5 offers a number of improvements over JUnit 4. In this article we will take a quick look at how exceptions are handled and verified in JUnit 4, and then see how the new assertThrows() in JUnit 5 improves the usability and readability when catching and verifying exceptions.

Handling and Verifying Exceptions in JUnit 4

In JUnit 4 there are two primary ways of handling exceptions. The most commonly used method is with the expected field in @Test. An alternative way of handling exceptions is by using a @Rule and ExpectedException. Below are examples of both: 

While both methods are capable of catching and verifying exceptions, each have issues that impact their usability and readability. Let’s step through some of these issues with expected and ExpectedException.

When using expected,  not only are you putting some of the assertion behavior into the definition of the test case, verifying fields within the thrown exception is a bit clunky. To verify the fields of an exception you’d have to add a try/catch within the test case, and within the catch block perform the additional assertions and then throw the caught exception.

When using ExpectedException you have to initially declare it with ​none(), no exception expected, which is a bit confusing. Within a test case you define the expected behavior before the method under test. This would be similar to if you were using a mock, but it’s not intuitive as a thrown exception is a “returned” value, not a dependency nor internal to the code under test.

These oddities significantly impacted the usability and readability of test cases in JUnit 4 that verified exception behavior. The latter is by no means a trivial problem as “easy to read” is probably one of, if not the, most import characteristics of test code. So it is not surprising then that exception handling behavior was heavily rewritten in JUnit 5.

Introducing assertThrows()

In JUnit 5, the above two methods of handling and verifying exceptions have been rolled into the much more straightforward and easier to use assertThrows(). assertThrows() requires two arguments, Class <T> and Executable, assertThrows() can also take an optional third argument of either String or Supplier<String> which can be used for providing a custom error message if the assertion fails. assertThrows() returns the thrown exception, which allows for further inspection and verification of the fields within the thrown exception.

Below is an example of assertThrows() in action:

As can be seen in the above, assertThrows()  is much cleaner and easier to use than either method in JUnit 4. Let’s take a bit closer look at assertThrows() and some of its  more subtle improvements as well.

The second argument, the Executable is where the requirement of Java 8 in JUnit 5 starts to show its benefits. Executable is a functional interface, which allows for, with the use of a lambda, directly executing the code under test within the declaration of assertThrows(). This makes it not only easier to check for if an exception thrown, but also allows assertThrows() to return the thrown exception so additional verification can be done.

Conclusion

assertThrows() offers significant improvements to usability and readability when verifying exception behavior for code under test. This is consistent with many of the changes made in JUnit 5, which have made the writing and reading of tests easier. If you haven’t yet made the switch to JUnit 5, I hope this seeing the improvements in exception handling and verification helps to build the case for making the switch.

The code used in this article can be found here: https://github.com/wkorando/junit-5-simple-demonstrator.

EDIT: An earlier version of this blog said that assertThrows()​ doesn’t support exception subtypes, that is incorrect.

What’s New in JUnit 5.4

It’s a new year and with that comes another release of the JUnit 5 framework! In this article we will look at some of the big new features released in JUnit 5.4.

Ordering Test Case Execution

I have been personally looking forward to this feature for sometime now. While unit tests by definition should be isolated from one another, JUnit covers a space larger than “just” unit testing. In my case, I have been wanting to be able to explicitly define test execution order to resolve an issue around an integration test scenario in a project demonstrating JUnit 5.

The goal of the integration test is to validate that the application can communicate with a Postgres database. In the test class, which is making use of TestContainers, three behaviors are being verified, reading, mapping, and writing to a database. For reading from the database, a simple count of the number of records is being used, which would obviously be impacted by writing a new record to the database. While tests in JUnit 5 are executed in a consistent order, it is “intentionally nonobvious” how that order is determined. With JUnit 5.4, we can finally define an explicit test execution order.

Let’s take a look at how to order test cases in a class (full class here):

To enable ordering tests cases in a class, the class must be annotated with the @TestMethodOrder extension and an ordering type of either AlphanumericOrderAnnotation, or Random must be provided.

  • Alphanumeric orders test execution based on the method name* of the test case.
  • OrderAnnotation allows for a custom defined execution order using @Order like shown above.
  • Random orders test cases pseudo-randomly, the random seed can be defined by setting the property junit.jupiter.execution.order.random.seed in your build file.
  • You can also create your own custom method orderer by implementing the interface org.junit.jupiter.api.MethodOrderer

*A test case’s @DisplayName, if defined, will not be used to determine ordering.

Order Only the Tests that Matter

When using OrderAnnotation you should note, and this can be seen in the code example above, you don’t have to define an execution order for every test case in a class. In the example above only one test has an explicit execution order, testCountNumberOfCustomersInDB, as that is the only test case that will be impacted by a change in state. By default JUnit will execute any tests without a defined execution order after all tests that do have a defined execution order. If you have multiple unordered tests, as is the case above, they will be executed in the default deterministic, but “nonobvious” execution order that JUnit 5 typically uses.

This design decision is not only helpful for the obvious reason of requiring less work, but it also helps prevent polluting tests with superfluous information. Adding an execution order to a test that does not need it, it could lead to confusion. If a test begins to fail, a developer or test automation specialist might spend time fiddling with execution order when the cause of the failure is unrelated to execution order. By leaving a test without a defined execution order it is stating this test is not impacted by state change. In short, it should be actively encouraged to omit @Order on test cases that do not require it.

Extension Ordering

The new ordering functionality isn’t limited to just ordering the execution of test cases. You can also order how programmatically registered extensions, i.e. extensions registered with @RegisterExentsion, are executed. This can be useful when a test(s) has complex setup/teardown behavior and that setup/teardown has separate domains. For example testing the behavior of how a cache and database are used.

While extensions by default execute in a consistent order, like test cases, that order is “intentionally nonobvious”. With @Order an explicit and consistent extension execution order can be defined. In the below example a simple extension is defined which prints out the value passed into its constructor:

Here is the console output from executing the above test class:

Executing beforeAll with value:C
Executing beforeAll with value:B
Executing beforeAll with value:A
Executing beforeEach with value:C
Executing beforeEach with value:B
Executing beforeEach with value:A
Executing afterEach with value:A
Executing afterEach with value:B
Executing afterEach with value:C
Executing beforeEach with value:C
Executing beforeEach with value:B
Executing beforeEach with value:A
Executing afterEach with value:A
Executing afterEach with value:B
Executing afterEach with value:C
Executing afterAll with value:A
Executing afterAll with value:B
Executing afterAll with value:C

Aggregate Artifact

A frequent question/concern I have heard when presenting on JUnit 5 has been the large number of dependencies that are required when using JUnit 5. With the 5.4 release the JUnit team will now start providing the junit-jupiteraggregate artifact. JUnit-Jupiter bundles junit-jupiter-api, junit-jupiter-params, so collectively this artifact should cover most of the needs when using JUnit 5. This change should help slim down the maven and gradle files of projects using JUnit 5, as well as make JUnit 5 easier to use in general. Below shows the “slimming” effect of the new aggregate artifact:

TempDir

@TempDir began its life originally as part of the JUnit-Pioneer third-party library. With the release of 5.4, @TempDir has been added as a native feature of the JUnit framework. @TempDir makes the process of validating some file I/O behavior easier by handling the setup and teardown of a temporary directory within the lifecycle of a test class. @TempDir can be injected in two ways, as a method argument or as a class field and must be used with either a Path or File type. @TempDir cannot be injected as a constructor argument. Let’s take a look at @TempDir in action:

Note: The same directory is shared across a test class even if you inject a@TempDir in multiple locations.

TestKit

TestKit was added in 5.4 as a way to perform meta-analysis on a test suite. TestKit can be used to check the number of; executed tests, passed tests, failed tests, skipped tests, as well as a few other behaviors. Let’s take a look at how you can check for tests being skipped when executing a test suite.

To use TestKit you will need to add the junit-platform-testkit dependency to your build file.

But That’s not All…

Another new feature added with 5.4 is the new Display Name Generator. Lee Turner already wrote a great article on the new display name generator, so rather than re-explaining it this article, check his: https://leeturner.me/blog/2019/02/building-a-camel-case-junit5-displaynamegenerator.html

This is only a highlight of some of the new features in JUnit 5.4, to view all the new features, changes, and bug fixes, checkout the release notes the JUnit team maintains: https://junit.org/junit5/docs/current/release-notes/

Also be sure to check out the JUnit 5 user guides for examples on how to use all the features in JUnit 5: https://junit.org/junit5/docs/current/user-guide/index.html

Conclusion

I have been continually impressed by the JUnit team’s steady work improving the JUnit 5 framework. In a little under a year and a half we have now seen four minor releases. As someone who has come to deeply appreciate and advocate for automated testing over the past couple of years, I am happy to see the JUnit team aggressively adding new features to JUnit 5 and taking in feedback from the community and other testing frameworks like Spock, TestNG, and others.

To view the code used in this article check out my project github page here: https://github.com/wkorando/WelcomeToJunit5

What’s New in JUnit 5.3

The JUnit team continues to make steady work refining and improving on JUnit 5. In this article we will take a look at some of the new features and changes that are in JUnit 5.3 which was released on September 3rd. Also in this article we will look at some recent changes and improvements to libraries and tools that are often used in conjunction with JUnit.

Parallel Test Execution

I covered parallel test execution in a previous blog article. If you want an technical and in-depth look at the parallel execution features in JUnit 5.3, give that article a read. Here is a high level overview of parallel test execution in JUnit 5.3:

  • Parallel test execution is enabled and configured through the build file (i.e. pom.xml, build.gradle)
  • Parallel test execution behavior can be modified at the method and class level with the annotations: @Execution and @ResourceLock.

Be sure to check out the user guide for more detailed information on configuring and using parallel test execution in JUnit 5.

AssertThrows Enhancements

Note: This feature has been reverted in 5.3.1

T̶h̶e̶ ̶a̶s̶s̶e̶r̶t̶T̶h̶r̶o̶w̶s̶ ̶a̶s̶s̶e̶r̶t̶i̶o̶n̶ ̶h̶a̶s̶ ̶b̶e̶e̶n̶ ̶u̶p̶d̶a̶t̶e̶d̶ ̶t̶o̶ ̶d̶i̶s̶p̶l̶a̶y̶ ̶t̶h̶e̶ ̶v̶a̶l̶u̶e̶ ̶r̶e̶t̶u̶r̶n̶e̶d̶ ̶f̶r̶o̶m̶ ̶a̶ ̶m̶e̶t̶h̶o̶d̶,̶ ̶i̶f̶ ̶a̶n̶y̶,̶ ̶w̶h̶e̶n̶ ̶a̶n̶ ̶e̶x̶c̶e̶p̶t̶i̶o̶n̶ ̶i̶s̶ ̶n̶o̶t̶ ̶t̶h̶r̶o̶w̶n̶.̶ ̶T̶h̶i̶s̶ ̶c̶a̶n̶ ̶b̶e̶ ̶h̶e̶l̶p̶f̶u̶l̶ ̶w̶h̶e̶n̶ ̶d̶e̶b̶u̶g̶g̶i̶n̶g̶ ̶a̶ ̶t̶e̶s̶t̶ ̶c̶a̶s̶e̶ ̶t̶h̶a̶t̶ ̶h̶a̶s̶ ̶s̶t̶a̶r̶t̶e̶d̶ ̶f̶a̶i̶l̶i̶n̶g̶.̶ ̶H̶e̶r̶e̶ ̶i̶s̶ ̶a̶ ̶s̶c̶r̶e̶e̶n̶s̶h̶o̶t̶ ̶o̶f̶ ̶w̶h̶a̶t̶ ̶t̶h̶i̶s̶ ̶d̶i̶s̶p̶l̶a̶y̶ ̶l̶o̶o̶k̶s̶ ̶l̶i̶k̶e̶ ̶i̶n̶ ̶E̶c̶l̶i̶p̶s̶e̶.̶ ̶I̶n̶ ̶t̶h̶i̶s̶ ̶e̶x̶a̶m̶p̶l̶e̶ ̶a̶n̶ ̶e̶m̶p̶t̶y̶ ̶L̶i̶s̶t̶ ̶i̶s̶ ̶b̶e̶i̶n̶g̶ ̶r̶e̶t̶u̶r̶n̶e̶d̶ ̶i̶n̶s̶t̶e̶a̶d̶ ̶o̶f̶ ̶t̶h̶e̶ ̶e̶x̶p̶e̶c̶t̶e̶d̶ ̶e̶x̶c̶e̶p̶t̶i̶o̶n̶.

Capturing Console Output

With JUnit 5.3 it is possible to capture System.out and System.err output using a TestExecutionListener. Capturing console output isn’t a frequent need, but might be necessary if working on more legacy code bases where statements are frequently being written to console and it would be nice to see what is being written in a report.

To start capturing console output it requires two steps:

1. Implementing a TestExecutionListener

2. Configuring surefire to start capturing console output.

Here is an example of each:

TestExecutionListener

Surefire Configuration

Library and Tool Updates

As mentioned in the introduction of this article, we will be taking a look at updates to some other projects often used with JUnit. Most of these changes are unrelated too (i.e. not dependent upon) JUnit 5.3, but I wanted to cover them here as they happened relatively recently.

Surefire Native Support

Back in June the maven surefire team released version 2.22.0 of their plugin. With this release the surefire (and failsafe 2.22.0) plugin offer native support for JUnit 5. This is a nice, because prior to the 2.22.0 release, using JUnit 5 required a lot of additional configuration which was frankly a bit of an eyesore. Prior to 2.22.0, to run JUnit 5 with surefire this configuration was required:

With 2.22.0 the default surefire configuration can look like this:

Note: For spring boot projects, you need only add maven-surefire-plugin.version to your properties, like this.

Note 2: JUnit 5.3 has a bug that prevents tests from being found when the reuseForks tag is set to false. Read more here.

Filtering Changes with Native Support

However with native support comes changes with how tag filtering behaves. When using the JUnit surefire-provider in 2.21.0 filtering by tags looked like this:

However this does not work when running maven-surefire-plugin 2.22.0. To pass to surefire which tags you want to filter on you must use the groups, for which tests to execute, and excludeGroups, for which tests should not be executed, tags. Here is the 2.22.0 equivalent example of the above:

Enhanced Mockito Dependency Injection Support

As the tweet from @sam_brannen states, you can now directly inject mockito mocks via constructor or method parameters. This can make help make tests more declarative as to what they depend upon and/or a little cleaner. These changes were added in 2.21.0¹, here is an example of those changes in action:

AssertJ Soft Assertions and opentest4J

I have covered AssertJ in a previous blog post. With 3.11.1 AssertJ will now use the MultipleFailuresError in the opentest4j library, if it’s available. opentest4j is a library created by the JUnit 5 team to help make the testing experience more consistent across projects for both users and library developers.

If you are not familiar with soft assertions, they are a way of asserting multiple values in a single test and instead of the test failing on the first assertion, all the failed assertions are instead collected and reported at the end of the test. This can help avoid the irritating loop of a test failing, fixing the failure, running the test again, and hitting another failure. Here’s a look at soft assertions in AssertJ:

Running this test will return both assertion failures, instead of just reporting that the first assert failed.

Conclusion

The JUnit team continues to make great progress on JUnit 5 adding new features, improving on existing ones, and fixing bugs as they are reported. This is only a highlight of some of the new features and changes coming in JUnit 5.3. For a full list of all the changes and improvements coming besure to check out the release notes and the JUnit 5 user guide on how to use the features as well.

1: Technically the changes were added in 2.20.3, but that is only available from the mockito’s maven repository, not maven central.