1. Introduction
Although executing tests serially works just fine most of the time, we may want to parallelize them to speed things up.
In this tutorial, we’ll cover how to parallelize tests using JUnit and Maven’s Surefire Plugin. First, we’ll run all tests in a single JVM process, then we’ll try it with a multi-module project.
2. Maven Dependencies
Let’s begin by importing the required dependencies. We’ll need to use JUnit 4.7 or later along with Surefire 2.16 or later:
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
<scope>test</scope>
</dependency>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.21.0</version>
</plugin>
In a nutshell, Surefire provides two ways of executing tests in parallel:
- Multithreading inside a single JVM process
- Forking multiple JVM processes
3. Running Parallel Tests
To run a test in parallel, we should use a test runner that extends org.junit.runners.ParentRunner.
However, even tests that don’t declare an explicit test runner work, as the default runner extends this class.
Next, to demonstrate parallel test execution, we’ll use a test suite with two test classes, each having a few methods. In fact, any standard implementation of a JUnit test suite would do.
3.1. Using Parallel Parameter
First, let’s enable parallel behaviour in Surefire using the parallel parameter. It states the level of granularity at which we’d like to apply parallelism.
The possible values are:
- methods – runs test methods in separate threads
- classes – runs test classes in separate threads
- classesAndMethods – runs classes and methods in separate threads
- suites – runs suites in parallel
- suitesAndClasses – runs suites and classes in separate threads
- suitesAndMethods – creates separate threads for suites and methods
- all – runs suites, classes as well as methods in separate threads
In our example, we use all:
<configuration>
<parallel>all</parallel>
</configuration>
Second, let’s define the total number of threads we want Surefire to create. We can do that in two ways:
Using threadCount which defines the maximum number of threads Surefire will create:
<threadCount>10</threadCount>
Or using useUnlimitedThreads parameter where one thread is created per CPU core:
<useUnlimitedThreads>true</useUnlimitedThreads>
By default, threadCount is per CPU core. We can use the parameter perCoreThreadCount to enable or disable this behavior:
<perCoreThreadCount>true</perCoreThreadCount>
3.2. Using Thread-Count Limitations
Let’s say we want to define the number of threads to create at the method, class, and suite level. We can do this with the threadCountMethods, threadCountClasses and threadCountSuites parameters.
Let’s combine these parameters with threadCount from the previous configuration:
<threadCountSuites>2</threadCountSuites>
<threadCountClasses>2</threadCountClasses>
<threadCountMethods>6</threadCountMethods>
Since we used all in parallel, we’ve defined the thread counts for methods, suites, and classes. However, it isn’t mandatory to define the leaf parameter. Surefire deduces the number of threads to use if leaf parameters are omitted.
For example, if threadCountMethods is omitted, then we need to make sure threadCount > threadCountClasses + threadCountSuites.
Sometimes we may want to limit the number of threads created for classes or suites, or methods even while we’re using an unlimited number of threads.
We can apply thread-count limitations in such cases as well:
<useUnlimitedThreads>true</useUnlimitedThreads>
<threadCountClasses>2</threadCountClasses>
3.3. Setting Timeouts
Sometimes we may need to ensure that test execution is time-bounded.
To do that, we can use the parallelTestTimeoutForcedInSeconds parameter. This will interrupt currently running tests and will not execute any of the queued tests after the timeout has elapsed:
<parallelTestTimeoutForcedInSeconds>5</parallelTestTimeoutForcedInSeconds>
Another option is to use parallelTestTimeoutInSeconds.
In this case, only the queued tests will be stopped from executing after a certain number of seconds:
<parallelTestTimeoutInSeconds>3.5</parallelTestTimeoutInSeconds>
Nevertheless, with both options, the tests will end with an error message when the timeout has elapsed.
3.4. Caveats
Surefire calls static methods annotated with @Parameters, @BeforeClass, and @AfterClass in the parent thread. Thus make sure to check for potential memory inconsistencies or race conditions before running tests in parallel.
Also, tests that mutate shared states are definitely not good candidates for running in parallel.
4. Test Execution in Multi-Module Maven Projects
Till now, we’ve focused on running tests in parallel within a Maven module.
But let’s say we have multiple modules in a Maven project. Since these modules are built sequentially, the tests for each module are also executed sequentially.
We can change this default behaviour by using Maven’s -T parameter, which builds modules in parallel. This can be done in two ways.
We can either specify the exact number of threads to use while building the project:
mvn -T 4 surefire:test
Or use the portable version and specify the number of threads to create per CPU core:
mvn -T 1C surefire:test
Either way, we can speed up tests and build execution times.
5. Forking JVMs
With the parallel test execution via the parallel option, concurrency happens inside the JVM process using threads.
Since threads are sharing the same memory space, this can be efficient in terms of memory and speed. However, we may encounter unexpected race conditions or other subtle concurrency-related test failures. As it turns out, sharing the same memory space can be both a blessing and a curse.
To prevent thread-level concurrency issues, Surefire provides another parallel test execution mode: forking and process-level concurrency. The idea of forked processes is actually quite simple. Instead of spawning multiple threads and distributing the test methods between them, surefire creates new processes and does the same distribution.
Since there’s no shared memory between different processes, we won’t suffer from those subtle concurrency bugs. Of course, this comes at the expense of more memory usage and a little less speed.
Anyway, to enable forking, we have to use the forkCount property and set it to any positive value:
<forkCount>3</forkCount>
Surefire will create at most three forks from the JVM and run the tests in them. The default value for forkCount is one, which means that maven-surefire-plugin creates one new JVM process to execute all tests in one Maven module.
The forkCount property supports the same syntax as -T. That is, if we append the C to the value, that value will be multiplied by the number of available CPU cores in our system. For instance:
<forkCount>2.5C</forkCount>
Then in a two-core machine, Surefire can create at most five forks for parallel test execution.
By default, Surefire will reuse the created forks for other tests. However, if we set the reuseForks property to false, it’ll destroy each fork after running one test class.
Also, to disable the forking, we can set the forkCount to zero.
6. Conclusion
To sum up, we started off by enabling multi-threaded behaviour and defining the degree of parallelism using the parallel parameter. Subsequently, we applied limitations on the number of threads Surefire should create. Later, we set timeout parameters to control test execution times.
Finally, we looked at how we can reduce build execution times and therefore test execution times in multi-module Maven projects.
As always, the code presented here is available on GitHub.