1. Overview
As both developers and administrators, we’re constantly looking for ways to streamline processes and speed up workflows. One powerful tool we can use is Jenkins, a popular open-source automation server. However, even with Jenkins, we sometimes wait for jobs to be completed sequentially. What if we could run the same job multiple times in parallel?
In this tutorial, we’ll go through methods to run identical Jenkins jobs simultaneously. First, we’ll look at the Jenkins parallel execution. After that, we’ll look at some implementations.
2. Understanding Jenkins Parallel Execution
To begin with, let’s discuss how Jenkins handles parallelization. In particular, we look at concurrent builds and the benefits of using parallel execution.
2.1. Concurrent Builds in Jenkins
When we think about Jenkins, we often picture a series of jobs running one after another. However, Jenkins is capable of much more. Concurrent builds enable multiple instances of the same job to run simultaneously. This feature is especially useful when we need to process a large number of similar tasks or run tests across different environments.
Further, to better understand concurrent builds, let’s consider a real-world scenario. Specifically, let’s imagine we’re developing a mobile app that needs to be tested on various device configurations. Instead of running tests sequentially for each device, we can leverage concurrent builds to test on multiple devices at once. This approach saves time and provides faster feedback on potential issues. Yet, there are other benefits as well.
2.2. Benefits of Running Jobs in Parallel
Usually, the advantages of parallel job execution in Jenkins are numerous and impactful. First and foremost, parallel execution significantly reduces the overall build time. By distributing the workload across multiple executors, we can complete tasks much faster than running them sequentially.
Moreover, parallel execution enhances resource utilization. When jobs run concurrently, idle resources are out to work, maximizing the efficiency of the Jenkins setup. This is particularly beneficial for teams with limited infrastructure, as it enables them to do more with less.
Lastly, parallel execution supports scalability. As projects grow and become more complex, the ability to run jobs in parallel becomes increasingly important. It enables us to handle larger workloads without necessarily expanding infrastructure.
3. Parallel Execution With Parameterized Trigger Plugin
After getting to know the basics, we begin with a Jenkins plugin that handles parallel jobs.
3.1. Setting up the Plugin
Firstly, we install the parameterized trigger plugin if it’s not already available in the Jenkins instance.
Let’s see a step-by-step of how we can install the plugin:
- Open the Jenkins dashboard
- Log in
- Navigate to Manage Jenkins
- Select Manage Plugins
- In the Available tab, search for Parameterized Trigger Plugin
- Check the box next to the plugin and click Install without restart
After the installation, we restart Jenkins to fully activate the plugin.
3.2. Configuring the Main Job
Now that we have the plugin installed, let’s set up the main job that triggers multiple instances of another job. The main job acts as an orchestrator, launching parallel executions of the target job.
Let’s begin with an example of how to configure the main job using a Jenkinsfile:
pipeline {
agent any
stages {
stage('Trigger Parallel Jobs') {
steps {
script {
def configs = ['config1', 'config2', 'config3']
def parallelSteps = [:]
configs.each { config ->
parallelSteps[config] = {
build job: 'TargetJob',
parameters: [string(name: 'CONFIG', value: config)],
wait: false
}
}
parallel parallelSteps
}
}
}
}
}
In this example, we’re creating a map of parallel steps, each triggering the TargetJob with a different configuration parameter. Specifically, the wait: false option ensures that the main job doesn’t wait for each triggered job to complete before moving on.
3.3. Creating and Configuring Child Jobs
Further, we need to set up the target job (or child job) that runs in parallel. This job should be parameterized to accept the configuration we’re passing from the main job.
There are several steps to this process:
- Create a new freestyle project in Jenkins
- In the General section, check This project is parameterized
- Add a string parameter named CONFIG
- Configure the rest of the job as needed, using the ${CONFIG} variable where appropriate
For example, in a Shell step, we can use echo to see the configuration:
echo "Running with configuration: ${CONFIG}"
# Rest of the job steps...
By setting up jobs this way, we can easily trigger multiple parallel executions of the target job, each with a different configuration.
Notably, while this method is powerful, it’s important to consider resource constraints. We must ensure the Jenkins instance has enough executors to handle the number of parallel jobs we’re triggering. Otherwise, some jobs can end up in the queue, defeating the purpose of parallel execution.
4. Implementing a Parallel Execution with Groovy Scripts
Moving on from the parameterized trigger plugin, let’s explore how we can achieve parallel execution using Groovy scripts with Jenkins pipelines. This method offers more flexibility and control over the parallel execution process. However, it can be more complex to implement, configure, and customize.
4.1. Writing a Groovy Script for Parallel Execution
Initially, the Groovy script enables us to define complex logic directly in the Jenkins pipeline.
Let’s see an example of a Groovy script that implements parallel execution:
def parallelJobs = [:]
def configurations = ['config1', 'config2', 'config3']
configurations.each { config ->
parallelJobs["Job for ${config}"] = {
node {
stage("Running ${config}") {
echo "Executing job for ${config}"
sh "echo 'Performing task for ${config}'"
sleep 10
}
}
}
}
parallel parallelJobs
In this script, we’re creating a map of closure definitions, each representing a job to be executed. The key advantage here is that we can define the entire job logic within the script, rather than triggering a separate job.
4.2. Integrating the Script Into Jenkins Pipeline
However, to use this Groovy script in a Jenkins pipeline, we can wrap it in a script step:
pipeline {
agent none
stages {
stage('Run Parallel Jobs') {
steps {
script {
def parallelJobs = [:]
def configurations = ['config1', 'config2', 'config3']
configurations.each { config ->
parallelJobs["Job for ${config}"] = {
node {
stage("Running ${config}") {
echo "Executing job for ${config}"
sh "echo 'Performing task for ${config}'"
sleep 10
}
}
}
}
parallel parallelJobs
}
}
}
}
}
We use the code snippet above to integrate the script into the Jenkins pipeline.
This approach offers several benefits:
- flexibility: we can more easily adjust the number of parallel jobs by modifying the configuration list
- self-contained: all the logic for parallel execution is in one place, making it easier to manage and version control
- dynamic allocation: using node blocks inside each parallel job, we enable Jenkins to allocate executors as they become available dynamically
However, it’s important to note that this method can make pipelines more complex, especially for larger jobs. It’s crucial to maintain good code organization and comments to keep the pipeline readable and maintainable.
Additionally, while this method enables great flexibility, it also requires more careful resource management. We should consider the total number of executors available in the Jenkins instance when defining the number of parallel jobs.
5. Using Parallel Test Executor Plugin
Additionally, the Parallel Test Executor plugin is particularly useful for distributing test workloads across multiple executors, but we can adapt it for other types of parallel jobs as well.
5.1. Setting up the Plugin
First, we need to install the parallel test executor plugin.
Let’s see a step-by-step of how we can achieve that:
- Open the Jenkins dashboard
- Log in
- Navigate to Manage Jenkins
- Select Manage Plugins
- In the Available tab, search for Parallel Test Executor
- Check the box next to the plugin and click Install without restart
Once installed, we can start using the plugin.
5.2. Implementing Parallel Test Execution
To use the parallel test executor plugin, we configure it in a pipeline script:
pipeline {
agent any
stages {
stage('Run Parallel Tests') {
steps {
parallelTestExecutor(
parallelism: 3,
testJob: 'TestJob',
testReportParallelism: 3,
testReportFiles: '**/target/surefire-reports/*.xml'
)
}
}
}
}
Let’s break down what’s happening here:
- parallelism: specifies that we want to split the tests across 3 executors
- testJob: name of the job that runs the tests
- testReportParallelism: tells Jenkins to use 3 executors to parse the test reports
- testReportFiles: the pattern to find the test report files
The plugin works by splitting the test execution across multiple executors, running them in parallel, and then aggregating the results. This can significantly speed up test execution, especially for large test suites.
Of course, to make this work, we need to set up the TestJob to accept a parameter that tells it which subset of tests to run:
pipeline {
agent any
parameters {
string(name: 'TEST_CASES', defaultValue: '', description: 'Test cases to run')
}
stages {
stage('Run Tests') {
steps {
sh "mvn test -Dtest=${params.TEST_CASES}"
}
}
}
}
In this setup, the parallel test executor automatically passes the appropriate TEST_CASES parameter to each parallel execution of TestJob.
While this plugin is designed for tests, we can adapt it for other types of parallel jobs by creating a job that accepts a parameter defining which subset of work to perform.
In addition, the key to effectively using this plugin is to have a well-structured test suite or workload that can be easily divided into independent parts. This enables Jenkins to distribute the work evenly across executors, maximizing the benefits of parallel execution.
6. Conclusion
In this article, we’ve explored three powerful methods to run the same Jenkins job multiple times in parallel.
Firstly, we looked at understanding what Jenkins parallel jobs are. After that, we delved into particular implementations of methods to run the same jobs multiple times in parallel. Each approach offers unique advantages and can significantly boost the efficiency of the CI/CD pipelines.