1. Overview
Continuous Integration (CI) and Continuous Delivery (CD) help ensure code quality and security, automate tests, and streamline deployments. However, as the number of projects grows, maintaining multiple CI configurations across multiple repositories becomes tedious and error-prone.
In this tutorial, we’ll learn how to manage GitLab CI/CD configuration efficiently by sharing the YAML configurations across multiple projects.
2. Defining Custom CI/CD Configuration
Let’s start by taking a look at the .gitlab-ci.yml configuration file within the ci-common project under the baeldung-developers GitLab group:
$ cat .gitlab-ci.yml
hello_world:
script:
- echo "Hello, world!"
.greeting:
script:
- echo "$GREETING"
GitLab CI/CD will not process the .greeting job unless we extend it with a regular job.
Now, we can reuse the pipeline configuration from the ci-common project in other projects by using the custom CI/CD configuration option under the respective project’s CI/CD settings:
<filepath>[@<full group path>/<project path>][:refname]
We can skip the group and project details in the path if our file belongs to the same project. Further, we can specify the refname only when the file is in a non-default branch.
Now, let’s customize this setting for a sample proj-use-custom-ci-config project:
Lastly, we must note that this approach’s simplicity comes at the cost of extensibility and visibility, as we must use the configuration as-is without any flexibility to modify it.
3. Using Custom Group-level Project Templates
We can use GitLab’s custom group-level project templates to bootstrap a new project with pre-defined CI/CD configuration. However, the feature is available only for GitLab Premium and Ultimate pricing tiers.
For this approach, we must define all our project templates under a subgroup dedicated to storing them. So, we’ve defined the “Baeldung Templates Subgroup” under the “Baeldung Developers” group.
Now, we can define the subgroup for templates under the “Baeldung Developers” group’s settings:
Further, let’s create the ci-common project under the “Baeldung Templates Subgroup” subgroup with the .gitlab-ci.yml configuration file as earlier.
Next, we can use the “Create from template” option for creating a new project with the predefined configuration:
Lastly, we must provide the necessary details, such as “Project name” and “Template” for creating the new project:
We’ve successfully reused the CI/CD configuration from an existing template project. Nonetheless, it’s important to note that any future changes made to the template will not be reflected in the projects using them.
4. Including Static YAMLs
In this section, we’ll learn about include and trigger keywords and how to share static GitLab CI/CD configuration files.
4.1. Enriching Project Pipeline
Let’s define the .gitlab-ci.yml configuration file in the proj-include-static-yaml sample project:
$ cat .gitlab-ci.yml
include:
- project: baeldung-developers/ci-common
file: .gitlab-ci.yml
We used the include keyword and referenced the .gitlab-ci.yml file from the ci-common project.
As expected, we can see the hello_word job running:
Moreover, we can modify or extend our CI/CD configuration by adding new definitions:
$ cat .gitlab-ci.yml
include:
- project: baeldung-developers/ci-common
file: .gitlab-ci.yml
hello_world:
script:
- echo "Hello, world! (modified)"
GitLab uses the “closest scope wins” algorithm to resolve the conflicting keys and produce an effective YAML configuration.
Next, let’s verify that the local definition of script took precedence over the external one:
Executing "step_script" stage of the job script
...
$ echo "Hello, world! (modified)"
Hello, world! (modified)
...
Job succeeded
Lastly, let’s write the hello_world_greeting job that extends the .greeting hidden job:
$ cat .gitlab-ci.yml
include:
- project: baeldung-developers/ci-common
file: .gitlab-ci.yml
hello_world_greeting:
extends: .greeting
variables:
GREETING: "Hello, world!"
We expect both two jobs, namely, hello_world and hello_world_greeting jobs, to run on pipeline execution:
Great! We’ve got this right.
4.2. Multi-project Downstream Pipeline
We can also share the .gitlab-ci.yml file from one project into another to create a multi-project downstream pipeline. This can help us orchestrate multiple pipelines running in their respective projects.
Let’s define the trigger_job_1 job in .gitlab-ci.yml under the proj-include-static-yaml project using the trigger keyword:
$ cat .gitlab-ci.yml
trigger_job_1:
trigger:
project: baeldung-developers/ci-common
We must note that we can’t provide custom YAML files to create multi-project downstream pipelines.
Now, let’s see an instance of this pipeline in action:
As expected, the hello_world job from the ci-common project was executed as part of the downstream pipeline.
4.3. Parent-child Downstream Pipeline
Parent-child pipelines are another interesting downstream pipeline scenario that allows us to share YAML files from one project into another. Contrary to the multi-project pipeline, such pipelines execute within the project that triggers them.
Further, we must use the include and trigger keywords to write the parent-child pipeline within .gitlab-ci.yml under the proj-include-static-yaml project:
$ cat .gitlab-ci.yml
trigger_job_2:
trigger:
include:
- project: baeldung-developers/ci-common
file: .gitlab-ci.yml
Although we’ve used the .gitlab-ci.yml file, we can use any other valid YAML file from the target project.
To this effect, let’s see a parent-child pipeline from the proj-include-static-yaml project:
Fantastic! We’ve got this running.
5. Using Custom Docker Image
In this section, we’ll learn how to share the GitLab CI/CD configurations using custom Docker images.
5.1. Project Setup
Let’s start by looking at the project structure of the docker-ci-files project, where we’ve got a few CI/CD YAML files:
$ tree ./
./
├── Dockerfile
└── jobs
├── hello-world.yml
└── test.yml
2 directories, 3 files
For simplicity, we have two YAML files containing job definitions and a Dockerfile to package them in a Docker image.
Now, let’s see the job definitions within the hello-world.yml file:
$ cat jobs/hello-world.yml
hello_world:
script:
- echo "Hello, world!"
In addition, let’s look at the content of the test.yml file:
$ cat jobs/test.yml
test_job:
script:
- echo "test job"
We’ve intentionally kept the configuration simple as we aren’t focusing on building the pipelines but on how to share them across projects.
Lastly, we use the COPY instruction within the Dockerfile to copy these configurations under the root (/) directory:
$ cat Dockerfile
FROM alpine:latest
COPY ./jobs/hello-world.yml ./hello-world.yml
COPY ./jobs/test.yml ./test.yml
That’s it. Next, we’ll learn how to share these YAML files across projects.
5.2. Building Custom Docker Image
Let’s create the .gitlab-ci.yml file and add a build-job GitLab job to it for building the Docker image and pushing it into a registry:
$ cat .gitlab-ci.yml
stages:
- build
build_job:
image: docker:stable
services:
- docker:dind
stage: build
script: |
echo "$DOCKER_PASSWORD" | docker login -u "$DOCKER_USER" --password-stdin $DOCKER_REGISTRY
docker build -t $DOCKER_REGISTRY/baeldung-developers/docker-ci-files:v1 .
docker push $DOCKER_REGISTRY/baeldung-developers/docker-ci-files:v1
We’ve defined DOCKER_AUTH_CONFIG, DOCKER_USER, DOCKER_PASSWORD, and DOCKER_REGISTRY for authentication as CI/CD variables for the group. Additionally, we used the docker:dind service to access the docker tools for build and publishing.
Now, as soon as we push any changes, we’ll see the build_job job running for a new pipeline:
Excellent! Our custom Docker image is ready for use.
5.3. Including YAML from Docker Image
Let’s create a new GitLab project, proj-use-docker-ci project, and add .gitlab-ci.yml to use the YAML files from our custom Docker image:
$ cat .gitlab-ci.yml
stages:
- prepare
- run
prepare_jobs:
stage: prepare
image: $DOCKER_REGISTRY/baeldung-developers/docker-ci-files:v1
script:
- cp /test.yml ${CI_PROJECT_DIR}/test.yml
- cp /hello-world.yml ${CI_PROJECT_DIR}/hello-world.yml
artifacts:
paths:
- test.yml
- hello-world.yml
run_jobs:
stage: run
needs: [prepare_jobs]
trigger:
strategy: depend
include:
- artifact: test.yml
job: prepare_jobs
- artifact: hello-world.yml
job: prepare_jobs
Firstly, we defined two stages, namely, prepare and run. Then, we used the image keyword in the prepare_jobs job to use our custom Docker image and copied the YAML files to the current directory, ${CI_PROJECT_DIR}, and shared them as artifacts to be used by other jobs within the pipeline. Further, we added the run_jobs job to include the YAML job artifacts from the prepare_jobs job for triggering a child pipeline. Lastly, we must note that we used the needs keyword to create an explicit dependency between the two jobs.
Now, let’s visualize our pipeline that used the artifacts from the custom docker-ci-files:v1 Docker image:
Excellent! We’ve added a powerful technique to our toolkit for sharing CI/CD configuration across projects.
6. Using CI/CD Components
In this section, let’s learn about the reusable GitLab CI/CD components, which are relatively new additions to the GitLab CI/CD toolkit.
6.1. Creating Component
Firstly, let’s create the ci-components project to store configuration files for our reusable components:
$ tree .
.
├── README.md
└── deploy_service
├── README.md
└── template.yml
2 directories, 3 files
We aim to abstract out the deployment steps for a service module using the deploy_service component.
Now, let’s look at the definition of the deploy_service component:
$ cat templates/deploy_service/template.yml
spec:
inputs:
stage:
default: deploy
services:
type: array
---
deploy_monolith:
stage: $[[ inputs.stage ]]
parallel:
matrix:
- SERVICE: $[[ inputs.services ]]
script: |
echo "deploying $SERVICE"
We’ve used the spec.inputs keyword to parameterize the “deploy monolith” job. Further, the job expects a list of services to deploy them in parallel using the parallel matrix strategy.
Next, let’s add a create-release job in .gitlab-ci.yml under the ci-components project:
$ cat .gitlab-ci.yml
stages: [release]
create_release:
stage: release
image: registry.gitlab.com/gitlab-org/release-cli:latest
script: echo "Creating release $CI_COMMIT_TAG"
rules:
- if: $CI_COMMIT_TAG
release:
tag_name: $CI_COMMIT_TAG
description: "Release $CI_COMMIT_TAG of components repository $CI_PROJECT_PATH"%
The job gets triggered whenever we create a new tag for the project.
So, let’s push the first tag as 1.0.0 to the project:
$ git tag 1.0.0
$ git push --tags
Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
To https://gitlab.com/baeldung-developers/ci-components.git
* [new tag] 1.0.0 -> 1.0.0
Finally, we’ll see that the create_release job runs for the project:
Our component is ready for use.
6.2. Deploying Monolith
Let’s add the .gitlab-ci.yml file to a monolith repository that uses the deploy_service component to trigger deployment for multiple services:
$ cat .gitlab-ci.yml
stages:
- deploy
include:
- component: $CI_SERVER_FQDN/baeldung-developers/ci-components/[email protected]
inputs:
services: [s1, s2, s3, s4, s5, s6, s7]
As expected by the deploy_service component, we’ve defined the services we want to deploy.
Further, we can notice the new pipelines for the monolith repository deploy each service in parallel:
Great! It looks like we’ve nailed this one!
7. Conclusion
In this article, we learned how to share GitLab CI/CD configurations across multiple projects. Further, we explored several strategies for sharing GitLab CI/CD configurations using include and trigger keywords, custom Docker images, and the reusable GitLab CI/CD components.
The sample GitLab pipelines used in this tutorial can be found over on GitHub.