1. Overview

Continuous Integration (CI) and Continuous Delivery (CD) help ensure code quality and security, automate tests, and streamline deployments. However, as the number of projects grows, maintaining multiple CI configurations across multiple repositories becomes tedious and error-prone.

In this tutorial, we’ll learn how to manage GitLab CI/CD configuration efficiently by sharing the YAML configurations across multiple projects.

2. Defining Custom CI/CD Configuration

Let’s start by taking a look at the .gitlab-ci.yml configuration file within the ci-common project under the baeldung-developers GitLab group:

$ cat .gitlab-ci.yml
hello_world:
  script:
    - echo "Hello, world!"

.greeting:
  script:
    - echo "$GREETING"

GitLab CI/CD will not process the .greeting job unless we extend it with a regular job.

Now, we can reuse the pipeline configuration from the ci-common project in other projects by using the custom CI/CD configuration option under the respective project’s CI/CD settings:

<filepath>[@<full group path>/<project path>][:refname]

We can skip the group and project details in the path if our file belongs to the same project. Further, we can specify the refname only when the file is in a non-default branch.

Now, let’s customize this setting for a sample proj-use-custom-ci-config project:

Custom Configuration - GitLab CI/CD

Lastly, we must note that this approach’s simplicity comes at the cost of extensibility and visibility, as we must use the configuration as-is without any flexibility to modify it.

3. Using Custom Group-level Project Templates

We can use GitLab’s custom group-level project templates to bootstrap a new project with pre-defined CI/CD configuration. However, the feature is available only for GitLab Premium and Ultimate pricing tiers.

For this approach, we must define all our project templates under a subgroup dedicated to storing them. So, we’ve defined the “Baeldung Templates Subgroup” under the “Baeldung Developers” group.

Now, we can define the subgroup for templates under the “Baeldung Developers” group’s settings:

GitLab - Project Templates

Further, let’s create the ci-common project under the “Baeldung Templates Subgroup” subgroup with the .gitlab-ci.yml configuration file as earlier.

Next, we can use the “Create from template” option for creating a new project with the predefined configuration:

GitLab - New Project Using Template

Lastly, we must provide the necessary details, such as “Project name” and “Template” for creating the new project:

GitLab - New Project Using Template Config

We’ve successfully reused the CI/CD configuration from an existing template project. Nonetheless, it’s important to note that any future changes made to the template will not be reflected in the projects using them.

4. Including Static YAMLs

In this section, we’ll learn about include and trigger keywords and how to share static GitLab CI/CD configuration files.

4.1. Enriching Project Pipeline

Let’s define the .gitlab-ci.yml configuration file in the proj-include-static-yaml sample project:

$ cat .gitlab-ci.yml
include:
  - project: baeldung-developers/ci-common
    file: .gitlab-ci.yml

We used the include keyword and referenced the .gitlab-ci.yml file from the ci-common project.

As expected, we can see the hello_word job running:

GitLab CI:CD - Include Static Yaml Default Run

Moreover, we can modify or extend our CI/CD configuration by adding new definitions:

$ cat .gitlab-ci.yml
include:
  - project: baeldung-developers/ci-common
    file: .gitlab-ci.yml

hello_world:
  script:
    - echo "Hello, world! (modified)"

GitLab uses the “closest scope wins” algorithm to resolve the conflicting keys and produce an effective YAML configuration.

Next, let’s verify that the local definition of script took precedence over the external one:

Executing "step_script" stage of the job script
...
$ echo "Hello, world! (modified)"
Hello, world! (modified)
...
Job succeeded

Lastly, let’s write the hello_world_greeting job that extends the .greeting hidden job:

$ cat .gitlab-ci.yml
include:
  - project: baeldung-developers/ci-common
    file: .gitlab-ci.yml

hello_world_greeting:
  extends: .greeting
  variables:
    GREETING: "Hello, world!"

We expect both two jobs, namely, hello_world and hello_world_greeting jobs, to run on pipeline execution:

GitLab CI/CD - Include Static Yaml Extend Job

Great! We’ve got this right.

4.2. Multi-project Downstream Pipeline

We can also share the .gitlab-ci.yml file from one project into another to create a multi-project downstream pipeline. This can help us orchestrate multiple pipelines running in their respective projects.

Let’s define the trigger_job_1 job in .gitlab-ci.yml under the proj-include-static-yaml project using the trigger keyword:

$ cat .gitlab-ci.yml
trigger_job_1:
  trigger:
    project: baeldung-developers/ci-common

We must note that we can’t provide custom YAML files to create multi-project downstream pipelines.

Now, let’s see an instance of this pipeline in action:

GitLab - Trigger Multi Project Downstream Pipeline

As expected, the hello_world job from the ci-common project was executed as part of the downstream pipeline.

4.3. Parent-child Downstream Pipeline

Parent-child pipelines are another interesting downstream pipeline scenario that allows us to share YAML files from one project into another. Contrary to the multi-project pipeline, such pipelines execute within the project that triggers them.

Further, we must use the include and trigger keywords to write the parent-child pipeline within .gitlab-ci.yml under the proj-include-static-yaml project:

$ cat .gitlab-ci.yml
trigger_job_2:
  trigger:
    include:
      - project: baeldung-developers/ci-common
        file: .gitlab-ci.yml

Although we’ve used the .gitlab-ci.yml file, we can use any other valid YAML file from the target project.

To this effect, let’s see a parent-child pipeline from the proj-include-static-yaml project:

GitLab - Trigger Child Downstream Pipeline

Fantastic! We’ve got this running.

5. Using Custom Docker Image

In this section, we’ll learn how to share the GitLab CI/CD configurations using custom Docker images.

5.1. Project Setup

Let’s start by looking at the project structure of the docker-ci-files project, where we’ve got a few CI/CD YAML files:

$ tree ./
./
├── Dockerfile
└── jobs
    ├── hello-world.yml
    └── test.yml
2 directories, 3 files

For simplicity, we have two YAML files containing job definitions and a Dockerfile to package them in a Docker image.

Now, let’s see the job definitions within the hello-world.yml file:

$ cat jobs/hello-world.yml
hello_world:
  script:
    - echo "Hello, world!"

In addition, let’s look at the content of the test.yml file:

$ cat jobs/test.yml
test_job:
  script:
    - echo "test job"

We’ve intentionally kept the configuration simple as we aren’t focusing on building the pipelines but on how to share them across projects.

Lastly, we use the COPY instruction within the Dockerfile to copy these configurations under the root (/) directory:

$ cat Dockerfile
FROM alpine:latest
COPY ./jobs/hello-world.yml ./hello-world.yml
COPY ./jobs/test.yml ./test.yml

That’s it. Next, we’ll learn how to share these YAML files across projects.

5.2. Building Custom Docker Image

Let’s create the .gitlab-ci.yml file and add a build-job GitLab job to it for building the Docker image and pushing it into a registry:

$ cat .gitlab-ci.yml
stages:
  - build

build_job:
  image: docker:stable
  services:
    - docker:dind
  stage: build
  script: |
    echo "$DOCKER_PASSWORD" | docker login -u "$DOCKER_USER" --password-stdin $DOCKER_REGISTRY
    docker build -t $DOCKER_REGISTRY/baeldung-developers/docker-ci-files:v1 .
    docker push $DOCKER_REGISTRY/baeldung-developers/docker-ci-files:v1

We’ve defined DOCKER_AUTH_CONFIG, DOCKER_USER, DOCKER_PASSWORD, and DOCKER_REGISTRY for authentication as CI/CD variables for the group. Additionally, we used the docker:dind service to access the docker tools for build and publishing.

Now, as soon as we push any changes, we’ll see the build_job job running for a new pipeline:

GitLab CI/CD - Build Custom Docker Image

Excellent! Our custom Docker image is ready for use.

5.3. Including YAML from Docker Image

Let’s create a new GitLab project, proj-use-docker-ci project, and add .gitlab-ci.yml to use the YAML files from our custom Docker image:

$ cat .gitlab-ci.yml
stages:
  - prepare
  - run

prepare_jobs:
  stage: prepare
  image: $DOCKER_REGISTRY/baeldung-developers/docker-ci-files:v1
  script:
    - cp /test.yml ${CI_PROJECT_DIR}/test.yml
    - cp /hello-world.yml ${CI_PROJECT_DIR}/hello-world.yml
  artifacts:
    paths:
      - test.yml
      - hello-world.yml

run_jobs:
  stage: run
  needs: [prepare_jobs]
  trigger:
    strategy: depend
    include:
      - artifact: test.yml
        job: prepare_jobs
      - artifact: hello-world.yml
        job: prepare_jobs

Firstly, we defined two stages, namely, prepare and run. Then, we used the image keyword in the prepare_jobs job to use our custom Docker image and copied the YAML files to the current directory, ${CI_PROJECT_DIR}, and shared them as artifacts to be used by other jobs within the pipeline. Further, we added the run_jobs job to include the YAML job artifacts from the prepare_jobs job for triggering a child pipeline. Lastly, we must note that we used the needs keyword to create an explicit dependency between the two jobs.

Now, let’s visualize our pipeline that used the artifacts from the custom docker-ci-files:v1 Docker image:

GitLab - YAML From Docker Image

Excellent! We’ve added a powerful technique to our toolkit for sharing CI/CD configuration across projects.

6. Using CI/CD Components

In this section, let’s learn about the reusable GitLab CI/CD components, which are relatively new additions to the GitLab CI/CD toolkit.

6.1. Creating Component

Firstly, let’s create the ci-components project to store configuration files for our reusable components:

$ tree .
.
├── README.md
└── deploy_service
    ├── README.md
    └── template.yml

2 directories, 3 files

We aim to abstract out the deployment steps for a service module using the deploy_service component.

Now, let’s look at the definition of the deploy_service component:

$ cat templates/deploy_service/template.yml 
spec:
  inputs:
    stage:
      default: deploy
    services:
      type: array
---
deploy_monolith:
  stage: $[[ inputs.stage ]]
  parallel:
    matrix:
      - SERVICE: $[[ inputs.services ]]
  script: |
    echo "deploying $SERVICE"

We’ve used the spec.inputs keyword to parameterize the “deploy monolith” job. Further, the job expects a list of services to deploy them in parallel using the parallel matrix strategy.

Next, let’s add a create-release job in .gitlab-ci.yml under the ci-components project:

$ cat .gitlab-ci.yml 
stages: [release]

create_release:
  stage: release
  image: registry.gitlab.com/gitlab-org/release-cli:latest
  script: echo "Creating release $CI_COMMIT_TAG"
  rules:
    - if: $CI_COMMIT_TAG
  release:
    tag_name: $CI_COMMIT_TAG
    description: "Release $CI_COMMIT_TAG of components repository $CI_PROJECT_PATH"%       

The job gets triggered whenever we create a new tag for the project.

So, let’s push the first tag as 1.0.0 to the project:

$ git tag 1.0.0
$ git push --tags                          
Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
To https://gitlab.com/baeldung-developers/ci-components.git
 * [new tag]         1.0.0 -> 1.0.0

Finally, we’ll see that the create_release job runs for the project:

GitLab Component - Create Release

Our component is ready for use.

6.2. Deploying Monolith

Let’s add the .gitlab-ci.yml file to a monolith repository that uses the deploy_service component to trigger deployment for multiple services:

$ cat .gitlab-ci.yml
stages:
  - deploy

include:
  - component: $CI_SERVER_FQDN/baeldung-developers/ci-components/[email protected]
    inputs:
      services: [s1, s2, s3, s4, s5, s6, s7]

As expected by the deploy_service component, we’ve defined the services we want to deploy.

Further, we can notice the new pipelines for the monolith repository deploy each service in parallel:

GitLab Component - Deploying Monolith

Great! It looks like we’ve nailed this one!

7. Conclusion

In this article, we learned how to share GitLab CI/CD configurations across multiple projects. Further, we explored several strategies for sharing GitLab CI/CD configurations using include and trigger keywords, custom Docker images, and the reusable GitLab CI/CD components.

The sample GitLab pipelines used in this tutorial can be found over on GitHub.