在同一个 GitLab 仓库中运行多个 CI/CD 管道

1. Overview

In GitLab, we use a .gitlab-ci.yml configuration file in our code repository to specify CI/CD pipelines. Since we can’t have multiple .gitlab-ci.yml files on the same branch, we might wonder if it’s possible to have distinct CI/CD workflows. Fortunately, GitLab CI/CD is quite flexible and offers several features we can use to achieve this.

In this tutorial, we’ll explore how to run multiple CI/CD pipelines in the same GitLab repository.

2. Controlling Pipelines and Jobs

In this section, we’ll learn about the workflow and rules keywords used by GitLab to decide when to create a pipeline and which CI pipeline jobs to run in a pipeline.

2.1. Pipeline Creation

Let’s look at the .gitlab-ci.yml configuration file in the main branch of the baeldung-developers/demo project:

$ cat .gitlab-ci.yml
hello_world:
  script:
    - echo "Hello, world!"

GitLab will create a pipeline against the main branch for any code change. The same applies to any other branch forked from the main branch.

Let’s create a dev branch from the main branch and see it in action:

A project with a high number of developers will result in a large number of pipelines. Such behavior can be undesirable, so we can limit the creation of a pipeline by defining rules for the pipeline workflow.

Next, let’s use the workflow keyword to add a rule that matches only the default branch of the project:

$ cat .gitlab-ci.yml
workflow:
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
    
hello_world:
  script:
    - echo "Hello, world!"

We used the CI_COMMIT_BRANCH and CI_DEFAULT_BRANCH CI/CD variables that GitLab predefines for us.

Lastly, we can add multiple rules to define the creation criteria for our pipeline:

workflow:
  rules: 
    - <rule1>
    - <rule2>
    - <rule3>

GitLab evaluates them in order until it finds a rule that satisfies the pipeline creation criteria. Otherwise, GitLab doesn’t create any pipeline.

2.2. Job Creation

Similar to the pipeline creation control, we can use the rules keyword to control which jobs should be part of the pipeline on creation.

Let’s define two mutually exclusive jobs, namely, job1 and job2, in the .gitlab-ci.yml file:

$ cat .gitlab-ci.yml
job1:
  rules:
    - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
  script:
    - echo "job started for the default branch"

job2:
  rules:
    - if: '$CI_PIPELINE_SOURCE != $CI_DEFAULT_BRANCH'
  script:
    - echo "job started for a non-default branch"

We expect that job1 will run only for the main branch, while job2 will run for all other branches.

Next, let’s verify the jobs in the pipeline for the main branch:

Now, let’s create a dev branch from the main branch and verify the pipeline:

It’s important to note that a valid GitLab pipeline must contain at least one job.

Lastly, we can imagine that these controls can generate different pipelines based on the rules defined for the pipeline workflow and individual jobs.

3. Using GitLab UI

GitLab defaults to creating pipelines only for push events, but we can activate other flows by defining appropriate rules. In this section, we will learn how to activate and run pipelines specifically from the GitLab UI.

3.1. On-demand Pipeline

Firstly, let’s see the .gitlab-ci.yml file that creates an on-demand pipeline from the GitLab UI:

$ cat .gitlab-ci.yml
workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "web"

test_job:
  script:
    - echo "job started with ui"

GitLab sets CI_PIPELINE_SOURCE=web for this flow, so we’ve added a rule to ensure pipeline creation for this scenario.

Now, we can use the “Run pipeline” option from the /Pipeline page:

Interestingly, we can define custom variables and choose the branch or tag to be used by the pipeline. So, it allows us to run different pipelines based on custom rules.

Lastly, let’s run the pipeline from the main branch:

As expected, we can see a successful run of the pipeline.

3.2. Scheduled Pipeline

Similar to the on-demand pipeline, we need to add a rule to enable the creation of scheduled pipelines. So, let’s check the .gitlab-ci.yml configuration:

$ cat .gitlab-ci.yml
workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"

test_job:
  script:
    - echo "job started with schedule"

For scheduled pipelines, GitLab sets CI_PIPELINE_SOURCE to schedule.

First, let’s create a new schedule from the “Pipeline schedules” page:

Besides defining variables and branch/tag, we need to specify a cron expression for the schedule. Additionally, we can activate the schedule at the creation time or edit it later.

Next, we’ll be able to see all the schedules that we’ve created on the “Pipeline schedules” page:

For our schedule, we can see that the “Next Run” is scheduled within 3 minutes.

Finally, we can see that a new pipeline runs after waiting for approximately 3 minutes:

It’s important to note that GitLab adds the “Scheduled” label to such pipelines.

4. Using GitLab APIs and Webhook

In this section, let’s learn how to use the GitLab APIs and Webhooks to create a pipeline.

4.1. With Create Pipeline API

Let’s start by looking at .gitlab-ci.yml with a rule to enable the pipeline creation from the API flow:

$ cat .gitlab-ci.yml
workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "api"

test_job:
  script:
    - echo "job started with api flow"

We must note that *GitLab sets CI_PIPELINE_SOURCE=api for this flow*.

Next, let’s use curl to invoke the create pipeline API for the main branch of the baeldung-developers/demo project:

$ curl --request POST --header "PRIVATE-TOKEN: <token>" "https://gitlab.com/api/v4/projects/baeldung-developers%2Fdemo/pipeline?ref=main"

We used the URL-encoded string for the project path. Further, we must replace with a project access token.

Alternatively, we can use the project ID instead of the project path to invoke the API:

$ curl --request POST --header "PRIVATE-TOKEN: <token>" "https://gitlab.com/api/v4/projects/61297443/pipeline?ref=main"

Now, we’ll see on the “Pipelines” page that a new pipeline is created, and GitLab shows the “api” label:

Great! We’ve got this one right.

4.2. With Trigger Pipeline API

Firstly, let’s set the rules in .gitlab-ci.yml to enable the creation of pipelines with trigger flow:

$ cat .gitlab-ci.yml
workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "trigger"

test_job:
  script:
    - echo "job started with trigger flow"

Now, let’s use curl to invoke the trigger pipeline endpoint for the main branch of the project:

$ curl -X POST \
     --fail \
     -F token=<trigger_token> \
     -F ref=main \
https://gitlab.com/api/v4/projects/61297443/trigger/pipeline

We must replace <trigger_token> with a pipeline trigger token. Moreover, GitLab doesn’t accept the URL-encoded path of the project in the trigger endpoint, so we’re limited to using project ID.

Lastly, we can visit the “Pipelines” page to see the recently triggered pipeline:

GitLab labels these pipelines with the “trigger token” keyword.

4.3. With GitLab Webhooks

We can define a Webhook to trigger the pipeline on the “Webhook” page:

Interestingly, GitLab allows us to mask the secret information, such as trigger_token, which is part of our endpoint.

Further, we can choose multiple trigger events, but for simplicity, let’s select the “Comments” event only:

Next, we need to visit a GitLab merge request or issue and add a new comment:

It’s important to note that GitLab won’t trigger the pipeline if we tick the “Make this an internal note” checkbox.

Lastly, let’s visit the “Pipelines” page to verify the newly created pipeline:

Excellent! We nailed this one.

5. Using GitLab ChatOps

With GitLab ChatOps, we can selectively run a standalone job within the pipeline by sending a slash command in Slack or Mattermost. We’ll cover this scenario with Slack integration.

First, let’s look at the rules required in .gitlab-ci.yml to enable pipeline creation using ChatOps flow:

$ cat .gitlab-ci.yml
workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "chat"

test_job:
  script:
    - echo "job started with chat ops flow"

GitLab sets CI_PIPELINE_SOURCE=chat for this scenario.

Next, we can send a Slack slash command to the channel integrated with the GitLab for the Slack app:

/gitlab baeldung-developers/demo run test_job

Now, we can visit the “Pipelines” page to see the pipeline in action:

Lastly, once the job finishes, the “GitLab” app sends the output of the job along with some metadata information as a reply:

Perfect! We’ve got this one working.

6. Conclusion

In this article, we learned how to run multiple CI/CD Pipelines in the same GitLab Repository. We explored how to control the creation of pipelines and jobs using the workflow and rules keywords,

Lastly, we learned how to create pipelines and jobs through different flows, such as GitLab UI, APIs, Webhooks, and ChatOps.