解决 Jenkins “Pipeline sh bad substitution” 错误

1. Introduction

Jenkins is a cornerstone tool in continuous integration and continuous deployment (CI/CD). It allows us to automate various stages of our software development lifecycle. However, as with any powerful tool, Jenkins is not without its quirks. One such common issue is the “Pipeline sh bad substitution” error encountered in Jenkins pipelines. This error can be frustrating, especially when it halts the progress of automated builds and deployments.

In this tutorial, we’ll look into the “bad substitution” error in Jenkins, understand why it occurs, and explore multiple solutions to resolve it. We’ll provide practical examples and tips to help us avoid this error in our Jenkins pipelines, ensuring smoother automation and more reliable builds. Let’s get started!

2. Understanding the “bad substitution” Error

The “bad substitution” error in Jenkins usually occurs when there’s a mismatch between the shell syntax and how Jenkins interprets variables. This error is often seen in the context of shell (sh) steps within a Jenkins pipeline script.

Let’s see a quick pipeline code where this error might arise:

sh 'curl -v --user user:password --data-binary ${buildDir}package${env.BUILD_NUMBER}.tar -X
  PUT "http://artifactory.mydomain.com/artifactory/release-packages/package${env.BUILD_NUMBER}.tar"'

In this example, the sh step attempts to use the env.BUILD_NUMBER variable. However, if not handled correctly, Jenkins throws a “bad substitution” error. This error generally means that the shell script is trying to substitute a variable in a way unsupported by the default shell (sh).

Here are some common causes of this error:

Incorrect quotation marks: Mixing single and double quotes improperly can lead to this issue
Shell discrepancies: Using sh instead of bash can cause compatibility issues with a certain syntax
Variable scope and syntax: Incorrect usage of environment variables and their syntax can result in substitution errors

When this error occurs, it can significantly impact pipeline execution. The affected stage will fail, potentially halting the entire pipeline and preventing subsequent stages from running. This can lead to delayed deployments, interrupted workflows, and considerable troubleshooting time.

3. Practical Description

Let’s assume we have a Jenkins pipeline that uploads a .tar file to an Artifactory server. The pipeline uses the build number to name the file dynamically.

Here is what our pipeline script might look like:

pipeline {
    agent any
    environment {
        buildDir = "/var/lib/jenkins/workspace/Package_Deploy_Pipeline/"
    }
    stages {
        stage('Upload') {
            steps {
                sh 'curl -v --user user:password --data-binary ${buildDir}package${env.BUILD_NUMBER}.tar -X
                  PUT "http://artifactory.mydomain.com/artifactory/release-packages/package${env.BUILD_NUMBER}.tar"'
            }
        }
    }
}

In this script, we use ${env.BUILD_NUMBER} to dynamically include the Jenkins build number in the filename. However, this often leads to a “bad substitution” error because of how the shell interprets the variables.

When the script runs, Jenkins throws the following error:

[Pipeline] sh
[Package_Deploy_Pipeline] Running shell script
/var/lib/jenkins/workspace/Package_Deploy_Pipeline@tmp/durable-4c8b7958/script.sh:
  2: /var/lib/jenkins/workspace/Package_Deploy_Pipeline@tmp/durable-4c8b7958/script.sh:
  Bad substitution
[Pipeline] } //node
[Pipeline] Allocate node : End
[Pipeline] End of Pipeline
ERROR: script returned exit code 2

Interestingly, if we hardcode the build number, the script works without any issues:

sh 'curl -v --user user:password --data-binary ${buildDir}package113.tar -X
  PUT "http://artifactory.mydomain.com/artifactory/release-packages/package113.tar"'

This scenario highlights the problem with variable substitution in the shell script. Understanding this specific issue helps us pinpoint the solution more effectively.

4. Diagnosing the Issue

To effectively resolve the “bad substitution” error, we need to systematically diagnose the problem and identify the root cause.

First, we should ensure that we can consistently reproduce the error. To do this, we should use a simplified version of our pipeline script to isolate the issue.

Let’s see a simplified example of our pipeline script for proper diagnosis:

pipeline {
    agent any
    environment {
        buildDir = "/var/lib/jenkins/workspace/Package_Deploy_Pipeline/"
    }
    stages {
        stage('Test') {
            steps {
                sh 'echo ${env.BUILD_NUMBER}'
            }
        }
    }
}

Jenkins pipelines are written in Groovy, which adds another layer of complexity when interacting with shell commands. If not handled correctly, Groovy’s string handling can interfere with the shell’s variable substitution.

In our scenario, we should run this script and verify that it produces the “bad substitution” error. If it does, we’ve successfully isolated the problem.

After reproducing the error, we should examine the shell script syntax within the sh step. The most common cause of the “bad substitution” error is the incorrect usage of quotation marks. When using single quotes (‘), the shell interprets everything literally and does not perform variable substitution.

After isolating the problem and identifying the root cause as single quotes, let’s now discuss multiple ways to fix the issue.

5. Using Double Quotes for Variable Substitution

One straightforward solution is to use double quotes (“) for variable interpolation in the shell script. This ensures that the shell correctly substitutes the variables.

Let’s see how we can modify our sample pipeline script to make use of double quotes:

pipeline {
    agent any
    environment {
        buildDir = "/var/lib/jenkins/workspace/Package_Deploy_Pipeline/"
    }
    stages {
        stage('Upload') {
            steps {
                sh "curl -v --user user:password
                  --data-binary ${buildDir}package${env.BUILD_NUMBER}.tar -X
                  PUT \"http://artifactory.mydomain.com/artifactory/release-packages/package${env.BUILD_NUMBER}.tar\""
            }
        }
    }
}

In this example, we wrap the entire sh command in double quotes. This allows for the proper interpolation of ${env.BUILD_NUMBER} and other variables.

This method is simple, straightforward, and easy to understand and implement.

On the other hand, it may lead to complex escaping if the command itself contains double quotes. There is also limited flexibility in handling more complex scripts.

6. Using Triple Quotes for Groovy String Interpolation

Another effective solution is to use triple double quotes (“””) for Groovy string interpolation. This method allows for cleaner handling of complex strings and reduces the need for escaping characters.

Let’s see how we can modify our sample pipeline script using triple-double quotes:

pipeline {
    agent any
    environment {
        buildDir = "/var/lib/jenkins/workspace/Package_Deploy_Pipeline/"
    }
    stages {
        stage('Upload') {
            steps {
                sh """curl -v --user user:password
                  --data-binary ${buildDir}package${env.BUILD_NUMBER}.tar -X PUT "http://artifactory.mydomain.com/artifactory/release-packages/package${env.BUILD_NUMBER}.tar""""
            }
        }
    }
}

Here, using triple quotes simplifies handling complex strings and avoids many common pitfalls associated with single and double quotes.

This approach is beneficial when dealing with long or complex shell commands and when the shell command contains multiple nested quotes. It helps to improve the pipeline script’s readability and maintainability. In short, we have cleaner and more readable code while reducing the need for escaping characters.

Contrastingly, this method has a slightly more complex syntax that may not be immediately familiar to all users.

7. Ensuring Correct Shell Usage

The shell (sh) used in the pipeline can also influence the occurrence of the “bad substitution” error. By default, Jenkins uses the sh shell, which may not support certain script syntax. Explicitly specifying the Bash shell can resolve these issues.

Let’s see an example of how to modify our script to use the Bash shell:

pipeline {
    agent any
    environment {
        buildDir = "/var/lib/jenkins/workspace/Package_Deploy_Pipeline/"
    }
    stages {
        stage('Upload') {
            steps {
                sh '''#!/bin/bash -xe
                curl -v --user user:password
                  --data-binary ${buildDir}package${env.BUILD_NUMBER}.tar -X PUT "http://artifactory.mydomain.com/artifactory/release-packages/package${env.BUILD_NUMBER}.tar"'''
            }
        }
    }
}

By adding #!/bin/bash -xe at the beginning of the sh command, we ensure that Bash interprets the script, which supports more complex syntax and variable substitution. This ensures compatibility with Bash-specific features and syntax and reduces the risk of encountering “bad substitution” errors due to shell discrepancies.

Lastly, it provides better debugging capabilities with the -xe options. This gives us greater control over the shell environment with improved compatibility and debugging.

However, it can be a slightly more complex setup, requiring familiarity with shell scripting and shebang usage.

8. Using withEnv for Environment Variables

Another robust method to handle environment variables in Jenkins pipelines is to use the withEnv block. This approach explicitly defines environment variables within the pipeline, ensuring they are correctly recognized and substituted in the shell scripts.

Let’s see an example of using the withEnv block to avoid the “bad substitution” error:

pipeline {
    agent any
    environment {
        buildDir = "/var/lib/jenkins/workspace/Package_Deploy_Pipeline/"
    }
    stages {
        stage('Upload') {
            steps {
                withEnv(["BUILD_NUMBER=${env.BUILD_NUMBER}"]) {
                    sh '''#!/bin/bash -xe
                    curl -v --user user:password
                      --data-binary ${buildDir}package${BUILD_NUMBER}.tar -X PUT "http://artifactory.mydomain.com/artifactory/release-packages/package${BUILD_NUMBER}.tar"'''
                }
            }
        }
    }
}

In this script, the withEnv block ensures that BUILD_NUMBER is correctly exported to the shell environment, allowing for proper substitution in the shell script.

With this method, we have explicit environment control by defining which environment variables are available to the shell script. Also, substitution errors are reduced by ensuring variables are properly exported. Lastly, this method is flexible as it can be combined with other methods for greater environmental control.

On the other hand, its syntax is slightly more complex as it requires additional configuration for each environment variable.

9. Combining Different Techniques

In some cases, we may need to combine different techniques to resolve the “bad substitution” error, especially in complex Jenkins pipelines. Combining double quotes with withEnv and explicit shell usage can provide a more robust solution.

Let’s see the combination of these methods in action:

pipeline {
    agent any
    environment {
        buildDir = "/var/lib/jenkins/workspace/Package_Deploy_Pipeline/"
    }
    stages {
        stage('Upload') {
            steps {
                withEnv(["BUILD_NUMBER=${env.BUILD_NUMBER}"]) {
                    sh """#!/bin/bash -xe
                    curl -v --user user:password
                      --data-binary ${buildDir}package${BUILD_NUMBER}.tar -X
                      PUT "http://artifactory.mydomain.com/artifactory/release-packages/package${BUILD_NUMBER}.tar""""
                }
            }
        }
    }
}

In this script, we use the withEnv block to define BUILD_NUMBER. We also use triple-double quotes for cleaner string handling. Then, we specify the Bash shell for improved compatibility and debugging.

This combination method is highly robust and flexible. Clear and maintainable pipeline scripts minimize the risk of errors. However, it can be more complex as it requires familiarity with multiple techniques and best practices.

Notably, we should note some best practices for robust Jenkins pipelines.

First, we should consistently use quotation marks and shell scripts. To enhance clarity, we should use withEnv to explicitly define environment variables. We should also enable shell debugging options (e.g., -xe) to troubleshoot issues more effectively.

Furthermore, certain operations might be restricted if we’re using the Groovy sandbox for security. We can temporarily disable the sandbox for testing purposes but remember to re-enable it for production use.

Lastly, we can incrementally build commands. We can start with a simple command and gradually build up to a complex one, testing at each step.

10. Conclusion

In this article, we explored the common causes of the Jenkins “Pipeline sh bad substitution” error and provided comprehensive solutions. We’ve learned about proper quoting techniques, environment variable handling, parameter substitution, and shell script considerations within Jenkins pipelines.

By following the best practices and prevention strategies discussed, we can significantly reduce the occurrence of these errors in our CI/CD workflows. Finally, attention to detail in pipeline scripting is crucial for maintaining robust and reliable automation processes.

Persistence

REST

Security