When you work with Jenkins pipelines it is pretty common that you create a pipeline and then after some time you need a similar pipeline with just a few small changes. Common practice in this case is to just copy the Jenkinsfile from the first job and make these small changes. But adopting this approach leaves a lot of code duplication in the two Jenkinsfiles. And since a Jenkinsfile is pipeline as code it should follow the same good practices as other pieces of code. And code repetition is not good practice.
What is a Jenkinsfile and infrastructure as code?
Pipeline as code is a concept defining all pipelines as source code. This allows better control over pipelines, it is easier to develop CI/CD pipelines and it simplifies collaboration and knowledge sharing. The pipelines can be defined in Jenkinsfiles. There are two ways to write Jenkinsfiles: declarative and scripted. In this blog post we will focus on the declarative approach.
Standard approach
Usually Jenkinsfiles are written as one big file with several steps, each one handling one or more tasks. If there are more pipelines, there is the same amount of Jenkinsfiles associated with them. It looks like this example:
pipeline {
agent any
stages {
stage('Checkout') {
steps {
<steps to check out repositories>
<steps to check out repositories>
}
}
stage('Build') {
steps {
<steps to build app/images>
<steps to build app/images>
<steps to build app/images>
<steps to build app/images>
}
}
stage('Deploy') {
steps {
<steps to deploy app>
<steps to deploy app>
<steps to deploy app>
}
}
stage('Test') {
steps {
<steps to test app>
<steps to test app>
}
}
}
}This will do the job but a lot of these stages or parts of stages will be repeated in other Jenkinsfiles and it is good practice to avoid this. So what is the solution?
One file to rule them all
First we need to have a file that will serve as a library for functions that will be used in the Jenkinsfile. Let's call this file ‘common.groovy’. Here is an example of what it should look like.
def checkoutRepos(String repoName, String branch) {
<checkout repo code>
}
def buildImages() {
<build images code>
}
def runTests() {
<run tests code>
}
return thisThis common Jenkinsfile needs to have at the very end line: return this. Remember this, it is very easy to miss, and very hard to find what is wrong if you forget it.
The next thing we will need to do is to include this in the Jenkinsfile in which we would like to use the functions stored in the common.groovy file. This can be done in a separate stage and this stage should be the very first stage of the pipeline:
stage("Include common") {
steps {
sh 'sudo rm common -rf; mkdir common'
dir ('common') {
git branch: 'main',
credentialsId: 'github-creds-id',
url: 'https://github.com/example-common.git'
}
script {
script {
common = load("${env.WORKSPACE}/common/common.groovy")
}
}
}
}To use these functions later in the pipeline you need to call them methods on a common object. Important: this needs to be done in the script block:
stage("Checkout") {
steps {
script {
common.checkoutRepo('repo_name', 'branch_name')
}
}
}Jenkinsfile examples
So now I would like to show some examples of how to use this and why it is a good idea to follow this pattern from the start when designing pipelines.
Maps iteration
Jenkinsfiles do not allow you to iterate over a map, so we need to create a function that converts a map to a string on which we will be able to iterate. This will be used in the next example.
@NonCPS
List<List<?>> mapToList(Map map) {
return map.collect { it ->
[it.key, it.value]
}
}One file code repetition
One of the situations when a common function is helpful is when you do something several times and just duplicate the same code. A good example is when you have more than one or two repositories to check out. Here is an example of how it will look:
stage("Checkout repos") {
# First repo checkout
sh "sudo rm <repo1_directory> -rf; mkdir <repo1_directory>"
dir ("<repo2>") {
git branch: "<branch>",
credentialsId: "<creds_id>",
url: "https://github.com/organisation/<repo1_name>.git"
}
# Second repo checkout
sh "sudo rm <repo2_directory> -rf; mkdir <repo2_directory>"
dir ("<repo2>") {
git branch: "<branch>",
credentialsId: "<creds_id>",
url: "https://github.com/organisation/<repo2_name>.git"
}
# Third repo checkout
sh "sudo rm <repo3_directory> -rf; mkdir <repo3_directory>"
dir ("<repo3>") {
git branch: "<branch>",
credentialsId: "<creds_id>",
url: "https://github.com/organisation/<repo3_name>.git"
}
}We can create a function that will handle the whole checkout process and take the repository name, credentials and branch as an argument. Here is example:
def checkoutRepo(String repo, String credentialsId, String branch) {
sh "sudo rm ${repo} -rf; mkdir ${repo}"
dir ("${repo}") {
git branch: "${branch}",
credentialsId: "${credentialsId}",
url: "https://github.com/organisation/${repo}.git"
}
}And then the above function can be used in a Jenkinsfile. This will simplify the whole Jenkinsfile, you will avoid code repetition, and reduce the number of lines of code. Here is an example of how to use it:
# Define map with repos and branches to be checked out
def cloneRepos = [
"repo1": "master",
"repo2": "main",
"repo3": "main"
]
pipeline {
agent any
stages {
stage("Include common") {...}
stage("Checkout repos") {
steps {
script {
for (repo in common.mapToList(cloneRepos)) {
common.checkoutRepo(repo[0], <github_creds_id>, repo[1])
}
}
}
}
}As you can see, the checkout stage is much simpler and has a lot fewer lines of code. The above example downloads three repositories, but it can be any amount you like. The function mapToList, which we created above, is also used to iterate over a map with repository information. If you want to check out only one repository, this can be done without a loop:
stage("Checkout repos") {
steps {
script {
common.checkoutRepo(<repo_name>, <github_creds_id>, <repo_branch>)
}
}Code repetition over a multiple files
You can also use functions defined in common in several different Jenkinsfiles. For example, if you have a deployment job for every environment (for example dev, stage and prod) most of the code is probably the same, only differing in the environment name that needs to be deployed. So you can create a function which will take one argument (or more if there are more variables) that will determine in which environment deployment should happen, and then use this function in different Jenkinsfiles, exactly as in the example above. Important: all Jenkinsiles need to include this common file.
Function with multiple arguments
Sometimes even if most of the code is the same it still differs in many places, like credentials, environment variables, etc. It is still a good way to go with creating one function for this task but instead of passing multiple arguments to function you can pass the map of arguments to have clearer calls to the function.
The function in this case should look like this:
def multipleArgsFunction(Map config) {
def values = ['key1', 'key2', 'key3', 'key4']
for (item in values) {
if (!config[item]) {
println "${item} variable is not set. Please check multipleArgsFunction function description and set appropriate variables."
error("${item} variable is not set. Please check multipleArgsFunction function description and set appropriate variables.")
}
}
sh """
./script1.sh ${config['key1']}
./script2.sh ${config['key2']}
./script3.sh ${config['key3']}
./script4.sh ${config['key4']}
"""
}This function needs to check if the map that is passed has all the necessary values, so we check them in lines 2-6. Simple loop checks of all keys defined in the values variable are set in the config map variable. Values from the config argument can be called config['key'] and this will return the value assigned to the key.
Example of calling the above function from the Jenkinsfile:
def config = [
"key1": "value1",
"key2": "value2",
"key3": "value3",
"key4": "value4",
]
common.multipleArgsFunction(config)Conclusion
Packing parts of Jenkinsfile stages in functions and moving them in a separate file is quite easy and straightforward, and including this function in several different Jenkinsfiles is just as easy. It is good to start writing a Jenkinsfile with this approach to have clean and easy to read code that is easily manageable.
But if you already have some Jenkinsfiles that have a lot of repeated code it is good to make the effort to refactor this code, to dispose of code repetition and move all things to functions and to common files.
