If you are working with the Python based JenkinsAPI library you might run into the No valid crumb was included in the request error. The error below will probably look familiar if you’ve run into this issue.
Traceback (most recent call last):
File "myscript.py", line 47, in <module>
deploy()
File "myscript.py", line 24, in deploy
jenkins.build_job('test')
File "/usr/local/lib/python3.6/site-packages/jenkinsapi/jenkins.py", line 165, in build_job
self[jobname].invoke(build_params=params or {})
File "/usr/local/lib/python3.6/site-packages/jenkinsapi/job.py", line 209, in invoke
allow_redirects=False
File "/usr/local/lib/python3.6/site-packages/jenkinsapi/utils/requester.py", line 143, in post_and_confirm_status
response.text.encode('UTF-8')
jenkinsapi.custom_exceptions.JenkinsAPIException: Operation failed. url=https://jenkins.example.com/job/test/build, data={'json': '{"parameter": [], "statusCode": "303", "redirectTo": "."}'}, headers={'Content-Type': 'application/x-www-form-urlencoded'}, status=403, text=b'<html>\n<head>\n<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>\n<title>Error 403 No valid crumb was included in the request</title>\n</head>\n<body><h2>HTTP ERROR 403</h2>\n<p>Problem accessing /job/test/build. Reason:\n<pre> No valid crumb was included in the request</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.z-SNAPSHOT</a><hr/>\n\n</body>\n</html>\n'
It is good practice to enable additional security in Jenkins by turning on the “Prevent Cross Site Forgery exploits” option in the security settings, so if you see this error it is a good thing. The below example shows this security feature in Jenkins.
The Fix
This error threw me off at first, but it didn’t take long to find a quick fix. There is a crumb_requester class in the jenkinsapi that you can use to create the crumbed auth token. You can use the following example as a guideline in your own code.
from jenkinsapi.jenkins import Jenkins
from jenkinsapi.utils.crumb_requester import CrumbRequester
JENKINS_USER = 'user'
JENKINS_PASS = 'pass'
JENKINS_URL = 'https://jenkins.example.com'
# We need to create a crumb for the request first
crumb=CrumbRequester(username=JENKINS_USER, password=JENKINS_PASS, baseurl=JENKINS_URL)
# Now use the crumb to authenticate against Jenkins
jenkins = Jenkins(JENKINS_URL, username=JENKINS_USER, password=JENKINS_PASS, requester=crumb)
...
The code looks very similar to creating a normal Jenkins authentication object, the only difference being that we create and then pass in a crumb for the request, rather than just a username/password combination. Once the crumbed authentication object has been created, you can continue writing your Python code as you would normally. If you’re interested in learning more about crumbs and CSRF you can find more here, or just Google for CSRF for more info.
This issue was slightly confusing/annoying, but I’d rather deal with an extra few lines of code and know that my Jenkins server is secure.
Having the ability to call Jenkins jobs remotely is pretty slick and adds some extra flexibility and allows for some interesting applications. For example, you could use remote builds to call a script from a chat app or from some other web application. I have chosen to write a quick bash script as a proof of concept, but this could easily be extended or written in a different language via one of its language specific libraries.
The instructions for the method I am using assume that you are using the Jenkins freestyle build as I haven’t experimented much yet with pipelines for remote builds yet.
The first step is to enable remote builds for the Jenkins job that will be triggered. There is an option in the job for “Trigger builds remotely” which allows the job to be called from a script.
The authentication token can be any arbitrary string you choose. Also note the URL below, you will need that later as part of the script to call this job.
With the authentication token configured and the Jenkins URL recorded, you can begin writing the script. The first step is to populate some variables for kicking off the job. Below is an example of how you might do this.
Be sure to fill in these variables with the correctly corresponding values. job_token should correspond to the string you entered above in the Jenkins job, auth should correspond to your github username/token combination. If you are not familiar with Github tokens you can find more information about setting them up here.
As part of the script, you will want to create a Jenkins “crumb” using your Github credentials that will be used to prevent cross-site scripting attacks. Here’s what the creation of the crumb looks like (borrowed from this Stackoverflow post).
In the example job above, I am using several Jenkins parameters as part of the build. The json name values correspond to the parameters. Notice that I am using variables for a few of the values above, make sure those variables are wrapped in singe quotes to correctly escape the json. The syntax for variables is slightly different but allows for some additional flexibility in the job configuration and also allows the script to be called with dynamic values.
If you call this script now, it should kick off a Jenkins job for you with all of the values you have provided.
Update: Thanks to Torben Kerr for adding instructions for setting up these properties at the multibranch build level rather then for each Jenkinsfile. You can check out his “fix” here.
As the Jenkins pipeline functionality continues to rapidly evolve – the project documentation (or lack thereof), has been a consistent pain point as a user. Invariably, the documentation is either out of date or completely missing. I expect the docs to improve as the project matures, but for now, the cake is a lie. I ran into this roadblock recently, looking for a way to limit the number of concurrent builds that happen in Jenkins, using the pipeline. In all of my anguish, I hope this post will help others in avoiding the tediousness of finding the seemingly simple functionality of limiting concurrent builds, as well as give some insight into strategies for figuring out how to find undocumented features in Jenkins.
While this feature is fairly obvious for old-style Jenkins jobs, a simple check box in the job configuration – finding the same functionality for pipelines is seemingly non existent. Through extensive Googling and Stack Overflowing, I discovered this feature was recently added to the Multibranch plugin. Specifically, I found an issue in the (awful) issue tracker used by Jenkins, which in turn led me to uncover some code in a semi recent PR that basically allows concurrency to be turned on or off. Of course when I tried to use the code from the PR it didn’t work right away. So I had to go deeper.
Eventually, I stumbled across a SO post that discusses how to use the properties functionality of pipelines. Equipped with this new piece of information, I finally had enough substance to start playing around with the code. To make the creation of pipelines easier, Jenkins also recently added a snippet generator, which allows users to build out sample snippets quickly.
To use the snippet generator, either drill into an existing pipeline style job using a similar URL as below:
Or create a new job, and click on the “Pipeline Syntax” link after it has been created to test out different snippets.
Inside the snippet generator there are a number of “steps” to choose from. From the information I had already gathered, I just selected the properties step to create the basic skeleton of what I wanted and was able to use the disableConcurrentBuilds() function I found earlier. Below is a snippet of what the code in your Jenkinsfile might actually look like:
node {
// This oneliner is what limits concurrent builds
properties([disableConcurrentBuilds()])
// Do stuff
...
}
Yep. That’s it. Just make sure to put the properties() function at the beginning of the node block, otherwise concurrency won’t be adjusted right away and could lead to problems. Another thing to note; the step to disable concurrency could just as easily be moved into workflow libraries and applied at the global level and applied at the beginning of all jobs if you wanted to limit concurrency for all pipeline builds, since the code is just Groovy. Finally, the code will disable concurrent builds on a per branch basis. Essentially, if you push many different branches it will still build all of them, it will just limit each branch to one build at a time and will queue up jobs for any commits that get pushed after the initial job has been created. I know that is a mouthful. Let me know in the comments if this explanation needs any clarification.
While I love open source software, sometimes project’s move so fast that certain areas of it get neglected. I am thankful for things like Github, because I was able use it to piece together all the other information I found to come up with a solution. But, I would argue having good documentation not only saves folks like me the time and energy of the crazy searches, it also makes it much easier for potentially new users to look at, and understand what is going on. I will be 100% honest and say that Jenkins pipelines are not for the faint of heart, and I’m sure there are many others who will agree with this sentiment. I know it is easier said than done, but anything right now would be an improvement in my opinion.
If you have worked with Jenkins for any extended length of time you quickly realize that the Jenkins server configurations can become complicated. If the server ever breaks and you don’t have a good backup of all the configuration files, it can be extremely painful to recreate all of the jobs that you have configured. And most recently if you have started using the Jenkins workflow libraries, all of your custom scripts and coding will disappear if you don’t back it up.
Luckily, backing up your Jenkins job configurations is a fairly simple and straight forward process. Today I will cover one quick and dirty way to backup configs using a Jenkins job.
There are some AWS plugins that will backup your Jenkins configurations but I found that it was just as easy to write a little bit of bash to do the backup, especially since I wanted to backup to S3, which none of the plugins I looked at handle. In genereal, the plugins I looked at either felt a little bit too heavy for what I was trying to accomplish or didn’t offer the functionality I was looking for.
If you are still interested in using a plugin, here are a few to check out:
Keep reading if none of the above plugins look like a good fit.
The first step is to install the needed dependencies on your Jenkins server. For the backup method that I will be covering, the only tools that need to be installed are aws cli, tar and rsync. Tar and rsync should already be installed and to get the aws cli you can download and install it with pip, from the Jenkins server that has the configurations you want to back up.
pip install awscli
After the prerequisites have been installed, you will need to create your Jenkins job. Click New Item -> Freestyle and input a name for the new job.
Then you will need to configure the job.
The first step will be figuring out how often you want to run this backup. A simple strategy would be to backup once a day. The once per day strategy is illustrated below.
Note the ‘H’ above means to randomize when the job runs over the hour so that if other jobs were configured they would try to space out the load.
The next step is to backup the Jenkins files. The logic is all written in bash so if you are familiar it should be easy to follow along.
# Delete all files in the workspace
rm -rf *
# Create a directory for the job definitions
mkdir -p $BUILD_ID/jobs
# Copy global configuration files into the workspace
cp $JENKINS_HOME/*.xml $BUILD_ID/
# Copy keys and secrets into the workspace
cp $JENKINS_HOME/identity.key.enc $BUILD_ID/
cp $JENKINS_HOME/secret.key $BUILD_ID/
cp $JENKINS_HOME/secret.key.not-so-secret $BUILD_ID/
cp -r $JENKINS_HOME/secrets $BUILD_ID/
# Copy user configuration files into the workspace
cp -r $JENKINS_HOME/users $BUILD_ID/
# Copy custom Pipeline workflow libraries
cp -r $JENKINS_HOME/workflow-libs $BUILD_ID
# Copy job definitions into the workspace
rsync -am --include='config.xml' --include='*/' --prune-empty-dirs --exclude='*' $JENKINS_HOME/jobs/ $BUILD_ID/jobs/
# Create an archive from all copied files (since the S3 plugin cannot copy folders recursively)
tar czf jenkins-configuration.tar.gz $BUILD_ID/
# Remove the directory so only the tar.gz gets copied to S3
rm -rf $BUILD_ID
Note that I am not backing up the job history because the history isn’t important for my uses. If the history IS important, make sure to add a line to backup those locations. Likewise, feel free to modify and/or update anything else in the script if it suits your needs any better.
The last step is to copy the backup to another location. This is why we installed aws cli earlier. So here I am just uploading the tar file to an S3 bucket, which is versioned (look up how to configure bucket versioning if you’re not familiar).
export AWS_DEFAULT_REGION="xxx"
export AWS_ACCESS_KEY_ID="xxx"
export AWS_SECRET_ACCESS_KEY="xxx"
# Upload archive to S3
echo "Uploading archive to S3"
aws s3 cp jenkins-configuration.tar.gz s3://<bucket>/jenkins-backup/
# Remove tar.gz after it gets uploaded to S3
rm -rf jenkins-configuration.tar.gz
Replace the AWS_DEFAULT_REGION with the region where the bucket lives (typically us-east-1), make sure to update the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to use an account with access to write to AWS S3 (not covered here). The final thing to note, <bucket> should be replaced to use your own bucket.
The backup process itself is usually pretty fast unless the Jenkins server has a massive amount of jobs and configurations. Once you have configured the job, feel free to run it once to test if it works. If the job worked and returns as completed, go check your S3 bucket and make sure the tar.gz file was uploaded. If you are using versioning there should just be one file, and if you choose the “show versions” option you will see something similar to the following.
If everything went okay with your backup and upload to s3 you are done. Common issues configuring this backup method are choosing the correct AWS bucket, region and credentials. Also, double check where all of your Jenkins configurations live in case there aren’t in a standard location.
The new Jenkins pipeline integration in 2.0 is pretty awesome and is a great way to add more automation to Jenkins.In this post we will set up Jenkins so that we can write our own custom libraries for Jenkins to use with the pipeline.
One huge benefit of the new pipeline integration in to the core components of Jenkins is the idea of a Jenkinsfile, which enables a way to automatically execute Jenkins jobs on the repo automatically, similar to the way TravisCI works. Simply place a Jenkinsfile in the root of the repo that you wish to automate, set up your webhook so that events to GitHub automatically trigger the Jenkins job and let Jenkins take care of the build.
There are a few very good guides for getting this functionality setup.
Unfortunately, while these guides/docs are very informative and useful they are somewhat confusing to new users and glaze over a few important steps and details that took me longer than it should have to figure out and understand. Therefore the focus will be on some of the details that the mentioned guides lack, especially setting up the built in Jenkins Git repo for accessing and working with the custom pipeline libraries.
Setup
There are many great resources out there already for getting a Jenkins server up and running. For the purposes of this post it will be assumed that you have already create and setup a Jenkins instance, using one of the other Digital Ocean [tutorials](https://www.digitalocean.com/community/tutorials/?q=jenkins).
The first step to getting the pipeline set up, again assuming you have already installed Jenkins 2.0 from one of the guides linked above, is to enable the Jenkins custom Git repo for housing unique pipeline libraries that we will write a little bit later on. There is a section in the workflow plugin tutorial that explains it, but is one of the confusing areas and is buried in the documentation so it will be covered in more detailed below.
In order to be able to clone, read and write to the built in Jenkins Git repo, you will need to add your SSH public key to the server. To do this, the first step will be to configure the server per the SSH plugin and configuring a port to connect to so that the repo can be cloned. This can be found in Jenkins -> Configuration. In this example I used port 2222.
As always security should be a concern, so make sure you have authentication turned on. In this example, I am using GitHub authentication but the process will be similar for other authentication methods. Also make sure you use best practices in locking down any external access by hardening SSH or using other ways of blocking access.
After the Git server has been configured, you will need to add the public key for your user in Jenkins. This was another one of the confusing parts initially. The section to add your personal SSH public key was buried in docs that were glossed over, the page can be found here. The location in Jenkins to add this key is located here,
Notice that we are using port 2222. This is the port we configured via the SSH plugin from above. The port number can be any port you like, just make sure to keep it consistent in the configuration and here.
Working with the pipeline
With the pipeline library repo cloned, we can write a simple function, and then add it back in to Jenkins. By default the repo is called workflowLibs. The basic structure of the repo is to place your custom functions is the vars directory. Within the vars directory you will create a .groovy file for the function and a matching .txt file any documentation of the command you want to add. In our example let’s create a hello function.
Create a hello.groovy and a hello.txt file. Inside the hello.groovy file, add something similar to the following.
echo ‘Hello world!’
Go ahead and commit the hello.groovy file, don’t worry about the hello.txt file.
git add hello.groovy
git commit -m “Hello world example”
git push (You may need to set your upstream)
Obviously this function won’t do much. Once you have pushed the new function you should be able to view it in the dropdown of pipeline functions in the Jenkins build configuration screen.
NOTE: There is a nice little script testing plugin built in as part of the pipeline now called snippet-generator. If you want to test out small scripts on your Jenkins server first before committing and pushing any local changes you can try out your script first in Jenkins. To do this open up any of your Jenkins job configurations and then click the link that says “Pipeline syntax” towards the bottom of the page.
Here’s what the snippet generator looks like.
From the snippet-generator you can build out quite a bit of the functionality that you might want to use as part of your pipeline.
Configuring the job
There are a few different options for setting up Jenkins pipelines. The most common types are the Pipeline or the Multibranch Pipeline. The Pipeline implements all of the scripting capabilities that have been covered so that the power of the Groovy scripting language can be leveraged in the Jenkins job as well as things like application life cycles (via stages) which makes CI/CD much easier.
The Multibranch Pipeline augments the functionality of the Pipeline by adding in the ability to index branches from SCM so that different branches can be easily built and tested. The Multibranch Pipeline is especially useful for larger projects that have many developers and feature branches being worked on because each one of the branches can be tracked separately so developers don’t need to worry as much about overlapping with others work.
Taking the pipeline functionality one step further, it is possible to create a Jenkinsfile with all of the needed pipeline code inside of a repo that so that it can be built automatically. The Jenkinsfile basically is used as a blueprint used to describe the how and what of a project for its build process and can leverage any custom groovy functions that you have written that are on the Jenkins server.
Using a the combination of a GitHub webhook and a Jenkinsfile in a Git repo it is easy to automatically tell your Jenkins server to kick off a build every time a commit or PR happens in GitHub.
Let’s take a look at what an example Jenkinsfile might look like.
node {
stage ‘Checkout’
// Checkout logic goes here
stage ‘Build’
// Build logic goes here
stage ‘Test’
// Test logic goes here
stage ‘Deploy’
// Deploy logic goes here
}
This Jenkinsfile defines various “stages”, which will run through a set of functions described in each stage every time a commit has been pushed or a PR has been opened for a given project. One workflow, as shown above, is to segment the job into build, test, push and deploy stages. Different projects might require different stages so it is nice to have granular control of what the job is doing on a per repo basis.
Bonus: Github Webhooks
Configuring webhooks in GitHub is pretty easy as well. SCM is fairly standard these days for storing source code and there are a number of different Git management tools so the steps should be very similar if using a tool other than GitHub. Setting up a webhook in GitHub can be configured to trigger a Jenkins pipeline build when either a commit is pushed to a branch, like master, or a PR is created for a branch. The advantages of using webhooks should be pretty apparent, as builds are created automatically and can be configured to output their results to various different communication channels like email or Slack or a number of other chat tools. The webhooks are the last step in automating the new pipeline features.
If you haven’t already, you will need to enable the GitHub plugin (https://wiki.jenkins-ci.org/display/JENKINS/GitHub+Plugin) in order to use the GitHub webhooks. No extra configuration should be needed out of the box after installing the plugin.
To configure the webhook, first make sure there is a Jenkinsfile in the root directory of the project. After the Jenkinsfile is in place you will need to set up the webhook. Navigate to the project settings that you would like to create a webhook for, select ‘Settings’ -> ‘Webhooks & services’ . From here there is a button to add a new webhook.
Change the Payload URL to point at the Jenkins server, update the Content type to application/x-www-form-urlencoded, and leave the secret section blank. All the other defaults should be fine.
After adding the webhook, create or update the associated job in Jenkins. Make sure the new job is configured as either a pipeline or multibranch pipeline type.
In the job configuration point Jenkins at the GitHub URL of the project.
Also make sure to select the build trigger to ‘Build when a change is pushed to GitHub’.
You may need to configure credentials if you are using private GitHub repos. This step can be done in Jenkins by navigating to ‘Manage Jenkins’ -> ‘Credentials’ -> ‘Global’. Then Choose ‘Add Credentials’ and select the SSH key used in conjunction with GitHub. After the credentials have been set up there should be an option when configuring jobs to use the SSH key to authenticate with GitHub.
Conclusion
Writing the Jenkinsfiles and custom libraries can take a little bit of time initially to get the hang of but are very powerful. If you already have experience writing Groovy, then writing these functions and files should be fairly straight forward.
The move towards pipelines brings about a number of nice features. First, you can keep track of your Jenkins job definition simply by adding a Jenkinsfile to a repo, so you get all of the nice benefits of history and version tracking and one central place to keep your build configurations. Because groovy is such a flexible language, pipelines give developers and engineers more options and creativity in terms of what their build jobs can do.
One gotcha of this process is that there isn’t a great workflow yet for working with the library functions, so there is a lot of trial and error to get custom functionality working correctly. One good way to debug is to set up a test job and watch for errors in the console output when you trigger a build for it. The combination of the snippet generator script tester though this process has become much easier.
Another thing that can be tricky is the Groovy sandbox. It is mostly and annoyance and I would not suggest turning it off, just be aware that it exists and often times needs to be worked around.
There are many more features and things that you can do with the Pipeline so I encourage readers to go out and explore some of the possibilities, the docs linked to above are a good place for exploring many of these features. As the pipeline matures, more and more plugins are adding the ability to be configured via the pipeline workflow, so if something isn’t possible right now it probably will be very soon.
