Fix the JenkinsAPI No valid crumb error

June 7, 2017 Josh Reichardt 1 Comment

If you are working with the Python based JenkinsAPI library you might run into the No valid crumb was included in the request error. The error below will probably look familiar if you’ve run into this issue.

Traceback (most recent call last):
 File "myscript.py", line 47, in <module>
 deploy()
 File "myscript.py", line 24, in deploy
 jenkins.build_job('test')
 File "/usr/local/lib/python3.6/site-packages/jenkinsapi/jenkins.py", line 165, in build_job
 self[jobname].invoke(build_params=params or {})
 File "/usr/local/lib/python3.6/site-packages/jenkinsapi/job.py", line 209, in invoke
 allow_redirects=False
 File "/usr/local/lib/python3.6/site-packages/jenkinsapi/utils/requester.py", line 143, in post_and_confirm_status
 response.text.encode('UTF-8')
jenkinsapi.custom_exceptions.JenkinsAPIException: Operation failed. url=https://jenkins.example.com/job/test/build, data={'json': '{"parameter": [], "statusCode": "303", "redirectTo": "."}'}, headers={'Content-Type': 'application/x-www-form-urlencoded'}, status=403, text=b'<html>\n<head>\n<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>\n<title>Error 403 No valid crumb was included in the request</title>\n</head>\n<body><h2>HTTP ERROR 403</h2>\n<p>Problem accessing /job/test/build. Reason:\n<pre> No valid crumb was included in the request</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.z-SNAPSHOT</a><hr/>\n\n</body>\n</html>\n'

It is good practice to enable additional security in Jenkins by turning on the “Prevent Cross Site Forgery exploits” option in the security settings, so if you see this error it is a good thing. The below example shows this security feature in Jenkins.

The Fix

This error threw me off at first, but it didn’t take long to find a quick fix. There is a crumb_requester class in the jenkinsapi that you can use to create the crumbed auth token. You can use the following example as a guideline in your own code.

from jenkinsapi.jenkins import Jenkins
from jenkinsapi.utils.crumb_requester import CrumbRequester

JENKINS_USER = 'user'
JENKINS_PASS = 'pass'
JENKINS_URL = 'https://jenkins.example.com'

# We need to create a crumb for the request first
crumb=CrumbRequester(username=JENKINS_USER, password=JENKINS_PASS, baseurl=JENKINS_URL)

# Now use the crumb to authenticate against Jenkins
jenkins = Jenkins(JENKINS_URL, username=JENKINS_USER, password=JENKINS_PASS, requester=crumb)

...

The code looks very similar to creating a normal Jenkins authentication object, the only difference being that we create and then pass in a crumb for the request, rather than just a username/password combination. Once the crumbed authentication object has been created, you can continue writing your Python code as you would normally. If you’re interested in learning more about crumbs and CSRF you can find more here, or just Google for CSRF for more info.

This issue was slightly confusing/annoying, but I’d rather deal with an extra few lines of code and know that my Jenkins server is secure.

Remote Jenkins builds using Github auth

May 22, 2017 Josh Reichardt Leave a comment

Having the ability to call Jenkins jobs remotely is pretty slick and adds some extra flexibility and allows for some interesting applications. For example, you could use remote builds to call a script from a chat app or from some other web application. I have chosen to write a quick bash script as a proof of concept, but this could easily be extended or written in a different language via one of its language specific libraries.

The instructions for the method I am using assume that you are using the Jenkins freestyle build as I haven’t experimented much yet with pipelines for remote builds yet.

The first step is to enable remote builds for the Jenkins job that will be triggered. There is an option in the job for “Trigger builds remotely” which allows the job to be called from a script.

The authentication token can be any arbitrary string you choose. Also note the URL below, you will need that later as part of the script to call this job.

With the authentication token configured and the Jenkins URL recorded, you can begin writing the script. The first step is to populate some variables for kicking off the job. Below is an example of how you might do this.

jenkins_url="https://jenkins.example.com"
job_name="my-jenkins-job"
job_token="xxxxx"
auth="username:token"
my_repo="some_git_repo"
git_tag="abcd123"

Be sure to fill in these variables with the correctly corresponding values. job_token should correspond to the string you entered above in the Jenkins job, auth should correspond to your github username/token combination. If you are not familiar with Github tokens you can find more information about setting them up here.

As part of the script, you will want to create a Jenkins “crumb” using your Github credentials that will be used to prevent cross-site scripting attacks. Here’s what the creation of the crumb looks like (borrowed from this Stackoverflow post).

crumb=$(curl -s 'https://'auth'@jenkins.example.com/crumbIssuer/api/xml?xpath=concat(//crumbRequestField,":",//crumb)')

Once you have your variables configured and your crumb all set up, you can test out the Jenkins job.

curl -X POST -H "$crumb" $jenkins_url/job/$job_name/build?token=$job_token \
  --user $auth \
  --data-urlencode json='
  {"parameter":
    [
      {"name":"parameter1", "value":"test1"},
      {"name":"parameter2", "value":"test2"},
      {"name":"git_repo", "value":"'$my_repo'"},
      {"name":"git_tag", "value":"'$git_tag'"}
    ]
  }'

In the example job above, I am using several Jenkins parameters as part of the build. The json name values correspond to the parameters. Notice that I am using variables for a few of the values above, make sure those variables are wrapped in singe quotes to correctly escape the json. The syntax for variables is slightly different but allows for some additional flexibility in the job configuration and also allows the script to be called with dynamic values.

If you call this script now, it should kick off a Jenkins job for you with all of the values you have provided.

Copy text to clipboard using Powershell

May 3, 2017May 3, 2017 Josh Reichardt 1 Comment

In day-to-day life as an administrator, it is very common to copy and paste text between files, editors, etc. There are a few features built into Windows and Powershell that can help make the copy pasta process easier.

The “clip” utility is a tool that has been built into Windows for a while now (I think since Windows 7), which allows you to redirect command output to the Windows system clipboard. Pairing the clip tool with the capabilities of Powershell piping and you have a very nice way of getting text out of the shell and into a text editor, or wherever else you may want.

The following commands are simple and quick examples of how to pipe output text of a Powershell command to the clipboard. You can use this method to output text from a file or from another Powershell command.

cat <file> | clip
Get-NetAdapater | clip

Then just hit CTRL+V in notepad (or your favorite text tool) and you will have the output of the previously clipped command. In the above example, cat is just an alias for Get-Content in Powershell, carried over from the Linux and Unix shells. Use the Get-Alias command to find at all of the various shortcuts for Powershell commands.

I tend to grab text out of files quite often, so writing out text to the clipboard and pasting elsewhere has saved me countless hours over the years of clicking around in GUI’s with a few keystrokes. Shortcuts like this are often missed or over looked by admins that are used to point and clicking, so make sure you try it out if you haven’t discovered this trick.

Monitoring email flow with MFM

May 1, 2017July 17, 2019 Josh Reichardt Leave a comment

This is a sponsored post by the folks over at EveryCloud. They have recently developed and released a new tool to help manage and troubleshoot email issues, which is starting to get some traction, especially among Exchange environments. As a mail admin in a previous life, I can sympathize with desire for better monitor tools. Here’s their post.

Managing mail flow is a challenge for every systems administrator, and the price of a mistake is very high. Any interruption in mail flow can spell disaster for a company, disrupting daily operations and leaving the management team, the IT team and the systems administrator scrambling for solutions.

While there are a number of mail flow solutions on the market, they tend to be quite pricey, making it difficult for systems administrators, especially those who work for small businesses and start-ups, to justify the cost.

For those who do not already know, the makers behind the EveryCloud mail flow monitor have recently launched a free service – Mail Flow Monitor (MFM). EveryCloud MFM tool is the only free round-trip mail flow monitor on the market, giving systems administrators the ability to observe their organizations’ email systems 24 hours a day, 7 days a week and 365 days a year, all without spending a penny.

Some of the features of Mail Flow Monitor include:

A full-featured round trip monitor, with start-to-finish email tracking and monitoring
Systems administrators can receive real-time text and email alerts whenever a delay or rejection occurs – to your cell phone as well an email or to your alternative email address.
Timely monitoring means issues can be addressed quickly, before they spiral out of control
The system sends a test email every few minutes to a monitoring mailbox on your server. You set up a forward to send the emails back and the Everycloud team does the rest.
MFM is cloud based, which means there is nothing to update or manage.
MSP’s and IT Resellers can create an account and manage as many customers as they wish via the EveryCloud Partner Area, all completely free!

When you consider that competing mail filtering solutions generally cost about $30 a month, it is easy to see the saving potential. That $360 annual cost savings may not seem like much, but since it is assessed on a domain level, the charges can add up quickly. In addition, the per-domain charges can make managing a complex IT operation difficult, an extra level of hard work that systems administrators do not need.

From the smallest startups to the largest multinational corporations, modern businesses live and die on their email. An unexpected email breakdown, significant bottleneck or major failure could make the firm’s email inaccessible and unreliable for hours or even days, and every minute of downtime is costing the company money.

Fix Google Analytics search queries in WordPress

April 28, 2017April 28, 2017 Josh Reichardt Leave a comment

I embarrassingly discovered the other day that I was not receiving metrics or analytics about keyword queries in the Google Analytics console. It turns out that problem was twofold. First, I didn’t have the SSL version of my site enabled in the Google webmaster tools and second, I was serving a cached version of my sitemap that was several months out of date.

To give you an idea of how this issue manifested itself, and how I discovered that there were issues in the first place – here’s what my search keywords were showing as in the Google Analytics console.

Clearly the data is less than useful. The solution to this problem is pretty easy to fix at least.

Fixing the webmaster properties

Open up the Google webmasters site (you should have this setup already, if not go ahead and get signed up and add your WordPress site). If you have recently updated your site to use https, make sure you add a new property in the webmaster tools for the https version of your site that matches your http version.

Doing this will tell google to keep track of search queries for the https version of your site, which should be the default after swtiching. After adding the new https property and indexing it, give it a day or two to start collecting metrics, and check back. Now when you check your search query traffic in the webmaster tools you should start to see all of the search results.

Also be sure to update properties to use https in both the Webmaster console as well as the Analytics console. For example, in the Analytics console under Admin -> Property settings -> default URL, there is an option to use http or https. Likewise, in the webmaster console there is an option for defaulting to http, which is buried in the Google Analytics interface under Admin -> Property settings -> Search Console. Make sure you update ALL of the site settings to use https.

NOTE: It can take some time for queries to begin showing up in the Google Analytics console (it took about two days for them to start showing up for my site after fixing all the https references).

Fixing sitemaps issues

If you find that Google isn’t indexing and using all of your posts and pages, the next thing to look at the sitemaps. A quick way to know if you your sitemaps file is doing its job is to pull open the sitemaps, which can be found under the Crawl -> Sitemaps menu.

The above shows what a healthy sitemap index looks like (after I corrected the problem). There is a button located in the top left of this view that can help you test your sitemap while you are updating your settings. First check for any items in the “issues” column. Also, if the “processed” date here isn’t recent then there is probably an issue. One last thing to check – if there are either no entries in this view or fewer then you expect, something is probably not working.

There are many more knobs and dials you can adjust in the webmaster tools, so if you haven’t played with them much I would recommend spending some time and poking around.

I should quickly mention that my solution assumes that you are using the Google XML Sitemaps plugin. If you’re not using this plugin, and you are either 1) new to WordPress or 2) don’t want to manage your sitemaps file manually, I suggest you enable it. It makes sitemap management so much easier.

After you have the plugin turned on, navigate to your blog settings for sitemaps, which can be located in Dashboard -> Settings -> XML-Sitemap. Clicking the popup should bring up a page similar to the following.

First, make sure everything looks correct in the settings. If you are setting this up for the first time you might need to configure some of the settings. For example, make sure the site name matches the listing, and the options to notify search indexers are all turned on.

When I was troubleshooting the search queries not getting set, I navigated to this menu and immediately noticed that the plugin was showing a warning about using a cached version of my sitemaps.xml file. To fix this warning, there should be an option to remove the cached versions.

Next, there should be an option near the top of the sitemap settings to “notify search engines about your sitemap”. After you have adjusted all the sitemap settings and cleared the cached sitemaps file, click that link to trigger a ping to the search indexers.

Be aware that the crawling process may take up to a few days to index and update so be patient.