In day-to-day life as an administrator, it is very common to copy and paste text between files, editors, etc. There are a few features built into Windows and Powershell that can help make the copy pasta process easier.
The “clip” utility is a tool that has been built into Windows for a while now (I think since Windows 7), which allows you to redirect command output to the Windows system clipboard. Pairing the clip tool with the capabilities of Powershell piping and you have a very nice way of getting text out of the shell and into a text editor, or wherever else you may want.
The following commands are simple and quick examples of how to pipe output text of a Powershell command to the clipboard. You can use this method to output text from a file or from another Powershell command.
cat <file> | clip
Get-NetAdapater | clip
Then just hit CTRL+V in notepad (or your favorite text tool) and you will have the output of the previously clipped command. In the above example, cat is just an alias for Get-Content in Powershell, carried over from the Linux and Unix shells. Use the Get-Alias command to find at all of the various shortcuts for Powershell commands.
I tend to grab text out of files quite often, so writing out text to the clipboard and pasting elsewhere has saved me countless hours over the years of clicking around in GUI’s with a few keystrokes. Shortcuts like this are often missed or over looked by admins that are used to point and clicking, so make sure you try it out if you haven’t discovered this trick.
Last week I encountered an interesting bug in Rancher that managed to cause some major problems across my Rancher infrastructure. Basically, the bug was causing of the Rancher agent clients to continuously bounce between disconnected/reconnected/finished and reconnecting states, which only manifested itself either after a 12 hour period or by deactivating/activating agents (for example adding a new host to an environment). The only way to temporarily fix the issue was to restart the rancher-server container.
With some help, we were eventually able to resolve the issue. I picked up a few nice lessons along the way and also became intimately acquainted with some of the inner workings of Rancher. Through this experience I learned some tips on how to effectively monitor the Rancher server environment that I would otherwise not have been exposed to, which I would like to share with others today.
All said and done, I view this experience as a positive one. Hitting the bug has not only helped mitigate this specific issue for other users in the future but also taught me a lot about the inner workings of Rancher. If you’re interested in the full story you can read about all of the details about the incident, including steps to reliably reproduce and how the issue was ultimately resolved here. It was a bug specific to Rancher v1.5.1-3, so upgrading to 1.5.4 should fix this issue if you come across it.
Before diving into the specifics for this post, I just want to give a shout out to the Rancher community, including @cjellik, @ibuildthecloud, @ecliptok and @shakefu. The Rancher developers, team and community members were extremely friendly and helpful in addressing and fixing the issue. Between all the late night messages in the Rancher slack, many many logs, countless hours debugging and troubleshooting I just wanted to say thank you to everyone for the help. The small things go a long way, and it just shows how great the growing Rancher community is.
Effective monitoring
I use Sysdig as the main source of container and infrastructure monitoring. To accomplish the metric collection, I run the Sysdig agent as a systemd service when a server starts up so when a server dies and goes away or a new one is added, Sysdig is automatically started up and begins dumping that metric data into the Sysdig Cloud for consumption through the web interface.
I have used this data to create custom dashboards which gives me a good overview about what is happening in the Rancher server environment (and others) at any given time.
The other important thing I discovered through this process, was the role that the Rancher database plays. For the Rancher HA setup, I am using an externally hosted RDS instance for the Rancher database and was able to fine found some interesting correlations as part of troubleshooting thanks to the metrics in Sysdig. For example, if the database gets stressed it can cause other unintended side effects, so I set up some additional monitors and alerts for the database.
Luckily Sysdig makes the collection of these additional AWS metrics seamless. Basically, Sysdig offers an AWS integration which pull in CloudWatch metrics and allows you to add them to dashboards and alert on them from Sysdig, which has been very nice so far.
Below are some useful metrics in helping diagnose and troubleshoot various Rancher server issues.
Memory usage % (server)
CPU % (server)
Heap used over time (server)
Number of network connections (server)
Network bytes by application (server)
Freeable memory over time (RDS)
Network traffic over time (RDS)
As you can see, there are quite a few things you can measure with metrics alone. Often though, this isn’t enough to get the entire picture of what is happening in an environment.
Logs
It is also important to have access to (useful) logs in the infrastructure in order to gain insight into WHY metrics are showing up the way they do and also to help correlate log messages and errors to what exactly is going on in an environment when problems occur. Docker has had the ability for a while now to use log drivers to customize logging, which has been helpful to us. In the beginning, I would just SSH into the server and tail the logs with the “docker logs” command but we quickly found that to be cumbersome to do manually.
One alternative to tailing the logs manually is to configure the Docker daemon to automatically send logs to a centralized log collection system. I use Logstash in my infrastructure with the “gelf” log driver as part of the bootstrap command that runs to start the Rancher server container, but there are other logging systems if Logstash isn’t the right fit. Here is what the relevant configuration looks like.
Just specify the public address of the Logstash log collector and optionally add tags. The extra tags make filtering the logs much easier, so I definitely recommend adding at least one.
Here are a few of the Logstash filters for parsing the Rancher logs. Be aware though, it is currently not possible to log full Java stack traces in Logstash using the gelf input.
if [tag] == "rancher-server" {
mutate { remove_field => "command" }
grok {
match => [ "host", "ip-(?<ipaddr>\d{1,3}-\d{1,3}-\d{1,3}-\d{1,3})" ]
}
# Various filters for Rancher server log messages
grok {
match => [ "message", "time=\"%{TIMESTAMP_ISO8601}\" level=%{LOGLEVEL:debug_level} msg=\"%{GREEDYDATA:message_body}\"" ]
match => [ "message", "%{TIMESTAMP_ISO8601} %{WORD:debug_level} (?<context>\[.*\]) %{GREEDYDATA:message_body}" ]
match => [ "message", "%{DATESTAMP} http: %{WORD:http_type} %{WORD:debug_level}: %{GREEDYDATA:message_body}" ]
}
}
There are some issues open for addressing this, but it doesn’t seem like there is much movement on the topic, so if you see a lot of individual messages from stack traces that is the reason.
One option to mitigate the problem of stack traces would be to run a local log collection agent (in a container of course) on the rancher server host, like Filebeat or Fluentd that has the ability to clean up the logs before sending it to something like Logstash, ElasticSearch or some other centralized logging. This approach has the added benefit of adding encryption to the logs, which GELF does not have (currently).
If you don’t have a centralized logging solution or just don’t care about rancher-server logs shipping to it – the easiest option is to tail the logs locally as I mentioned previously, using the json-file log format. The only additional configuration I would recommend to the json-file format is to turn on log rotation which can be accomplished with the following configuration.
Adding these logging options will ensure that the container logs for rancher-server will never full up the disk on the server.
Bonus: Debug logs
Additional debug logs can be found inside of each rancher-server container. Since these debug logs are typically not needed in day to day operations, they are sort of an easter egg, tucked away. To access these debug logs, they are located in /var/lib/cattle/logs/ inside of the rancher-server container. The easiest way to analyze the logs is to get them off the server and onto a local machine.
Below is a sample of how to do this.
docker exec -it <rancher-server> bash
cd /var/lib/cattle/logs
cp cattle-debug.log /tmp
Then from the host that the container is sitting on you can docker cp the logs out of the container and onto the working directory of the host.
From here you can either analyze the logs in a text editor available on the server, or you can copy the logs over to a local machine. In the example below, the server uses ssh keys for authentication and I chose to copy the logs from the server into my local /tmp directory.
With a local copy of the logs you can either examine the logs using your favorite text editor or you can upload them elsewhere for examination.
Conclusion
With all of our Rancher server metrics dumping into Sysdig Cloud along with our logs dumping into Logstash it has made it easier for multiple people to quickly view and analyze what was going on with the Rancher servers. In HA Rancher environments with more than one rancher-server running, it also makes filtering logs based on the server or IP much easier. Since we use 2 hosts in our HA setup we can now easily filter the logs for only the server that is acting as the master.
As these container based grow up, they also become much more complicated to troubleshoot. With better logging and monitoring systems in place it is much easier to tell what is going on at a glance and with the addition of the monitoring solution we can be much more proactive about finding issues earlier and mitigating potential problems much faster.
Microsoft has been making a lot of inroads in the Open Source and Linux communities lately. Linux and Unix purists undoubtedly have been skeptical of this recent shift. Canonical for example, has caught flak for partnering with Microsoft recently. But the times are changing, so instead of resenting this progress, I chose to embrace it. I’ll even admit that I actually like many of the Open Source contributions Microsoft has been making – including a flourishing Github account, as well as an increasingly rich, and cross platform platform set of tools that includes Visual Studio Code, Ubuntu/Bash for Windows, .NET Core and many others.
If you want to take the latest and greatest in Powershell v6 for a spin on a Linux system, I recommend using a Docker container if available. Otherwise just spin up an Ubuntu (14.04+) VM and you should be ready. I do not recommend trying out Powershell for any type of workload outside of experimentation, as it is still in alpha for Linux. The beta v6 release (with Linux support) is around the corner but there is still a lot of ground that needs to be covered to get there. Since Powershell is Open Source you can follow the progress on Github!
If you use the Docker method, just pull and run the container:
docker run -it --rm ubuntu:16.04 bash
Then add the Microsoft Ubuntu repo:
# apt-transport-https is needed for connecting to the MS repo
apt-get update && apt-get install curl apt-transport-https
If it worked, you should see a message for Powershell and a new command prompt:
# powershell
PowerShell
Copyright (C) 2016 Microsoft Corporation. All rights reserved.
PS />
Congratulations, you now have Powershell running in Linux. To take it for a spin, try a few commands out.
Write-Host "Hello Wordl!"
This should print out a hello world message. The Linux release is still in alpha, so there will surely be some discrepancies between Linux and Windows based systems, but the majority of cmdlets should work the same way. For example, I noticed in my testing that the terminal was very flaky. Reverse search (ctrl+r) and the Get-History cmdlet worked well, but arrow key scrolling through history did not.
You can even run Powershell on OSX now if you choose to. I haven’t tried it yet, but is an option for those that are curious. Needless to say, I am looking for the
Does the Windows built-in version of “curl” confuse or intimidate you? Maybe you come from a Linux or Unix background, and yearn for some of your favorite go-to tools? Newer versions of Powershell include a cmdlet for interacting with the web called Invoke-WebRequest, which is useful, but is not a great drop in replacement for those with experience in non Windows environments. The Powershell cmdlets are a move in the right direction to unifying CLI experiences but there are still many folks that have become attached to curl over the years, including myself. It is worth noting that a Windows compatible version of curl has existed for a long time, however it has always been a nuisance dealing with the zip file, just as using SSH has always been a hassle on Windows. It has always been possible to use the *nix equivalent tools, it is just clunky.
I found a low effort solution for adding curl to my Windows CLI flow, that acts as a nice middle ground between learning Invoke-WebRequest and installing curl binaries directly, which I’d like to share. This alias trick is a simple way to use curl for working with API’s and other various web testing in Windows environments without getting tangled in managing versions, and dealing with vulnerabilities. Just download the latest Docker image to update curl to the newest version, and don’t worry about its implementation across different systems.
Prerequisites are light. First, make sure to have the Docker for Windows app installed (stable or beta are both fine) as well as a semi-recent version of Powershell.
Next step. If you haven’t set up a Powershell profile, there are also lots of links and resources about how to do it. I even wrote about it recently, so I am skipping that step as well. Start by adding the following snippet to your Powershell profile (by default located in C:\Users\<user>\Documents\WindowsPowerShell\Microsoft.PowerShell_profile.ps1) and saving.
# Curl alias using docker
function Docker-Curl {
docker run --rm byrnedo/alpine-curl $args
}
# Aliases
New-Alias dcurl Docker-Curl
Then source you terminal and run the curl command that was just created.
dcurl -h
One issue you might notice from the snippet above is that the Docker image is not an “official” image. If this bothers you (security concerns, etc.), it is really easy to create your own, secure image. There are lots of examples of how to create minimal images with Curl pre-installed. Just be aware that your custom image will need to be maintained and occasionally rebuilt/published to guard against future vulnerabilities. For brevity, I have skipped this process, but here’s an example of creating a custom image.
Optional
To update curl, just run the docker pull command.
docker pull apline-curl
Now you have the best of both worlds. The built-in Invoke-WebRequest cmdlet provided by Powershell is available, as well as the venerable curl command.
My number one case for using curl in a container is that it has been in existence for such a long time (less bugs and edge cases) and it can be used for nearly any web related task. It is also much handier to use curl for those with a background using *nix systems, rather than digging around in unfamiliar Powershell docs for similar functionality. Having the ability to run some of my favorite tools in an easy, reproducible way on Windows has been a refreshing experience while sliding back into the Windows world.
If you have worked with Jenkins for any extended length of time you quickly realize that the Jenkins server configurations can become complicated. If the server ever breaks and you don’t have a good backup of all the configuration files, it can be extremely painful to recreate all of the jobs that you have configured. And most recently if you have started using the Jenkins workflow libraries, all of your custom scripts and coding will disappear if you don’t back it up.
Luckily, backing up your Jenkins job configurations is a fairly simple and straight forward process. Today I will cover one quick and dirty way to backup configs using a Jenkins job.
There are some AWS plugins that will backup your Jenkins configurations but I found that it was just as easy to write a little bit of bash to do the backup, especially since I wanted to backup to S3, which none of the plugins I looked at handle. In genereal, the plugins I looked at either felt a little bit too heavy for what I was trying to accomplish or didn’t offer the functionality I was looking for.
If you are still interested in using a plugin, here are a few to check out:
Keep reading if none of the above plugins look like a good fit.
The first step is to install the needed dependencies on your Jenkins server. For the backup method that I will be covering, the only tools that need to be installed are aws cli, tar and rsync. Tar and rsync should already be installed and to get the aws cli you can download and install it with pip, from the Jenkins server that has the configurations you want to back up.
pip install awscli
After the prerequisites have been installed, you will need to create your Jenkins job. Click New Item -> Freestyle and input a name for the new job.
Then you will need to configure the job.
The first step will be figuring out how often you want to run this backup. A simple strategy would be to backup once a day. The once per day strategy is illustrated below.
Note the ‘H’ above means to randomize when the job runs over the hour so that if other jobs were configured they would try to space out the load.
The next step is to backup the Jenkins files. The logic is all written in bash so if you are familiar it should be easy to follow along.
# Delete all files in the workspace
rm -rf *
# Create a directory for the job definitions
mkdir -p $BUILD_ID/jobs
# Copy global configuration files into the workspace
cp $JENKINS_HOME/*.xml $BUILD_ID/
# Copy keys and secrets into the workspace
cp $JENKINS_HOME/identity.key.enc $BUILD_ID/
cp $JENKINS_HOME/secret.key $BUILD_ID/
cp $JENKINS_HOME/secret.key.not-so-secret $BUILD_ID/
cp -r $JENKINS_HOME/secrets $BUILD_ID/
# Copy user configuration files into the workspace
cp -r $JENKINS_HOME/users $BUILD_ID/
# Copy custom Pipeline workflow libraries
cp -r $JENKINS_HOME/workflow-libs $BUILD_ID
# Copy job definitions into the workspace
rsync -am --include='config.xml' --include='*/' --prune-empty-dirs --exclude='*' $JENKINS_HOME/jobs/ $BUILD_ID/jobs/
# Create an archive from all copied files (since the S3 plugin cannot copy folders recursively)
tar czf jenkins-configuration.tar.gz $BUILD_ID/
# Remove the directory so only the tar.gz gets copied to S3
rm -rf $BUILD_ID
Note that I am not backing up the job history because the history isn’t important for my uses. If the history IS important, make sure to add a line to backup those locations. Likewise, feel free to modify and/or update anything else in the script if it suits your needs any better.
The last step is to copy the backup to another location. This is why we installed aws cli earlier. So here I am just uploading the tar file to an S3 bucket, which is versioned (look up how to configure bucket versioning if you’re not familiar).
export AWS_DEFAULT_REGION="xxx"
export AWS_ACCESS_KEY_ID="xxx"
export AWS_SECRET_ACCESS_KEY="xxx"
# Upload archive to S3
echo "Uploading archive to S3"
aws s3 cp jenkins-configuration.tar.gz s3://<bucket>/jenkins-backup/
# Remove tar.gz after it gets uploaded to S3
rm -rf jenkins-configuration.tar.gz
Replace the AWS_DEFAULT_REGION with the region where the bucket lives (typically us-east-1), make sure to update the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to use an account with access to write to AWS S3 (not covered here). The final thing to note, <bucket> should be replaced to use your own bucket.
The backup process itself is usually pretty fast unless the Jenkins server has a massive amount of jobs and configurations. Once you have configured the job, feel free to run it once to test if it works. If the job worked and returns as completed, go check your S3 bucket and make sure the tar.gz file was uploaded. If you are using versioning there should just be one file, and if you choose the “show versions” option you will see something similar to the following.
If everything went okay with your backup and upload to s3 you are done. Common issues configuring this backup method are choosing the correct AWS bucket, region and credentials. Also, double check where all of your Jenkins configurations live in case there aren’t in a standard location.
