Kubernetes CLI Tricks

May 7, 2018May 13, 2018 Josh Reichardt 1 Comment

Kubernetes is complicated, as you’ve probably already discovered if you’ve used Kubernetes before. Likewise, the Kubectl command line tool can pretty much do anything but can feel cumbersome, clunky and generally overwhelming for those that are new to the Kubernetes ecosystem. In this post I want to take some time to describe a few of the CLI tools that I have discovered that help ease the pain of working with and managing Kubernetes from the command line.

There are many more tools out there and the list keeps growing, so I will probably revisit this post in the future to add more cool stuff as the community continues to grow and evolve.

Where to find projects?

As a side note, there are a few places to check for tools and projects. The first is the CNCF Cloud Native Landscape. This site aims to keep track of all the various different projects in the Cloud/Kubernetes world. An entire post could be written about all of the features and and filters but at the highest level it is useful for exploring and discovering all the different evolving projects. Make sure to check out the filtering capabilities.

The other project I have found to be extremely useful for finding different projects is the awesome-kubernetes repo on Github. I found a number of tools mentioned in this post because of the awesome-kubernetes project. There is some overlap between the Cloud Native Landscape and awesome-kubernetes but they mostly compliment each other very nicely. For example, awesome-kubernetes has a lot more resources for working with Kubernetes and a lot of the smalller projects and utilities that haven’t made it into the Cloud Native Landscape. Definitely check this project out if you’re looking to explore more of the Kubernetes ecosystem.

Kubectl tricks

These are various little tidbits that I have found to help boost my productivity from the CLI.

Tab completion – The first thing you will probably want to get working when starting. There are just too many options to memorize and tab completion provides a nice way to look through all of the various commands when learning how Kubernetes works. To install (on OS X) run the following command.

brew install bash-completion

In zsh, adding the completion is as simple as running source <(kubectl completion bash). The same behavior can be accomplished in zsh using source <(kubectl completion zsh).

Aliases and shortcuts – One distinct flavor of Kubernetes is how cumbersome the CLI can be. If you use Zsh and something like oh-my-zsh, there is a default set of aliases that work pretty well, which you can find here. There are a many posts about aliases out there already so I won’t go into too much detail about them. I will say though that aliasing k to kubectl is one of the best time savers I have found so far. Just add the following snippet to your bash/zsh profile for maximum glory.

alias k=kubectl

kubectl –export – This is a nice hidden feature that basically allows users to switch Kubernetes from imperative (create) to declarative (apply). The --export flag will basically take an existing object and strip out unwanted/unneeded metadata like statuses and timestamps and present a clear version of what’s running, which can then be exported to a file and applied to the cluster. The biggest advantage of using declarative configs is the ability to mange and maintain them in git repos.

kubectl top – In newer versions, there is the top command, which gives a high level overview of CPU and memory utilization in the cluster. Utilization can be filtered at the node level as well as the pod level to give a very quick and dirty view into potential bottlenecks in the cluster. In older versions, Heapster needs to be installed for this functionaliity to work correctly, and in newer versions needs metrics-server to be running.

kubectl explain – This is a utility built in to Kubectl that basically provides a man page for what each Kubernetes resource does. It is a simple way to explore Kubernetes without leaving the terminal

kubectx/kubens

This is an amazing little utility for quickly moving between Kubernetes contexts and namespaces. Once you start working with multiple different Kubernetes clusters, you notice how cumbersome it is to switch between environments and namespaces. Kubectx solves this problem by providing a quick and easy way to see what environments and namespaces a user is currently in and also quickly switch between them. I haven’t had any issues with this tool and it is quickly becoming one of my favorites.

stern

Dealing with log output using Kubectl is a bit of a chore. Stern (and similarly kail) offer a much nicer user experience when dealing with logs. These tools allow users the ability to do things like show logs for multiple containers in pod, use regex matching to tail logs for specific containers, give nice colored output for distinguishing between logs, filter logs by namespaces and a bunch of other nice features.

Obviously for a full setup, using an aggregated/centralized logging solution with something like Fluenctd or Logstash would be more ideal, but for examining logs in a pinch, these tools do a great job and are among my favorites. As an added bonus, I don’t have to copy/paste log names any more.

yq is a nice little command line tool for parsing yaml files, which works in a similar way to the venerable jq. Parsing, reading, updating yaml can sometimes be tricky and this tool is a great and lightweight way to manipulate configurations. This tool is especially useful for things like CI/CD where a tag or version might change that is nested deep inside yaml.

There is also the lesser known jsonpath option that allows you to interact with the json version of a Kubernetes object, baked into kubectl. This feature is definitely less powerful than jq/yq but works well when you don’t want to overcomplicate things. Below you can see we can use it to quickly grab the name of an object.

kubectl get pods -o=jsonpath='{.items[0].metadata.name}'

Working with yaml and json for configuration in general seems to be an emerging pattern for almost all of the new projects. It is definitely worth learning a few tools like yq and jq to get better at parsing and manipulating data using these tools.

ksonnet / jsonnet

Similar to the above, ksonnet and jsonnet are basically templating tools for working with Kubernetes and json objects. These two tools work nicely for managing Kubernetes manifests and make a great fit for automating deployments, etc. with CI/CD.

ksonnet and jsonnet are gaining popularity because of their ease of use and simplicity compared to a tool like Helm, which also does templating but needs a system level permission pod running in the Kubernetes cluster. Jsonnet is all client side, which removes the added attack vector but still provides users with a lot of flexibility for creating and managing configs that a templating language provides.

More random Kubernetes tricks

Since 1.10, kubectl has the ability to port forward to resource name rather than just a pod. So instead of looking up pods that are running and connecting to one all the time, you can just grab the service name or deployment and just port forward to it.

port-forward TYPE/NAME [LOCAL_PORT:]REMOTE_PORT
k port-forward deployment/mydeployment 5000:6000

New in 1.11, which will be dropping soonish, there is a top level command called api-resource, which allows users to view and interact with API objects. This will be a nice troubleshooting tool to have if for example you are wanting to see what kinds of objects are in a namespace. The following command will show you these objects.

k api-resources --verbs=list --namespace -o name | xargs -n 1 kubectl get -o name -n foo

Another handy trick is the ability to grab a base64 string and decode it on the fly. This is useful when you are working with secrets and need to quickly look at what’s in the secret. You can adapt the following command to accomplish this (make sure you have jq installed).

k get secret my-secret --namespace default -o json | jq -r '.data | .["secret-field"]' | base64 --decode

Just replace .["secret-field"] to use your own field.

UPDATE: I just recently discovered a simple command line tool for decoding base64 on the fly called Kubernetes Secret Decode (ksd for short). This tool looks for base64 and renders it out for you automatically so you don’t have to worry about screwing around with jq and base64 to extract data out when you want to look at a secret.

k get secret my-secret --namespace default -o json | ksd

That command is much cleaner and easier to use. This utility is a Go app and there are binaries for it on the releases page, just download it and put it in your path and you are good to go.

Conclusion

The Kubernetes ecosystem is a vast world, and it only continues to grow and evolve. There are many more kubectl use cases and community to tools that I haven’t discovered yet. Feel free to let me know any other kubectl tricks you know of, and I will update them here.

I would love to grow this list over time as I get more acquainted with Kubernetes and its different tools.

Add Office 365 Calendars to Google Calendar

November 3, 2017 Josh Reichardt 1 Comment

I ran into a scenario recently where I wanted to be able to combine both my work (Office 365) and personal (Google Calendar) calendars, which I found to be a really painful process. Likewise, the information on the web turned out to be equally painful. The main motivation for doing this is so that I can sync all of my calendars to my (Android phone), so that when I’m away from the computer I can easily see what is on both my personal schedule as well as the work calendar.

Since I started using Office 365 I have found a number of small annoyances, and this post servers to illustrate a good example of one. Office 365 is a good product, there are just certain things that are needlessly confusing and complicated as well as a few things that could just work better out of the box. My opinion is probably also skewed, due to the fact that I haven’t used any of the Microsoft Office online offerings for a while now.

The first step is to publish/export the calendar on the Office 365 side of things. The setting to export the calendar is completely buried in the O365 web app but can be located by navigating to Settings -> Your app settings -> Calendar

This should open a navigation menu on the left with more options. From here you can find Shared Calendars -> Publish Calendars settings.

From here you can select which Office 365 calendar you’d like to publish and also the permissions to share. Once those options have been chosen, you can save and a link will be generated for both the HTML and ICS versions of the calendar.

It turns out that only the ICS link will work on the Google Calendar side of things. I did have some trouble when I was playing around with the HTML version, which turns out is because Google doesn’t cooperate with HTML calendars, only ICS.

Now you can import the ICS calendar on the Google Calendar side of things. There are a few ways to do this. Pull open the settings in Google Calendar and either use the Import & Export feature to upload the ICS file generated above or add the ICS URL to the shared calendar using the Add Calendar -> From URL option. Note again that the HTML version of the URL won’t work.

Using either the ICS web link or importing the calendar using the ICS file should work. And after adding the calendar in Google you should see all of the events pop up and if you look at your calendar on your phone, the events should be present. The sync can sometimes take a few minutes, so if they don’t show up right away just wait a bit (or sync them manually) and if you created a new calendar you might need to turn it on.

NOTE: If you add the calendar using the ICS import method, you will not have any way to delete “unmerge” the calendars. Basically, you will have to manually delete the items that were imported from the ICS file if you want to get rid of them permanently. If you create a work calendar in Google Calendars and import the ICS there, it will probably be a lot more manageable.

This solutions feels a little bit clunky but it is good enough for me, and there doesn’t seem to be a good work around for sharing the Office 365 calendar with Google. If there are any good Android apps that can add calendars from different providers let me know, that would probably work just as well but I didn’t really need to look after putting this workaround together.

Writing For Tech

October 23, 2017October 23, 2017 Josh Reichardt Leave a comment

As my career has progressed, I have discovered writing to be an invaluable skill to develop and polish as an engineer. The skill of writing well translates to a number of areas outside of tech including things like writing good emails, networking and chat using real time collaboration tools (IRC, Slack, etc.), writing documentation, writing specs, or even just asking for help in online formats like message boards or communicating on social media sites.

For example, when asking for help in a technical public forum, e.g. GitHub issues or Stack Overflow, knowing exactly what the problem you are having and describing it in a way that makes sense to others (who often don’t speak English as a first language) is much more difficult than it looks. It takes time and practice to learn how to craft questions well and to frame technical problems in easy to understand ways. In my own experience, people are almost always happy to help but I’ve seen so many bad questions on Stack Overflow.

There are two books that I recently read that have had a tremendous impact on how I think about and approach writing, which has helped me grow as a writer, engineer, and technical collaborator, which I’d like to share with readers today. These books have been around for a long time so if you’ve already heard of them or it has been some time since reading them, I encourage you to reread or at least skim through them again.

The first book, On Writing Well: The Classic Guide to Writing Nonfiction is a great book and really forced me think a lot about my writing and what I could be doing better. Instead of focusing on a lot of the mechanics and building blocks of writing (it does touch on these a little bit), On Writing Well focuses mainly on the style and how to make your writing better by making it more interesting and less wasteful. The book teaches readers that often times, more is less in writing, and it focuses on teaching lessons of simplicity as well brevity, boiling things down to their simplest forms and avoiding certain traps and pitfalls.

The second book is called The Elements of Style, Fourth Edition. There are some really good tricks and tidbits in this book that 100% improved my writing fundamentals and mechanics, even without much practice outside of reading the book. I would highly recommend this book for anybody that is interesting in improving the foundations of their writing, from things like improving vocabulary to improving the structure and overall quality of their writing. The book is fairly short so doesn’t take long to work through and is a great tool for improvement and you will more than likely find some tips that are immediately useful in your own writing style.

Getting better at writing is a process, just like learning any other skill. The more time you spend thinking about it and practicing, the better you will get. Obviously in my own personal experience, having this blog has been a great way for me to learn and grow my writing skills. Not every blog post is a success in my eyes but I have learned lessons from doing things over and over again and discovered things that work or don’t work.

One lesson from On Writing Well that has really stuck with me is the idea that your writing should be written for yourself. Instead of thinking about things that other people want, or what you think they want, just write about things that are interesting or that have personal meaning and the writing process will be much more rewarding. Applying this concept makes the process of writing much more enjoyable and keeps the gears turning.

Another idea from the book that stuck with me is that everybody has their own style of writing and none of them are bad. So if you feel pressure to write or create a certain way, don’t. Your writing process works best for you and that is fine, you just need to find it if you don’t know what it is already. One of the most important lessons in writing that I have discovered over the years is that I’m not really interested in writing my blog posts according to any set of formulas or criteria.

In my own writing process, I usually like to find a problem that is interesting or challenging to me, sit down and just start writing. This process helps me internalize and understand the problem I am attempting to solve better, as well gives me a platform to help others. I attribute my own process and writing style to a lot of practice and just using the lessons I have learned to eventually build up my own style, which works for me.

Good luck and happy writing.

Top Five Reasons to Use a Hybrid Cloud

September 12, 2017September 19, 2017 Josh Reichardt Leave a comment

Guest post by Aventis Systems

Cloud computing is increasing in popularity as business users have become more comfortable with cloud capabilities. According to a recent VMware report, some 15% of workloads currently reside in the public cloud, and 50% are projected to be running in the public cloud by 2030.

Some business customers will move completely to the public cloud, drawn by its ability to help them respond to changing business needs, align costs and stay on the cutting edge of innovation. But a complete move isn’t the best option for all customers. Some processes still simply run more efficiently and securely on on-premise hardware.

For many businesses, there is no one-size-fits-all solution. Ultimately, using a mix of public clouds and private infrastructures is the best way for many companies to make the most of their resources while optimizing performance and productivity.

What Is a Hybrid Cloud?
A hybrid cloud is a combination of a private cloud platform designed for use by a specific organization and a public cloud provider like Amazon Web Services (AWS) or Google Cloud. These public clouds are shared by customers all over the world and are a cheaper alternative to buying physical servers.

Though the public and private cloud platforms operate independently from one another, they can communicate over an encrypted connection.

A hybrid approach enables data and applications to move between the public and private infrastructures. These are independent platforms, so businesses can store protected data on the private cloud while still leveraging applications that rely on that protected data on the public cloud.

In other words, your sensitive data stays out of the public cloud and on the private platform. The challenge is integrating the different public and private clouds and technologies in a way that is seamless for business users.

Here are some of the top benefits of a hybrid cloud approach, according to the experts at Aventis Systems:

1. Workload Flexibility
With hybrid cloud technology, your IT team has the flexibility to match resources with the infrastructure that best serves the needs of your business.

For example, an integrated hybrid cloud approach with VMware Cloud on AWS enables you to decide where to most effectively run workloads based on cost, risk and changing business needs. With the flexibility to move workloads onsite or offsite as needed, IT is able to better serve the business as a whole.

The ability for organizations to easily transition applications without having to re-platform them — along with the ability to effortlessly access and leverage native cloud services — enables businesses to create a flexible infrastructure in a constantly evolving IT landscape. When new technology becomes available or new trends emerge, businesses are agile enough to take advantage of them quickly.

2. Consistency and Scalability
VMware Cloud on AWS enables companies to leverage operational consistency, along with scalability, on one streamlined platform. By maintaining security and networking policies, along with consistent resource utilization both on- and off-premise, businesses can benefit most from a hybrid infrastructure.

Customers can strategically leverage and allocate company resources to get the most out of system functionality, while becoming better positioned for growth. As your business and capacity needs grow, a hybrid cloud infrastructure offers an easy way to scale to fit these complex needs.

3. Improved Security
Maintaining secure customer transaction data and personal information with a hybrid cloud infrastructure also offers a major benefit over an exclusively public platform. The hybrid approach enables specified servers to be isolated from specific security threats by allowing devices to be configured to communicate with them on a private network.

Where some compliance requirements prevent businesses from running payments in the cloud, for example, a hybrid cloud platform allows you to house secure customer data on a dedicated server, while maintaining the flexibility and convenience of online transactions.

4. Maximized Skillsets and Cost Optimization
Not only are hybrid clouds less expensive to manage, with VMware Cloud on AWS, business customers can also reap the benefits of utilizing their existing IT investments.

Hybrid cloud offerings integrate with your existing IT and use many of the same tools as those used on-premise. You can leverage the resources you already have without having to adopt new tools or acquire new hardware.

Additionally, an integrated hybrid cloud approach enables customers to better align their costs to business needs. Upfront costs can be balanced with recurring expenses, depending on the requirements.

5. Innovation
With a hybrid cloud approach, your business will have access to all of the resources on the public cloud without the burden of big upfront investments. With access to all the newest technologies and innovations, you can stay on the forefront of the latest capabilities.

As businesses become more comfortable and reliant on cloud capabilities, more and more companies will look for the right mix of public cloud and on-premise infrastructure models to increase efficiency and performance.

Remote Jenkins builds using Github auth

May 22, 2017 Josh Reichardt Leave a comment

Having the ability to call Jenkins jobs remotely is pretty slick and adds some extra flexibility and allows for some interesting applications. For example, you could use remote builds to call a script from a chat app or from some other web application. I have chosen to write a quick bash script as a proof of concept, but this could easily be extended or written in a different language via one of its language specific libraries.

The instructions for the method I am using assume that you are using the Jenkins freestyle build as I haven’t experimented much yet with pipelines for remote builds yet.

The first step is to enable remote builds for the Jenkins job that will be triggered. There is an option in the job for “Trigger builds remotely” which allows the job to be called from a script.

The authentication token can be any arbitrary string you choose. Also note the URL below, you will need that later as part of the script to call this job.

With the authentication token configured and the Jenkins URL recorded, you can begin writing the script. The first step is to populate some variables for kicking off the job. Below is an example of how you might do this.

jenkins_url="https://jenkins.example.com"
job_name="my-jenkins-job"
job_token="xxxxx"
auth="username:token"
my_repo="some_git_repo"
git_tag="abcd123"

Be sure to fill in these variables with the correctly corresponding values. job_token should correspond to the string you entered above in the Jenkins job, auth should correspond to your github username/token combination. If you are not familiar with Github tokens you can find more information about setting them up here.

As part of the script, you will want to create a Jenkins “crumb” using your Github credentials that will be used to prevent cross-site scripting attacks. Here’s what the creation of the crumb looks like (borrowed from this Stackoverflow post).

crumb=$(curl -s 'https://'auth'@jenkins.example.com/crumbIssuer/api/xml?xpath=concat(//crumbRequestField,":",//crumb)')

Once you have your variables configured and your crumb all set up, you can test out the Jenkins job.

curl -X POST -H "$crumb" $jenkins_url/job/$job_name/build?token=$job_token \
  --user $auth \
  --data-urlencode json='
  {"parameter":
    [
      {"name":"parameter1", "value":"test1"},
      {"name":"parameter2", "value":"test2"},
      {"name":"git_repo", "value":"'$my_repo'"},
      {"name":"git_tag", "value":"'$git_tag'"}
    ]
  }'

In the example job above, I am using several Jenkins parameters as part of the build. The json name values correspond to the parameters. Notice that I am using variables for a few of the values above, make sure those variables are wrapped in singe quotes to correctly escape the json. The syntax for variables is slightly different but allows for some additional flexibility in the job configuration and also allows the script to be called with dynamic values.

If you call this script now, it should kick off a Jenkins job for you with all of the values you have provided.