Virtual environments are really useful for maintaining different packages and for separating different environments without getting your system messy. In this post I will go over some of the various virtual environment tricks that I seem to always forget if I haven’t worked with Python in awhile.
This post is meant to be mostly a reference for remembering commands and syntax and other helpful notes. I’d also like to mention that these steps were all tested on OSX, I haven’t tried on Windows so don’t know if it is any different.
Working with virtual environments
There are a few pieces in order to get to get started. First, the default version of Python that ships with OSX is 2.7, which is slowly moving towards extinction. Unfortunately, it isn’t exactly obvious how to replace this version of Python on OSX.
Just doing a “brew install python” won’t actually point the system at the newly installed version. In order to get Python 3.x working correctly, you need to update the system path and place Python3 there.
You will want to put the above line into your bashrc or zshrc (or whatever shell profile you use) to get the brew installed Python onto your system path by default.
Another thing I discovered – in Python 3 there is a built in command for creating virtual environments, which alleviates the need to install the virtualenv package.
Here is the command in Python 3 the command to create a new virtual environment.
python -m venv test
Once the environment has been created, it is very similar to virtualenv. To use the environment, just source it.
source test/bin/activate
To deactivate the environment just use the “deactivate” command, like you would in virutalenv.
The virtualenv package
If you like the old school way of doing virtual environments you can still use the virtualenv package for managing your environments.
Once you have the correct version of Python, you will want to install the virtualenv package on your system globally in order to work with virtual environments.
sudo pip install virtualenvwrapper
You can check to see if the package was installed correctly by running virtualenv -h. There are a number of useful virtualenv commands listed below for working with the environments.
Make a new virtual env
mkvirtualenv test-env
Work on a virtual env
workon test-env
Stop working on a virtual env
(when env is actiave) deactive
List environments
lsvirtualenv
Remove a virtual environment
rmvirtualenv test-env
Create virtualenv with specific version of python
mkvirtualenv -p $(which pypy) test2-env
Look at where environments are stored
ls ~/.virtualenvs
I’ll leave it here for now. The commands and tricks I (re)discovered were enough to get me back to being productive with virtual environments. If you have some other tips or tricks feel free to let me know. I will update this post if I find anything else that is noteworthy.
Kubernetes is complicated, as you’ve probably already discovered if you’ve used Kubernetes before. Likewise, the Kubectl command line tool can pretty much do anything but can feel cumbersome, clunky and generally overwhelming for those that are new to the Kubernetes ecosystem. In this post I want to take some time to describe a few of the CLI tools that I have discovered that help ease the pain of working with and managing Kubernetes from the command line.
There are many more tools out there and the list keeps growing, so I will probably revisit this post in the future to add more cool stuff as the community continues to grow and evolve.
Where to find projects?
As a side note, there are a few places to check for tools and projects. The first is the CNCF Cloud Native Landscape. This site aims to keep track of all the various different projects in the Cloud/Kubernetes world. An entire post could be written about all of the features and and filters but at the highest level it is useful for exploring and discovering all the different evolving projects. Make sure to check out the filtering capabilities.
The other project I have found to be extremely useful for finding different projects is the awesome-kubernetes repo on Github. I found a number of tools mentioned in this post because of the awesome-kubernetes project. There is some overlap between the Cloud Native Landscape and awesome-kubernetes but they mostly compliment each other very nicely. For example, awesome-kubernetes has a lot more resources for working with Kubernetes and a lot of the smalller projects and utilities that haven’t made it into the Cloud Native Landscape. Definitely check this project out if you’re looking to explore more of the Kubernetes ecosystem.
Kubectl tricks
These are various little tidbits that I have found to help boost my productivity from the CLI.
Tab completion – The first thing you will probably want to get working when starting. There are just too many options to memorize and tab completion provides a nice way to look through all of the various commands when learning how Kubernetes works. To install (on OS X) run the following command.
brew install bash-completion
In zsh, adding the completion is as simple as running source <(kubectl completion bash). The same behavior can be accomplished in zsh using source <(kubectl completion zsh).
Aliases and shortcuts – One distinct flavor of Kubernetes is how cumbersome the CLI can be. If you use Zsh and something like oh-my-zsh, there is a default set of aliases that work pretty well, which you can find here. There are a many posts about aliases out there already so I won’t go into too much detail about them. I will say though that aliasing k to kubectl is one of the best time savers I have found so far. Just add the following snippet to your bash/zsh profile for maximum glory.
alias k=kubectl
kubectl –export – This is a nice hidden feature that basically allows users to switch Kubernetes from imperative (create) to declarative (apply). The --export flag will basically take an existing object and strip out unwanted/unneeded metadata like statuses and timestamps and present a clear version of what’s running, which can then be exported to a file and applied to the cluster. The biggest advantage of using declarative configs is the ability to mange and maintain them in git repos.
kubectl top – In newer versions, there is the top command, which gives a high level overview of CPU and memory utilization in the cluster. Utilization can be filtered at the node level as well as the pod level to give a very quick and dirty view into potential bottlenecks in the cluster. In older versions, Heapster needs to be installed for this functionaliity to work correctly, and in newer versions needs metrics-server to be running.
kubectl explain – This is a utility built in to Kubectl that basically provides a man page for what each Kubernetes resource does. It is a simple way to explore Kubernetes without leaving the terminal
This is an amazing little utility for quickly moving between Kubernetes contexts and namespaces. Once you start working with multiple different Kubernetes clusters, you notice how cumbersome it is to switch between environments and namespaces. Kubectx solves this problem by providing a quick and easy way to see what environments and namespaces a user is currently in and also quickly switch between them. I haven’t had any issues with this tool and it is quickly becoming one of my favorites.
Dealing with log output using Kubectl is a bit of a chore. Stern (and similarly kail) offer a much nicer user experience when dealing with logs. These tools allow users the ability to do things like show logs for multiple containers in pod, use regex matching to tail logs for specific containers, give nice colored output for distinguishing between logs, filter logs by namespaces and a bunch of other nice features.
Obviously for a full setup, using an aggregated/centralized logging solution with something like Fluenctd or Logstash would be more ideal, but for examining logs in a pinch, these tools do a great job and are among my favorites. As an added bonus, I don’t have to copy/paste log names any more.
yq is a nice little command line tool for parsing yaml files, which works in a similar way to the venerable jq. Parsing, reading, updating yaml can sometimes be tricky and this tool is a great and lightweight way to manipulate configurations. This tool is especially useful for things like CI/CD where a tag or version might change that is nested deep inside yaml.
There is also the lesser known jsonpath option that allows you to interact with the json version of a Kubernetes object, baked into kubectl. This feature is definitely less powerful than jq/yq but works well when you don’t want to overcomplicate things. Below you can see we can use it to quickly grab the name of an object.
kubectl get pods -o=jsonpath='{.items[0].metadata.name}'
Working with yaml and json for configuration in general seems to be an emerging pattern for almost all of the new projects. It is definitely worth learning a few tools like yq and jq to get better at parsing and manipulating data using these tools.
Similar to the above, ksonnet and jsonnet are basically templating tools for working with Kubernetes and json objects. These two tools work nicely for managing Kubernetes manifests and make a great fit for automating deployments, etc. with CI/CD.
ksonnet and jsonnet are gaining popularity because of their ease of use and simplicity compared to a tool like Helm, which also does templating but needs a system level permission pod running in the Kubernetes cluster. Jsonnet is all client side, which removes the added attack vector but still provides users with a lot of flexibility for creating and managing configs that a templating language provides.
More random Kubernetes tricks
Since 1.10, kubectl has the ability to port forward to resource name rather than just a pod. So instead of looking up pods that are running and connecting to one all the time, you can just grab the service name or deployment and just port forward to it.
port-forward TYPE/NAME [LOCAL_PORT:]REMOTE_PORT
k port-forward deployment/mydeployment 5000:6000
New in 1.11, which will be dropping soonish, there is a top level command called api-resource, which allows users to view and interact with API objects. This will be a nice troubleshooting tool to have if for example you are wanting to see what kinds of objects are in a namespace. The following command will show you these objects.
k api-resources --verbs=list --namespace -o name | xargs -n 1 kubectl get -o name -n foo
Another handy trick is the ability to grab a base64 string and decode it on the fly. This is useful when you are working with secrets and need to quickly look at what’s in the secret. You can adapt the following command to accomplish this (make sure you have jq installed).
k get secret my-secret --namespace default -o json | jq -r '.data | .["secret-field"]' | base64 --decode
Just replace .["secret-field"] to use your own field.
UPDATE: I just recently discovered a simple command line tool for decoding base64 on the fly called Kubernetes Secret Decode (ksd for short). This tool looks for base64 and renders it out for you automatically so you don’t have to worry about screwing around with jq and base64 to extract data out when you want to look at a secret.
k get secret my-secret --namespace default -o json | ksd
That command is much cleaner and easier to use. This utility is a Go app and there are binaries for it on the releases page, just download it and put it in your path and you are good to go.
Conclusion
The Kubernetes ecosystem is a vast world, and it only continues to grow and evolve. There are many more kubectl use cases and community to tools that I haven’t discovered yet. Feel free to let me know any other kubectl tricks you know of, and I will update them here.
I would love to grow this list over time as I get more acquainted with Kubernetes and its different tools.
There are quite a few use cases for monitoring outside of Kubernetes, especially for previously built infrastructure and otherwise legacy systems. Additional monitoring adds an extra layer of complexity to your monitoring setup and configuration, but fortunately Prometheus makes this extra complexity easier to manage and maintain, inside of Kubernetes.
In this post I will describe a nice clean way to monitor things that are internal to Kubernetes using Prometheus and the Prometheus Operator. The advantage of this approach is that it allows the Operator to manage and monitor infrastructure, and it allows Kubernetes to do what it’s good at; make sure the things you want are running for you in an easy to maintain, declarative manifest.
If you are already familiar with the concepts in Kubernetes then this post should be pretty straight forward. Otherwise, you can pretty much copy/paste most of these manifests into your cluster and you should have a good way to monitor things in your environment that are external to Kubernetes.
Below is an example of how to monitor external network devices using the Prometheus SNMP exporter. There are many other exporters that can be used to monitor infrastructure that is external to Kubernetes but currently it is recommended to set up these configurations outside of the Prometheus Operator to basically separate monitoring concerns (which I plan on writing more about in the future).
At this point you would have a pod in your cluster, attached to a static IP address. To see if it worked you can check to make sure a service IP was created. The service is basically what the Operator uses to create targets in Prometheus.
kubectl get sv
From this point you can 1) set up your own instance of Prometheus using Helm or by deploying via yml manifests or 2) set up the Prometheus Operator.
Today we will walk through option 2, although I will probably cover option 1 at some point in the future.
Setting up the Prometheus Operator
The beauty of using the Prometheus Operator is that it gives you a way to quickly add or change Prometheus specific configuration via the Kubernetes API (custom resource definition) and some custom objects provided by the operator, including AlertManager, ServiceMonitor and Prometheus objects.
The first step is to install Helm, which is a little bit outside of the scope of this post but there are lots of good guides on how to do it. With Helm up and running you can easily install the operator and the accompanying kube-prometheus manifests which give you access to lots of extra Kubernetes metrics, alerts and dashboards.
After a few moments you can check to see that resources were created correctly as a quick test.
kubectl get pods -n monitoring
NOTE: You may need to manually add the “prometheus” service account to the monitoring namespace after creating everything. I ran into some issues because Helm didn’t do this automatically. You can check this with kubectl get events.
Prometheus Operator configuration
Below are steps for creating custom objects (CRDs) that the Prometheus Operator uses to automatically generate configuration files and handle all of the other management behind the scenes.
These objects are wired up in a way that configs get reloaded and Prometheus will automatically get updated when it sees a change. These object definitions basically convert all of the Prometheus configuration into a format that is understood by Kubernetes and converted to Prometheus configuration with the operator.
First we make a servicemonitor for monitoring the the snmp exporter.
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
k8s-app: snmp-exporter
prometheus: kube-prometheus # tie servicemonitor to correct Prometheus
name: snmp-exporter
spec:
jobLabel: k8s-app
selector:
app: snmp-exporter
namespaceSelector:
matchNames:
- monitoring
endpoints:
- interval: 60s
port: http-metrics
params:
module:
- if_mib # Select which SNMP module to use
target:
- 1.2.3.4 # Modify this to point at the SNMP target to monitor
path: "/snmp"
targetPort: 9116
Next, we create a custom alert and tie it our Prometheus Operator. The alert doesn’t do anything useful, but is a good demo for showing how easy it is to add and manage alerts using the Operator.
Create an alert-example.yml configuration file, add it as a configmap to k8s and mount it in as a configuration with the ruleSelector label selector and the prometheus operator will do the rest. Below shows how to hook up a test rule into an existing Prometheus (kube-prometheus) alert manager, handled by the prometheus-operator.
kind: ConfigMap
apiVersion: v1
metadata:
name: josh-test
namespace: monitoring
labels:
role: alert-rules # Standard convention for organizing alert rules
prometheus: kube-prometheus # tie to correct Prometheus
data:
test.rules.yaml: |
groups:
- name: test.rules # Top level description in Prometeheus
rules:
- alert: TestAlert
expr: vector(1)
Once you have created the rule definition via configmap just use kubectl to create it.
kubectl create -f alert-example.yml -n monitoring
Testing and troubleshooting
You will probably need to port forward the pod to get access to the IP and port in the cluster
kubectl port-forward snmp-exporter-<name> 9116
Then you should be able to visit the pod in your browser (or with curl).
localhost:9116
The exporter itself does a lot more so you will probably want to play around with it. I plan on covering more of the details of other external exporters and more customized configurations for the SNMP exporter.
For example, if you want to do any sort of monitoring past basic interface stats, etc. you will need to generate and build your own set of MIBs to gather metrics from your infrastructure and also reconfigure your ServiceMonitor object in Kubernetes to use the correct MIBs so that the Operator updates the configuration correctly.
Conclusion
The amount of options for how to use Prometheus is one area of confusion when it comes to using Prometheus, especially for newcomers. There are lots of ways to do things and there isn’t much direction on how to use them, which can also be viewed as a strength since it allows for so much flexibility.
In some situations it makes sense to use an external (non Operator managed Prometheus) when you need to do things like manage and tune your own configuration files. Likewise, the Prometheus Operator is a great fit when you are mostly only concerned about managing and monitoring things inside Kubernetes and don’t need to do much external monitoring.
That said, there is some support for external monitoring using the Prometheus Operator, which I want to write about in a different post. This support is limited to a handful of different external exporters (for the time being) so the best advice is to think about what kind of monitoring is needed and choose the best solution for your own use case. It may turn out that both types of configurations are needed, but it may also end up being just as easy to use one method or another to manage Prometheus and its configurations.
Sometimes when provisioning a server you may want to configure and provision storage as part of the bootstrapping and booting process. For example, the other day I ran into an issue where I needed to define a disk, partition it, mount it to a specified location and then create a few directories in it. It turned out to be surprisingly not straight forward to provision this storage and I learned quite a few things that I thought were worth sharing.
I’d just like to mention that ignition works like magic. If you aren’t familiar, Ignition is basically a tool to help provision and configure servers, very similar to cloud-config except by default Ignition only runs once, on first boot. The magic of Ignition is that it injects itself into initramfs before the OS ever eve boots and manipulating the system. Ignition can be read in from remote URL so that it can easily be provisioned in bare metal infrastructures. There were several pieces to this puzzle.
The first was getting down all of the various ignition configuration components in Terraform. Nothing was particularly complicated, there was just a lot of trial and error to get everything working. Terraform has some really nice documentation for working with Ignition configurations, I’d recommend starting there and just playing around to figure out some of the various bits and pieces of configuration that Ignition can do. There is some documentation on Ignition troubleshooting as well which I found to be helpful when things weren’t working correctly.
Below each portion of the Ignition configuration gets declared inside of a “ignition_config” block. The Ignition configuration then points towards each invidual component that we want Ignition to configure. e.g. systemd, filesystem, directories, etc.
This part of the setup is pretty straight forward. Create a data block with the needed ignition configuration to mount the disk to the correct location, format the device if it hasn’t already been formatted and create the desired directory and then create the Systemd unit to configure the mount point for the OS. Here’s what each of the data blocks might look like.
data "ignition_filesystem" "data_fs" {
name = "data"
mount {
device = "/dev/xvdb1"
format = "ext4"
}
}
data "ignition_directory" "data_dir" {
filesystem = "data"
path = "/data"
uid = 500
gid = 500
}
data "ignition_disk" "data_disk" {
device = "/dev/xvdb"
partition {
number = 1
start = 0
size = 0
}
}
Next, create the Systemd unit.
data "ignition_systemd_unit" "mount_data" {
content = "${file("./data.mount")}"
name = "data.mount"
}
Another challenge was getting the Systemd unit to mount the disk correctly. I don’t work with Systemd frequently so initially had some trouble figuring this part out. Basically, Systemd expects the service/unit definition name to EXACTLY match what’s declared inside the “Where” clause of the service definition.
For example, the following configuration needs to be named data.mount because that is what is defined in the service.
After all the kinks have been worked out of the Systemd unit(s) and other above Terraform Ignition configuration you should be able to deploy this and have Ignition provision disks for you automatically when the OS comes up. This can be extended as much as needed for getting initial disks set up correctly and is a huge step in automating your infrastructure in a nice repeatable way.
There is currently an open issue with Ignition currently where it breaks when attempting to re-provision a previously configured disk on a new machine. Basically the Ignition process chokes because it sees the device has already been partitioned and formatted and can’t do it again. I ran into this scenario where I was trying to create a basically floating persistent data EBS volume that gets attached to servers in an autoscaling group and wanted to allow the volume to be able to move around freely if the server gets killed off.
If you’ve ever encountered the following error (or similar) when setting up an AWS load balancer to write its logs to an s3 bucket using Terraform then you are not alone. I decided to write a quick note about this problem because it is the second time I have been bitten by this and had to spend time Googling around for an answer. The AWS documentation for creating and attaching the policy makes sense but the idea behind why you need to do it is murky at best.
For reference, here are the docs for how to manually create the policy by going through the AWS console. This method works fine for manually creating and attaching to the policy to the bucket. The problem is that it isn’t obvious why this needs to happen in the first place and also not obvious to do in Terraform after you figure out why you need to do this. Luckily Terraform has great support for IAM, which makes it easy to configure the policy and attach it to the bucket correctly.
Below is an example of how you can create this policy and attach it to your load balancer log bucket.
Notice that you don’t need to explicitly define the principal like you do when setting up the policy manually. Just use the ${data.aws_elb_service_account.main.arn} variable and Terraform will figure out the region that the bucket is in and pick out the correct parent ELB ID to attach to the policy. You can verify this by checking the table from the link above and cross reference it with the Terraform output for creating and attaching the policy.
You shouldn’t need to update anything in the load balancer config for this to work, just rerun the failed command again and it should work. For completeness here is what that configuration might look like.
This process is easy enough but still begs the question of why this seemingly unnecessary process needs to happen in the first place? After searching around for a bit I finally found this:
When Amazon S3 receives a request—for example, a bucket or an object operation—it first verifies that the requester has the necessary permissions. Amazon S3 evaluates all the relevant access policies, user policies, and resource-based policies (bucket policy, bucket ACL, object ACL) in deciding whether to authorize the request.
Okay, so it basically looks like when the load balancer gets created, the load balancer gets associated with an AWS owned ID, which we need to explicitly give permission to, through IAM policy:
If the request is for an operation on an object that the bucket owner does not own, in addition to making sure the requester has permissions from the object owner, Amazon S3 must also check the bucket policy to ensure the bucket owner has not set explicit deny on the object
Note
A bucket owner (who pays the bill) can explicitly deny access to objects in the bucket regardless of who owns it. The bucket owner can also delete any object in the bucket.
There we go. There is a little bit more information in the link above but now it makes more sense.
