Build a Pine64 Kubernetes Cluster with k3os

Kubernetes (k3os) arm64 cluster with custom 3D printed case

The k3os project was recently announced and I finally got a chance to test it out. k3os greatly simplifies the steps needed to create a Kubernetes cluster along with its counterpart, k3s, to reduce the overhead of running Kubernetes clusters. Paired with Rancher for the UI, all of these components make for an even better option. You can even run Rancher in your (arm64) k3os cluster via the Rancher Helm chart now.

Instead of using Etcd, k3s opts to use SQLite by default and does some other magic to reduce extra Kubernetes bloat and simplify management. Check here for more about k3s and how it works and how to run it.

k3os replaces some complicated OS components with much simpler ones. For example, instead of using Systemd it uses OpenRC, instead of Docker it uses containerd, it also leverages connman for configuring network components and it doesn’t use a package manager.

The method I am showing in this post uses the k3os overlay installation, which is detailed here. The reason for this choice is because the pine64 boards use u-boot to boot the OS and so special steps are needed to accommodate for the way this process is handled. The upside of this method is that these instructions should pretty much work for any of the Pi form factor boards, including the newly released Raspberry Pi4, with minimal changes.

Setup

If you haven’t downloaded and imaged your Pine64 yet, I like to use the ayufan images, which can be found here. You can easily write these images to a microSD card on OSX using something like Etcher.

Assuming the node is connected to your network, you can SSH into it.

ssh [email protected] # or use the ip, password is rock64

When using the overlay installation, the first step is to download the k3os rootfs and lay it down on the host. This step applies to all nodes in the cluster.

curl -sL https://github.com/rancher/k3os/releases/download/v0.2.1/k3os-rootfs-arm64.tar.gz | tar --strip-components=1 -zxvf - -C 

The above command is installing v0.2.1 which is the most current version as of writing this, so make sure to check if there is a newer version available.

After installing, lay down the following configurations into /k3os/system/config.yaml, modifying as needed. After the machine is rebooted this path will become read only so if you need to change the configuration again you will need to edit /etc/fstab to change the location to be writable again.

Server node

ssh_authorized_keys:
- ssh-rsa <your-public-ssh-key-to-login>
hostname: k3s-master

k3os:
  data_sources:
  - cdrom
  dns_nameservers:
  - 192.168.1.1 # update this to your local or public DNS server if needed
  ntp_servers:
  - 0.us.pool.ntp.org
  - 1.us.pool.ntp.org
  password: rancher
  token: <TOKEN>

The k3s config will be written to /etc/rancher/k3s/k3s.yaml on this node so make sure to grab it if you want to connect the cluster from outside this node. Reboot the machine to boot to the new filesystem and you should be greeted with the k3os splash screen.

Agent node

The agent uses nearly the same config, with the addition of the server_url. Just point the agent nodes to the server/master and you should be good to go. Again, reboot after creating the config and the host should boot to the new filesystem and everything should be ready.

ssh_authorized_keys:
- ssh-rsa <your-public-ssh-key-to-login>
hostname: k3s-node-1

k3os:
  data_sources:
  - cdrom
  dns_nameservers:
  - 192.168.1.1 # update this to your local or public DNS server if needed
  ntp_servers:
  - 0.us.pool.ntp.org
  - 1.us.pool.ntp.org
  password: rancher
  server_url: https://<server-ip-or-hostname>:6443
  token: <TOKEN>

You can do a lot more with the bootstrap configurations, such as setting labels or environment variables. Some folks in the community have had luck getting the wifi configuration working on the RPi4’s out of the box, but I haven’t been able to get it to work yet on my Pine64 cluster. Check the docs for more details on the various configuration options.

After the nodes have been rebooted and configs applied, the cluster “should just work”. You can check that the cluster is up using k3s using the kubectl passthrough command (checking from the master node below).

k3s-master [~]$ k3s kubectl get nodes
NAME         STATUS   ROLES    AGE     VERSION
k3s-master   Ready    <none>   6d1h    v1.14.1-k3s.4
k3s-node-1   Ready    <none>   7m10s   v1.14.1-k3s.4

NOTE: After installing the overlay filesystem there will be no package manager and no obvious way to upgrade the kernel so use this guide only for testing purposes. The project is still very young and a number of things still need to be added, including update mechanisms and HA. Be sure to follow the k3os issue tracker and Rancher Slack (#k3os channel) for updates and developments.

Conclusion

This is easily the best method I have found so far for getting a Kubernetes cluster up and running, minus the few caveats mentioned above, which I believe will be resolved very soon. I have been very impressed with how simple and easy it has been to configure and use. The next step for me is to figure out how to run Rancher and start working on some configurations for running workloads on the cluster. I will share more on that project in another post.

There are definitely some quirks to getting this setup working for the Pi and Pine64 based boards, but aren’t major problems by any means.

References

This post was heavily inspired by this gist for getting the overlay installation method working on Raspberry Pi.

Read More

Building Multiarch Conda-Forge Recipes

As part of my adventures in building a 100% Arm64 Kubernetes cluster, I recently tried to build an Arm64 Jupyterhub Docker imageto run in the cluster. To my surprise, there doesn’t seems to be any “official” Jupyterhub arm64 Docker images out there, so I decided to set out and create one.

In the process of building my image, I almost immediately hit a stumbling block in that the Docker image uses the Conda package manager and several Conda packages for its build. The problem is that several of these packages have not yet been built to work on alternate architectures, e.g. Arm64, and others. So I went off down into the rabbit hole of seeing how hard it would be to add this support for these packages in order to get the Jupyterhub Docker image working.

The first stop on this journey was to conda-forge to look at the multiarch support. If you aren’t familiar (I wasn’t), conda-forge bills itself as a large Github community for creating and building Conda packages.

The first thing to look at when adding support to an existing package is getting familiar with conda-smithy, which is the tool responsible for setting up and building all of the various conda-forge “recipes”. There are generic instructions for using conda-smithy here.

As a fun side note, there is no “native” Arm64 build infrastructure for creating packages. The current builds use QEMU to emulate aarch64 (arm64) using Azure pipelines. This has some issues so while I was down in the rabbit hole I decided to contribute a PR to help get native arm64 builds added. The work isn’t yet complete, it still needs to be hooked up to CI, so if you want to help out feel free to let me know or just open a PR in the conda-smithy repo.

Multiarch support

With the housekeeping out of the way, we can now look at how to actually add the multiarch support for a package.

First, fork and clone the desired recipe.  In this example I am adding arm64 support to the pycurl recipe as it is one of the Conda package dependencies that I need to build Jupyterhub for Arm64.

git clone https://github.com/conda-forge/pycurl-feedstock.git

Edit conda-forge.yml and add the following line to the bottom.

provider: {linux_aarch64: default, linux_ppc64le: default}

If you are just adding support for new architectures like I am here, you will need to bump the build number.  This can be found recipe/meta.yml, and there are also instruction for doing this.

…
build:
  number: 0
…

Just change this value to 1. Next, install conda smithy if you don’t have it already.

conda install conda-smithy

And then you can render out all the new files needed for the various builds.

conda-smithy rerender

Add the generated files to a new (forked) branch of the recipe.

git add .
git commit -m "Add multiarch support"
git push

Then open up a PR to the conda-forge repo with the details.  Once the PR has been open a series of checks should kick off to build the recipe for the various architectures.

CI checks

If everything is green you are good to go.  Maintainers are usually pretty good about merging in changes, but if you need to, you can ping an admin to get help.

You can also tell the build to rerun if it fails using the “@conda-forge-admin, please rerender” command.

You can find more details about what all the bot can do here.

Conclusion

Conda-forge provides some nifty tools for large scale automation and makes it super easy for outsiders to contribute to the community. If you find a missing, outdated or package lacking multiarch support on the Anaconda repo (which includes packages contributed by conda-forge along with many others), definitely think about contributing. The process of adding changes is easy and the conda-forge community is growing all the time.

Read More

Set up Drone on arm64 Kubernetes clusters

Continuing with the multiarch and Kubernetes narratives that I have been writing about for awhile now, in this post I will be showing off some of the capabilities of the Drone.io continuous integration tool running on my arm64 Kubernetes homelab cluster.

As arm64 continues to gain popularity, more and more projects are adding support for it, including Drone.io. With its semi recent announcement, Drone can now run on a variety of different architectures now. Likewise, the Drone maintainers have also been working towards a 1.0 release, which brings first class support for Kubernetes, among other things.

I won’t touch on too many of the specifics of Drone, but if you’re interested in learning more, you can check out the website. I will mostly be focusing on how to get things running in Kubernetes, which turns out to be exactly the same for both amd64 and arm64 architectures. There were a few things I discovered along the way to get the Kubernetes integrations working but for the most part things “just work”.

I started off by grabbing the Helm chart and modifying it to suit my needs. I find it easiest to template the chart and then save it off locally so I can play around with it.

Note: the below example assumes you already have helm installed locally.

git clone [email protected]:helm/charts.git && cd charts
helm template --name drone --namespace cicd \
   --set 'sourceControl.provider=github' \
   --set 'sourceControl.github.clientID=XXXXXXXX' \
   --set 'sourceControl.secret=drone-server-secrets' \
   --set 'server.host=drone.example.com' \
   --set 'server.kubernetes.enabled=false' \
   stable/drone > /tmp/manifest.yaml

Obviously you will want to set the configurations to match your own settings, like domain name and oauth settings (I used Github).

After saving out the manifest, the first issue I ran into is that port 9000 is still referenced in the Helm chart, which was used for communication between the client and server in the older releases, but is no longer used. So I just completely removed the references to the port in my Frankenstein configuration. If you are just using the Kubernetes configuration mentioned below, you won’t run into these problems connecting the server and agent, but if you use the agent you will.

There is some server config that will need to adjusted as well to get things working. For example, the oauth settings will need to be created on the Github side first in order for any of this to work. Also, the drone server host will need to be accessible from the internet so any firewall rules will need to be added or adjusted to allow traffic.

 env:
  # Webhook setings
  - name: DRONE_ALWAYS_AUTH
    value: "false"
  - name: DRONE_SERVER_HOST
    value: "drone.example.com"
  - name: DRONE_SERVER_PROTO
    #value: http
    value: https
  # Agent config
  - name: DRONE_RPC_SECRET
    valueFrom:
      secretKeyRef:
        name: drone
        key: secret
  # Server config
  - name: DRONE_DATABASE_DATASOURCE
    value: "/var/lib/drone/drone.sqlite"
  - name: DRONE_DATABASE_DRIVER
    value: "sqlite3"
  - name: DRONE_LOGS_DEBUG
    value: "true"
  - name: DRONE_LOGS_PRETTY
    value: "true"
  - name: DRONE_USER_CREATE
    value: "username:<github_user>,machine:false,admin:true,token:abc123"
  # Github config
  - name: DRONE_GITHUB_CLIENT_ID
    value: abcd
  - name: DRONE_GITHUB_SERVER
    value: https://github.com
  - name: DRONE_GITHUB_CLIENT_SECRET
    valueFrom:
      secretKeyRef:
        name: client-secret-drone
        key: secret

Add the DRONE_USER_CREATE env var to bootstrap an admin account when starting the Drone server. This will allow your user to do all of the admin things using the CLI tool.

The secrets so should get generated when you dump the Helm chart, so feel free to update those with any values you may need.

Note: if you have double checked all of your settings but builds aren’r being triggered, there is a good chance that the webhook is the problem. There is a really good post about how to troubleshoot these settings here.

Running Drone with the Kubernetes integration

This option turned out to be the easier approach. Just add the following configuration to the drone server deployment environment variables, updating according to your environment. For example, the namespace I am deploying to is called “cicd”, which will need to be updated if you choose a different namespace.

- name: DRONE_KUBERNETES_ENABLED
  value: "true"
- name: DRONE_KUBERNETES_NAMESPACE
  value: cicd
- name: DRONE_KUBERNETES_SERVICE_ACCOUNT
  value: drone-pipeline

The main downside to this method is that it creates Kubernetes jobs for each build. By default, once these builds are done, the will exit and not clean themselves up, so if you do a lot of builds then your namespace will get clogged up. There is a way to set TTLs on finished to clean themselves up in newer versions of Kubernetes via the TTLAfterFinished flag but this functionality isn’t default in Kubernetes yet and is a little bit out of the scope of this post.

Running Drone with the agent configuration

The agent uses the sidecar pattern to run a Docker in Docker (dind) container to connect to the Docker socket in order to allow the Drone agent to do its builds.

The main downside of using this approach is that there seems to be an issue (sometimes) where the Drone components can’t talk to the Docker socket, you can find a better description of this problem and more details here. The problem seems to be a race condition where the docker socket is not being able to be mounted before the agent comes up, but I still haven’t totally solved the problem there yet. The advice for getting around this is to run the agent on a dedicated stand alone host to avoid race conditions and other pitfalls.

That being said, if you still want to use this method you will need to add an additional deployment to the config for the drone agent. If you use the agent you can disregard the above Kubernetes environment variable configurations and instead set appropriate environment variables in the agent. Below is the working snipped I used for deploying the agent to my test cluster.

---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: drone-agent
  labels:
    app: drone
    component: agent
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: drone
        component: agent
    spec:
      serviceAccountName: drone
      containers:
      - name: agent
        image: "docker.io/drone/agent:1.0.0-rc.6"
        imagePullPolicy: IfNotPresent
        ports:
        - name: http
          containerPort: 3000
          protocol: TCP
        env:
          # This value should point to the Drone server service
          - name: DRONE_RPC_SERVER
            value: http://drone.cicd
          - name: DRONE_RPC_SECRET
            valueFrom:
              secretKeyRef:
                name: drone
                key: secret
          - name: DOCKER_HOST
            value: tcp://localhost:2375
          - name: DRONE_LOGS_DEBUG
            value: "true"
          # Uncomment this for additional trace logs
          #- name: DRONE_LOGS_TRACE
          #  value: "true"
          - name: DRONE_LOGS_PRETTY
            value: "true"

      - name: dind
        image: "docker.io/library/docker:18.06-dind"
        imagePullPolicy: IfNotPresent
        env:
        - name: DOCKER_DRIVER
          value: overlay2

        securityContext:
          privileged: true

        volumeMounts:
          - name: docker-graph-storage
            mountPath: /var/lib/docker
      volumes:
      - name: docker-graph-storage
        emptyDir: {}

I have gotten the agent to work, I just haven’t had very much success getting it working consistently. I would avoid using this method unless you have to or as mentioned above, get a dedicated host for running builds on.

Testing it out

After wiring everything up, the rest is easy. Add a file called .drone.yml to a repository that you would like to automate builds for. You can find out more about the various capabilities here.

For my use case I wanted to tell drone to build and publish an arm64 based Docker image whenever a change to master occurs. You can look at my drone configuration here to get a better idea of the multiarch options as well as authenticating to a Docker registry.

After adding the .drone.yml to your repo and triggering a build you should see something similar in your local Drone instance.

Sample Drone build

If everything worked correctly and is green then you should be good to go. Obviously there is a lot of overhead that Kubernetes brings but the actual Drone setup if really straight forward. Just set stuff up on the Github side, translate it into Kubernetes configurations and add some other Drone specific config options and you should have a nice CI/CD environment ready to go.

Conclusion

I am very excited to see Drone grow and mature and use it more in the future. It is simple yet flexible and it fits nicely into the new paradigm of using containers for everything.

The new 1.0 YAML syntax is really nice as well, as it basically maps to the conventions that Kubernetes has chosen, so if you are familiar with that syntax you should feel at home. You can check out the available plugins here, which cover about 90% of the use cases you would see in the wild.

One downside is that YAML syntax errors are really hard to debug, and there isn’t much in the way of output to figure out where your problems are. The best approach I have found is to run the .drone.yml file through the Drone CLI lint/fmt tool before committing and building.

The Drone CLI tool is really powerful and could probably be its own post. There are some links in the references that show off some of its other features.

References

There are a few cool Drone resources I would definitely recommend checking out if you are interested running Drone in your own environment. The docs reference is a great place to start and is great for finding information about how to configure and tweak various Drone settings.

https://docs.drone.io/reference/

Here is a link to the CLI reference.

https://github.com/drone/awesome-drone

I also definitely recommend checking out the jsonnet extension docs, which can be used to help improve automation workflows. The second link show an good example of how it works and the third link shows some practical applications of using jsonnet to help manage complicated CI pipelines.

https://docs.drone.io/extend/config/jsonnet/
https://github.com/drone/drone/blob/master/.drone.jsonnet
https://medium.com/dazn-tech/simplify-your-ci-pipeline-configuration-with-jsonnet-5a96cd9ccc51

Here is a link for various cool drone stuff, including blog posts and tools.

https://docs.drone.io/cli/

Read More

Kubernets plugins

Manage Kubernetes Plugins with Krew

There have been quite a few posts recently describing how to write custom plugins, now that the mechanism for creating and working with them has been made easier in upstream Kubernetes (as of v1.12). Here are the official plugin docs if you’re interested in learning more about how it all works.

One neat thing about the new plugin architecture is that they don’t need to be written in Go to be recognized by kubectl. There is a document in the Kubernetes repo that describes how to write your own custom plugin and even a helper library for making it easier to write plugins.

Instead of just writing another tutorial about how to make your own plugin, I decided to show how easy it is to grab and experiment with existing plugins.

Installing krew

If you haven’t heard about it yet, Krew is a new tool released by the Google Container Tools team for managing Kubernetes plugins. As far as I know this is the first plugin manager offering available, and it really scratches my itch for finding a specific tool for a specific job (while also being easy to use).

Krew basically builds on top of the kubectl plugin architecture for making it easier to deal with plugins by providing a sort of framework for keeping track of things and making sure they are doing what they are supposed to.

The following kubectl-compatible plugins are available:

/home/jmreicha/.krew/bin/kubectl-krew
/home/jmreicha/.krew/bin/kubectl-rbac_lookup
...

You can manage plugins without Krew, but if you work with a lot of plugins complexity and maintenance generally start to escalate quickly if you are managing everything manually. Below I will show you how easy it is to deal with plugins instead using Krew.

There are installation instructions in the repo, but it is really easy to get going. Run the following commands in your shell and you are ready to go.

(
  set -x; cd "$(mktemp -d)" &&
  curl -fsSLO "https://storage.googleapis.com/krew/v0.2.1/krew.{tar.gz,yaml}" &&
  tar zxvf krew.tar.gz &&
  ./krew-"$(uname | tr '[:upper:]' '[:lower:]')_amd64" install \
    --manifest=krew.yaml --archive=krew.tar.gz
)

# Then append the following to your .zshrc or bashrc
export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"

# Then source your shell to pick up the path
source ~/.zshrc # or ~/.bashrc

You can use the kubectl plugin list command to look at all of your plugins.

Test it out to make sure it works.

kubectl krew help

If everything went smoothly you should see some help information and can start working with the plugin manager. For example, if you want to check currently available plugins you can use Krew.

kubectl krew update
kubectl krew search

Or you can browse around the plugin index on GitHub. Once you find a plugin you want to try out, just install it.

kubectl krew install view-utilization

That’s it. Krew should take care of downloading the plugin and putting it in the correct path to make it usable right away.

kubectl view-utilization

Some plugins require additional tools to be installed beforehand as dependencies but should tell you which ones are required when they are installed the first time.

Installing plugin: view-secret
CAVEATS:
\
 |  This plugin needs the following programs:
 |  * jq
/
Installed plugin: view-secret

When you are done with a plugin, you can install it just as easily as it was installed.

kubectl krew uninstall view-secret

Conclusion

I must say I am a really big fan of this new model for managing and creating plugins, and I think it will encourage the community to develop even more tools so I’m looking forward to seeing what people come up with.

Likewise I think Krew is a great fit for this because it is super easy to get installed and started with, which I think is important for gaining widespread adoption in the community. If you have an idea for a Kubectl plugin please consider adding it to the krew-index. The project maintainers are super friendly and are great about feedback and getting things merged.

Read More

Kubernetes Tips and Tricks

I have been getting more familiar with Kubernetes in the past few months and have uncovered some interesting capabilities that I had no idea existed when I started, which have come in handy in helping me solve some interesting and unique problems.  I’m sure there are many more tricks I haven’t found, so please feel free to let me know of other tricks you may know of.

Semi related; if you haven’t already checked it out, I wrote a post awhile ago about some of the useful kubectl tricks I have discovered.  The CLI has improved since then so I’m sure there are more and better tricks now but it is still a good starting point for new users or folks that are just looking for more ideas of how to use kubectl.  Again, let me know of any other useful tricks and I will add them.

Kubernetes docs

The Kubernetes community has somewhat of a love hate relationship with the documentation, although that relationship has been getting much better over time and continues to improve.  Almost all of the stuff I have discovered is scattered around the documentation, the main issue is that it a little difficult to find unless you know what you’re looking for.  There is so much information packed into these docs and so many features that are tucked away that aren’t obvious to newcomers.  The docs have been getting better and better but there are still a few gaps in examples and general use cases that are missing.  Often the “why” of using various features is still sometimes lacking.

Another point I’d like to quickly cover is the API reference documentation.  When you are looking for some feature or functionality and the main documentation site fails, this is the place to go look as it has everything that is available in Kubernetes.  Unfortunately the API reference is also currently a challenge to use and is not user friendly (especially for newcomers), so if you do end up looking through the API you will have to spend some time to get familiar with things, but it is definitely worth reading through to learn about capabilities you might not otherwise find.

For now, the best advice I have for working with the docs and testing functionality is trial and error.  Katacoda is an amazing resource for playing around with Kubernetes functionality, so definitely check that out if you haven’t yet.

Simple leader election

Leader election built on Kubernetes is really neat because it buys you a quick and dirty way to do some pretty complicated tasks.  Usually, implementing leader election requires extra software like ZooKeeper, etcd, Consul or some other distributed key/value store for keeping track of consensus, but it is built into Kubernetes, so you don’t have much extra work to get it working.

Leader election piggy backs off the same etcd Kubernetes uses as well as Kubernetes annotations, which give users a robust way to do distributed tasks without having to recreate the wheel for doing complicated leader elections.

Basically, you can deploy the leader-elector as a sidecar with any app you deploy.  Then, any container in the pod that’s interested in who is the master can can check by visiting the http endpoint (localhost:4044 by default) and they will get back some json with the current leader.

Shared process namespace across namespaces

This is a beta feature currently (as of 1.13) so is enabled now by default.  This one is interesting because it allows you to share a PID between containers.  Unfortunately the docs don’t really tell you why this feature is useful.

Basically, if you add shareProcessNamespace: true to your pod spec, you turn on the ability to share a PID across containers. This allows you to do things like changing a configuration in one container, sending a SIGHUP, and then reloading that configuration in another container.

For example, running a sidecars that controls configuration files or for reaping orphaned zombie processes.

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  shareProcessNamespace: true
  containers:
  - name: nginx
    image: nginx
  - name: shell
    image: busybox
    securityContext:
      capabilities:
        add:
        - SYS_PTRACE
    stdin: true
    tty: true

Custom termination messages

Custom termination messages can be useful when debugging tricky situations.

You can actually customize pod terminations by using the terminationMessagePolicy which can control how terminations get outputted. For example, by using FallbackToLogsOnError you can tell Kubernetes to use container log output if the termination message is empty and the container exited with error.

Likewise, you can specify the terminationMessagePath spec to customize the path to a log file for specifying successes and failures when a pod terminates.

apiVersion: v1
kind: Pod
metadata:
  name: msg-path-demo
spec:
  containers:
  - name: msg-path-demo-container
    image: debian
    terminationMessagePath: "/tmp/my-log"

Container lifecycle hooks

Lifecycle hooks are really useful for doing things either after  a container has started (such as joining a cluster) or for running commands/code for cleanup when a container is stopped (such as leaving a cluster).

Below is a straight forward example taken from the docs that writes a message after a pod starts and sends a quit signal to nginx when the pod is destroyed.

apiVersion: v1
kind: Pod
metadata:
  name: lifecycle-demo
spec:
  containers:
  - name: lifecycle-demo-container
    image: nginx
    lifecycle:
      postStart:
        exec:
          command: ["/bin/sh", "-c", "echo Hello from the postStart handler > /usr/share/message"]
      preStop:
        exec:
          command: ["/usr/sbin/nginx","-s","quit"]

Kubernetes downward API

This one is probably more known, but I still think it is useful enough to add to the list.  The downward API basically allows you to grab all sorts of useful metadata information about containers, including host names and IP addresses.  The downward API can also be used to retrieve information about resources for pods.

The simplest example to show off the downward API is to use it to configure a pod to use the hostname of the node as an environment variable.

apiVersion: v1
kind: Pod
spec:
  containers:
    - name: test-container
      image: k8s.gcr.io/busybox
      command: [ "sh", "-c"]
      args:
      - while true; do
          echo -en '\n';
          printenv MY_NODE_NAME
          sleep 10;
        done;
      env:
        - name: MY_NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName

Injecting a script into a container from a configmap

This is a useful trick when you want to add a layer on top of a Docker container but don’t necessarily want to build either a custom image or update an existing image.  By injecting the script as a configmap directly into the container you can augment a Docker image to do basically any extra work you need it to do.

The only caveat is that in Kubernetes, configmaps are by default not set to be executable.

In order to make your script work inside of Kubernetes you will simply need to add defaultMode: 0744 to your configmap volume spec. Then simply mount the config as volume like you normally would and then you should be able to run you script as a normal command.

...
volumeMounts:
- name: wrapper
mountPath: /scripts
volumes:
- name: wrapper
configMap:
name: wrapper
defaultMode: 0744
...

Using commands as liveness/readiness checks

This one is also pretty well known but often forgotten.  Using commands a health checks is a nice way to check that things are working.  For example, if you are doing complicated DNS things and want to check if DNS has updated you can use dig.  Or if your app updates a file when it becomes healthy, you can run a command to check for this.

readinessProbe:
exec:
command:
- cat
- /tmp/healthy
initialDelaySeconds: 5
periodSeconds: 5

Host aliases

Host aliases in Kubernetes offer a simple way to easily update the /etc/hosts file of a container.  This can be useful for example if a localhost name needs to be mapped to some DNS name that isn’t handled by the DNS server.

apiVersion: v1
kind: Pod
metadata:
  name: hostaliases-pod
spec:
  restartPolicy: Never
  hostAliases:
  - ip: "127.0.0.1"
    hostnames:
    - "foo.local"
    - "bar.local"
  containers:
  - name: cat-hosts
    image: busybox
    command:
    - cat
    args:
    - "/etc/hosts"

Conclusion

As mentioned, these are just a few gems that I have uncovered, I’m sure there are a lot of other neat tricks out there.  As I get more experience using Kubernetes I will be sure to update this list.  Please let me know if there are things that should be on here that I missed or don’t know about.

Read More