In this post I will describe some of the interesting discoveries I have made for a recent side project I have been working on, which include a few nice discoveries to automate and manage WordPress deployments with Kubernetes.
Automating WordPress
I am very happy with the patterns that have emerged from this project. One thing that I have always struggled with (I’m sure others have as well) in the world of WP has been finding a good way to create completely reproducible and immutable code and configurations. For example, managing plugins and themes has been a painful experience because WP was designed back in a time before configuration as code and infrastructure as code. Due to this different paradigm, WP is usually stood up once, then managed via it
Tools have evolved since then and a perfect example of one of the bridges between the old and new way is the wp-cli. The wp-cli is basically a way to automate all kinds of things you would otherwise do in the UI. For example, wp-cli provides a way to manage plugin and themes, which as I mentioned has been notoriously difficult to do in the past.
The next step forward is the combination of a tool called bedrock and its accompanying way of modifying it and building your own Docker image. The roots/bedrock method provides the wp-cli in the build scripts so if needed, you can automate tasks using extra entrypoint scripts and/or wp-cli commands, which is just a nice extra touch and shows that the maintainers of the project are putting a lot of effort into it.
A few other bells and whistles include a way to build custom plugins into Docker images for portability rather than relying on some external persistent storage solution which can quickly add overhead and complexity to a project, as well as modern tools like PHP Composer and Packagist which provide a way to install packages (Composer) and a way to manage WP plugins via the Composer package manager (packagist).
Sidenote
There are several other ways of deploying WP into Kubernetes, unfortunately most of these methods do not address multitenancy. If multitenancy is needed, a much more complicated approach is needed involving either NFS or some other many -> many volume mapping.
Deploying with Kubernetes
The tricky part to all of this is the fact that I was unable to find any examples of others using Kubernetes to deploy bedrock managed Docker containers. There is a docker-compose.yaml file in the repo that works perfectly well, but the next step beyond that doesn’t seem to be a topic that has been covered much.
Luckily it is mostly straight forward to bring the docker-compose configuration into Kubernetes, there are just a few minor adjustments that need to be made. The below link should provide the basic scaffolding needed to bring bedrock into a Kubernetes cluster. This method will even expose a way to create and manage WP multisite, another notoriously difficult aspect of WP to manage.
There are a couple of things to note with this configuration. You will need build/maintain your own Docker image based off the roots/bedrock repo linked above. You will also need to have some knowledge of Kubernetes and a working Kubernetes cluster in place. The configuration will require certificates, and DNS so cert-manager and external-dns will most likely need to be deployed into the cluster.
Finally, in the configuration the password, domain (example.com), environment variables for configuring the database and Docker image will need to be updated to reflect your own environment. This method assumes that the WordPress database has already been split out to another location, so will require the Kubernetes cluster to be able to communicate with wherever the database is hosted.
To see some of the magic, change the number of replicas in the Kubernetes manifest configuration from 1 to 2, and you should be able to see a new, completely identical container come up with all the correct configurations and code and start taking traffic.
Conclusion
Switching to the immutable infrastructure approach with WP nets a big win. By adopting these new methods and workflows you can control everything with code, which removes the need for manually managing WP instances and instead allows you to create workflows and pipelines to do all of the heavy lifting.
These benefits include much more visibility in controlling changes, because now Git becomes the central source of truth which allows you to get a better picture of the what, when and why than any other system I have found. This new paradigm also enables the use of Continuous Integration as it is intended – the automatic builds and deploys because of Docker and Kubernetes integrations producing immutable artifacts (Docker), and deployments (Kubernetes manifests) create a clean and simple way to manage the aspects of running the WordPress site.
Continuing with the multiarch and Kubernetes narratives that I have been writing about for awhile now, in this post I will be showing off some of the capabilities of the Drone.io continuous integration tool running on my arm64 Kubernetes homelab cluster.
As arm64 continues to gain popularity, more and more projects are adding support for it, including Drone.io. With its semi recent announcement, Drone can now run on a variety of different architectures now. Likewise, the Drone maintainers have also been working towards a 1.0 release, which brings first class support for Kubernetes, among other things.
I won’t touch on too many of the specifics of Drone, but if you’re interested in learning more, you can check out the website. I will mostly be focusing on how to get things running in Kubernetes, which turns out to be exactly the same for both amd64 and arm64 architectures. There were a few things I discovered along the way to get the Kubernetes integrations working but for the most part things “just work”.
I started off by grabbing the Helm chart and modifying it to suit my needs. I find it easiest to template the chart and then save it off locally so I can play around with it.
Note: the below example assumes you already have helm installed locally.
Obviously you will want to set the configurations to match your own settings, like domain name and oauth settings (I used Github).
After saving out the manifest, the first issue I ran into is that port 9000 is still referenced in the Helm chart, which was used for communication between the client and server in the older releases, but is no longer used. So I just completely removed the references to the port in my Frankenstein configuration. If you are just using the Kubernetes configuration mentioned below, you won’t run into these problems connecting the server and agent, but if you use the agent you will.
There is some server config that will need to adjusted as well to get things working. For example, the oauth settings will need to be created on the Github side first in order for any of this to work. Also, the drone server host will need to be accessible from the internet so any firewall rules will need to be added or adjusted to allow traffic.
Add the DRONE_USER_CREATE env var to bootstrap an admin account when starting the Drone server. This will allow your user to do all of the admin things using the CLI tool.
The secrets so should get generated when you dump the Helm chart, so feel free to update those with any values you may need.
Note: if you have double checked all of your settings but builds aren’r being triggered, there is a good chance that the webhook is the problem. There is a really good post about how to troubleshoot these settings here.
Running Drone with the Kubernetes integration
This option turned out to be the easier approach. Just add the following configuration to the drone server deployment environment variables, updating according to your environment. For example, the namespace I am deploying to is called “cicd”, which will need to be updated if you choose a different namespace.
The main downside to this method is that it creates Kubernetes jobs for each build. By default, once these builds are done, the will exit and not clean themselves up, so if you do a lot of builds then your namespace will get clogged up. There is a way to set TTLs on finished to clean themselves up in newer versions of Kubernetes via the TTLAfterFinished flag but this functionality isn’t default in Kubernetes yet and is a little bit out of the scope of this post.
Running Drone with the agent configuration
The agent uses the sidecar pattern to run a Docker in Docker (dind) container to connect to the Docker socket in order to allow the Drone agent to do its builds.
The main downside of using this approach is that there seems to be an issue (sometimes) where the Drone components can’t talk to the Docker socket, you can find a better description of this problem and more details here. The problem seems to be a race condition where the docker socket is not being able to be mounted before the agent comes up, but I still haven’t totally solved the problem there yet. The advice for getting around this is to run the agent on a dedicated stand alone host to avoid race conditions and other pitfalls.
That being said, if you still want to use this method you will need to add an additional deployment to the config for the drone agent. If you use the agent you can disregard the above Kubernetes environment variable configurations and instead set appropriate environment variables in the agent. Below is the working snipped I used for deploying the agent to my test cluster.
I have gotten the agent to work, I just haven’t had very much success getting it working consistently. I would avoid using this method unless you have to or as mentioned above, get a dedicated host for running builds on.
Testing it out
After wiring everything up, the rest is easy. Add a file called .drone.yml to a repository that you would like to automate builds for. You can find out more about the various capabilities here.
For my use case I wanted to tell drone to build and publish an arm64 based Docker image whenever a change to master occurs. You can look at my drone configuration here to get a better idea of the multiarch options as well as authenticating to a Docker registry.
After adding the .drone.yml to your repo and triggering a build you should see something similar in your local Drone instance.
If everything worked correctly and is green then you should be good to go. Obviously there is a lot of overhead that Kubernetes brings but the actual Drone setup if really straight forward. Just set stuff up on the Github side, translate it into Kubernetes configurations and add some other Drone specific config options and you should have a nice CI/CD environment ready to go.
Conclusion
I am very excited to see Drone grow and mature and use it more in the future. It is simple yet flexible and it fits nicely into the new paradigm of using containers for everything.
The new 1.0 YAML syntax is really nice as well, as it basically maps to the conventions that Kubernetes has chosen, so if you are familiar with that syntax you should feel at home. You can check out the available plugins here, which cover about 90% of the use cases you would see in the wild.
One downside is that YAML syntax errors are really hard to debug, and there isn’t much in the way of output to figure out where your problems are. The best approach I have found is to run the .drone.yml file through the Drone CLI lint/fmt tool before committing and building.
The Drone CLI tool is really powerful and could probably be its own post. There are some links in the references that show off some of its other features.
References
There are a few cool Drone resources I would definitely recommend checking out if you are interested running Drone in your own environment. The docs reference is a great place to start and is great for finding information about how to configure and tweak various Drone settings.
I also definitely recommend checking out the jsonnet extension docs, which can be used to help improve automation workflows. The second link show an good example of how it works and the third link shows some practical applications of using jsonnet to help manage complicated CI pipelines.
Recently I have been experimenting with different ways of building multi architecture Docker images. As part of this process I wrote about Docker image manifests and the different ways you can package multi architecture builds into a single Docker image. Packaging the images is only half the problem though. You basically need to create the different Docker images for the different architectures first, before you are able to package them into manifests.
There are several ways to go about building the Docker images for various architectures. In the remainder of this post I will be showing how you can build Docker images natively against arm64 only as well as amd64/arm64 simultaneously using some slick features provided by the folks at Shippable. Having the ability to automate multi architecture builds with CI is really powerful because it avoids having to use other tools or tricks which can complicate the process.
Shippable recently announced integrated support for arm64 builds. The steps for creating these cross platform builds is fairly straight forward and is documented on their website. The only downside to using this method is that currently you must explicitly contact Shippable and requests access to use the arm64 pool of nodes for running jobs, but after that multi arch builds should be available.
For reference, here is the full shippable.yml file I used to test out the various types of builds and their options.
Arm64 only builds
After enabling the shippable_shared_aarch64 node pool (from the instruction above) you should have access to arm64 builds, just add the following block to your shippable.yml file.
runtime:
nodePool: shippable_shared_aarch64
The only other change that needs to be made is to point the shippable.yaml file at the newly added node pool and you should be ready to build on arm64. You can use the default “managed” build type in Shippable to create builds.
Below I have a very simple example shippable.yml file for building a Dockerfile and pushing its image to my Dockerhub account. The shippable.yml file for this build lives in the GitHub repo I configured Shippable to track.
Once you have a shippable.yml file in a repo that you would like to track and also have things set up on the Shippable side, then every time a commit/merge happens on the master branch (or whatever branch you set up Shippable to track) an arm64 Docker image gets built and pushed to the Dockerhub.
Docs for settings up this CI style job can be found here. There are many other configuration settings available to tune so I would encourage you to read the docs and also play around with the various options.
Parallel arm64 and amd64 builds
The approach for doing the simultaneous parallel builds is a little bit different and adds a little bit more complexity, but I think is worth it for the ability to automate cross platform builds. There are a few things to note about the below configuration. You can use templates in either style job. Also, notice the use of the shipctl command. This tool basically allows you to mimic some of the other functionality that exists in the default runCI jobs, including the ability to login to Docker registries via shell commands and manage other tricky parts of the build pipeline, like moving into the correct directory to build from.
Most of the rest of the config is pretty straight forward. The top level jobs directive lets you create multiple different jobs, which in turn allows you to set the runtime to use different node pools, which is how we build against amd64 and arm64. Jobs also allow for setting different environment variables among other things. The full docs for jobs shows all of the various capabilities of these jobs.
As you can see, there is a lot more manual configuration going on here than the first job.
I decided to use the top level templates directive to basically DRY the configuration so that it can be reused. I am also setting environment variables per job to ensure the correct architecture gets built and pushed for the various platforms. Otherwise the configuration is mostly straight forward. The confusion with these types of jobs if you haven’t set them up before mostly comes from figuring out where things get configured in the Shippable UI.
Conclusion
I must admit, Shippable is really easy to get started with, has good support and has good documentation. I am definitely a fan and will recommend and use their products whenever I get a chance. If you are familiar with Travis then using Shippable is easy. Shippable even supports the use of Travis compatible environment variables, which makes porting over Travis configs really easy. I hope to see more platforms and architectures supported in the future but for now arm64 is a great start.
There are some downside to using the parallel builds for multi architecture builds. Namely there is more overhead in setting up the job initially. With the runSh (and other unmanaged jobs) you don’t really have access to some of the top level yml declarations that come with managed jobs, so you will need to spend more time figuring out how to wire up the logic manually using shell commands and the shipctl tool as depicted in my above example. This ends up being more flexible in the long run but also harder to understand and get working to begin with.
Another downside of the assembly line style jobs like runSh is that they currently can’t leverage all the features that the runCI job can, including the matrix generation (though there is a feature request to add it in the future) and report parsing.
The last downside when setting up unmanaged jobs is trying to figure out how to wire up the different components on the Shippable side of things. For example you don’t just create a runCI job like the first example. You have to first create an integration with the repo that you are configuring so that shippable can make an rSync and serveral runSh jobs to connect with the repo and be able to work correctly.
Overall though, I love both of the runSh and runCI jobs. Both types of jobs lend themselves to being flexible and composable and are very easy to work with. I’d also like to mention that the support has been excellent, which is a big deal to me. The support team was super responsive and helpful trying to sort out my issues. They even opened some PRs on my test repo to fix some issues. And as far as I know, there are no other CI systems currently offering native arm64 builds which I believe will become more important as the arm architecture continues to gain momentum.
As part of my recent project to build an ARM based Kubernetes cluster (more on that in a different post) I have run into quite a few cross platform compatibility issues trying to get containers working in my cluster.
After a little bit of digging, I found that support was added in version 2.2 of the Docker image specification for manifests, which all Docker images to built against different platforms, including arm and arm64. To add to this, I just recently discovered that in newer versions of Docker, there is a manifest sub-command that you can enable as an experimental feature to allow you to interact with the image manifests. The manifest command is great for exploring Docker images without having to pull and run and test them locally or fighting with curl to get this information about an image from a Docker registry.
Enable the manifest command in Docker
First, make sure to have a semi recent version of Docker installed, I’m using 18.03.1 in this post.
Edit your docker configuration file, usually located in ~/.docker/config.json. The following example assumes you have authentication configured, but really the only additional configuration needed is the { “experimental”: “enabled” }.
After adding the experimental configuration to the client you should be able to access the docker manifest commands.
docker manifest -h
To inspect a manifest just provide an image to examine.
docker manifest inspect traefik
This will spit out a bunch of information about the Docker image, including schema, platforms, digests, etc. which can be useful for finding out which platforms different images support.
As you can see above image (traefik) supports arm and arm64 architectures. This is a really handy way for determining if an image works across different platforms without having to pull an image and trying to run a command against it to see if it works. The manifest sub command has some other useful features that allow you to create, annotate and push cross platform images but I won’t go into details here.
Manifest tool
I’d also like to quickly mention the Docker manifest-tool. This tool is more or less superseded by the built-in Docker manifest command but still works basically the same way, allowing users to inspect, annotate, and push manifests. The manifest-tool has a few additional features and supports several registries other than Dockerhub, and even has a utility script to see if a given registry supports the Docker v2 API and 2.2 image spec. It is definitely still a good tool to look at if you are interested in publishing multi platform Docker images.
Downloading the manifest tool is easy as it is distributed as a Go binary.
I’d also like to touch quickly on the mquery tool. If you’re only interested in seeing if a Docker image uses manifest as well as high level multi-platform information you can run this tool as a container.
docker run --rm mplatform/mquery traefik
Here’s what the output might look like. Super simple but useful for quickly getting platform information.
This can be useful if you don’t need a solution that is quite as heavy as manifest-tool or enabling the built in Docker experimental support.
You will still need to figure out how to build the image for each architecture first before pushing, but having the ability to use one image for all architectures is a really nice feature.
There is work going on in the Docker and Kubernetes communities to start leveraging the features of the 2.2 spec to create multi platform images using a single name. This will be a great boon for helping to bring ARM adoption to the forefront and will help make the container experience on ARM much better going forward.
If you haven’t heard yet, Rancher recently revealed news that they will be building out a new v2.0 of their popular container orchestration and management platform to be built specifically to run on top of Kubernetes. In the container realm, Kubernetes has recently become a clear favorite in the battle of orchestration and container management. There are still other options available, but it is becoming increasingly clear that Kubernetes has the largest community, user base and overall feature set so a lot of the new developments are building onto Kubernetes rather than competing with it directly. Ultimately I think this move to build on Kubernetes will be good for the container and cloud community as companies can focus more narrowly now on challenges tied specifically around security, networking, management, etc, rather than continuing to invent ways to run containers.
With Minikube and the Docker for Mac app, testing out this new Rancher 2.0 functionality is really easy. I will outline the (rough) process below, but a lot of the nuts and bolts are hidden in Minikube and Rancher. So if you’re really interested in learning about what’s happening behind the scenes, you can take a look at the Minikube and Rancher logs in greater detail.
Speaking of Minkube and Rancher, there are a few low level prerequisites you will need to have installed and configured to make this process work smoothly, which are listed out below.
Prerequisites
Tested on OSX
Get Minikube working – use the Kubernetes/Minikube notes as a reference (you may need to bump memory to 4GB)
I won’t cover the installation of these perquisites, but I have blogged about a few of them before and have provided links above for instructions on getting started if you aren’t familiar with any of them.
Get Rancher 2.0 working locally
The quick start guide on the Rancher website has good details for getting this working – http://rancher.com/docs/rancher/v2.0/en/quick-start-guide/. On OSX you can use the Docker for Mac app to get a current version of Docker and compose. After Docker is installed, the following command will start the Rancher container for testing.
docker run -d --restart=unless-stopped -p 8080:8080 --name rancher-server rancher/server:preview
Check that you can access the Rancher 2.0 UI by navigating to http://localhost:8080 in your browser.
If you wanted to dummy a host name to make access a little bit easier you could just add an extra entry to /etc/hosts.
Import Minikube
You can import an existing cluster into the Rancher environment. Here we will import the local Minikube instance we got going earlier so we can test out some of the new Rancher 2.0 functionality. Alternately you could also add a host from a cloud provider.
In Rancher go to Hosts, Use Existing Kubernetes.
Then grab the IP address that your local machine is using on your network. If you aren’t familiar, on OSX you can reach into the terminal and type “ifconfig” and pull out the IP your machine is using. Also make sure to set the port to 8080, unless you otherwise modified the port map earlier when starting Rancher.
Registering the host will generate a command to run that applies configuration on the Kubernetes cluster. Just copy this kubectl command in Rancher and run it against your Minikube machine.
The above command will join Minikube into the Rancher environment and allow Rancher to manage it. Wait a minute for the Rancher components (mainly the rancher-agent continer/pod) to bootstrap into the Minikube environment. Once everything is up and running, you can check things with kubectl.
kubectl get pods --all-namespaces | grep rancher
Alternatively, to verify this, you can open the Kubernetes dashboard with the “minikube dashboard” command and see the rancher-agent running.
On the Rancher side of things, after a few minutes, you should see the Minikube instance show up in the Rancher UI.
That’s it. You now have a working Rancher 2.0 instance that is connected to a Kubernetes cluster (Minikube). Getting the environment to this point should give you enough visibility into Rancher and Kubernetes to start tinkering and learning more about the new features that Rancher 2.0 offers.
The new Rancher 2.0 UI is nice and simplifies a lot of the painful aspects of managing and administering a Kubernetes cluster. For example, on each host, there are metrics for memory, cpu, disk, etc. as well as specs about the server and its hardware. There are also built in conveniences for dealing with load balancers, secrets and other components that are normally a pain to deal with. While 2.0 is still rough around the edges, I see a lot of promise in the idea of building a management platform on top Kubernetes to make administrative tasks easier, and you can still exec to the container for the UI and check logs easily, which is one of my favorite parts about Rancher. The extra visualization is a nice touch for folks that aren’t interested in the CLI or don’t need to know how things work at a low level.
When you’re done testing, simply stop the rancher container and start it again whenever you need to test. Or just blow away the container and start over if you want to start Rancher again from scratch.
