CoreOS Tips and Tricks

One thing that was never clear to me when I started learning CoreOS were techniques for rapidly testing out different CoreOS features.  I will spend some time walking folks through a few of tips and tricks that I have learned so far along the way learning about CoreOS.

The folks at CoreOS have an awesome repo for testing out features locally, called coreos-vagrant.  If you haven’t heard of it or used it, go check it out.  Another great resource for getting started with the CoreOS Vagrant project are the docs on the CoreOS website, you should be able to find most of the use cases there.

So in this post I will be going over some of what is already detailed in the docs and README but will additionally fill readers in with a few extra tips and tricks I have discovered so far along the way.  I am surprised by all of the hidden secrets I frequently discover buried in CoreOS and its documentation.  It is always fun to find new features and capabilities of the OS that you didn’t know existed.


So to get started we need to briefly cover Vagrant.  Vagrant has made things soooo much easier to test.  If you haven’t heard of Vagrant, definitely go check it out and get familiar with it.  It is basically an interface for controlling VM’s and their various components locally.

When I first starting testing things out with CoreOS I would spin boxes up in either Digital Ocean or AWS with a cloud-config that I would agonize over because I was afraid of screwing up small details or provisioning the server incorrectly.  It is also more of a hassle to provision a cloud server because it involves some additional authentication keys for command line tools or manually creating instances via GUI tools.  However, when testing VM’s you often destroy and recreate instances and so that additional overhead can become tedious.

Using Vagrant I can quickly and easily make changes to a configuration or even test out entirely different CoreOS versions in minutes and not care about getting small details wrong since a) it doesn’t cost me anything extra to run the instance locally and b) I can blow out and reprovision in a few seconds.

I think this local Vagrant approach also makes you a better CoreOS citizen because it forces you to look at what you’re doing and fix issues more often because you are iterating more frequently and therefore testing more features and options of CoreOS out (at least this has been my experience so far).


Cloud-config was initially a painful part of the learning process for me but I have grown to love it.  I like to test out new cloud-configs quite a bit and at first it was frustrating to screw up configs because that meant I had to redo the entire server bootstrap process in DO or AWS.  Terraform makes the provisioning process less painful but it is still a little bit of a hassle, especially when you are smoke testing configs to make sure they work.

Luckily it is dead simple to set up cloud-configs using Vagrant locally.  The repo comes with a “user-data.sample” file that you can copy to “user-data” and away you go, make any modifications you may need or config changes you want to test out.  The local testing via cloud-config discovery alone was a game changer for me.

To fix a problem with your cloud-config you can simply edit the user-data file that was copied in to place on the server and then rerun cloud-init to fix the provisioning.  Below is an example of how to do this cloud-init provisioning.

Before you provision any of you cloud configs though, I recommend testing them out by running them through the CoreOS cloud-config validator tool to help identify any potential problems your config might have before you even run it.  There is an experimental validation flag option in the cloud-init binary shipped with the OS if you want to try it out as well.  Most of the time I find it just as easy to copy the config in to the online checker but there are definitely scenario’s and use cases where it might be a good idea to test locally, I just haven’t needed to yet.

Next, if you have an existing config on your server and would like to modify the existing content and reprovision the server with the updated cloud-config, without destroying and recereating the server, use the following.

sudo /usr/bin/coreos-cloudinit --from-file /path/to/user-data

Then you can watch the logs to make sure they are doing what you expect.

journalctl -b _EXE=/usr/bin/coreos-cloudinit

If you don’t want to muck around with the cloud-config stuff on the server you can easily blow up the server, modify the user-data file on the host and just reprovision the Vagrant machine.  Obviously this method will take a little bit longer but it isn’t a significant penalty and is also is easier to keep track of since you know exactly what user-data values are being passed in the Vagrant machine from the host and can more easily stay on top of the changes you are making.


The config section in Vagrant gives you a great deal of flexibility when testing CoreOS out locally.  For example, you can control most options that CoreOS gets provisioned with, including the version release with this,

$image_version = "723.1.0"

You can specify any version inside the quotes to bootstrap the CoreOS instance.  This is handy for testing out new alpha features or things are broken in one release.  Quickly changing versions gives you and easy way to check if they are fixed yet by either rolling back or forward easily.

In the config.rb file you can also specify server level details for things like the hostname,


How many instances to provision,


Custom memory or cpu’s for the instance,

$vm_memory = 1024
$vm_cpus = 1

Shared folders, forwarded ports etc.  Granted these are Vagrant level configurations, it still makes working with CoreOS much easier in my opinion.

Additionally, there is an option to provision the instance with an etcd/2 discovery token to bootstrap etcd when the server gets created.  If you have ever dealt with testing out etcd, this is an option way for quickly bringing servers up and down without ever having to worry about reissuing the discovery tokens, etc.

Tips and Tricks

I have found a few other tips and tricks along the way that can be used when testing CoreOS locally or after it has been deployed.

The first tip is getting the OS version to update manually (without reprovisioning via Vagrant).  For most testing puproses I usually turn off automatic reboots using the following key in my cloud configs.

    group: alpha
    reboot-strategy: off

This will tell CoreOS to try to use the latest alpha (if a version is not specified in your config.rb) and tell CoreOS to not reboot.

Sometimes it is easier to just manually updated the OS than destroy the VM and specify a new version.  To update manually you can run the following commnads.

update_engine_client -check_for_update
journalctl -f (this will follow the update progress)
sudo reboot (after the updated version is downloaded)

After you see that the newest release has been downloaded you can reboot the server and it should boot up with the newest updates.

Another cool trick is to customize the toolbox on CoreOS.  I’ve written about this before but figured I might as well mention it again since it is a useful trick.

By default the toolbox runs Fedora, but we are mainly an Ubuntu/Debian shop so are much more comfortable using the tools bundled with those distros.  It is pretty simple to configure the toolbox to automatically use Debian when the instance is provisioned using the following key in your cloud-config.

  - path: /home/core/.toolboxrc
    owner: core
    content: |

When you run the “toolbox” command it will look for Ubuntu instead of the default Fedora image.

Another trick I have used a few times is overriding the update strategy on a server that has already been provisioned using environment variables.

As I have discovered, much of the configuration that takes place happens via environment variables.  So to update the reboot strategy you can modify the /etc/coreos/update.conf file.  The contents should look something like this:


If you’d like to have the server use alpha images change the key to GROUP=alpha, etc. for the keys inside the configuration.  After making your changes, you will need to restart the update-engine service.

sudo systemctl restart update-engine

The system should pick up the changes you made and you should be good to go.

The last trick I will highlight in this post is how to get “drop in” services working.  This is a core part of how systemd (especially on CoreOS) works, but so few realize how it works.  By creating a drop in you are simply extending a service to read in extra bits of configuration.  For example, the following unit file extends the system etcd2 service.

Create the following file,


The etcd2 service will look in this location for its extra configuration.

# General settings

Inside the unit file we are just setting some extra environment variables that etcd2 can then use as flags to instruct it how to run.

There are obviously a lot more CoreOS tricks.  I have just highlighted a few of my favorites here.  I suggest looking at the CoreOS docs, there is a lot of good information over there.  Feel free to comment with your own tricks and I will be sure to try them out and get them added here.

Read More

CoreOS etcd2 encryption

Etcd 2.1.1 Encryption and Authentication

New to etcd 2.1.0 is the ability to use authentication to secure your etcd resources.  Encryption and authentication are relatively new additions so I thought I would write a quick blog post to help remember how to get these components up and running as well as help others because some of the ideas were a little confusing to me at first.

I pieced together most of the information for this post together from a few different sources.

The first were a pair of great tutorials (1, 2) for getting etcd encryption up and going.  The second resource used was the etcd-ca project by CoreOS for creating a CA and issuing certs, there are other ways of doing it but this was a straight forward method.  The third resource I recommend look at is the Security page in the CoreOS docs that shows examples of how to piece all of the commands and certs together.  The last resource readers might find useful is the etcd2 docs for the different flags and configuration options.  This resource was helpful for finding out all the various options that I needed to enable to get etcd2 working properly.


To use the authentication feature you will need to have etcd 2.1.0 or greater, which means you will need to be running a version of CoreOS that has the correct binary, which means you will either need CoreOS v752.1.0 or above, OR the correct binary version/Docker image.

Authentication is still an “experimental” feature so it may change at any time, therefore I have decided not to get in to any of the details of how it works.  If you are interested you can check out the docs on users and auth.

Running the CA server

At first I was conernced about running a CA server because I’ve had painful experiences in the past with CA’s but the etcd-ca tool makes this process easy and straight forward.  There are a few other CA resources in the etcd2 encryption docs but I won’t cover them here.

The easiest way to use the etcd-ca tool is to run it in a Docker container and write the certs out to the host via a shared voulme.  The following steps will pull the repo and build the binary for running the tool.

docker pull golang
docker run -i -t $(pwd):/go golang /bin/bash
git clone
cd etcd-ca
cd ./bin

Create the certs

After the etcd-ca binary has been built we can start creating certs.  The first thing necessary is to create the CA certs which will be used to sign all other certs.

./etcd-ca init

After creating the CA signing cert we will create a certificate for the etcd server that will be authenticating to.

./etcd-ca new-cert -ip <etcd_server_ip> <hostname>
./etcd-ca sign <hostname>
./etcd-ca chain <hostname>
./etcd-ca export --insecure <hostname> | tar xvf -

Replace <etcd_server_ip> with the public address of the etcd server and <hostname> with the hostname of the etcd server.  In this example, something like core01 would be a good name.

Optional – Client cert

This is not necessary in all scenarios for setting up encryption for etcd but if you are interested in having clients authenticate with their own cert it isn’t that much effort to add.

./etcd-ca new-cert -ip <etcd_server_ip> client
./etcd-ca sign client
./etcd-ca export --insecure client | tar xvf -

Note:  You may need to move the above keys from the server/clientkey files generated to the correct filename.  Also to note, if you screw up any of the certs or for any reason need to recreate them you can simply delete the certificates from the .etcd-ca/ hidden folder that contains all of the certificates.

Etcd cloud-config

The following cloud-config will configure etcd2 to use the certs we configured above.

There is currently an issue parsing a few of the etcd2 command line flags so the workaround (for now) is to split the configuration up in to a base config and then to add env vars as a a drop in.

  - path: /etc/systemd/system/etcd2.service.d/30-configuration.conf
  permissions: '0644'
  content: |
  # General settings
  # Encrytpion
  - path: /home/core/ca.crt
  permissions: '0644'
  content: |
  ca cert content

  - path: /home/core/server.crt
  permissions: '0644'
  content: |
  server cert content

  - path: /home/core/server.key
  permissions: '0644'
  content: |
  server key content

  - path: /home/core/client.crt
  permissions: '0644'
  content: |
  client cert content

  - path: /home/core/client.key
  permissions: '0644'
  content: |
  client key content

    name: etcd
    advertise-client-urls: https://$public_ipv4:2379
    initial-advertise-peer-urls: https://$private_ipv4:2380
    listen-peer-urls: https://$private_ipv4:2380
    - name: etcd2.service
    command: start

If you don’t want to bootstrap a node with cloud-config and instead are just interested in testing out testing encryption on an existing how you can use the following commands.  You will still need to make sure you follow the steps above to generate all of the necessary certs!

Manually start etcd2 with server certificate:

etcd2 -name infra0 -data-dir infra0 \ -cert-file=/home/core/server.crt -key-file=/home/core/server.key \ -advertise-client-urls=https://<server_ip>:2379 -listen-client-urls=https://<server_ip>:2379

and to test the connection use the following curl command.

curl --cacert /home/core/ca.crt -XPUT -d value=bar -v

Manually start etcd2 with client certificate:

Etcd2 -name infra0 -data-dir infra0 \ -client-cert-auth -trusted-ca-file=/home/core/ca.crt -cert-file=/home/core/server.crt -key-file=/home/core/server.key \ -advertise-client-urls https://<server_ip>:2379 -listen-client-urls https://<server_ip>:2379

Similar to the above command you will just need to add the client certs to authenticate.

curl --cacert /home/core/ca.crt --cert /home/core/client.crt --key /home/core/client.key \ -L https://<server_ip>:2379/v2/keys/foo -XPUT -d value=bar -v

Another way to test the certs out is by using the etcdctl tool by addding a few flags.

etcdctl --ca-file ca.crt --cert-file client.crt --key-file client.key --peers https://<server_ip>:2379 set /foo bar

etcdctl --ca-file ca.crt --cert-file client.crt --key-file client.key --peers get /foo

Encrypting etcd was a confusing process to me at first due to the complexity of encryption but after working through the above examples, most of the process made sense.  I seem to have a hard time wrapping my head around all of the different parts so hopefully I have effectively showed how the encryption component works.

The etcd-ca tool is very nice for testing because it is simple and straightforward but lacks a few features of a full fledged CA.  I suggest looking at using something like Openssl for a production type scenario.  Especially if things like certificate revocations are important.

Read More

Change CoreOS default toolbox

This is a little trick that allows you to override the default base OS in the CoreOS “toolbox“.  The toolbox is a neat trick to allow you to debug and troubleshoot issues inside containers on CoreOS without having to do any outside work of setting up a container.

The default toolbox OS defaults to Fedora, which we’re going to change to Ubuntu.  There is a custom configuration file that will get read in via the .toolboxrc file, located at /home/core/.toolboxrc by default.  To keep things simple we will only be changing the few pieces of the config to get the toolbox to behave how we want.  More can be changed but we don’t really need to override anything else.


That’s pretty cool, but what if we want to have this config file be in place for all servers?  We don’t want to have to manually write this config file for every server we log in to.

To fix this issue we will add a simple configuration in to the user-data file that gets fed in to the CoreOS cloud-config when the server is created.  You can find more information about the CoreOS cloud-configs here.

The bit in the cloud config that needs to change is the following.

  - path: /home/core/.toolboxrc
    owner: core
    content: |

If you are already using cloud-config then this change should be easy, just add the bit starting with -path to your existing -write_files section.  New servers using this config will have the desired toolbox defaults.

This approach gives us an automated, reproducible way to clone our custom toolbox config to every server that uses cloud-config to bootstrap itself.  Once the config is in place simply run the “toolbox” command and it should use the custom values to pull the desired Ubuntu image.

Then you can run your Ubuntu commands and debugging tools from within the toolbox.  Everything else will be the same, we just use Ubuntu now as our default toolbox OS.  Here is the post that gave me the idea to do this originally.

Read More