New to etcd 2.1.0 is the ability to use authentication to secure your etcd resources. Encryption and authentication are relatively new additions so I thought I would write a quick blog post to help remember how to get these components up and running as well as help others because some of the ideas were a little confusing to me at first.
I pieced together most of the information for this post together from a few different sources.
The first were a pair of great tutorials (1, 2) for getting etcd encryption up and going. The second resource used was the etcd-ca project by CoreOS for creating a CA and issuing certs, there are other ways of doing it but this was a straight forward method. The third resource I recommend look at is the Security page in the CoreOS docs that shows examples of how to piece all of the commands and certs together. The last resource readers might find useful is the etcd2 docs for the different flags and configuration options. This resource was helpful for finding out all the various options that I needed to enable to get etcd2 working properly.
Requirements
To use the authentication feature you will need to have etcd 2.1.0 or greater, which means you will need to be running a version of CoreOS that has the correct binary, which means you will either need CoreOS v752.1.0 or above, OR the correct binary version/Docker image.
Authentication is still an “experimental” feature so it may change at any time, therefore I have decided not to get in to any of the details of how it works. If you are interested you can check out the docs on users and auth.
Running the CA server
At first I was conernced about running a CA server because I’ve had painful experiences in the past with CA’s but the etcd-ca tool makes this process easy and straight forward. There are a few other CA resources in the etcd2 encryption docs but I won’t cover them here.
The easiest way to use the etcd-ca tool is to run it in a Docker container and write the certs out to the host via a shared voulme. The following steps will pull the repo and build the binary for running the tool.
docker pull golang docker run -i -t $(pwd):/go golang /bin/bash git clone https://github.com/coreos/etcd-ca cd etcd-ca ./build cd ./bin
Create the certs
After the etcd-ca binary has been built we can start creating certs. The first thing necessary is to create the CA certs which will be used to sign all other certs.
./etcd-ca init
After creating the CA signing cert we will create a certificate for the etcd server that will be authenticating to.
./etcd-ca new-cert -ip <etcd_server_ip> <hostname> ./etcd-ca sign <hostname> ./etcd-ca chain <hostname> ./etcd-ca export --insecure <hostname> | tar xvf -
Replace <etcd_server_ip> with the public address of the etcd server and <hostname> with the hostname of the etcd server. In this example, something like core01 would be a good name.
Optional – Client cert
This is not necessary in all scenarios for setting up encryption for etcd but if you are interested in having clients authenticate with their own cert it isn’t that much effort to add.
./etcd-ca new-cert -ip <etcd_server_ip> client ./etcd-ca sign client ./etcd-ca export --insecure client | tar xvf -
Note: You may need to move the above keys from the server/clientkey files generated to the correct filename. Also to note, if you screw up any of the certs or for any reason need to recreate them you can simply delete the certificates from the .etcd-ca/ hidden folder that contains all of the certificates.
Etcd cloud-config
The following cloud-config will configure etcd2 to use the certs we configured above.
There is currently an issue parsing a few of the etcd2 command line flags so the workaround (for now) is to split the configuration up in to a base config and then to add env vars as a a drop in.
write_files:
- path: /etc/systemd/system/etcd2.service.d/30-configuration.conf
permissions: '0644'
content: |
[Service]
# General settings
Environment=ETCD_NAME=etcd-config
Environment=ETCD_VERBOSE=1
# Encrytpion
Environment=ETCD_CLIENT_CERT_AUTH=1
Environment=ETCD_TRUSTED_CA_FILE=/home/core/ca.crt
Environment=ETCD_PEER_KEY_FILE=/home/core/server.key
Environment=ETCD_PEER_CERT_FILE=/home/core/server.crt
Environment=ETCD_CERT_FILE=/home/core/server.crt
Environment=ETCD_KEY_FILE=/home/core/server.key
- path: /home/core/ca.crt
permissions: '0644'
content: |
-----BEGIN CERTIFICATE-----
ca cert content
-----END CERTIFICATE-----
- path: /home/core/server.crt
permissions: '0644'
content: |
-----BEGIN CERTIFICATE-----
server cert content
-----END CERTIFICATE-----
- path: /home/core/server.key
permissions: '0644'
content: |
-----BEGIN RSA PRIVATE KEY-----
server key content
-----END RSA PRIVATE KEY-----
- path: /home/core/client.crt
permissions: '0644'
content: |
-----BEGIN CERTIFICATE-----
client cert content
-----END CERTIFICATE-----
- path: /home/core/client.key
permissions: '0644'
content: |
-----BEGIN RSA PRIVATE KEY-----
client key content
-----END RSA PRIVATE KEY-----
coreos:
etcd2:
name: etcd
discovery: https://discovery.etcd.io/a1c999ec1a23039996419e0a20cb1e35
advertise-client-urls: https://$public_ipv4:2379
initial-advertise-peer-urls: https://$private_ipv4:2380
listen-client-urls: https://0.0.0.0:2379
listen-peer-urls: https://$private_ipv4:2380
units:
- name: etcd2.service
command: start
If you don’t want to bootstrap a node with cloud-config and instead are just interested in testing out testing encryption on an existing how you can use the following commands. You will still need to make sure you follow the steps above to generate all of the necessary certs!
Manually start etcd2 with server certificate:
etcd2 -name infra0 -data-dir infra0 \ -cert-file=/home/core/server.crt -key-file=/home/core/server.key \ -advertise-client-urls=https://<server_ip>:2379 -listen-client-urls=https://<server_ip>:2379
and to test the connection use the following curl command.
curl --cacert /home/core/ca.crt https://172.17.8.101:2379/v2/keys/foo -XPUT -d value=bar -v
Manually start etcd2 with client certificate:
Etcd2 -name infra0 -data-dir infra0 \ -client-cert-auth -trusted-ca-file=/home/core/ca.crt -cert-file=/home/core/server.crt -key-file=/home/core/server.key \ -advertise-client-urls https://<server_ip>:2379 -listen-client-urls https://<server_ip>:2379
Similar to the above command you will just need to add the client certs to authenticate.
curl --cacert /home/core/ca.crt --cert /home/core/client.crt --key /home/core/client.key \ -L https://<server_ip>:2379/v2/keys/foo -XPUT -d value=bar -v
Another way to test the certs out is by using the etcdctl tool by addding a few flags.
etcdctl --ca-file ca.crt --cert-file client.crt --key-file client.key --peers https://<server_ip>:2379 set /foo bar etcdctl --ca-file ca.crt --cert-file client.crt --key-file client.key --peers https://172.17.8.101:2379 get /foo
Encrypting etcd was a confusing process to me at first due to the complexity of encryption but after working through the above examples, most of the process made sense. I seem to have a hard time wrapping my head around all of the different parts so hopefully I have effectively showed how the encryption component works.
The etcd-ca tool is very nice for testing because it is simple and straightforward but lacks a few features of a full fledged CA. I suggest looking at using something like Openssl for a production type scenario. Especially if things like certificate revocations are important.
