One thing I have run into recently and have been thinking about a little bit lately, is a simple way to hide environment variables that contain sensitive information. For example, when working in a local environment, if you need access to a secret like an oauth token or some authentication method to an API, the first inclination is usually to just hard code the secret contents into your local bash/zsh profile so that it can be read anytime you need access to it. This method obviously will work but if the filesystem itself isn’t encrypted, the secret can easily be leaked and for a small amount of effort I believe I have found an effective way of shrinking the visibility of these secrets.
Inspired by the aws-vault tool which is a simple but secure way of storing local AWS credentials in environment variables using a local password store, in this post I will show you a quick and dirty way to add an extra layer of security to your (other) local environment by injecting sensitive secrets stored in an encrypted location (password store) into your local terminal. This method works for both OSX and Linux and is just a few lines of configuration and examples for both OSes are shown below.
In OSX the keychain is a good starting place for storing and retrieving secrets and in Linux the combination of GPG and the standard unix password manager “pass” work well together. Pass also works on OSX if you aren’t a fan of keychain.
Below are steps for storing and retrieving local secrets using the Linux pass tool. There are installation instructions and full documentation for how to use the tool in the link above. It should also be noted that the system needs to have GPG installed in order to write and read secrets.
One you have GPG configured, create the password store. I am skipping most of the GPG configuration because there is a lot to know, the command below should be enough to get things started. If you already have GPG set up and configured you can skip the setup.
Set up GPG and pass.
gpg2 --full-gen-key # follow prompts to create a gpg store with defaults
pass init <email> # use the same email address used with gpg
pass git init # optionally set pass up as a git repo
Pass allows for hierarchies but in the example we are just going to put the secret at the top level. The command above will open the default editor. After closing the editor, the password will be written to an encrypted file in ~/.password-store. Once you have added the password you can show the contents of the newly added secret.
To read a secret into the terminal.
#pass show <secret>
pass show mysecret
You can also quickly list all of your secrets.
pass ls
Now that we have a created secret, we can write a little bash function to pull out the contents of the password and export them as an environment variable when the shell gets sourced. Put the following snippet into your ~/.bashrc, ~/.zshrc or ~/.bashprofile to read secrets.
get_password () {
pass show "$1"
}
A similar result can be achieved in OSX using the “security” command line tool.
In your shell configuration file you can simply export the result of calling the get_password() function into an environment variable.
export MYSECRET="$(get_password mysecret)"
Source the shell profile to pickup the new changes. After that, you should now see the contents of the secret inside an environment variable in your terminal.
source ~/.bashrc
env | grep MYSECRET
Conclusion
Obviously this isn’t a perfect way to secure your environment since the secret is available to anyone who is able to connect to this user so make sure you practice good security in as many other ways as possible.
What this method does do though is cuts down the amount of sensitive information that can be gleaned from a user account by ensuring that shell secrets are encrypted at rest and unavailable as clear text.
Kubernetes (k3os) arm64 cluster with custom 3D printed case
The k3os project was recently announced and I finally got a chance to test it out. k3os greatly simplifies the steps needed to create a Kubernetes cluster along with its counterpart, k3s, to reduce the overhead of running Kubernetes clusters. Paired with Rancher for the UI, all of these components make for an even better option. You can even run Rancher in your (arm64) k3os cluster via the Rancher Helm chart now.
Instead of using Etcd, k3s opts to use SQLite by default and does some other magic to reduce extra Kubernetes bloat and simplify management. Check here for more about k3s and how it works and how to run it.
k3os replaces some complicated OS components with much simpler ones. For example, instead of using Systemd it uses OpenRC, instead of Docker it uses containerd, it also leverages connman for configuring network components and it doesn’t use a package manager.
The method I am showing in this post uses the k3os overlay installation, which is detailed here. The reason for this choice is because the pine64 boards use u-boot to boot the OS and so special steps are needed to accommodate for the way this process is handled. The upside of this method is that these instructions should pretty much work for any of the Pi form factor boards, including the newly released Raspberry Pi4, with minimal changes.
Setup
If you haven’t downloaded and imaged your Pine64 yet, I like to use the ayufan images, which can be found here. You can easily write these images to a microSD card on OSX using something like Etcher.
Assuming the node is connected to your network, you can SSH into it.
ssh rock64@rock64 # or use the ip, password is rock64
When using the overlay installation, the first step is to download the k3os rootfs and lay it down on the host. This step applies to all nodes in the cluster.
curl -sL https://github.com/rancher/k3os/releases/download/v0.2.1/k3os-rootfs-arm64.tar.gz | tar --strip-components=1 -zxvf - -C
The above command is installing v0.2.1 which is the most current version as of writing this, so make sure to check if there is a newer version available.
After installing, lay down the following configurations into /k3os/system/config.yaml, modifying as needed. After the machine is rebooted this path will become read only so if you need to change the configuration again you will need to edit /etc/fstab to change the location to be writable again.
Server node
ssh_authorized_keys:
- ssh-rsa <your-public-ssh-key-to-login>
hostname: k3s-master
k3os:
data_sources:
- cdrom
dns_nameservers:
- 192.168.1.1 # update this to your local or public DNS server if needed
ntp_servers:
- 0.us.pool.ntp.org
- 1.us.pool.ntp.org
password: rancher
token: <TOKEN>
The k3s config will be written to /etc/rancher/k3s/k3s.yaml on this node so make sure to grab it if you want to connect the cluster from outside this node. Reboot the machine to boot to the new filesystem and you should be greeted with the k3os splash screen.
Agent node
The agent uses nearly the same config, with the addition of the server_url. Just point the agent nodes to the server/master and you should be good to go. Again, reboot after creating the config and the host should boot to the new filesystem and everything should be ready.
ssh_authorized_keys:
- ssh-rsa <your-public-ssh-key-to-login>
hostname: k3s-node-1
k3os:
data_sources:
- cdrom
dns_nameservers:
- 192.168.1.1 # update this to your local or public DNS server if needed
ntp_servers:
- 0.us.pool.ntp.org
- 1.us.pool.ntp.org
password: rancher
server_url: https://<server-ip-or-hostname>:6443
token: <TOKEN>
You can do a lot more with the bootstrap configurations, such as setting labels or environment variables. Some folks in the community have had luck getting the wifi configuration working on the RPi4’s out of the box, but I haven’t been able to get it to work yet on my Pine64 cluster. Check the docs for more details on the various configuration options.
After the nodes have been rebooted and configs applied, the cluster “should just work”. You can check that the cluster is up using k3s using the kubectl passthrough command (checking from the master node below).
k3s-master [~]$ k3s kubectl get nodes
NAME STATUS ROLES AGE VERSION
k3s-master Ready <none> 6d1h v1.14.1-k3s.4
k3s-node-1 Ready <none> 7m10s v1.14.1-k3s.4
NOTE: After installing the overlay filesystem there will be no package manager and no obvious way to upgrade the kernel so use this guide only for testing purposes. The project is still very young and a number of things still need to be added, including update mechanisms and HA. Be sure to follow the k3os issue tracker and Rancher Slack (#k3os channel) for updates and developments.
Conclusion
This is easily the best method I have found so far for getting a Kubernetes cluster up and running, minus the few caveats mentioned above, which I believe will be resolved very soon. I have been very impressed with how simple and easy it has been to configure and use. The next step for me is to figure out how to run Rancher and start working on some configurations for running workloads on the cluster. I will share more on that project in another post.
There are definitely some quirks to getting this setup working for the Pi and Pine64 based boards, but aren’t major problems by any means.
References
This post was heavily inspired by this gist for getting the overlay installation method working on Raspberry Pi.
As part of my adventures in building a 100% Arm64 Kubernetes cluster, I recently tried to build an Arm64 Jupyterhub Docker imageto run in the cluster. To my surprise, there doesn’t seems to be any “official” Jupyterhub arm64 Docker images out there, so I decided to set out and create one.
In the process of building my image, I almost immediately hit a stumbling block in that the Docker image uses the Conda package manager and several Conda packages for its build. The problem is that several of these packages have not yet been built to work on alternate architectures, e.g. Arm64, and others. So I went off down into the rabbit hole of seeing how hard it would be to add this support for these packages in order to get the Jupyterhub Docker image working.
The first stop on this journey was to conda-forge to look at the multiarch support. If you aren’t familiar (I wasn’t), conda-forge bills itself as a large Github community for creating and building Conda packages.
The first thing to look at when adding support to an existing package is getting familiar with conda-smithy, which is the tool responsible for setting up and building all of the various conda-forge “recipes”. There are generic instructions for using conda-smithy here.
As a fun side note, there is no “native” Arm64 build infrastructure for creating packages. The current builds use QEMU to emulate aarch64 (arm64) using Azure pipelines. This has some issues so while I was down in the rabbit hole I decided to contribute a PR to help get native arm64 builds added. The work isn’t yet complete, it still needs to be hooked up to CI, so if you want to help out feel free to let me know or just open a PR in the conda-smithy repo.
Multiarch support
With the housekeeping out of the way, we can now look at how to actually add the multiarch support for a package.
First, fork and clone the desired recipe. In this example I am adding arm64 support to the pycurl recipe as it is one of the Conda package dependencies that I need to build Jupyterhub for Arm64.
If you are just
adding support for new architectures like I am here, you will need to bump the
build number. This can be found
recipe/meta.yml, and there are also instruction for doing this.
…
build:
number: 0
…
Just change this value to 1. Next, install conda smithy if you don’t have it already.
conda install conda-smithy
And then you can
render out all the new files needed for the various builds.
conda-smithy rerender
Add the generated files to a new (forked) branch of the recipe.
Then open up a PR to
the conda-forge repo with the details.
Once the PR has been open a series of checks should kick off to build
the recipe for the various architectures.
If everything is green you are good to go. Maintainers are usually pretty good about merging in changes, but if you need to, you can ping an admin to get help.
You can also tell
the build to rerun if it fails using the “@conda-forge-admin, please
rerender” command.
You can find more details about what all the bot can do here.
Conclusion
Conda-forge provides some nifty tools for large scale automation and makes it super easy for outsiders to contribute to the community. If you find a missing, outdated or package lacking multiarch support on the Anaconda repo (which includes packages contributed by conda-forge along with many others), definitely think about contributing. The process of adding changes is easy and the conda-forge community is growing all the time.
The problem is described in more detail in the link below and there are a tons of posts in this thread but it was an arduous process and there wasn’t a lot of other information about this problem so I figured I would write up a quick summary of how to fix this for the case where the instructions aren’t quite enough.
Bascially, you need to run the lxrun command to reset Windows subsystem for Linux. The intructions were basically the following.
lxrun /uninstall /full /y
Reboot
Open an admin prompt or through explorer, delete all the content under the %localappdata%\lxss directory.
Install using bash.exe or "LxRun.exe /install"
However, something strange happened and every time I tried to do this I got a neat little error.
PS C:\Users\jmreicha\AppData\Local> LxRun /install /y
Warning: lxrun.exe is only used to configure the legacy Windows Subsystem for Linux distribution.
Distributions can be installed by visiting the Microsoft Store:
https://aka.ms/wslstore
This will install Ubuntu on Windows, distributed by Canonical and licensed under its terms available here:
https://aka.ms/uowterms
Error: 0x80070005
Likewise, I couldn’t reinstall the Windows subsystem using other methods, I just got the same exact error. I was poking around a little bit more and found that the files the subsystem uses get saved off into %localappdata%\lxss. By default this directory isn’t displayed in Windows so in order to see this folder you need to uncheck the “Hide protected operating system files (Recommended)” option in Windows Explorer View Options.
After showing OS files I was able to see the lxss directory and so I tried to delete the files, but got an interesting error message saying the files were no longer there and that Windows couldn’t remove them.
Item Not Found
The only way I found to get around this problem was to rename the folder (hence the _old suffix in the screen shot).
After moving the directory to another name I was able to run the lxrun /install command and successfully reinstall Ubuntu.
PS C:\Users\jmreicha\AppData\Local> LxRun /install
Warning: lxrun.exe is only used to configure the legacy Windows Subsystem for Linux distribution.
Distributions can be installed by visiting the Microsoft Store:
https://aka.ms/wslstore
This will install Ubuntu on Windows, distributed by Canonical and licensed under its terms available here:
https://aka.ms/uowterms
Type "y" to continue: y
Downloading from the Microsoft Store... 100%
Extracting filesystem, this will take a few minutes...
Conclusion
I saw various renditions of the solution sprinkled around the internet but none of them seemed to mention how to handle the case where Windows couldn’t locate a file and therefore couldn’t actually delete the lxss directory and recreate everything.
I haven’t figured out how to hard delete everything in the newly renamed “lxss_old” directory, so if anyone has any input it would be appreciated. As far as I know its contents are still consuming some amount of space on the system.
Continuing with the multiarch and Kubernetes narratives that I have been writing about for awhile now, in this post I will be showing off some of the capabilities of the Drone.io continuous integration tool running on my arm64 Kubernetes homelab cluster.
As arm64 continues to gain popularity, more and more projects are adding support for it, including Drone.io. With its semi recent announcement, Drone can now run on a variety of different architectures now. Likewise, the Drone maintainers have also been working towards a 1.0 release, which brings first class support for Kubernetes, among other things.
I won’t touch on too many of the specifics of Drone, but if you’re interested in learning more, you can check out the website. I will mostly be focusing on how to get things running in Kubernetes, which turns out to be exactly the same for both amd64 and arm64 architectures. There were a few things I discovered along the way to get the Kubernetes integrations working but for the most part things “just work”.
I started off by grabbing the Helm chart and modifying it to suit my needs. I find it easiest to template the chart and then save it off locally so I can play around with it.
Note: the below example assumes you already have helm installed locally.
Obviously you will want to set the configurations to match your own settings, like domain name and oauth settings (I used Github).
After saving out the manifest, the first issue I ran into is that port 9000 is still referenced in the Helm chart, which was used for communication between the client and server in the older releases, but is no longer used. So I just completely removed the references to the port in my Frankenstein configuration. If you are just using the Kubernetes configuration mentioned below, you won’t run into these problems connecting the server and agent, but if you use the agent you will.
There is some server config that will need to adjusted as well to get things working. For example, the oauth settings will need to be created on the Github side first in order for any of this to work. Also, the drone server host will need to be accessible from the internet so any firewall rules will need to be added or adjusted to allow traffic.
Add the DRONE_USER_CREATE env var to bootstrap an admin account when starting the Drone server. This will allow your user to do all of the admin things using the CLI tool.
The secrets so should get generated when you dump the Helm chart, so feel free to update those with any values you may need.
Note: if you have double checked all of your settings but builds aren’r being triggered, there is a good chance that the webhook is the problem. There is a really good post about how to troubleshoot these settings here.
Running Drone with the Kubernetes integration
This option turned out to be the easier approach. Just add the following configuration to the drone server deployment environment variables, updating according to your environment. For example, the namespace I am deploying to is called “cicd”, which will need to be updated if you choose a different namespace.
The main downside to this method is that it creates Kubernetes jobs for each build. By default, once these builds are done, the will exit and not clean themselves up, so if you do a lot of builds then your namespace will get clogged up. There is a way to set TTLs on finished to clean themselves up in newer versions of Kubernetes via the TTLAfterFinished flag but this functionality isn’t default in Kubernetes yet and is a little bit out of the scope of this post.
Running Drone with the agent configuration
The agent uses the sidecar pattern to run a Docker in Docker (dind) container to connect to the Docker socket in order to allow the Drone agent to do its builds.
The main downside of using this approach is that there seems to be an issue (sometimes) where the Drone components can’t talk to the Docker socket, you can find a better description of this problem and more details here. The problem seems to be a race condition where the docker socket is not being able to be mounted before the agent comes up, but I still haven’t totally solved the problem there yet. The advice for getting around this is to run the agent on a dedicated stand alone host to avoid race conditions and other pitfalls.
That being said, if you still want to use this method you will need to add an additional deployment to the config for the drone agent. If you use the agent you can disregard the above Kubernetes environment variable configurations and instead set appropriate environment variables in the agent. Below is the working snipped I used for deploying the agent to my test cluster.
I have gotten the agent to work, I just haven’t had very much success getting it working consistently. I would avoid using this method unless you have to or as mentioned above, get a dedicated host for running builds on.
Testing it out
After wiring everything up, the rest is easy. Add a file called .drone.yml to a repository that you would like to automate builds for. You can find out more about the various capabilities here.
For my use case I wanted to tell drone to build and publish an arm64 based Docker image whenever a change to master occurs. You can look at my drone configuration here to get a better idea of the multiarch options as well as authenticating to a Docker registry.
After adding the .drone.yml to your repo and triggering a build you should see something similar in your local Drone instance.
If everything worked correctly and is green then you should be good to go. Obviously there is a lot of overhead that Kubernetes brings but the actual Drone setup if really straight forward. Just set stuff up on the Github side, translate it into Kubernetes configurations and add some other Drone specific config options and you should have a nice CI/CD environment ready to go.
Conclusion
I am very excited to see Drone grow and mature and use it more in the future. It is simple yet flexible and it fits nicely into the new paradigm of using containers for everything.
The new 1.0 YAML syntax is really nice as well, as it basically maps to the conventions that Kubernetes has chosen, so if you are familiar with that syntax you should feel at home. You can check out the available plugins here, which cover about 90% of the use cases you would see in the wild.
One downside is that YAML syntax errors are really hard to debug, and there isn’t much in the way of output to figure out where your problems are. The best approach I have found is to run the .drone.yml file through the Drone CLI lint/fmt tool before committing and building.
The Drone CLI tool is really powerful and could probably be its own post. There are some links in the references that show off some of its other features.
References
There are a few cool Drone resources I would definitely recommend checking out if you are interested running Drone in your own environment. The docs reference is a great place to start and is great for finding information about how to configure and tweak various Drone settings.
I also definitely recommend checking out the jsonnet extension docs, which can be used to help improve automation workflows. The second link show an good example of how it works and the third link shows some practical applications of using jsonnet to help manage complicated CI pipelines.