Multiarch Docker builds using Shippable

September 23, 2018 Josh Reichardt Leave a comment

Recently I have been experimenting with different ways of building multi architecture Docker images. As part of this process I wrote about Docker image manifests and the different ways you can package multi architecture builds into a single Docker image. Packaging the images is only half the problem though. You basically need to create the different Docker images for the different architectures first, before you are able to package them into manifests.

There are several ways to go about building the Docker images for various architectures. In the remainder of this post I will be showing how you can build Docker images natively against arm64 only as well as amd64/arm64 simultaneously using some slick features provided by the folks at Shippable. Having the ability to automate multi architecture builds with CI is really powerful because it avoids having to use other tools or tricks which can complicate the process.

Shippable recently announced integrated support for arm64 builds. The steps for creating these cross platform builds is fairly straight forward and is documented on their website. The only downside to using this method is that currently you must explicitly contact Shippable and requests access to use the arm64 pool of nodes for running jobs, but after that multi arch builds should be available.

For reference, here is the full shippable.yml file I used to test out the various types of builds and their options.

Arm64 only builds

After enabling the shippable_shared_aarch64 node pool (from the instruction above) you should have access to arm64 builds, just add the following block to your shippable.yml file.

runtime:
  nodePool: shippable_shared_aarch64

The only other change that needs to be made is to point the shippable.yaml file at the newly added node pool and you should be ready to build on arm64. You can use the default “managed” build type in Shippable to create builds.

Below I have a very simple example shippable.yml file for building a Dockerfile and pushing its image to my Dockerhub account. The shippable.yml file for this build lives in the GitHub repo I configured Shippable to track.

language: none

runtime:
  nodePool:
    - shippable_shared_aarch64
    - default_node_pool

build:

  ci:
    - sed -i 's|registry.fedoraproject.org/||' Dockerfile.fedora-28
    - docker build -t local/freeipa-server -f Dockerfile.fedora-28 .
    - tests/run-master-and-replica.sh local/freeipa-server

  post_ci:
    - docker tag local/freeipa-server jmreicha/freeipa-server:test
    - docker push jmreicha/freeipa-server:test

integrations:
  hub:
    - integrationName: dockerhub
      type: dockerRegistryLogin

Once you have a shippable.yml file in a repo that you would like to track and also have things set up on the Shippable side, then every time a commit/merge happens on the master branch (or whatever branch you set up Shippable to track) an arm64 Docker image gets built and pushed to the Dockerhub.

Docs for settings up this CI style job can be found here. There are many other configuration settings available to tune so I would encourage you to read the docs and also play around with the various options.

Parallel arm64 and amd64 builds

The approach for doing the simultaneous parallel builds is a little bit different and adds a little bit more complexity, but I think is worth it for the ability to automate cross platform builds. There are a few things to note about the below configuration. You can use templates in either style job. Also, notice the use of the shipctl command. This tool basically allows you to mimic some of the other functionality that exists in the default runCI jobs, including the ability to login to Docker registries via shell commands and manage other tricky parts of the build pipeline, like moving into the correct directory to build from.

Most of the rest of the config is pretty straight forward. The top level jobs directive lets you create multiple different jobs, which in turn allows you to set the runtime to use different node pools, which is how we build against amd64 and arm64. Jobs also allow for setting different environment variables among other things. The full docs for jobs shows all of the various capabilities of these jobs.

templates: &build-test-push
  - export HUB_USERNAME=$(shipctl get_integration_field "dockerhub" "username")
  - export HUB_PASSWORD=$(shipctl get_integration_field "dockerhub" "password")
  - docker login --username $HUB_USERNAME --password $HUB_PASSWORD
  - cd $(shipctl get_resource_state "freeipa-container-gitRepo")
  - sed -i 's|registry.fedoraproject.org/||' Dockerfile.fedora-27
  - sed -i 's/^# debug:\s*//' Dockerfile.fedora-27
  - docker build -t local/freeipa-server -f Dockerfile.fedora-27 .
  - tests/run-master-and-replica.sh local/freeipa-server
  - docker tag local/freeipa-server jmreicha/freeipa-server:$arch
  - docker push jmreicha/freeipa-server:$arch

resources:
    - name: freeipa-container-gitRepo
      type: gitRepo
      integration: freeipa-container-gitRepo
      versionTemplate:
          sourceName: jmreicha/freeipa-container
          branch: master

jobs:
  - name: build_amd64
    type: runSh
    runtime:
      nodePool: default_node_pool
      container: true
    integrations:
      - dockerhub
    steps:
      - IN: freeipa-container-gitRepo
      - TASK:
          runtime:
            options:
              env:
                - privileged: --privileged
                # Also look at using SHIPPABLE_NODE_ARCHITECTURE env var
                - arch: amd64
          script:
            - *build-test-push

  - name: build_arm64
    type: runSh
    runtime:
      nodePool: shippable_shared_aarch64
      container: true
    integrations:
      - dockerhub
    steps:
      - IN: freeipa-container-gitRepo
      - TASK:
          runtime:
            options:
              env:
                - privileged: --privileged
                - arch: arm64
          script:
            - *build-test-push

As you can see, there is a lot more manual configuration going on here than the first job.

I decided to use the top level templates directive to basically DRY the configuration so that it can be reused. I am also setting environment variables per job to ensure the correct architecture gets built and pushed for the various platforms. Otherwise the configuration is mostly straight forward. The confusion with these types of jobs if you haven’t set them up before mostly comes from figuring out where things get configured in the Shippable UI.

Conclusion

I must admit, Shippable is really easy to get started with, has good support and has good documentation. I am definitely a fan and will recommend and use their products whenever I get a chance. If you are familiar with Travis then using Shippable is easy. Shippable even supports the use of Travis compatible environment variables, which makes porting over Travis configs really easy. I hope to see more platforms and architectures supported in the future but for now arm64 is a great start.

There are some downside to using the parallel builds for multi architecture builds. Namely there is more overhead in setting up the job initially. With the runSh (and other unmanaged jobs) you don’t really have access to some of the top level yml declarations that come with managed jobs, so you will need to spend more time figuring out how to wire up the logic manually using shell commands and the shipctl tool as depicted in my above example. This ends up being more flexible in the long run but also harder to understand and get working to begin with.

Another downside of the assembly line style jobs like runSh is that they currently can’t leverage all the features that the runCI job can, including the matrix generation (though there is a feature request to add it in the future) and report parsing.

The last downside when setting up unmanaged jobs is trying to figure out how to wire up the different components on the Shippable side of things. For example you don’t just create a runCI job like the first example. You have to first create an integration with the repo that you are configuring so that shippable can make an rSync and serveral runSh jobs to connect with the repo and be able to work correctly.

Overall though, I love both of the runSh and runCI jobs. Both types of jobs lend themselves to being flexible and composable and are very easy to work with. I’d also like to mention that the support has been excellent, which is a big deal to me. The support team was super responsive and helpful trying to sort out my issues. They even opened some PRs on my test repo to fix some issues. And as far as I know, there are no other CI systems currently offering native arm64 builds which I believe will become more important as the arm architecture continues to gain momentum.

Kubernetes CLI Tricks

May 7, 2018May 13, 2018 Josh Reichardt 1 Comment

Kubernetes is complicated, as you’ve probably already discovered if you’ve used Kubernetes before. Likewise, the Kubectl command line tool can pretty much do anything but can feel cumbersome, clunky and generally overwhelming for those that are new to the Kubernetes ecosystem. In this post I want to take some time to describe a few of the CLI tools that I have discovered that help ease the pain of working with and managing Kubernetes from the command line.

There are many more tools out there and the list keeps growing, so I will probably revisit this post in the future to add more cool stuff as the community continues to grow and evolve.

Where to find projects?

As a side note, there are a few places to check for tools and projects. The first is the CNCF Cloud Native Landscape. This site aims to keep track of all the various different projects in the Cloud/Kubernetes world. An entire post could be written about all of the features and and filters but at the highest level it is useful for exploring and discovering all the different evolving projects. Make sure to check out the filtering capabilities.

The other project I have found to be extremely useful for finding different projects is the awesome-kubernetes repo on Github. I found a number of tools mentioned in this post because of the awesome-kubernetes project. There is some overlap between the Cloud Native Landscape and awesome-kubernetes but they mostly compliment each other very nicely. For example, awesome-kubernetes has a lot more resources for working with Kubernetes and a lot of the smalller projects and utilities that haven’t made it into the Cloud Native Landscape. Definitely check this project out if you’re looking to explore more of the Kubernetes ecosystem.

Kubectl tricks

These are various little tidbits that I have found to help boost my productivity from the CLI.

Tab completion – The first thing you will probably want to get working when starting. There are just too many options to memorize and tab completion provides a nice way to look through all of the various commands when learning how Kubernetes works. To install (on OS X) run the following command.

brew install bash-completion

In zsh, adding the completion is as simple as running source <(kubectl completion bash). The same behavior can be accomplished in zsh using source <(kubectl completion zsh).

Aliases and shortcuts – One distinct flavor of Kubernetes is how cumbersome the CLI can be. If you use Zsh and something like oh-my-zsh, there is a default set of aliases that work pretty well, which you can find here. There are a many posts about aliases out there already so I won’t go into too much detail about them. I will say though that aliasing k to kubectl is one of the best time savers I have found so far. Just add the following snippet to your bash/zsh profile for maximum glory.

alias k=kubectl

kubectl –export – This is a nice hidden feature that basically allows users to switch Kubernetes from imperative (create) to declarative (apply). The --export flag will basically take an existing object and strip out unwanted/unneeded metadata like statuses and timestamps and present a clear version of what’s running, which can then be exported to a file and applied to the cluster. The biggest advantage of using declarative configs is the ability to mange and maintain them in git repos.

kubectl top – In newer versions, there is the top command, which gives a high level overview of CPU and memory utilization in the cluster. Utilization can be filtered at the node level as well as the pod level to give a very quick and dirty view into potential bottlenecks in the cluster. In older versions, Heapster needs to be installed for this functionaliity to work correctly, and in newer versions needs metrics-server to be running.

kubectl explain – This is a utility built in to Kubectl that basically provides a man page for what each Kubernetes resource does. It is a simple way to explore Kubernetes without leaving the terminal

kubectx/kubens

This is an amazing little utility for quickly moving between Kubernetes contexts and namespaces. Once you start working with multiple different Kubernetes clusters, you notice how cumbersome it is to switch between environments and namespaces. Kubectx solves this problem by providing a quick and easy way to see what environments and namespaces a user is currently in and also quickly switch between them. I haven’t had any issues with this tool and it is quickly becoming one of my favorites.

stern

Dealing with log output using Kubectl is a bit of a chore. Stern (and similarly kail) offer a much nicer user experience when dealing with logs. These tools allow users the ability to do things like show logs for multiple containers in pod, use regex matching to tail logs for specific containers, give nice colored output for distinguishing between logs, filter logs by namespaces and a bunch of other nice features.

Obviously for a full setup, using an aggregated/centralized logging solution with something like Fluenctd or Logstash would be more ideal, but for examining logs in a pinch, these tools do a great job and are among my favorites. As an added bonus, I don’t have to copy/paste log names any more.

yq is a nice little command line tool for parsing yaml files, which works in a similar way to the venerable jq. Parsing, reading, updating yaml can sometimes be tricky and this tool is a great and lightweight way to manipulate configurations. This tool is especially useful for things like CI/CD where a tag or version might change that is nested deep inside yaml.

There is also the lesser known jsonpath option that allows you to interact with the json version of a Kubernetes object, baked into kubectl. This feature is definitely less powerful than jq/yq but works well when you don’t want to overcomplicate things. Below you can see we can use it to quickly grab the name of an object.

kubectl get pods -o=jsonpath='{.items[0].metadata.name}'

Working with yaml and json for configuration in general seems to be an emerging pattern for almost all of the new projects. It is definitely worth learning a few tools like yq and jq to get better at parsing and manipulating data using these tools.

ksonnet / jsonnet

Similar to the above, ksonnet and jsonnet are basically templating tools for working with Kubernetes and json objects. These two tools work nicely for managing Kubernetes manifests and make a great fit for automating deployments, etc. with CI/CD.

ksonnet and jsonnet are gaining popularity because of their ease of use and simplicity compared to a tool like Helm, which also does templating but needs a system level permission pod running in the Kubernetes cluster. Jsonnet is all client side, which removes the added attack vector but still provides users with a lot of flexibility for creating and managing configs that a templating language provides.

More random Kubernetes tricks

Since 1.10, kubectl has the ability to port forward to resource name rather than just a pod. So instead of looking up pods that are running and connecting to one all the time, you can just grab the service name or deployment and just port forward to it.

port-forward TYPE/NAME [LOCAL_PORT:]REMOTE_PORT
k port-forward deployment/mydeployment 5000:6000

New in 1.11, which will be dropping soonish, there is a top level command called api-resource, which allows users to view and interact with API objects. This will be a nice troubleshooting tool to have if for example you are wanting to see what kinds of objects are in a namespace. The following command will show you these objects.

k api-resources --verbs=list --namespace -o name | xargs -n 1 kubectl get -o name -n foo

Another handy trick is the ability to grab a base64 string and decode it on the fly. This is useful when you are working with secrets and need to quickly look at what’s in the secret. You can adapt the following command to accomplish this (make sure you have jq installed).

k get secret my-secret --namespace default -o json | jq -r '.data | .["secret-field"]' | base64 --decode

Just replace .["secret-field"] to use your own field.

UPDATE: I just recently discovered a simple command line tool for decoding base64 on the fly called Kubernetes Secret Decode (ksd for short). This tool looks for base64 and renders it out for you automatically so you don’t have to worry about screwing around with jq and base64 to extract data out when you want to look at a secret.

k get secret my-secret --namespace default -o json | ksd

That command is much cleaner and easier to use. This utility is a Go app and there are binaries for it on the releases page, just download it and put it in your path and you are good to go.

Conclusion

The Kubernetes ecosystem is a vast world, and it only continues to grow and evolve. There are many more kubectl use cases and community to tools that I haven’t discovered yet. Feel free to let me know any other kubectl tricks you know of, and I will update them here.

I would love to grow this list over time as I get more acquainted with Kubernetes and its different tools.

Mount a volume using Ignition and Terraform

April 1, 2018 Josh Reichardt Leave a comment

Sometimes when provisioning a server you may want to configure and provision storage as part of the bootstrapping and booting process. For example, the other day I ran into an issue where I needed to define a disk, partition it, mount it to a specified location and then create a few directories in it. It turned out to be surprisingly not straight forward to provision this storage and I learned quite a few things that I thought were worth sharing.

I’d just like to mention that ignition works like magic. If you aren’t familiar, Ignition is basically a tool to help provision and configure servers, very similar to cloud-config except by default Ignition only runs once, on first boot. The magic of Ignition is that it injects itself into initramfs before the OS ever eve boots and manipulating the system. Ignition can be read in from remote URL so that it can easily be provisioned in bare metal infrastructures. There were several pieces to this puzzle.

The first was getting down all of the various ignition configuration components in Terraform. Nothing was particularly complicated, there was just a lot of trial and error to get everything working. Terraform has some really nice documentation for working with Ignition configurations, I’d recommend starting there and just playing around to figure out some of the various bits and pieces of configuration that Ignition can do. There is some documentation on Ignition troubleshooting as well which I found to be helpful when things weren’t working correctly.

Below each portion of the Ignition configuration gets declared inside of a “ignition_config” block. The Ignition configuration then points towards each invidual component that we want Ignition to configure. e.g. systemd, filesystem, directories, etc.

data "ignition_config" "staging_rancher_host_stateful" {
  systemd = [
     "${data.ignition_systemd_unit.mount_data.id}",
  ]

  filesystems = [
    "${data.ignition_filesystem.data_fs.id}",
  ]

  directories = [
    "${data.ignition_directory.data_dir.id}",
  ]

  disks = [
    "${data.ignition_disk.data_disk.id}",
  ]
}

This part of the setup is pretty straight forward. Create a data block with the needed ignition configuration to mount the disk to the correct location, format the device if it hasn’t already been formatted and create the desired directory and then create the Systemd unit to configure the mount point for the OS. Here’s what each of the data blocks might look like.

data "ignition_filesystem" "data_fs" {
   name = "data"

  mount {
    device = "/dev/xvdb1"
    format = "ext4"
  }
}

data "ignition_directory" "data_dir" {
  filesystem = "data"
  path = "/data"
  uid = 500
  gid = 500
}

data "ignition_disk" "data_disk" {
  device = "/dev/xvdb"

  partition {
    number = 1
    start = 0
    size = 0
  }
}

Next, create the Systemd unit.

data "ignition_systemd_unit" "mount_data" {
  content = "${file("./data.mount")}"
  name = "data.mount"
}

Another challenge was getting the Systemd unit to mount the disk correctly. I don’t work with Systemd frequently so initially had some trouble figuring this part out. Basically, Systemd expects the service/unit definition name to EXACTLY match what’s declared inside the “Where” clause of the service definition.

For example, the following configuration needs to be named data.mount because that is what is defined in the service.

[Unit]
Description=Mount /data
Before=local-fs.target

[Mount]
What=/dev/xvdb1
Where=/data
Type=ext4

[Install]
WantedBy=local-fs.target

After all the kinks have been worked out of the Systemd unit(s) and other above Terraform Ignition configuration you should be able to deploy this and have Ignition provision disks for you automatically when the OS comes up. This can be extended as much as needed for getting initial disks set up correctly and is a huge step in automating your infrastructure in a nice repeatable way.

There is currently an open issue with Ignition currently where it breaks when attempting to re-provision a previously configured disk on a new machine. Basically the Ignition process chokes because it sees the device has already been partitioned and formatted and can’t do it again. I ran into this scenario where I was trying to create a basically floating persistent data EBS volume that gets attached to servers in an autoscaling group and wanted to allow the volume to be able to move around freely if the server gets killed off.

Bash tricks

December 22, 2017February 18, 2018 Josh Reichardt Leave a comment

Update 2/18/18 – add some handy alt shortcuts

Bash is great. As I have discovered over the years, Bash contains many different layers, like a good movie or a fine wine. It is fun to explore and expose these different layers and find uses for them. As my experience level has increased, I have (slowly) uncovered a number of these features of Bash that make life easier and worked to incorporate them in different ways into my own workflows and use them within my own style.

The great thing about fine arts, Bash included, is that there are so many nuances and for Bash, a huge number of features and uses, which makes the learning process that much more fun.

It does take a lot of time and practice to get used to the syntax and to become effective with these shortcuts. I use this page as a reference whenever I think of something that sounds like it would be useful and could save time in a script or a command. At first, it may take more time to look up how to use these shortcuts, but eventually, with practice and drilling will become second nature and become real time savers.

Shell shortcuts

Navigating the Bash shell is easy to do. But it takes time to learn how to do well. Below are a number of shortcuts that make the navigation process much more efficient. I use nearly all of the shortcuts daily (except Ctrl + t and Ctrl + xx, which I only recently discovered). In a similar vein, I wrote a separate post long ago about setting up CLI shortcuts on iterm that can further augment the capabilities of the CLI.

This is a nice reference with more examples and features

Ctrl + a => Return to the start of the command you’re typing
Ctrl + e => Go to the end of the command you’re typing
Ctrl + u => Cut everything before the cursor to a special clipboard
Ctrl + k => Cut everything after the cursor to a special clipboard
Ctrl + y => Paste from the special clipboard that Ctrl + u and Ctrl + k save their data to
Ctrl + t => Swap the two characters before the cursor (you can actually use this to transport a character from the left to the right, try it!)
Ctrl + w => Delete the word / argument left of the cursor
Ctrl + l => Clear the screen
Ctrl + _ => Undo previous key press
Ctrl + xx => Toggle between current position and the start of the line

There are some nice Alt key shortcuts in Linux as well. You can map the alt key in OSX pretty easily to unlock these shortcuts.

Alt + l => Uncapitalize the next word that the cursor is under (If the cursor is in the middle of the the word it will capitalize the last half of the word).
Alt + u => Capitalize the word that the cursor is under
Alt + t => Swap words or arguments that the cursor is under with the previous
Alt + . => Paste the last word of the previous command
Alt + b => Move backward one word
Alt + f => Move forward one word
Alt + r => Undo any changes that have been done to the current command

Argument tricks

Argument tricks can help to grow the navigation capabilities that Bash shortcuts provide and can even further speed up your effectiveness in the terminal. Below is a list of special arguments that can be passed to any command that can be expanded into various commands.

Repeating

!! => Repeat the previous (full) command
!foo => Repeat the most recent command that starts with ‘foo‘ (e.g. !ls)
!^ => Repeat the first argument of the previous command
!$ => Repeat the last argument of the previous command
!* => Repeat all arguments of last command
!:<number> => Repeat a specifically positioned argument
!:1-2 => Repeat a range of arguments

Printing

!$:p => Print out the word that !$ would substitute
!*:p => Print out the previous command except for the last word
!foo:p =>Print out the command that !foo would run

Special parameters

When writing scripts , there are a number of special parameters you can feed into the shell. This can be convenient for doing lots of different things in scripts. Part of the fun of writing scripts and automating things is discovering creative ways to fit together the various pieces of the puzzle in elegant ways. The “special” parameters listed below can be seen as pieces of the puzzle, and can be very powerful building blocks in your scripts.

Here is a full reference from the Bash documentation

$* => Expand parameters. Expands to a single word for each parameter separated by IFS delimeter – think spaces
$@ => Expand parameters. Each parameter expand to a separate word, enclosed by “” – think arrays
$# => Expand the number of parameters of a command
$? => Expand the exit status of the previous command
$$ => Expand the pid of the shell
$! => Expand the pid of the most recent command
$0 => Expand the name of the shell or script
$_ => Expand the last previous argument

Conclusion

There are some many crevices and cracks of Bash to explore, I keep finding new and interesting things about Bash that lead down new paths and help my skills grow. I hope some of these tricks give you some ideas that can help and improve your own Bash style and workflows in the future.

Templated Nginx configuration with Bash and Docker

September 21, 2017August 29, 2018 Josh Reichardt Leave a comment

Shoutout to @shakefu for his Nginx and Bash wizardry in figuring a lot of this stuff out. I’d like to take credit for this, but he’s the one who got a lot of it working originally.

Sometimes it can be useful to template Nginx files to use environment variables to fine tune and adjust control for various aspects of Nginx. A recent example of this idea that I recently worked on was a scenario where I setup an Nginx proxy with a very bare bones configuration. As part of the project, I wanted a quick and easy way to update some of the major Nginx configurations like the port it uses to listen for traffic, the server name, upstream servers, etc.

It turns out that there is a quick and dirty way to template basic Nginx configurations using Bash, which ended up being really useful so I thought I would share it. There are a few caveats to this method but it is definitely worth the effort if you have a simple setup or a setup that requires some changes periodically. I stuck the configuration into a Dockerfile so that it can be easily be updated and ported around – by using the nginx:alpine image as the base image the total size all said and done is around 16MB. If you’re not interested in the Docker bits, feel free to skip them.

The first part of using this method is to create a simple configuration file that will be used to substitute in some environment variables. Here is a simple template that is useful for changing a few Nginx settings. I called it nginx.tmpl, which will be important for how the template gets rendered later.

events {}

http {
  error_log stderr;
  access_log /dev/stdout;

  upstream upstream_servers {
    server ${UPSTREAM};
  }

  server {
    listen ${LISTEN_PORT};
    server_name ${SERVER_NAME};
    resolver ${RESOLVER};
    set ${ESC}upstream ${UPSTREAM};

    # Allow injecting extra configuration into the server block
    ${SERVER_EXTRA_CONF}

    location / {
       proxy_pass ${ESC}upstream;
    }
  }
}

The configuration is mostly straight forward. We are basically just using this configuration file and inserting a few templated variables denoted by the ${VARIABLE} syntax, which are just environment variables that get inserted into the configuration when it gets bootstrapped. There are a few “tricks” that you may need to use if your configuration starts to get more complicated. The first is the use of the ${ESC} variable. Nginx uses the ‘$’ for its variables, which also is used by the template. The extra ${ESC} basically just gives us a way to escape that $ so that we can use Nginx variables as well as templated variables.

The other interesting thing that we discovered (props to shakefu for this magic) was that you can basically jam arbitrary server block level configurations into an environment variable. We do this with the ${SERVER_EXTRA_CONF} in the above configuration and I will show an example of how to use that environment variable later.

Next, I created a simple Dockerfile that provides some default values for some of the various templated variables. The Dockerfile aslso copies the templated configuration into the image, and does some Bash magic for rendering the template.

FROM nginx:alpine

ENV LISTEN_PORT=8080 \
  SERVER_NAME=_ \
  RESOLVER=8.8.8.8 \
  UPSTREAM=icanhazip.com:80 \
  UPSTREAM_PROTO=http \
  ESC='$'

COPY nginx.tmpl /etc/nginx/nginx.tmpl

CMD /bin/sh -c "envsubst < /etc/nginx/nginx.tmpl > /etc/nginx/nginx.conf && nginx -g 'daemon off;' || cat /etc/nginx/nginx.conf"

There are some things to note. First, not all of the variables in the template need to be declared in the Dockerfile, which means that if the variable isn’t set it will be blank in the rendered template and just won’t do anything. There are some variables that need defaults, so if you ever run across that scenario you can just add them to the Dockerfile and rebuild.

The other interesting thing is how the template gets rendered. There is a tool built into the shell called envsubst that substitutes the values of environment variables into files. In the Dockerfile, this tool gets executed as part of the default command, taking the template as the input and creating the final configuration.

/bin/sh -c "envsubst < /etc/nginx/nginx.tmpl > /etc/nginx/nginx.conf

Nginx gets started in a slightly silly way so that daemon mode can be disabled (we want Nginx running in the foreground) and if that fails, the rendered template gets read to help look for errors in the rendered configuration.

&& nginx -g 'daemon off;' || cat /etc/nginx/nginx.conf"

To quickly test the configuration, you can create a simple docker-compose.yml file with a few of the desired environment variables, like I have below.

version: '3'
services:
  nginx_proxy:
    build:
      context: .
      dockerfile: Dockerfile
    # Only test the configuration
    #command: /bin/sh -c "envsubst < /etc/nginx/nginx.tmpl > /etc/nginx/nginx.conf && cat /etc/nginx/nginx.conf"
    volumes:
      - "./nginx.tmpl:/etc/nginx/nginx.tmpl"
    ports:
      - 80:80
    environment:
    - SERVER_NAME=_
    - LISTEN_PORT=80
    - UPSTREAM=test1.com
    - UPSTREAM_PROTO=https
    # Override the resolver
    - RESOLVER=4.2.2.2
    # The following would add an escape if it isn't in the Dockerfile
    # - ESC=$$

Then you can bring up Nginx server.

docker-compose up

The configuration doesn’t get rendered until the container is run, so to test the configuration only, you could add in a command in the docker-compose file that renders the configuration and then another command that spits out the rendered configuration to make sure it looks right.

If you are interested in adding additional configuration you can use the ${SERVER_EXTRA_CONF} as eluded to above. An example of this extra configuration can be assigned to the environment variable. Below is an arbitrary snippet that allows for connections to do long polling to Nginx, which basically means that Nginx will try to hold the connection open for existing connections for longer.

error_page 420 = @longpoll;
if ($arg_wait = "true") { return 420; }
}
location @longpoll {
# Proxy requests to upstream
proxy_pass $upstream;
# Allow long lived connections
proxy_buffering off;
proxy_read_timeout 900s;
keepalive_timeout 160s;
keepalive_requests 100000;

The above snipped would be a perfectly valid environment variable as far as the container is concerned, it will just look a little bit weird to the eye.

That’s all I’ve got for now. This minimal templated Nginx configuration is handy for testing out simple web servers, especially for proxies and is also nice to port around using Docker.