Getting a usable and productive dev environment working with Docker on OS X is not exactly trivial, although it is getting much better. If you have spent any time working with docker-machine and Docker on OS X you’ve probably run across some type of roadblock to getting your dev environment working.
If you have used docker-machine you are probably familiar with the Virtualbox driver, the driver that ships as by default. Obviously it works out of the box but if you have used Virtualbox for any amount of time you have probably discovered some of its quirks. My biggest gripe thus far with Vbox is that their shared folders technology to sync files between the host and VM is slooooow. In fact, I have written about my own workaround here.
I have run in to some other performance issues using VBox. This write up is a very detailed comparison of the performance between VBox and VMWare. The tl;dr of the post is that that the VMWare hypervisor has better performance. To Oracle’s credit though, many of these performance issues have actually been addressed in the vbox 5.0.0 release. So if you aren’t running on 5.x definitely make the jump. The Docker Toolbox ships the newer release so there is no reason not to upgrade.
Making the jump to VBox 5.x may, and most likely should solve your problems but I have been curious about what other options are out there. Recently, as of July 2015, the xhyve hypervisor project has been available on OS X. xhyve is a port of the byhve project, which aims to bring high performance virtualization with a light footprint to OS X. It is still very young but shows a lot of potential.
Even younger than the xhyve project itself is the xhyve driver for docker-machine. It is so young that it is still not an officially supported driver yet, though it looks like it is well on its way. Definitely keep an eye on the xhyve and docker-machine xhyve projects if you are looking for an alternative to either VBox or VMWare. The xhyve docker-machine driver project has recently closed a ticket to be added to brew so it is much less complicated to get working.
Xhyve installation
I will be going over the bare minimum installation instructions to getting everything working. If you are interested in more of the details on how to get the xhyve driver, I suggest taking a look at this awesome blog post. The post goes in to depth on how to install and use the docker-machine xhyve driver if you are interested in a more in depth look at how to get things working.
Make sure you have brew installed first. You will also need to have brew cask installed. After you have brew installed you should be able to get it from the command line with the following command.
brew tap caskroom/cask
Once you have cask installed you should be able to install the remaining components.
brew update brew install xhyve brew cask install dockertoolbox
This might take a little bit depending on how fast your internet connection is. After you have the toolbox installed, go grab the docker-machine xhyve driver.
brew install docker-machine-driver-xhyve
If you have the dockertoolbox installed already you might some errors in the output. This just means there was a version conflict somewhere. As of docker-machine version 0.5.6_1, support has been added for the xhyve driver.
There is currently a caveat to using this driver where you need to change some permissions. This should hopefully be fixed in the future but is at least something to be aware of.
sudo chown root:wheel $(brew --prefix)/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve sudo chmod u+s $(brew --prefix)/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve
You will also need to clean out your /etc/exports file if you have made changes.
sudo mv /etc/exports{,.backup} && touch /etc/exportsThen create the machine.
docker-machine create --driver xhyve --xhyve-experimental-nfs-share test
If you can interact with the Docker daemon you should be in business.
Benchmark results
The remainder of the post describes the benchmark and performance results of the VBox driver and the xhyve driver. If you are only interested in getting the xhyve driver working then feel free to skim through the benchmarks, but be sure to take a look at the conclusion for the final verdict.
Below are the specs of the OS X machine that was used to run my benchmarks.
- OS X 10.10.5
- Virtualbox 5.0.12
- xhyve 0.2.0
- docker-machine-xyhve 0.2.2
Many of the ideas I used for the benchmark tests were taken from the post linked above. It shows in great detail the methodology that was used to benchmark each of the drivers, which is useful because it gives some really good insight into the tools that were used and how the tests were performed.
Benchmarking on the boot2docker VM is tricky because it is mostly a read only file system and there is no package manger. Therefore I relied on running the benchmarks inside containers, using a few different methodologies for my testing. The first was borrowed from the simple-container-benchmarks project on Dockerhub. This benchmark test gives a good idea of the overall write performance and CPU performance of a container running inside the VM. For network performance I used the iperf3 image located on Dockerhub.
Below are the results of a few random runs for both the VBox driver as well as the xhyve driver. I have left out the specific commands here as they are included in the links to each benchmark. Use the links to each project for specific instructions on how to run the benchmarks yourself if you are interested. The results were interesting because I was expecting the xhyve driver to outperform the VBox driver.
Virtualbox results
container benchmark results (FS write and CPU)
Client mode...
Target: 172.17.0.2
------------------------------
Performance benchmarks
------------------------------
dockerhost: tcp://192.168.99.100:2376
host: 172.17.0.2 a8b790317264
eth0: 172.17.0.2
date: Sat Jan 23 02:24:20 UTC 2016
------------------------------
FS write performance
------------------------------
1073741824 bytes (1.1 GB) copied, 2.39743 s, 448 MB/s
1073741824 bytes (1.1 GB) copied, 2.35377 s, 456 MB/s
1073741824 bytes (1.1 GB) copied, 1.9075 s, 563 MB/s
1073741824 bytes (1.1 GB) copied, 2.37838 s, 451 MB/s
1073741824 bytes (1.1 GB) copied, 2.03373 s, 528 MB/s
1073741824 bytes (1.1 GB) copied, 1.94024 s, 553 MB/s
1073741824 bytes (1.1 GB) copied, 1.99546 s, 538 MB/s
1073741824 bytes (1.1 GB) copied, 2.00287 s, 536 MB/s
1073741824 bytes (1.1 GB) copied, 1.5292 s, 702 MB/s
1073741824 bytes (1.1 GB) copied, 1.92617 s, 557 MB/s
------------------------------
CPU performance
------------------------------
268435456 bytes (268 MB) copied, 22.6775 s, 11.8 MB/s
268435456 bytes (268 MB) copied, 22.1466 s, 12.1 MB/s
268435456 bytes (268 MB) copied, 30.7552 s, 8.7 MB/s
268435456 bytes (268 MB) copied, 22.2861 s, 12.0 MB/s
268435456 bytes (268 MB) copied, 22.5571 s, 11.9 MB/s
268435456 bytes (268 MB) copied, 21.9901 s, 12.2 MB/s
268435456 bytes (268 MB) copied, 21.8232 s, 12.3 MB/s
268435456 bytes (268 MB) copied, 31.3903 s, 8.6 MB/s
268435456 bytes (268 MB) copied, 28.1219 s, 9.5 MB/s
268435456 bytes (268 MB) copied, 31.0172 s, 8.7 MB/s
------------------------------
System info
------------------------------
total used free shared buffers cached
Mem: 1019960 313288 706672 113104 7808 132532
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 2
On-line CPU(s) list: 0,1
Thread(s) per core: 1
Core(s) per socket: 2
Socket(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 58
Stepping: 9
CPU MHz: 2294.770
BogoMIPS: 4589.54
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 3072K
iperf results
Connecting to host 172.17.0.3, port 5201 [ 4] local 172.17.0.4 port 39476 connected to 172.17.0.3 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 2.09 GBytes 17.9 Gbits/sec 2321 1.03 MBytes [ 4] 1.00-2.00 sec 2.46 GBytes 21.1 Gbits/sec 496 980 KBytes [ 4] 2.00-3.00 sec 2.24 GBytes 19.3 Gbits/sec 339 1.77 MBytes [ 4] 3.00-4.00 sec 2.54 GBytes 21.8 Gbits/sec 1355 389 KBytes [ 4] 4.00-5.00 sec 2.10 GBytes 18.0 Gbits/sec 106 495 KBytes [ 4] 5.00-6.00 sec 3.00 GBytes 25.7 Gbits/sec 217 411 KBytes [ 4] 6.00-7.00 sec 2.60 GBytes 22.4 Gbits/sec 440 1.72 MBytes [ 4] 7.00-8.00 sec 2.06 GBytes 17.7 Gbits/sec 0 1.72 MBytes [ 4] 8.00-9.00 sec 2.07 GBytes 17.8 Gbits/sec 0 1.72 MBytes [ 4] 9.00-10.00 sec 2.51 GBytes 21.6 Gbits/sec 876 713 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 23.7 GBytes 20.3 Gbits/sec 6150 sender [ 4] 0.00-10.00 sec 23.7 GBytes 20.3 Gbits/sec receiver iperf Done.
xhyve results
container benchmark results (FS write and CPU)
Client mode...
Target: 172.17.0.2
------------------------------
Performance benchmarks
------------------------------
dockerhost:
host: 172.17.0.2 2c8d9ba61eae
eth0: 172.17.0.2
date: Sat Jan 23 02:08:15 UTC 2016
------------------------------
FS write performance
------------------------------
1073741824 bytes (1.1 GB) copied, 8.24671 s, 130 MB/s
1073741824 bytes (1.1 GB) copied, 5.89179 s, 182 MB/s
1073741824 bytes (1.1 GB) copied, 6.05392 s, 177 MB/s
1073741824 bytes (1.1 GB) copied, 5.37728 s, 200 MB/s
1073741824 bytes (1.1 GB) copied, 4.824 s, 223 MB/s
1073741824 bytes (1.1 GB) copied, 5.90409 s, 182 MB/s
1073741824 bytes (1.1 GB) copied, 5.22375 s, 206 MB/s
1073741824 bytes (1.1 GB) copied, 5.07298 s, 212 MB/s
1073741824 bytes (1.1 GB) copied, 5.89058 s, 182 MB/s
1073741824 bytes (1.1 GB) copied, 4.80828 s, 223 MB/s
------------------------------
CPU performance
------------------------------
268435456 bytes (268 MB) copied, 25.478 s, 10.5 MB/s
268435456 bytes (268 MB) copied, 31.3984 s, 8.5 MB/s
268435456 bytes (268 MB) copied, 24.698 s, 10.9 MB/s
268435456 bytes (268 MB) copied, 31.1973 s, 8.6 MB/s
268435456 bytes (268 MB) copied, 23.3705 s, 11.5 MB/s
268435456 bytes (268 MB) copied, 23.3973 s, 11.5 MB/s
268435456 bytes (268 MB) copied, 23.7405 s, 11.3 MB/s
268435456 bytes (268 MB) copied, 23.6118 s, 11.4 MB/s
268435456 bytes (268 MB) copied, 23.5606 s, 11.4 MB/s
268435456 bytes (268 MB) copied, 24.3341 s, 11.0 MB/s
------------------------------
System info
------------------------------
total used free shared buffers cached
Mem: 1020028 291632 728396 70356 6420 89824
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 2
On-line CPU(s) list: 0,1
Thread(s) per core: 1
Core(s) per socket: 1
Socket(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 58
Stepping: 9
CPU MHz: 2294.450
BogoMIPS: 4607.99
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 3072K
iperf results
Connecting to host 172.17.0.2, port 5201 [ 4] local 172.17.0.3 port 49244 connected to 172.17.0.2 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 2.29 GBytes 19.7 Gbits/sec 0 1.90 MBytes [ 4] 1.00-2.00 sec 2.84 GBytes 24.4 Gbits/sec 567 953 KBytes [ 4] 2.00-3.00 sec 2.16 GBytes 18.6 Gbits/sec 327 667 KBytes [ 4] 3.00-4.00 sec 2.32 GBytes 19.9 Gbits/sec 166 1.52 MBytes [ 4] 4.00-5.00 sec 2.63 GBytes 22.6 Gbits/sec 565 769 KBytes [ 4] 5.00-6.00 sec 2.71 GBytes 23.3 Gbits/sec 608 583 KBytes [ 4] 6.00-7.00 sec 2.67 GBytes 22.9 Gbits/sec 217 1.40 MBytes [ 4] 7.00-8.00 sec 2.98 GBytes 25.6 Gbits/sec 782 498 KBytes [ 4] 8.00-9.00 sec 2.80 GBytes 24.0 Gbits/sec 359 1.01 MBytes [ 4] 9.00-10.00 sec 2.43 GBytes 20.9 Gbits/sec 883 467 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 25.8 GBytes 22.2 Gbits/sec 4474 sender [ 4] 0.00-10.00 sec 25.8 GBytes 22.2 Gbits/sec receiver iperf Done.
Conclusion
I was on on the fence about VBox performance but the proof is in the pudding here with the test results. The VBox driver had significantly better FS write performance (almost 2x). CPU performance was about equal overall, and network throughput was also very similar. I suspect CPU performance would be in favor of xhyve if these tests were run using VBox 4.x. Regardless, equal CPU performance, similar network throughput and significantly better FS writes tip the scale in favor of the VBox driver.
As frustrating as it can be at times to use Vbox, many of its past performance issues have been fixed as of the v5.0 release. The shared folder issue still exists but is largely taken care of by the great, easy to use tools that the Docker community has written, docker-machine-nfs is my favorite.
Surprisingly, or maybe not THAT surprisingly, xhyve actually performs worse that Virtualbox at this point. xyhve itself is still a super young project and the docker-machine xhyve driver is still super young so there is definitely some room for growth. That said, it was very straightforward to get xhyve and the docker-drive installed and configured, so I believe it is just a matter of time before the xhyve driver matures to a point where it can replace other drivers. One down side of the xhyve driver is that it also suffers from the host to VM shared folder issue and the current best work around is to use the –nfs-share flag that the xhyve docker-machine driver offers.
I will definitely have my eye on the xhyve project moving forward because it looks to be a great alternative to other virtualization technologies for OS X once it reaches a point of maturity. For now, VBox works more than sufficiently, has been around for a long time, is pretty much ubiquitous across platforms and the developers have shown that they are still actively working on improving the project with the recent 5.0 release.