The Grumpy Troll

Ramblings of a grumpy troll.

boot2docker xhyve DNS

Using macOS with Docker can be “interesting”. When I got started, I followed the useful advice at https://pilsniak.com/how-to-install-docker-on-mac-os-using-brew/. This approach appealed to me, especially the use of xhyve. Because sometimes I just make life difficult for myself.

Thus my initial setup was:

brew install docker docker-machine xhyve docker-machine-driver-xhyve
f=/usr/local/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve
sudo chown root:wheel $f; sudo chmod u+s $f
  # because yay, more setuid root binaries; it's written in Go, which is
  # something at least.

docker-machine create default --driver xhyve --xhyve-experimental-nfs-share
eval "$(docker-machine env default)"

tmutil addexclusion $HOME/.docker ; sudo tmutil addexclusion -p $HOME/.docker

(Warning: that docker-machine create line will fail for releases of the xhyve driver after v0.3.3; you’ll need to add an area to be shared to the end of the command-line; use /Users to match previous behavior.)

Excluding the Docker VM area from regular backups is a useful addition. Note that the usual docker-machine eval guidance from the maintainers is wrong: it will result in exporting an empty environment variable named export. Wrap the command-substitution in double-quotes to preserve newlines. This is why decent tools emitting export commands for eval will insert semi-colons at the end of each line, and avoid comments: to make life easier for sloppy command-line usage and missing quotes.

This gives us a VM running the fairly minimal OS named boot2docker, which runs Docker and exposes that to Docker on the host macOS via some environment variables pointing the client tool at the VM instead of a local socket.

The biggest problem I had was that my DNS server as seen by the VM kept getting reset to be the IP of my laptop on that virtual network, and my laptop’s DNS resolver was not set to be exposed over the network. I’m reluctant to do so, because I do attend conferences and use untrusted WiFi, but exposing the DNS server to only also listen to an arbitrary list of dynamically created virtual networks, but not to the WiFi, is a fragile pain I don’t want. So the laptop is not itself a suitable DNS server for the VM running boot2docker.

Alas, the docker-machine-driver-xhyve driver’s DHCP server simply doesn’t set a DNS field and ignores whatever’s in the config elsewhere, so the VMs end up with the IP address of the server being used as the DNS server IP too. Always.

In addition, the latest tagged version of that driver, 0.3.3, predates a fix from August wherein that driver was also stomping over bootlocal.sh on each boot, removing the ability of users to work around the first issue. My grateful thanks to Hugues Alary for committing the fix, which I found in git when I went looking to figure out why boot2local.sh changes weren’t persisting.

Really, the xhyve driver is neat, but not yet ready for prime time. In fairness, it does call itself version 0.3.3. That’s pretty clear. So let’s show how to stick with it.

docker-machine stop
brew unlink docker-machine-driver-xhyve
brew install --HEAD docker-machine-driver-xhyve
f=/usr/local/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve
sudo chown root:wheel $f; sudo chmod u+s $f

sudo true
  # to get the password in the cache, because I/O is messed up through docker-machine

docker-machine start
# json: cannot unmarshal bool into Go struct field Driver.Virtio9p of type []string

Ah, JSON config changes. I have Docker images I really don’t want to have to reload. So instead of nuking the machine and recreating, we study commit dff9c59d of the driver, bring up a temporary second machine to have a template to compare against, and then:

C="$HOME/.docker/machine/machines/default/config.json"
jq < "$C" > "$C.new" '
  .Driver.Virtio9p = null |
  .Driver.NFSShares = [.Driver.Virtio9pFolder] |
  .Driver.NFSShare = false |
  .Driver.NFSSharesRoot = "/xhyve-nfsshares" |
  .Driver.Virtio9pRoot = "/xhyve-virtio9p" |
  del(.Driver.Virtio9pFolder)
  '
ls -ld "$C" "$C.new"
chmod 0600 "$C.new"
mv "$C.new" "$C"

That should get us a bootable VM. At this point, we can once more boot boot2docker, and bootlocal.sh won’t be stomped over on each boot (but has cruft left in it).

sudo true
docker-machine start

docker-machine ssh
  sudo -Hi
    cd /var/lib/boot2docker
    cp /usr/share/udhcpc/default.script ./dhcp.script
    vi dhcp.script
        # near top, after other checks but before the 'case':
      [ -f /var/lib/boot2docker/dns-servers ] && dns="$(cat /var/lib/boot2docker/dns-servers)"
    vi dns-servers
        # whatever IPs you want, separated by whitespace; no comments allowed
        # for Google's Public DNS servers:
      8.8.4.4 8.8.8.8
    vi bootlocal.sh
        # everything in this script is now junk, left behind by old versions of
        # the xhyve driver, so bring it down to:
      #!/bin/sh
      sed -i -e 's,/sbin/udhcpc,& -s /var/lib/boot2docker/dhcp.script,' /etc/rc.d/dhcp.sh
      # and to trigger one run now, because we're too late
      (
        DEVICE=eth0
        /sbin/udhcpc -s /var/lib/boot2docker/dhcp.script -b -i $DEVICE -x hostname:$(/bin/hostname) -p /var/run/udhcpc.$DEVICE.pid
      )
  exit; exit

docker-machine stop
sudo true
docker-machine start

And lo, working DNS.

-The Grumpy Troll

Categories: docker macOS boot2docker VMs containers DNS