bhyve Container Linux Virtual Machines

Download the raw container linux image from CoreOS.

fetch https://stable.release.core-os.net/amd64-usr/current/coreos_production_image.bin.bz2
fetch https://stable.release.core-os.net/amd64-usr/current/coreos_production_image.bin.bz2.DIGESTS
fetch https://stable.release.core-os.net/amd64-usr/current/coreos_production_image.bin.bz2.DIGESTS.asc
fetch https://stable.release.core-os.net/amd64-usr/current/coreos_production_image.bin.bz2.DIGESTS.sig
fetch https://stable.release.core-os.net/amd64-usr/current/coreos_production_image.bin.bz2.sig

Probably good practice to Verify the image, not described here.

Optionally create a ZFS volumne for the disk drive

zfs create -V 32G -s -o reservation=none -o volmode=dev zroot/coreos

bzcat the image into your new drive, if no ZFS then simply un-bzip the file.

bzcat coreos_production_image.bin.bz2 > /dev/zvol/zroot/coreos

Optionally snapshot if you are using ZFS

zfs snap zroot/coreos@vanilla

At this point you can boot the container linux VM, but it will not be accessible.

bhyve -c 1 -m 4G -A -H -P \
  -s 0,hostbridge \
  -s 3,ahci-hd,/dev/zvol/zroot/coreos \
  -s 31,lpc \
  -l com1,stdio \
  -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
  coreos

There are a couple of options here for accessing and configuring the VM:

1. Add ssh key(s) to ~core/.ssh/authorized_keys
2. Configure via hosted configuration

For both options you can boot the VM from alternate media to configure.

This example uses SystemRescueCD, I've also added a network interface so I can copy things over to the VM:

bhyve -c 1 -m 4G -A -H -P \
  -s 0,hostbridge \
  -s 2,ahci-cd,/usr/home/amarks/distfiles/systemrescuecd-x86-5.2.2.iso \
  -s 3,ahci-hd,/dev/zvol/zroot/coreos \
  -s 4,virtio-net,tap1 \
  -s 29,fbuf,w=1280,h=1024,tcp=0.0.0.0:5900 \
  -s 30,xhci,tablet \
  -s 31,lpc \
  -l com1,stdio \
  -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
  coreos

From your live disk, mount the root partition

mount /dev/sda9 /media

Copy your public key(s) over to the VM and add them to ~core/.ssh/authorized_keys

Restart the VM without the live disc and ssh using public key authentication.

ssh -l core <ip address>

A web-server is required to host a configuration file. I'm using the same FreeBSD host system to host the Container Linux configuration.

pkg install nginx

Eeither write the configuration in json, or write it in a yml format then convert it to json using https://github.com/coreos/container-linux-config-transpiler.

YML:

passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - "ssh-rsa AAAAB3NzaC1...

JSON:

{
    "ignition": {
        "config": {},
        "security": {
            "tls": {}
        },
        "timeouts": {},
        "version": "2.2.0"
    },
    "networkd": {},
    "passwd": {
        "users": [
            {
                "name": "core",
                "sshAuthorizedKeys": [
                    "ssh-rsa AAAA..."
                ]
            }
        ]
    },
    "storage": {},
    "systemd": {}
}

This part is also done from the Linux Live ISO.

To add kernel parameters to instruct container Linux to use the hosted configuration, add a grub.cfg to the OEM partition of the Container Linux VM.

ls -l /dev/disk/by-partlabel

lrwxrwxrwx 1 root root 10 Aug 19 13:42 BIOS-BOOT -> ../../sda2
lrwxrwxrwx 1 root root 10 Aug 19 13:42 EFI-SYSTEM -> ../../sda1
lrwxrwxrwx 1 root root 10 Aug 19 13:42 OEM -> ../../sda6
lrwxrwxrwx 1 root root 10 Aug 19 13:43 OEM-CONFIG -> ../../sda7
lrwxrwxrwx 1 root root 10 Aug 19 13:42 ROOT -> ../../sda9
lrwxrwxrwx 1 root root 10 Aug 19 13:42 USR-A -> ../../sda3
lrwxrwxrwx 1 root root 10 Aug 19 13:42 USR-B -> ../../sda4

mount /dev/sda6 /media

/media/grub.cfg

linux_append="coreos.autologin coreos.first_boot=1 coreos.config.url=http://<IP Address>/config.json"

Container Linux specific options are documented here: https://coreos.com/os/docs/latest/booting-with-pxe.html

With your web-server running and the hosted configuration in place, reboot the VM without the Linux Live ISO.

Enable dockerd, either manually or via the hosted configuration, and you can use the docker client command on another machine to run containers on this VM.