====== bhyve Container Linux Virtual Machines ======

===== Creating The Container Linux VM =====

Download the raw container linux image from %%CoreOS%%.

<code>
fetch https://stable.release.core-os.net/amd64-usr/current/coreos_production_image.bin.bz2
fetch https://stable.release.core-os.net/amd64-usr/current/coreos_production_image.bin.bz2.DIGESTS
fetch https://stable.release.core-os.net/amd64-usr/current/coreos_production_image.bin.bz2.DIGESTS.asc
fetch https://stable.release.core-os.net/amd64-usr/current/coreos_production_image.bin.bz2.DIGESTS.sig
fetch https://stable.release.core-os.net/amd64-usr/current/coreos_production_image.bin.bz2.sig
</code>

Probably good practice to Verify the image, not described here.

Optionally create a ZFS volumne for the disk drive

<code>
zfs create -V 32G -s -o reservation=none -o volmode=dev zroot/coreos
</code>

bzcat the image into your new drive, if no ZFS then simply un-bzip the file.

<code>
bzcat coreos_production_image.bin.bz2 > /dev/zvol/zroot/coreos
</code>

Optionally snapshot if you are using ZFS

<code>
zfs snap zroot/coreos@vanilla
</code>

At this point you can boot the container linux VM, but it will not be accessible.

<code>
bhyve -c 1 -m 4G -A -H -P \
  -s 0,hostbridge \
  -s 3,ahci-hd,/dev/zvol/zroot/coreos \
  -s 31,lpc \
  -l com1,stdio \
  -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
  coreos
</code>

===== Accessing the Container Linux VM =====

There are a couple of options here for accessing and configuring the VM:

  - Add ssh key(s) to ~core/.ssh/authorized_keys
  - Configure via hosted configuration

Either option can be accomplished by booting from Linux ISO.

==== Boot from Linux ISO ====
This example uses %%SystemRescueCD%%, I've also added a network interface so I can copy things over to the VM:

<code>
bhyve -c 1 -m 4G -A -H -P \
  -s 0,hostbridge \
  -s 2,ahci-cd,/usr/home/amarks/distfiles/systemrescuecd-x86-5.2.2.iso \
  -s 3,ahci-hd,/dev/zvol/zroot/coreos \
  -s 4,virtio-net,tap1 \
  -s 29,fbuf,w=1280,h=1024,tcp=0.0.0.0:5900 \
  -s 30,xhci,tablet \
  -s 31,lpc \
  -l com1,stdio \
  -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
  coreos
</code>

==== Adding ssh key(s) ====
This section is accomplished running from the Linux ISO.

From your live disk, mount the root partition

<code>
mount /dev/sda9 /media
</code>

Copy your public key(s) over to the VM and add them to ~core/.ssh/authorized_keys

Restart the VM without the live disc and ssh using public key authentication.

<code>
ssh -l core <ip address>
</code>

==== Hosting A Configuration ====
This section is accomplished from any host on the same network as the VM.

A web-server is required to host a configuration file. I'm using the same %%FreeBSD%% host system to host the Container Linux configuration.

<code>
pkg install nginx
</code>

Either write the configuration in json, or write it in a yml format then convert it to json using https://github.com/coreos/container-linux-config-transpiler.

YML:

<code>
passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - "ssh-rsa AAAAB3NzaC1...
</code>

JSON:

<code>
{
    "ignition": {
        "config": {},
        "security": {
            "tls": {}
        },
        "timeouts": {},
        "version": "2.2.0"
    },
    "networkd": {},
    "passwd": {
        "users": [
            {
                "name": "core",
                "sshAuthorizedKeys": [
                    "ssh-rsa AAAA..."
                ]
            }
        ]
    },
    "storage": {},
    "systemd": {}
}
</code>

==== Configure Kernel Parameters for hosted configuration ====

This section is accomplished running from the Linux ISO.

To add kernel parameters to instruct container Linux to use the hosted configuration, add a grub.cfg to the OEM partition of the Container Linux VM.

<code>
ls -l /dev/disk/by-partlabel

lrwxrwxrwx 1 root root 10 Aug 19 13:42 BIOS-BOOT -> ../../sda2
lrwxrwxrwx 1 root root 10 Aug 19 13:42 EFI-SYSTEM -> ../../sda1
lrwxrwxrwx 1 root root 10 Aug 19 13:42 OEM -> ../../sda6
lrwxrwxrwx 1 root root 10 Aug 19 13:43 OEM-CONFIG -> ../../sda7
lrwxrwxrwx 1 root root 10 Aug 19 13:42 ROOT -> ../../sda9
lrwxrwxrwx 1 root root 10 Aug 19 13:42 USR-A -> ../../sda3
lrwxrwxrwx 1 root root 10 Aug 19 13:42 USR-B -> ../../sda4

mount /dev/sda6 /media
</code>

/media/grub.cfg

<code>
linux_append="coreos.autologin coreos.first_boot=1 coreos.config.url=http://<IP Address>/config.json"
</code>

Container Linux specific options are documented here:
https://coreos.com/os/docs/latest/booting-with-pxe.html

With your web-server running and the hosted configuration in place, reboot the VM without the Linux Live ISO.

===== Where to go from here =====

  - Enable dockerd, either manually or via the hosted configuration, and you can use the docker client command on another machine to run containers on this VM.
  - Clone the ZFS disk and make clusters of Container Linux VMs running dockerd, swarm, rkt, flannel, kubernetes, etc.