This is an old revision of the document!
AndrewMarks
This is a work in progress
bhyve Container Linux Virtual Machines
Creating a single Container Linux VM
Download the raw container linux image from CoreOS.
fetch https://stable.release.core-os.net/amd64-usr/current/coreos_production_image.bin.bz2 fetch https://stable.release.core-os.net/amd64-usr/current/coreos_production_image.bin.bz2.DIGESTS fetch https://stable.release.core-os.net/amd64-usr/current/coreos_production_image.bin.bz2.DIGESTS.asc fetch https://stable.release.core-os.net/amd64-usr/current/coreos_production_image.bin.bz2.DIGESTS.sig fetch https://stable.release.core-os.net/amd64-usr/current/coreos_production_image.bin.bz2.sig
Verify the image
Optionally create a ZFS volumne for the disk drive
zfs create -V 32G -s -o refreservation=none -o volmode=dev zroot/coreos
bzcat the image into your new drive (this can be a file if not using ZFS)
bzcat coreos_production_image.bin.bz2 > /dev/zvol/zroot/coreos
Optionally snapshot if you are using ZFS
zfs snap zroot/coreos@vanilla
At this point you can boot the container linux VM, but it will not be “provisioned”, will not be accessible, and is of little use. Read on for how to provision an instance of this VM.
bhyve -c 1 -m 4G -A -H -P \ -s 0,hostbridge \ -s 3,ahci-hd,/dev/zvol/zroot/coreos \ -s 31,lpc \ -l com1,stdio \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \ coreos
Provisioning
There are a couple of options here for accessing and configuring the VM: 1. You can manually boot of a linux ISO and add your ssh key to ~core/.ssh/authorized_keys 1. You can provision the VM via a network-hosted config
Network Hosted Config
I'm using nginx to host my configuration. You can possibly use FTP if you don't want to install any ports or packages, I have not tried it.
pkg install nginx
The old yml based cloud config is deprecated, you know how to “transpile” your yml to json using the container-linux-config-transpiler. It builds fine with go and gmake installed. Perhaps a port is in order.
https://github.com/coreos/container-linux-config-transpiler
Write your provisioning configuration in yml then convert to json and host on some network.
/usr/local/www/nginx/coreos.yml:
ct < /usr/local/www/nginx/coreos.yml > /usr/local/www/nginx/coreos.ign
To point your VM at your cloud config, hit 'e' immediately after starting the VM, then add the following paramters to the last line of the boot prompt.
coreos.first_boot=1 coreos.config.url=http://<host_or_ip>/coreos.ign
then ctl-x to boot with these paramters
Boot from ISO
bhyve -c 1 -m 4G -A -H -P \ -s 0,hostbridge \ -s 2,ahci-cd,/usr/home/amarks/distfiles/systemrescuecd-x86-5.2.2.iso \ -s 3,ahci-hd,/dev/zvol/zrt/core01 \ -s 4,virtio-net,tap1 \ -s 29,fbuf,w=1280,h=1024,tcp=0.0.0.0:5900 \ -s 30,xhci,tablet \ -s 31,lpc \ -l com1,stdio \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \ core01
Configure Kernel Boot Options for "Cloud Provisioning"
mount /dev/sda1
tar xf /media/coreos/grub/grub.cfg.tar
Important parts of grub.cfg
# Anything else the OEM adds should use this variable. set linux_append=""
# Search for the OEM partition, load additional configuration if found.
if [ "$secure_boot" = "0" ]; then
search --no-floppy --set oem --part-label OEM --hint "$root"
if [ -n "$oem" -a -f "($oem)/grub.cfg" ]; then
source "($oem)/grub.cfg"
fi
fi
# Assemble the options applicable to all the kernels below set linux_cmdline="rootflags=rw mount.usrflags=ro consoleblank=0 $linux_root $linux_console $first_boot $randomize_disk_guid $e xtra_options $oem $linux_append"
This shows that linux_append should be used to add arguments to the Linux command line, and if there is a grub.cfg at the root of the OEM partition, it will be sourced, so that is where we can set linux_append.
Find the OEM partition using /dev/disk/by-partlabel
ls -l /dev/disk/by-partlabel
lrwxrwxrwx 1 root root 10 Aug 19 13:42 BIOS-BOOT -> ../../sda2 lrwxrwxrwx 1 root root 10 Aug 19 13:42 EFI-SYSTEM -> ../../sda1 lrwxrwxrwx 1 root root 10 Aug 19 13:42 OEM -> ../../sda6 lrwxrwxrwx 1 root root 10 Aug 19 13:43 OEM-CONFIG -> ../../sda7 lrwxrwxrwx 1 root root 10 Aug 19 13:42 ROOT -> ../../sda9 lrwxrwxrwx 1 root root 10 Aug 19 13:42 USR-A -> ../../sda3 lrwxrwxrwx 1 root root 10 Aug 19 13:42 USR-B -> ../../sda4
/dev/sda6 is labeled OEM
mount /dev/sda6 /media vi /media/grub.conf
Container Linux specific options are documented here: https://coreos.com/os/docs/latest/booting-with-pxe.html
linux_append="coreos.autologin coreos.first_boot=1 coreos.config.url=http://<IP Address>/config.json"
Shutdown
shutdown -h now