===== FreeBSD Containers and Orchestration =====

%%FreeBSD%% introduced its container, [[https://en.wikipedia.org/wiki/OS-level_virtualization|OS-level virtualization]] primitive in 1999 in the form of a security-oriented isolation framework and subsystem called [[Jails]]. Similar to [[https://en.wikipedia.org/wiki/OpenVZ|OpenVZ Containers]] in 2005, Solaris Zones, LXC, [[Docker]] and other implementations, %%FreeBSD%% Jails allow isolation of applications or entire stacks with their own processes, filesystems and users, whilst using the same host operating system kernel.

===== Container Tools: Base System =====

Out of the box, %%FreeBSD%% provides:

  * [[https://man.freebsd.org/jail(8)|jail command line utility]]
  * [[https://man.freebsd.org/jail.conf(5)|jail.conf: configuration file for jails]]
  * [[https://man.freebsd.org/rc.conf(5)|rc.conf integration for jail management]]

===== Container Tools: Third-Party =====

These third-party tools aim to simplify and speed up the process of creating and managing %%FreeBSD%% jail-based containers on individual hosts, and many include additional support and integration for %%FreeBSD%% features such as [[bhyve]] virtualization, [[ZFS]], Virtual Networks ([[Jails/VNET|VNET]]), Templating, Import/Export among others.

==== AppJail ====

Jail framework written in C and posix shell to create, deploy and maintain jail-based containers.

  * [[https://github.com/DtxdF/AppJail|AppJail GitHub Repository]]
  * [[https://github.com/AppJail-makejails|Makejails]]
  * [[https://github.com/DtxdF/AppJail/wiki|Wiki]]
  * [[https://freshports.org/sysutils/appjail|AppJail FreeBSD Port/Package: sysutils/appjail]]
  * [[https://freshports.org/sysutils/appjail-devel|AppJail (devel) FreeBSD Port/Package: sysutils/appjail-devel]]

Supports:

  * Supervisor (healthcheckers)
  * Parallel startup (Healthcheckers, jails & NAT).
  * ZFS support.
  * RACCT/RCTL support.
  * NAT support.
  * Port forwarding.
  * IPv4 and IPv6 support.
  * DHCP and SLAAC support.
  * Virtual networks.
  * Bridge support.
  * VNET support.
  * Text file to make jails: Makejail.
  * Netgraph support.
  * LinuxJails support.
  * Supports thin and thick jails.
  * TinyJails - Experimental feature to create a very stripped down jail that is very useful to distribute.
  * Startup order control.
  * Jail dependency support.
  * InitScript for interactive use of jails.
  * Commands to import/export jails (ZFS & Tarballs).
  * Table interface to easily integrate AppJail with scripts.
  * Images.
  * Dynamic DEVFS ruleset management.
  * Orchestration with [[https://freshports.org/sysutils/py-director|AppJail Director]].

==== podman, buildah ====

A %%FreeBSD%% port of the [[https://github.com/containers]] stack. Install sysutils/podman-suite from %%FreeBSD%% Ports/Packages.

  * Suitable for testing and evaluation.
  * Container storage using the zfs and vfs storage drivers. ZFS is strongly preferred since its use of snapshots and clones makes it more efficient than vfs.
  * Podman provides a CLI which is a drop-in replacement for docker.
  * Supports docker-style networking using a port of [[https://github.com/containernetworking/plugins]].
  * Container images use the same formats and infrastructure as containerd and can be shared between the two implementations.

==== runj / containerd / nerdctl ====

Experimental %%FreeBSD%% [[Jails|Jail]] execution runtime implementation and %%FreeBSD%% OCI specification development.

  * First Release: 2021((https://samuel.karp.dev/blog/2021/03/runj-a-new-oci-runtime-for-freebsd-jails/))
  * Latest Release: 2022 (No release tags)
  * nerdctl provides a Docker-compatible CLI for containerd

Supports:

  * [[https://www.freshports.org/sysutils/containerd|containerd]] support((https://samuel.karp.dev/blog/2021/05/running-freebsd-jails-with-containerd-1-5/))
  * Linux Jail support
  * Network Support ([[https://samuel.karp.dev/blog/2022/09/experimental-networking-support-for-freebsd-jails-with-runj/|Experimental]])

  * [[https://www.freshports.org/sysutils/runj|runj FreeBSD Port/Package: sysutils/runj]]
  * [[https://github.com/samuelkarp/runj/|runj GitHub Repository]]

  * [[https://www.freshports.org/sysutils/nerdctl|nerdctl FreeBSD Port/Package: sysutils/nerdctl]]
  * [[https://github.com/containerd/nerdctl|nerdctl GitHub Repository]]

==== Bastille ====

Command-line (shell) tool and automation framework for jail-based containers.

  * First Release: 2018 ((https://bastillebsd.org/history/))
  * Latest Release: 2022 (0.9.2022*)

Supports:

  * %%FreeBSD%% Features: ZFS, VNET
  * Other Features: Template Creation, Import, Export
  * Has a container / template registry

  * [[https://bastillebsd.org/|BastilleBSD Homepage]]
  * [[https://www.freshports.org/sysutils/bastille|Bastille Port/Package: sysutils/bastille]]
  * [[https://github.com/BastilleBSD/bastille|Bastille GitHub Repository]]

==== pot ====

  * First Release: 2018 ((https://github.com/bsdpot/pot/blob/master/README.md#pot))
  * Latest Release: 2022 (0.15.3)

Command-line (shell) tool for jail-based containers.

  * [[https://github.com/pizzamig/pot/|pot GitHub Repository]]
  * [[https://www.freshports.org/sysutils/pot|pot Port/Package: sysutils/pot]]

Supports:

  * %%FreeBSD%% Features: ZFS, pf, rctl.
  * Orchestration with HashiCorp Nomad ((See [[https://github.com/pizzamig/pot/|pot homepage]], [[https://www.freshports.org/sysutils/nomad/|sysutils/nomad]] and [[https://www.freshports.org/sysutils/nomad-pot-driver/|sysutils/nomad-pot-driver]]))

==== cbsd ====

Command-line and TUI (shell) tool for jail-based containers.

  * First Release: ~2013 ((https://www.bsdstore.ru/en/about.html#FAQ))
  * Last Release: 2022 (13.1.4)

Supports:

  * %%FreeBSD%% Features: ZFS, VNET, bhyve, [[Xen]]
  * Other Features: Templates and Profiles

  * [[https://www.bsdstore.ru/en/about.html|cbsd homepage]]
  * [[https://www.freshports.org/sysutils/cbsd|cbsd Port/Package: sysutils/cbsd]] 
  * [[https://github.com/cbsd/cbsd|CBSD GitHub Repository]]

None of the above orchestration tools have a "run a plain vanilla linux inside the container" feature, but the start/stop/templating feature of them does not prevent a manually created plain-vanilla-linux-jail.

==== iocage ====

Command-line (Python, originally shell) tool for jail-based containers.

  * First Release: ~2016 ((If someone can identify an authoritative initial history/announcement, please update))
  * Latest Release: 2019 (1.2)

Supports:

  * %%FreeBSD%% Features: ZFS (required), VNET 
  * Other Features: Template Creation, Import, Export

  * [[https://github.com/iocage/iocage|iocage Homepage]]
  * [[https://www.freshports.org/sysutils/iocage|iocage Port/Package: sysutils/iocage]]

==== ioc ====

Re-implementation (Python) of iocage.

  * First Release: Unknown, but ~2017 ((If someone can identify an authoritative initial history/announcement, please update))
  * Latest Release: 2019 (0.8.2)

  * [[https://github.com/bsdci/ioc|ioc GitHub Repository]]
  * [[https://www.freshports.org/sysutils/ioc|ioc Port/Package:sysutils/ioc]] 

==== iocell ====

Fork of original (shell) iocage

  * First Release: ~2016 ("v2.0.0") ((If someone can identify an authoritative initial history/announcement, please update))
  * Latest Release: 2017 (2.1.2)

  * [[https://github.com/bartekrutkowski/iocell|iocell GitHub Repository]]
  * [[https://www.freshports.org/sysutils/iocell|iocell Port/Package: sysutils/iocell]] 

==== ezjail ====

Command-line (shell) tool for jail-based containers.

  * First Release: 2005 ((https://erdgeist.org/arts/software/ezjail/#id5))
  * Latest Release: 2015 (3.4.2)

Supports:

  * %%FreeBSD%% Features: ZFS, VNET (via manual scripting)
  * Other Features: Template Creation, Import, Export

  * [[https://erdgeist.org/arts/software/ezjail/|ezjail Homepage]]
  * [[https://www.freshports.org/sysutils/ezjail|ezjail Port/Package: sysutils/ezjail]]
  * [[https://erdgeist.org/gitweb/ezjail/|ezjail Git Repository (gitweb)]]

===== Container Orchestration =====

The container tools above can be used as a base for multi-host orchestration, by exporting containers from one host, and either moving the container to another host, or by utilising shared storage (eg: SAN) or ZFS to switch container datasets from one host to another and importing it on the target host.

Additionally the following %%FreeBSD%% container orchestration tools are available:

  * [[https://www.nomadproject.io/plugins/drivers/community/jail-task-driver|HashiCorp Nomad]] (jail-task driver)
  * [[https://www.freshports.org/sysutils/nomad|Nomad Port/Package: sysutils/nomad]]

===== Container Orchestration Distributions =====

XXX: add some %%FreeBSD%% derived distributions which have their focus (not only) on container virtualisation.
  * [[https://clonos.tekroutine.com/|CloneOS]] - %%WebUI%% for bhyve, jails, xen, ...