===== Jails =====
%%FreeBSD%% jail(8) provides lightweight, kernel-level containers for the secure isolation of one or more processes up to a complete userland.
Jails were introduced by [[http://people.freebsd.org/~phk/|Poul-Henning Kamp]] in March 2000 with %%FreeBSD%% 4.0-RELEASE.
===== Additional Resources =====
* [[https://docs.freebsd.org/en/books/handbook/jails/|FreeBSD Jails Handbook Chapter]]
* [[https://man.freebsd.org/cgi/man.cgi?query=jail&sektion=8&manpath=FreeBSD|jail(8) Manual Page]]
* [[https://lists.freebsd.org/subscription/freebsd-jail|freebsd-jail Mailing List]]
* [[https://lists.freebsd.org/subscription/freebsd-virtualization|freebsd-virtualization Mailing List]]
* [[https://en.wikipedia.org/wiki/FreeBSD_jail|FreeBSD Jail Wikipedia Page]]
* [[https://freebsdfoundation.org/freebsd-project/resources/introduction-to-freebsd-jails/|An Introduction to FreeBSD Jails]]
* [[https://klarasystems.com/articles/freebsd-jails-the-beginning-of-freebsd-containers/|FreeBSD Jails – Deep Dive into the Beginning of FreeBSD Containers]]
* [[https://klarasystems.com/articles/virtualize-your-network-on-freebsd-with-vnet/|Virtualise your network on FreeBSD with VNET]]
* [[Jails/VNET|VNET Jails]]
* [[https://freebsdfoundation.org/wp-content/uploads/2020/03/Jail-vnet-by-Examples.pdf|Jail vnet By Examples (PDF)]]
* [[JailingGUIApplications|Jailing GUI Applications]]
* [[https://man.freebsd.org/cgi/man.cgi?rctl(8)|rctl(8) Manual Page]]
* [[JailResourceLimits|Jail Resource Limits]]
* [[Hierarchical_Resource_Limits|Hierarchical Resource Limits]]
===== Jail-specific and Jail-aware Utilities =====
* [[https://man.freebsd.org/cgi/man.cgi?query=jls|jls(8)]]
* [[https://man.freebsd.org/cgi/man.cgi?query=jexec|jexec(8)]]
* [[https://man.freebsd.org/cgi/man.cgi?query=sockstat|sockstat(1)]]
* [[https://man.freebsd.org/cgi/man.cgi?query=ifconfig|ifconfig(8)]]
* [[https://man.freebsd.org/cgi/man.cgi?query=pkg|pkg(7)]]
* [[https://man.freebsd.org/cgi/man.cgi?query=route|route(8)]]
* [[https://man.freebsd.org/cgi/man.cgi?query=tzsetup|tzsetup -sC /var/jails/myjail UTC]]
===== Feature Requests =====
Feature requests, directly within jail(8) or supporting resources:
* MAC Policy on IP Address in Jail [[SummerOfCode2019Projects/MACPolicyIPAddressJail|GSoC Page]] and [[https://reviews.freebsd.org/D20967|Review D20967]]
* Add support for nice value for rc.d/jail + rc.conf [[https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=124248|PR124248]]
* Do not keep JID incrementing forever [[http://www.freebsd.org/cgi/query-pr.cgi?pr=122270|PR122270]] (Reverted?)
* Ability to specify a different root for the virtual machine
* Ability to specify a different network address
* Ability to specify a completely different network universe
* Ability to dedicate an interface to it
* Ability to have a separate PID space for it
* Ability to specify a separate UID space for it
* Ability to have different mount tables for it
* SNMP monitoring of jail resources (Did Stef Walter have a patch?)
* Expose certain sysctl values inside jails
* Unprivileged jail_attach / a way to do jail_attach as non-root [[https://www.freshports.org/sysutils/jailme/|jailme port]] - [[https://web.archive.org/web/20130622043638/http://people.collaborativefusion.com/~wmoran/code/jailme.html|More information from Bill Moran]]
* %%FreeVPS%% features developed by Alex Lyashkov: Per prison process count limit, Separated UID hash, Separated %%SysV%% IPC, File handles usage limit (Status of these?)
===== VNET-based networking for Jails =====
==== Configuration ====
The VNET is set up using an
[[https://www.freebsd.org/cgi/man.cgi?query=if_epair&apropos=0&sektion=0&manpath=FreeBSD+13.0-RELEASE+and+Ports&arch=default&format=html|if_epair(4)]]
interface bridged with the actual network interface, in my case ''%%re0%%''.
Just like if_tap(4) interfaces, epairs can be used by one jail at a
time, so if you need to run more than one jail at the same time,
you have to make more epairs.
If you're using tap interfaces for bhyve VMs, you can just ''%%addm%%''
them into the bridge.
Add the following lines to ''%%/etc/rc.conf%%'':
cloned_interfaces="bridge0"
ifconfig_bridge0="addm re0 up"
Add or edit the kld_list parameter to add the following module in ''%%/etc/rc.conf%%'':
kld_list="if_bridge if_epair"
Apply changes:
# service netif restart && service routing restart
The jail needs to inherit ''%%/dev/bpf*%%'' from the host in order for
networking to work at all. Make a new ''%%/etc/devfs.rules%%'' ruleset:
[devfsrules_jails=5]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add path 'bpf*' unhide
Restart devfs(8):
# service devfs restart
We'll name this jail ''%%foo%%'' and give it the other end of the epair
as its network interface. Its IP address will be acquired using
DHCP. The reason I'm manually calling dhclient(8) is because adding
''%%ifconfig_epair0b="DHCP"%%'' in the jail's ''%%/etc/rc.conf%%'' doesn't
work. Options are detailed in
[[https://www.freebsd.org/cgi/man.cgi?query=jail.conf&apropos=0&sektion=0&manpath=FreeBSD+13.0-RELEASE+and+Ports&arch=default&format=html|jail.conf(5)]].
''%%/etc/jail.conf%%'':
path = "/usr/local/jail/$name";
host.hostname="$name";
exec.clean;
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
allow.mount;
allow.raw_sockets = 1;
mount.devfs;
devfs_ruleset="5";
vnet;
sysvmsg=new;
sysvsem=new;
sysvshm=new;
foo {
vnet.interface = "epair0b";
exec.start += "dhclient epair0b";
}
**Adding additional jails**
There are a few things that should always be kept in mind:
* Please also remember to increment the epair devices.
* Jails can have dependencies via the depend parameter in ''%%/etc/jail.conf%%'', and these are respected when doing parallel startup.
==== Installation ====
# mkdir -p /usr/local/jail/foo
# bsdinstall jail /usr/local/jail/foo
...
# ifconfig epair0 create
# ifconfig bridge0 addm epair0a
# ifconfig epair0a up
# service jail start foo
Test to see if the jail has networking:
# jexec foo ping google.com
On jail shutdown, destroy the epair:
# service jail onestop foo
# ifconfig epair0a destroy
==== Delete jail ====
Deleting jails isn't as straight forward, so I'm leaving this here as well:
# service jail stop foo
# chflags -R noschg /usr/local/jail/foo
# rm -rf /usr/local/jail/foo
===== Netgraph-based networking for jails =====
==== Introduction ====
There are three ways to do networking with jails:
(i) using IP addresses (aliases) from the host,
(ii) using if_bridge(4) and epair(4) interfaces, and
(iii) using netgraph.
I will start this how-to with the third method as the second method has already been explained by [[MasonLoringBliss/JailsEpair|Mason Loring Bliss]].
==== Netgraph Terminology ====
* **Node** Any object that can connect to another object. This can be a bridge, a virtual interface, but also a VLAN interface or a BPF object.
* **Edge** The connection between two nodes. You could consider it to be a virtual network cable.
* **Hook** In netgraph, edges do not really exist //per se//. Instead, an edge is simply an association of two hooks, one from each node. A node's hooks define how that node can be connected. For a bridge node, the hooks are the virtual interfaces, though for a virtual interface node, hooks can be each protocol supported by that interface.
* **Path** Every netgraph node is addressable via an ASCII string called a //node address// or //path//. Many nodes have //names//. When a node has a name, it can always be addressed usingthe //absolute address// consisting of the name followed by a colon. If a node does not have a name, you can construct one from the node's unique ID number by enclosing the number in square brackets. //Relative addressing// is also possible when two nodes are indirectly connected. A relative address uses the names of consecutive hooks on the path from the source node to the target node. Relative and absolute addressing can be combined.
This is quite confusing. See the example at [[https://people.freebsd.org/~julian/netgraph.html]].
Let's start off with a small network consisting of two jails, called "test1" and "test2", which are bridged to an new, virtual interface on the host machine. We thus need three new virtual interfaces. We will use **ngctl** to manage all netgraph nodes, hooks, and edges. This tool can be used interactively or you can supply the command on the command line. Below we will use it interactively. Note that this command must be run as the root user.
# ngctl
mkpeer eiface ether ether
mkpeer eiface ether ether
mkpeer eiface ether ether
name ngeth0: ng_host
name ngeth1: ng_test1
name ngeth2: ng_test2
We create three interfaces of type ''%%eiface%%'' as we need Ethernet interfaces (as opposed to L3 interfaces when using type ''%%iface%%''). The man page state they are named ''%%ngeth0%%'', etc. We rename them to indicate their purpose.
Note that when referencing a node name, we must append it with a colon. That is because we're specifying a //path//, not just a //node//.
Next we need to bridge these interfaces together. We can either create the bridge first and then connect it to the three interfaces, or while creating the bridge, we can already connect it to an interface.
/*
* OPTION A:
* To only create the bridge but not connect it to any existing nodes
*/
mkpeer bridge link0 link0
/*
* OPTION B:
* Create a new node of type bridge and connect it to the
* path 'ng_host:'. Use its hook called 'ether' to connect it to the
* hook 'link0' of the new node (the bridge).
*/
mkpeer ng_host: bridge ether link0
/*
* As opposed to the Ethernet interfaces, a bridge does not have a
* name by default. So we name it here.
*/
name [1e]: br0
/*
* Now connect the nodes together. The first connection is only needed
* when option A was used.
*/
connect br0: ng_host: link0 ether
connect br0: ng_test1: link1 ether
connect br0: ng_test2: link2 ether
The above configuration is pretty straightforward, the exception being the **name** command.
Since the bridge does not have a name, we can't use it to construct a path.
Instead we must use its unique ID (between brackets).
To determine this ID, we need the **list** command which I ommitted from the configuration above.
Let's show it here.
+ list
There are 6 total nodes:
Name: ng_host Type: eiface ID: 00000007 Num hooks: 1
Name: ng_test1 Type: eiface ID: 00000009 Num hooks: 0
Name: ngctl85802 Type: socket ID: 0000002b Num hooks: 0
Name: Type: bridge ID: 0000001e Num hooks: 1
Name: ng_test2 Type: eiface ID: 0000000e Num hooks: 0
+
The above output shows the unique ID for the bridge (preceding zeros can be ommitted) as well as the number of hooks used, i.e. connections made.
I thus created the bridge by directly connecting it to the ''%%ng_host%%'' interface.
==== Configuring the jails ====
I assume you know how to setup a jail, i.e. create a base jail, clone it, etc. If not, please read the //Jails// chapter of the %%FreeBSD%% Handbook.
The below configuration only includes the networking-specific configuration for the two jails. Although shown per jail, you can easily move these two lines to the default configuration using the variable ''%%$name%%'' to customize the interface name.
test1 {
vnet;
vnet.interface = ng_test1;
}
test2 {
vnet;
vnet.interface = ng_test2;
}
Once the jails have been started, you can manually configure IP addresses on their interfaces, as well as on the host interface ''%%ng_host%%''.
NAT to be covered in the future.
==== Additional Netgraph resources ====
* [[MasonLoringBliss/JailsEpair|Using jails with if_bridge(4) and epair(4)]] (Mason Loring Bliss)
* [[VIMAGE/VNETSamples|Sample (and simple) VNET configurations in shell]] (Bjoern Zeeb)
===== Jail Management Tools =====
* **%%AppJail%%**
* Simple and easy-to-use tool for creating portable jails.
* [[freshports>sysutils/appjail]]
* [[freshports>sysutils/appjail-devel]]
* **Bastille**
* Pure shell system for automating deployment and management of containerized applications in jails
* [[freshports>sysutils/bastille]]
* **BSDploy**
* Container management and orchestration for %%FreeBSD%% jails using ansible
* [[freshports>sysutils/bsdploy]]
* **cbsd**
* %%FreeBSD%% jail and bhyve management utility
* [[freshports>sysutils/cbsd]]
* **crate**
* An application containerizer utilizing jails
* [[freshports>sysutils/crate]]
* **ezjail**
* A framework to easily create, manipulate and run %%FreeBSD%% jails
* [[freshports>sysutils/ezjail]]
* **focker**
* A jail-based image orchestration tool in the vein of [[Docker]]
* [[freshports>sysutils/py-focker]]
* **iocage**
* Utility for managing jails
* [[freshports>sysutils/iocage]]
* **iocell**
* iocage fork focused on supporting //classic iocage// experience and adding fixes and features requested by the iocage users
* [[freshports>sysutils/iocell]]
* **jadm**
* %%FreeBSD%% jail framework with zfs, vnet and jail.conf support
* [[freshports>sysutils/jadm]]
* **jailadmin**
* A system for managing a set of named jails
* [[freshports>sysutils/jailadmin]]
* **jailaudit**
* Script for generating portaudit reports for jails
* [[freshports>ports-mgmt/jailaudit]]
* **jailctl**
* Jail management tool
* [[freshports>sysutils/jailctl]]
* **jailer**
* Manage %%FreeBSD%% jail startup, shutdown and console
* [[freshports>sysutils/jailer]] (This port expired on: 2014-03-10)
* **jailme**
* A setuid version of jexec to allow normal users access to jails
* [[freshports>sysutils/jailme]]
* **jailrc**
* Improved jail startup/shutdown script
* [[freshports>sysutils/jailrc]]
* **jailutils**
* Several utilities for managing jails
* [[freshports>sysutils/jailutils]]
* **jest**
* REST API for creating and managing jails
* [[freshports>sysutils/jest]]
* **jkill**
* Shutdowns a running jail and all its processes
* [[freshports>sysutils/jkill]]
* **jps**
* Wrapper to ps(1) that maps pids to jails
* [[freshports>sysutils/jps]]
* **jruls**
* A top-alike utility to display and update %%FreeBSD%% jails resource usage
* [[freshports>sysutils/jruls]]
* **jtop**
* Wrapper to top(1) that maps pids to jails
* [[freshports>sysutils/jtop]]
* **mkjail**
* Easy jail creation, updates and upgrades
* [[freshports>sysutils/mkjail]]
* **mod_jail**
* Apache 1.3.x/2.0.xx module to enable an easy alternative to mod_chroot
* [[freshports>www/mod_jail]] (This port expired on: 2017-04-30)
* **p5-BSD-Jail-Object**
* An object oriented perl interface to jail(2)
* [[freshports>sysutils/p5-BSD-Jail-Object]]
* **ploy**
* Command-line tool to provision, manage, and control servers, including jails
* [[freshports>sysutils/py-ploy]]
* **pot**
* Container framework for %%FreeBSD%%, based on jails, ZFS and pf
* [[freshports>sysutils/pot]]
* **qjail**
* Utility to deploy large number of jails quickly
* [[freshports>sysutils/qjail]]
* **quickjail**
* utility to create transient single-command jails
* [[freshports>sysutils/quickjail]]
* **runj**
* Proof-of-concept OCI-compatible runtime for %%FreeBSD%% jails
* [[freshports>sysutils/runj]]