Work In Progress
Contributed by JohnNielsen
This document is intended to serve as a walk-through for getting a Centos 5.5 jail running on a host running FreeBSD 8-STABLE.
I have been working on a project to provide an arbitrary number of dev/test/demo environments for a 3-tier web application. The environments need to be independent, lightweight, easily replicated, network-reachable, and resemble the production environment as closely as possible. Being able to rapidly provision new environments is a plus. In this case the production environment runs CentOS 5, Python, Apache, and PostgreSQL.
Fully virtualized environments as provided by VMware or VirtualBox might have worked, but I wanted something more flexible and lightweight in terms of resource allocation and consumption. The virtual disks required by such VM's are also not ideal since they need to be a fixed size and generally need to be copied in full to be cloned.
I decided that I really wanted to use ZFS and FreeBSD jails. That combination meets all of the requirements except for being similar to the production stack running CentOS. Unless maybe CentOS could run inside a FreeBSD jail with Linux emulation…
It turns out that it can with a modest amount of effort. Now that I've been through that once I'd like to share what I've learned so anyone else who likes this crazy idea can quickly implement it. With a successful proof-of-concept in hand I will be using this solution for my project.
This is not a requirement but it makes managing jail filesystems very easy. Each jail can have its own ZFS, and the filesystems can be cloned, snapshotted, backed up, compressed, etc. with very little effort. The remainder of this document assumes you have a zpool named “tank” and a (parent) ZFS named “tank/jails” mounted on /jails. The (first) jail will go into a (child) ZFS called “tank/jails/centos”. Adjust as necessary if you use different names or mountpoints.
Linprocfs in 8-STABLE needs a hack for openpty (used by e.g. SSH in the jail to get a pty) to work.
There was a (different) commit to 9-CURRENT to address this (http://svn.freebsd.org/viewvc/base?view=revision&revision=204825) but it does NOT work on 8.x (at least I couldn't get fdescfs to behave for me). The fix does work for FreeBSD 9.x but you do need to enable fdescfs in the jail. See: http://lists.freebsd.org/pipermail/freebsd-emulation/2010-March/007493.html (same thread as initial patch above). Reading the whole thread could be enlightening).
Version 4 works better with the CentOS packages than version 3 (archivers/rpm), which is what you get by default.
To get things rolling we will extract the squashfs environment used by the CentOS installer. We then use that environment to install the selected set of packages.
cd /tmp fetch http://archives.fedoraproject.org/pub/archive/fedora/linux/releases/10/Everything/i386/os/Packages/squashfs-tools-3.4-1.i386.rpm cd /compat/linux rpm2cpio -q < /tmp/squashfs-tools-3.4-1.i386.rpm | cpio -id
Remember that even on an amd64 (64-bit) host only 32-bit Linux emulation is supported, so get the “i386” DVD. Use BitTorrent or find a mirror at centos.org that hosts the whole thing on the web (only some of them do). The file I used is called CentOS-5.5-i386-bin-DVD.iso. He also suggested simply downloading the content of centos/5.5/os/i386 from any mirror instead of using an ISO image.
mdconfig -a -t vnode -f /tmp/CentOS-5.5-i386-bin-DVD.iso mkdir /media/centos mount -t cd9660 /dev/md0 /media/centos
This is the environment that runs when you boot from the DVD. For our purposes it provides Linux-native versions of bash and rpm, along with the libraries they need.
cd /jails /compat/linux/usr/sbin/unsquashfs -dest centos-stage2 /media/centos/images/stage2.img
cd centos-stage2 mkdir dist umount /dev/md0 mount -t cd9660 /dev/md0 /usr/jails/centos-stage2/dist
mkdir stage zfs set mountpoint=/usr/jails/centos-stage2/stage tank/centos
chroot /usr/jails/centos-stage2 /usr/bin/bash
mkdir -p /stage/var/lib/rpm rpm --root /stage --initdb
This can be tricky without automatic dependency resolution. If someone knows of a good way to provide that at this point let me know. My approach was to iteratively copy the packages I knew I would want or need into a “bootstrap” directory, then add packages to supply the dependencies rpm complained about. Lucky for you, I saved my list of packages.
I recommend copying the RPM packages in the list and others you need into the bootstrap directory to eliminate the need for both the RPM list and the DVD image in the future. It also makes the RPM command a lot simpler.
mkdir bootstrap
for rpm in `cat /centos5.5_bootstrap_rpm_list.txt`; do
cp /dist/CentOS/${rpm} bootstrap
done
cd bootstrap rpm --root /stage -Uvh *.rpm
You should not have any dependency errors. If you do, check the contents of the bootstrap directory against the list and try again. You can do this repeatedly, since nothing is actually installed until all dependencies are satisfied.
You will now have a minimal but reasonable set of packages installed. If you want to add more packages, you can use yum after the jail is running. (Yum of course provides automatic dependency resolution and will use the default CentOS repositories to retrieve the appropriate packages.)
exit umount /dev/md0
mkdir -p /usr/jails/centos zfs set mountpoint=/usr/jails/centos tank/centos
If you plan to have non-linux jails then some of these settings will need to be set per-jail instead of globally as below. Replace the interface and IP with the appropriate values for your system.
jail_enable="YES" jail_list="centos" jail_sysvipc_allow="YES" # INSECURE (shared memory isn't jailed) but required for e.g. postgresql jail_interface="em0" jail_devfs_enable="YES" jail_fdescfs_enable="YES" # XXX this only works for me on FreeBSD 9.x and newer jail_exec_start="/bin/sh /etc/rc.d/rc 3" jail_exec_stop="/bin/sh /etc/rc.d/rc 0" jail_centos_rootdir="/jails/centos" jail_centos_hostname="centos.your.domain" jail_centos_ip="1.2.3.4" jail_centos_fstab="/jails/fstab_centos" # for linprocfs
This must be done for every jail.
echo "linproc /jails/centos/proc linprocfs rw 0 0" >> /jails/fstab_centos
Otherwise you have to gesticulate around devfs and you may not get logging from the jail.
SYSLOGD_OPTIONS="-m 0 -p /var/run/log"
This is used by the host to look up the uid of the jail owner, so it must exist. Take care not to clobber the exising (CentOS) etc/passwd.
echo "root::0:0::0:0:Charlie &:/root:/bin/bash" > /jails/centos/etc/master.passwd pwd_mkdb -d /jails/centos/tmp -p /jails/centos/etc/master.passwd cd /jails/centos/tmp mv master.passwd pwd.db spwd.db ../etc
cp /etc/resolv.conf /jails/centos/etc
echo "NETWORKING=yes" >> /jails/centos/etc/sysconfig/network echo "HOSTNAME=centos.your.domain" >> /jails/centos/etc/sysconfig/network echo "1.2.3.4 localhost centos centos.your.domain" >> /jails/centos/etc/hosts
chroot /jails/centos /bin/bash cd /etc pwconv grpconv passwd root # this is the password you will use for su, etc. within the jail touch /etc/fstab touch /etc/mtab
cd /sbin mv consoletype consoletype.orig ln -s /bin/true consoletype cd /bin mv umount umount.prev ln -s /bin/true umount
chkconfig --list # (look through the output) chkconfig gpm off chkconfig lvm2-monitor off chkconfig mcstrans off chkconfig netfs off chkconfig network off # (really) chkconfig rawdevices off
In my setup that just leaves crond, sendmail, sshd and syslog. Your needs may vary.
exit
With the changes to /etc/rc.conf above the jail will start automatically at reboot.
/etc/rc.d/jail start centos
tail -n300 /var/log/messages|grep "linux:" cat /var/log/jail_centos_console.log
With any luck, you should be in business at this point. Use “jexec <num> bash” to look around inside the jail, where <num> is obtained from running “jls”. SSH (as root) should work as well.
RPM and yum will be installed and functional, so installing new packages or updates to existing ones is much easier than bootstrapping the initial set.
Most things should work as you would expect.
zfs snapshot tank/centos@somelabel zfs clone tank/centos@somelabel tank/jails/newjail
rm -f /jails/newjail/etc/ssh/ssh_host_* rm -f /jails/newjail/var/lib/rpm/__*
echo "linproc /jails/newjail/proc linprocfs rw,late 0 0" >> /jails/fstab_newjail
jail_newjail_rootdir="/jails/newjail" jail_newjail_hostname="newjail.your.domain" jail_newjail_ip="1.2.3.5" jail_newjail_fstab="/jails/fstab_newjail"
/etc/rc.d/jail start newjail