Package Building for the FreeBSD.org Cluster

The basic idea is that we need to make certain that when port upgrades happen on cluster machines, we need to be able to somewhat reasonably guarantee that the ports being upgraded are compatible with the existing ports, and nothing is missed. poudriere handles this quite nicely and reliably.

The cluster package build repository is separate from the default clusteradm subversion repository. It is located at: svn+ssh://pb00.FreeBSD.org/var/svn/pkgbuild/keg

This repository contains the /usr/local/etc/poudriere.d/ directory on pb00.FreeBSD.org.

This directory contains all the magic needed for FreeBSD.org cluster machine package builds as noted in the Local Configuration and Local Scripts sections.

As we already know, pre-built packages were never one-size-fits-all. What I have tended to do when setting up such package build systems is to do package builds based on either a per-machine basis, or based on a set of machine roles. Both of these have their pros and cons:

• With per-role builds, time spent building packages is reduced, however you lose flexibility with machine-independent configuration.
• With per-machine builds, more time is spent building (potentially duplicate) packages, however each can have independent configurations.

My preference has always been per-machine package builds.

/usr/local/etc/poudriere.d/ is set up according to what poudriere expects to find, and where it will look. These files are all under revision control, for the inevitable case of needing to look back to see what changed with a port build options, or needing to revert an entire set of changed options.

Each build has its own:

• options/ directory
• make.conf file
• pkglist.txt file

These are linked to a particular build by prefixing the jail name and target machine name, for example:

• 10-amd64-pb00-make.conf
• 10-amd64-pb00-options/
• 10-amd64-pb00-pkglist.txt

In this case, 10-amd64 is the name of the jail used for the port build and pb00 is the FreeBSD.org cluster machine that will be the consumer of the resulting packages.

The idea for jail names is effectively ${BRANCH_VERSION}-${ARCH}

The idea for the “sets” poudriere will build is effectively the machine hostname, less “freebsd.org”. In cases where there are multiple machines per site, or a machine hostname is site specific, the “.” is replaced with “-”, for example: cook-isc

There are two scripts right now to help automate the maintenance of package builds:

• clusterbuild.sh: Runs the package build for all configured systems.
• regen.sh: Updates the ports tree and, if TERM is set, runs make config for the ports in the pkglist.txt files (and dependencies).

There are a few configuration files that are used by the scripts:

• jails.conf: List of jails to map to build “sets”
• sets-${jail}.conf: List of target machines to build using a particular build jail.

There is also a pkgbootstrap.sh script that is intended to fetch the sample pkg.conf file and update as necessary, as well as the public pkg.cert file from pbdist.FreeBSD.org, and put them into the correct locations. After fetching the files, the SHA256 is calculated and verified against known values.

pb00.FreeBSD.org has a single jail for web services, pbdist.FreeBSD.org. Its web root directory is a read-only nullfs(5) mount from pb00.FreeBSD.org, containing an empty index.html file, a sample pkg.conf file, and the public certificate file (pkg.cert) for the SSL key used to sign the packages.

Note: If ever changing the pkg.cert (or the signing key) or pkg.conf, the SHA256 sums must be updated in pkgbootstrap.sh.

poudriere will automatically create the repo.txz using the signing key after a package build completes. In general, there should be nothing to do here, but if manually signing a package build is needed, this can be done as:

# pkg repo /poudrieredata/data/packages/${JAIL}-cluster-${TARGET_HOST}/ /root/ssl/pkg.key

A “bootstrapping a system for cluster pkgng builds” example:

root@pb00:~ # fetch http://pbdist.freebsd.org/pkgbootstrap.sh
pkgbootstrap.sh                               100% of 1465  B 1346 kBps 00m00s
root@pb00:~ # sh ./pkgbootstrap.sh 
/tmp/pkg.cert                                 100% of 1491  B 4282 kBps 00m00s
/tmp/pkg.conf                                 100% of  479  B 1480 kBps 00m00s
SHA256 for pkg.conf and pkg.cert match.
Creating /etc/ssl/ directory for pkg.cert...
Installing the public certificate file...
Done.
Running FreeBSD:10 amd64
This hostname is: pb00
Installing pkg.conf...
Done.
Cleaning up.
Done.
root@pb00:~ # grep ^PACKAGESITE /usr/local/etc/pkg.conf
PACKAGESITE         : https://pbdist.FreeBSD.org/10-amd64-cluster-pb00
root@pb00:~ # pkg update
Updating repository catalogue
digests.txz
100% 2280     2.2KB/s   2.2KB/s   00:00    
packagesite.txz
100% 7580     7.4KB/s   7.4KB/s   00:00    
Incremental update completed, 0 packages processed:
0 packages updated, 0 removed and 21 added.