====== Poudriere: Getting Started - Tutorial ======

To put it simply, [[https://www.freshports.org/ports-mgmt/poudriere|Poudriere]] is a tool that builds pkg repositories. It's the recommended tool for building ports, and it's used by the %%FreeBSD%% project to build the official ''%%pkg%%'' repositories.

Because of its flexibility and the fact it doesn't build a port and install it right away, but builds a repository first, it's perceived as complex and heavy. The reality is, though, it's a simple collection of shell scripts that automate port building in isolated environments using jails, and is applicable for simple systems as well as for complex deployments where a variety of arch, ABI or options sets are required.

<callout type="info">

This tutorial covers the basic setup + typical workflow example, in a single easy-to-read document. However, the official Poudriere source repository on [[/GitHub|GitHub]], and the poudriere(8) manage have a lot of information and examples, and the reader is advised to peruse those as well:

  * [[https://github.com/freebsd/poudriere/wiki]]
  * [[https://www.freebsd.org/cgi/man.cgi?query=poudriere&sektion=8&manpath=freebsd-release-ports|poudriere(8)]] manpage
  * [[https://www.freebsd.org/doc/handbook/ports-poudriere.html|Building packages with Poudriere]] section in the official %%FreeBSD%% Handbook.
</callout>

===== Getting started: configuration =====

While Poudriere almost works out of the box (plus all the setup required for jails, ports trees and package lists, done in this tutorial), it is very much configurable, and not just for some advanced use. The main config file is located at ''%%/usr/local/etc/poudriere.conf%%'', and the repo-specific configuration (done by Poudriere automatically for options etc) is located at ''%%/usr/local/etc/poudriere.d/%%''.

It //almost// works out of the box, because it requires at least //one// option be explicitly set in ''%%poudriere.conf%%'', and that is the ''%%FREEBSD_HOST%%'' config that for some reason does not default to the recommended default (see comments in the config file). All other options needn't be modified to just get started using Poudriere.

The config file is extensively self-documenting, so see the file for more information, but we'll mention a few worth taking a special look at, before we begin:

  * **USE_TMPFS** -- Define where to use ''%%tmpfs%%'' in the build process, default being ''%%yes%%'', which means Poudriere will build each port (having the wrkdir) in a ''%%tmpfs%%''. For memory-constrained systems that might not be preferable.

  * **CCACHE_DIR** -- Define the path to (and thus enable) CCACHE files. Default being off, it is very useful to enable CCACHE (create a directory for it, and specify the path to ''%%CCACHE_DIR%%'' var). The reason for this is that Poudriere will rebuild any reverse dependency of any port that changed the version string. This is important for isolated (re-)builds guaranteeing some build integrity for a dependency graph of any port. The side-effect is frequent rebuilds of ports that have no change in source code. CCACHE thus helps a lot with this, as it allows re-use of previously compiled objects to be re-used without compilation, and drastically improves compilation times.

  * **PARALLEL_JOBS** -- define how many ports to build in parallel, default being the number of detected "cpus", via ''%%hw.ncpu%%'' sysctl.

  * **ALLOW_MAKE_JOBS** -- define makefile parallelism, per port build process, default off.

The overall default is such that ports are built in parallel, but with no parallelism within each build process. Since not all ports can be built in parallel, especially with dependencies where a port must wait until all build-deps are built first, the production configuration depends on many things. The author's "sweet spot" is to split those in two, so eg. for a 4-core system, build 2 ports in parallel, allow make parallelism, and set ''%%MAKE_JOBS_NUMBER%%'' in ''%%make.conf%%''.

==== Per-repository configuration and make.conf ====

A bit advanced topic at this stage in the tutorial, but it's important to mention that the main poudriere configuration can be made repository-specific. As the [[https://www.freebsd.org/cgi/man.cgi?query=poudriere&sektion=8&manpath=freebsd-release-ports|poudriere(8)]] manpage shows, Poudriere is using configuration files (and ''%%make.conf%%'') in a specific order of patterns tried, so repository-specific configurations are possible with repository-specific naming patterns for those files under ''%%/usr/local/etc/poudriere.d/%%''. The manpage explains how.

===== Basic workflow =====

The most simple use case for Poudriere -- to build a package repository for use on a single machine -- consists of the following simple steps.

==== 1. Create a builder jail ====

Poudriere builds ports in isolated environment using jails. It manages jails on its own, so no previous jail configuration is needed for it. It is also possible to run Poudriere in a jail itself.

Pretty much all the steps below require the user to specify which builder jail is used. That is because Poudriere builds repositories, but differentiates them according to:

  * Which builder jail is used

  * Which ports tree is used

  * Which user-defined set is used

Poudriere uses this information to build a directory where the repository is stored, directory the name of which includes the above information. So for example, if we have:

  * Builder jail called "111x64" (arbitrary name for a 11.1 64-bit base jail)

  * Ports tree called "default"

Poudriere will build the directory called ''%%111x64-default%%''. If a set named "py35" is used, then the directory will be ''%%111x64-default-py35%%''. This allows complex combinations of repositories, but for a simple use case, we need just one jail, one tree and probably no sets to get started.

Poudriere thus requires a builder jail, so we define one, for 11.1 64-bit base, and call it "111x64" (the name is arbitrary, but must be consistently used to identify that jail, for ''%%options%%'', ''%%bulk%%'' and other commands):

<code>
poudriere jail -c -j 111x64 -v 11.1-RELEASE
</code>

The version is important because that's the %%FreeBSD%% base, with libs against which ports will be built (so it defines ''%%libc%%'', base ''%%openssl%%'' and other base facilities that ports might need).

<callout type="info">
**Repository names**

It is crucial to remember that Poudriere builds pkg repositories, which are directories under the configured ''%%${POUDRIERE_DATA}/packages%%'' directory (see below). Each repository directory name is constructed from ''%%<jailname>-<ports_tree>%%'' and if sets are used, ''%%<jailname>-<ports_tree>-<setname>%%''. This is why for most commands we must always specify ''%%-j%%'', ''%%-p%%'' and if sets are used ''%%-z%%'' flags, as those three actually define **the** repository for which we're running the command.
</callout>

==== 2. Create the ports tree ====

The second requirement is of course the ports tree. Poudriere manages ports tree on its own, so it's possible to have several, using various methods to track them: svn, ftp or, very important, a local directory which is useful if you already have a ''%%/usr/ports%%'' populated and don't want it managed again elsewhere by Poudriere. Let's assume we want Poudriere to manage the ports tree (so we don't have a system-wide ''%%/usr/ports%%'' populated), and let's give it a name "default":

<code>
poudriere ports -c -p default
</code>

But wait, what if we //do// want to use our existing ''%%/usr/ports%%'' tree? Say, we're tracking it with svn and adding our own patches or modifications. Easy, let's then instruct Poudriere to "create" a tree using a local directory. Let's call it "local" (the names are really arbitrary, but must be consistently used to identify that tree, for ''%%options%%'', ''%%bulk%%'' and other commands):

<code>
poudriere ports -c p local -F -M /usr/ports -f none
</code>

However, in that case poudriere will not manage the tree with the ''%%ports%%'' command, and a separate command is needed to update it, like running ''%%portsnap%%'' or ''%%svn up%%'', depending on how the tree is managed by the user.

==== 3. Create the list of ports wanted in the repo ====

Poudriere can build individual ports, by giving the port origin to the ''%%bulk%%'' command, but its more useful option is to define a **persistent list** of packages we want built, and have Poudriere manage dependencies, updates and upgrades automatically. Let's say we want Firefox in our repo. All that's needed is a file containing just the Firefox origin:

<code>
# /usr/local/etc/poudriere.d/packages-default

www/firefox
</code>

The file location, and name, is completely arbitrary, but it would be wise to follow the naming patterns Poudriere uses for ''%%make.conf%%'', options and other things (see below), as they're important in the order at which Poudriere selects them to build a repo. See the [[https://www.freebsd.org/cgi/man.cgi?query=poudriere&sektion=8&manpath=freebsd-release-ports|poudriere(8)]] manpage, section CUSTOMIZATION on how Poudriere selects correct config files for jail-ports_tree-set combinations.

The list of packages is the only file that Poudriere does not automatically source based on a pattern, so its location and name really is at the choice of user.

==== 4. Optionally change default port options ====

If we don't change any ports options, Poudriere will automatically use default options, in that it will use defaults to create a specific options directory, the poudriere-managed options directory under ''%%/usr/local/etc/poudriere.d/%%'', which is also subject to the above-mentioned naming patterns. So if we do nothing, defaults will be used, but then that's probably not why we're building from ports.

Ideally, we want Poudriere to go through all the options recursively, and give us an options dialog for any port that has new or changed options. Poudriere has the ''%%options%%'' command for this, which (because it can differentiate options per jail, ports tree or set), requires us to specify jail and ports tree, and of course the list of ports we're configuring. So, simply:

<code>
poudriere options -j 111x64 -p default -f /usr/local/etc/poudriere.d/packages-default
</code>

See the manpage on how to do more advanced things with options including resetting or forcing a dialog just for one port, etc... If sets are used, ''%%options%%'' will require a ''%%-z%%'' flag too.

==== 5. Configure pkg to use the repo built by Poudriere locally ====

As mentioned before, Poudriere really builds a package repository, so the actual installation, removal or update/upgrade of packages built by Poudriere is done with [[https://www.freebsd.org/cgi/man.cgi?query=pkg&apropos=0&sektion=8&manpath=FreeBSD+11.1-RELEASE+and+Ports&arch=default&format=html|pkg(8)]].

To access that repository, we have to configure ''%%pkg%%'' to use it. The [[https://www.freebsd.org/cgi/man.cgi?query=pkg.conf&apropos=0&sektion=5&manpath=FreeBSD+11.1-RELEASE+and+Ports&arch=default&format=html|pkg.conf(5)]] manpage tells us we can either modify the system configs under ''%%/etc/pkg/%%'' or have a local config (preferable!) under ''%%/usr/local/etc/pkg/repos/%%''. Now, ''%%pkg%%'' requires a (arbitrarily named) ''%%.conf%%'' files at those paths. While the file names are not important, the repository names configured in them **are** important. This allows us to set a local repository configuration and disable the default official repo configured in ''%%/etc/pkg/FreeBSD.conf%%'':

<code>
# /usr/local/etc/pkg/repos/Local.conf

Poudriere: {
  url: "file:///usr/local/poudriere/data/packages/111x64-default"
}

FreeBSD: {
  enabled: no
}
</code>

The path given above is constructed from Poudriere's configured location for repositories, and the repo we're configuring in this tutorial: based on "111x64" named jail and "default" named ports tree. Poudriere stores repositories under ''%%${POUDRIERE_DATA}/packages%%'' directory, where ''%%POUDRIERE_DATA%%'' is configured in ''%%/usr/local/etc/poudriere.conf%%'', and the default is ''%%/usr/local/poudriere%%''.

==== 6. Build the repository ====

With all the previous steps done to initialize our Poudriere builder, the actual building is as simple as:

<code>
poudriere bulk -j 111x64 -p default -f /usr/local/etc/poudriere.d/packages-default
</code>

As mentioned before for other commands, the ''%%bulk%%'' command requires us to specify which builder jail is used, which ports tree, which list of packages, at minimum. Poudriere will build all the ports we specified in the file given by ''%%-f%%'', as well as all the dependencies required to build and run Firefox.

==== 7. Use the repository ====

Once configured, we really just use ''%%pkg%%'' to install, update, upgrade or remove the packages. Our configured and locally built repository is essentially not different from the official //pkg// repositories, except that we've used our own list of ports we want (the official repos build all 30k+ ports in the tree), and we used our own options (the official repos use default options). So to continue the above example with Firefox, we just:

<code>
pkg install firefox
</code>

===== Production use workflow =====

So far we've covered the steps required to set up and build a pkg repository. However, what next? How do we keep our packages updated so we can simply update our systems with ''%%pkg upgrade%%'' ? The (production) workflow is then as simple as: updating the ports tree, handling any changed/new options and simply re-building the repo for any changes. By that we mean building packages that have updated including any dependencies:

<code>
poudriere ports -u
poudriere options -j 111x64 -p default -f /usr/local/etc/poudriere.d/packages-default
poudriere bulk -j 111x64 -p default -f /usr/local/etc/poudriere.d/packages-default
</code>

These three commands are all that's needed. Now, it must be noted that the ''%%options%%'' step might require user interaction. Poudriere will go through options of each port and if there's anything new, will present the dialog for us. It will also present a dialog for any port that became a new dependency, or any port that doesn't have an entry in the poudriere-managed options directory.

One extra step may be needed, and that's the package cleanup. Any package previously built in the repository that is no longer needed -- because it wasn't specified in the ''%%-f%%'' package list, and is suddenly no longer a dependency --  will remain in the repo unless we explicitly clean it with ''%%pkgclean%%'' command, so the typical workflow might require that additional step:

<code>
poudriere pkgclean -j 111x64 -p default -f /usr/local/etc/poudriere.d/packages-default
</code>

It is very important to give the ''%%-f%%'' list of packages that we //do// want installed, so it can determine which package is no longer needed in that dependency graph. The ''%%-j%%'' and ''%%-p%%'' options are needed because, like we said, Poudriere defines a repository based on jail + ports tree used. It's also worth noting that, like the ''%%options%%'' step, this requires user interaction as Poudriere will collate the list of packages to remove, but ask the user for confirmation, unless ''%%-y%%'' option is given to ''%%pkgclean%%''.

Additionally, if the ports tree repo was configured not to be managed by poudriere (like in step #2 above when the tree was defined to be the already existing ''%%/usr/ports%%''), it must be then updated separately (in place of ''%%poudriere ports -u%%'' given in the above example).

==== Updating jails ====

Running ''%%poudriere bulk%%'' will always automatically update and upgrade any package in the list given by ''%%-f%%'' (as well as any dependencies). But, what about builder jails? The builder jails have to be updated separately, just like a base system needs to be updated running ''%%freebsd-update fetch install%%''. With Poudriere, updating a jail named "111x64" is as simple as:

<code>
poudriere jail -u -j 111x64
</code>

However, unlike with, say, ''%%freebsd-update -r 12.0-RELEASE upgrade%%'', Poudriere does not upgrade jails ot new major version, and it would be against the idea of it, and designed workflow if it did. To "upgrade", one merely has to **create** a jail for the new major base version, and use the new jail name for ''%%-j%%'' in all poudriere commands that need to specify the (new) jail. This means, of course, that poudriere will create a new repository, named after the new jail.

**Note:** Updating jails (to a new patch level) will trigger rebuild of **all** packages defined in the repo (in the package list given by ''%%-f%%''), the next time ''%%bulk%%'' is used for the ''%%-j%%'' jail that has been updated.

===== Sets =====

A more advanced feature of Poudriere are sets, or the ''%%-z%%'' flag given to most poudriere commands. Sets allow building different repositories based on some variation. Since poudriere will already build a repository based on jail and ports tree, namely ''%%-j%%'' and ''%%-p%%'' flags, it is further possible to vary things for the same jail and/or for the same ports tree.

As an example, the user might want to build two repositories, one with Python 3.x in ''%%DEFAULT_VERSIONS%%'', and one with Python 2.7 in ''%%DEFAULT_VERSIONS%%''. Let's ignore the fact that right now 2.7 is default and no specific configuration is needed for it, and so let's create two ''%%make.conf%%'' files, for those two sets, both of which will use the same builder jail ''%%-j 111x64%%'' and same default port tree ''%%-p default%%''. Given the filename patterns tried first, we can thus define two ''%%make.conf%%'' files:

<code>
# /usr/local/etc/poudriere.d/py27-make.conf
DEFAULT_VERSIONS= python=2.7
</code>

<code>
# /usr/local/etc/poudriere.d/py36-make.conf
DEFAULT_VERSIONS= python=3.6
</code>

Then in all the ''%%options%%'', ''%%bulk%%'' and ''%%pkgclean%%'' commands we supply sets "py27" and "py36" in order to build two repositories. For example bulk:

<code>
poudriere bulk -j 111x64 -p default -f /usr/local/etc/poudriere.d/packages-default -z py27
poudriere bulk -j 111x64 -p default -f /usr/local/etc/poudriere.d/packages-default -z py36
</code>

With the set name of "py36", poudriere will source ''%%/usr/local/etc/poudriere.d/py36-make.conf%%''  (as it tries %%<%%setname>-make.conf), and similarly for the set name of "py27". The created pkg repositories will thus be at:

  * ''%%${POUDRIERE_DATA}/packages/111x64-default-py27/%%''

  * ''%%${POUDRIERE_DATA}/packages/111x64-default-py36/%%''

Which is at ''%%${POUDRIERE_DATA}/packages/<jailname>-<ports_tree>-<setname>%%''.

Of course, we've used here one builder jail and one ports tree, but all these could be combined to create a really complex and flexible array of individual package repositories. Each is at a path that can then be given to ''%%pkg.conf%%'' in various (optional) priorities.

**Note:** With set-specific ''%%make.conf%%'' as given in this example, or in other words with **multiple** ''%%make.conf%%'' existing on the system, Poudriere will source each in the order explained in the manpage. That means it will use multiple at the same time. It will first try ''%%/usr/local/etc/poudriere.d/make.conf%%'' and source it if found. It will then try ''%%/usr/local/etc/poudriere.d/<setname>-make.conf%%'', thatis for example ''%%/usr/local/etc/poudriere.d/py27-make.conf%%'' and source it too, **additionally**. Which means that individual repository-specific ''%%make.conf%%'' files should only override whatever is given in the ''%%make.conf%%'' files sourced **first**.

----

{{tag>HowTo Ports}}