Building on non-FreeBSD hosts

Currently, FreeBSD can only be built on a FreeBSD host. However, most CI systems do not include FreeBSD build slaves by default. If we allow FreeBSD to be compiled on a non-FreeBSD host (current target is Linux and macOS) it would be much easier to use these CI systems. Furthermore, it would allow developers who do not have a FreeBSD system at work to use that to work on FreeBSD.

• Some build tools depend on host headers being compatible with target headers (e.g. localedef). This would require some refactoring of the xlocale headers to fix properly.
• The version of clang contained in the FreeBSD source tree cannot be bootstrapped with the makefiles from base since the config headers depend on FreeBSD
  • Could build the bootstrap clang using CMake+ninja (which should also be much faster than the current Makefile solution), but that will be a lot of effort
  • Current solution is to use upstream clang
    • However, some features such as ifunc-noplt are not in upstream clang
• The build system often falls back to invoking binaries in $PATH
  • This can cause strange build errors if that binary was not bootstrapped and an incompatible versions is executed
  • can also happen when building on FreeBSD (e.g. running dtc from an old FreeBSD version)
  • Building with a clean $PATH D16815 fixes this problem since it requires all bootstrap tools to be listed explicitly

A mostly working prototype can be found at github.com/CTSRD-CHERI/cheribsd in the branch freebsd-crossbuild. This branch is regularly merged with HEAD.

Since Linux and macOS don't include a version of bmake by default, building should be done using the script tools/build/make.py which will bootstrap bmake before attempting to build. Additionally it will set the required XCC,XLD etc variables that are needed to build with an ExternalToolchain. If you have a recent version of bmake installed using homebrew or another package manager you can also set the required variables manually (the easiest way to obtain them is by running tools/build/make.py --debug)

Usage:

MAKEOBJDIRPREFIX=... ./tools/build/make.py buildworld TARGET=foo TARGET_ARCH=bar ...

tools/build/make.py should work out-of-the box on macOS with clang and libarchive installed via homebrew:

brew install clang libarchive

You will need a recent version of clang (ideally 10.0) and the libarchive + libbz2 headers.

See tools/build/cross-build/docker/<distribution>/Dockerfile for the required packages for some distributions.

The GitHub repo also contains a script tools/build/cross-build/docker-make.py that will attempt to build inside a docker container. It also contains dockerfiles with clang+other build tools preinstalled for openSUSE, Ubuntu, Arch Linux.

MAKEOBJDIRPREFIX=... ./tools/build/cross-build/docker-make.py --local-docker-image <opensuse|ubuntu|arch> buildworld TARGET=foo TARGET_ARCH=bar ...