Separation of Ports Build Process from Local Installation

• Student: TheronTarigo
• Mentor: BakulShah

This project aims to provide the capability of the FreeBSD ports infrastructure to safely and cleanly build ports and all their dependencies without superuser privileges, jails, or touching the installed system in anyway, in the interest of improving the safety, reliability, and repeatability of ports building without the administrative and resource overhead of a separated build host or jail.

This will bring the building capability of ports tree in line with what is already possible for FreeBSD base system: separate build vs. install workflow.

Currently this level of separation can only be accomplished in practice through chroot or Jail. This project will eliminate the need for cooperation of the root user since /usr/local will not need to be touched. To enable this, file accesses will be redirected to a location controlled by the ports build process through use of a userspace library to intercept file accesses.

This implementation takes the possibility of cross-building of ports into consideration as a long-term goal (far beyond end of GSoC due to the significant challenges and history of ports not taking this into consideration).

Where possible, the need for the userspace file interception hack should be eliminated, by switching to use of build tools supporting sysroot (this is analogous to desirability of sysroot-capable compiler vs. user-mode emulation for cross-arch compilation). The interception tool serves to cover otherwise uncooperative software.

User (no special privileges, no ahead-of-time preparation by system admin) may do:

make -C /usr/ports/<category>/<port> package

→ The port and its dependencies are built and installed within a pristine environment (File access attempts to /usr/local never occur).

→ The packaged port is identical (with exception of dates and hostnames) to the package from official FreeBSD repository, assuming the same SVN revision has been used.

The building as done according to the first goal is done efficiently.

Independent ports are built concurrently.

• File path redirection library, capable of emulating chroot behavior on any dynamically-linked executable (BSD replacement for “fakechroot”) (Nominally completed for amd64 arch; bugs may be found)
• bsd.port.mk coordinates usage of temporary dependency location (Completed)
• Most ports are buildable without any interaction with real ${PREFIX} (In progress)
• Packages built by modified process are identical to packages build by existing ports framework

Running the entire ports framework in a chroot (as is trivially possible) defeats the purpose of the project, since it only moves the problem of isolating each port's build environment from a physical host to a virtual one: Ports framework itself, not just a blanket chroot created by system administrator, should configure the build-environment of each port.

The commands needed for the ports infrastructure are logically separated as follows:

• Ports-framework commands, such as dependency calculation, fetching(*), patching(*), and packaging
  • (*) fetching and patching may require tools from ports in unusual cases
• Port-building commands, such as configure, make, (staged)install, tests

However, the existing implementation of the ports framework does not make this a clear distinction, since no such distinction has been needed for the existing capabilities of the ports framework. The distinction becomes important when the building processes of ports must occur in a specially prepared environment managed by the ports framework itself. Places where setup of this environment is needed roughly follows existing usage of ${CONFIGURE_ENV} / ${MAKE_ENV} / etc.

Userspace file redirection library

• https://github.com/therontarigo/freebsd-user-namespace

FreeBSD ports framework

• https://github.com/therontarigo/freebsd-ports branch separated
• Changes made by this project: https://github.com/freebsd/freebsd-ports/compare/master...therontarigo:separated

• Ports are built using PORTS_SEPARATED_BUILD=1 set (otherwise normal ports behavior is left alone as the default).
• Port building is started from unprivileged user. All compilation work is run as this user.
  • Superuser is requested for configuring and switching into chroot (switches back to user after entering chroot). This will be obviated by integration of userspace file redirection tool.
• Ports built as dependencies are installed to a working directory, not system-wide.
• Dependency resolution is patched to use this working directory.
• All port-building commands (configure scripts, compilers, interpreters) run under file redirection to see dependency install area instead of local system.
  • Current implementation of file redirection: freebsd-user-namespace, as port devel/userns.
• All programs invoked by name from the port Makefile's targets must be specified in BUILD_DEPENDS, explicitly or indirectly through USES. This is a good practice for port authoring; SEPARATED_BUILD mode enforces this by restricting the path so that /usr/local/bin is not used and the required commands are linked to a wrapper to run within file redirection environment.

• Anything using standard configure/make procedure
• Builds using Autoreconf; requires patch to Mk/Uses/autoreconf.mk
• Builds using Cmake; requires patch to Mk/Uses/cmake.mk
• USES=perl; requires patch to Mk/Uses/perl5.mk
• USES=python; requires patch to Mk/Uses/python.mk

• Any port which attempts to run a tool using an absolute path to ${LOCALBASE} directly from its Makefile. This requires patch for each port. Example: ${PYTHON_CMD} starts with /usr/local; a port makefile needs to use ${BUILD_PYTHON_CMD}.

• Familiarized myself with the port dependency mechanisms in /usr/ports/Mk/Scripts.
• Hacked on bsd.port.mk to check assumptions and to get an idea of what additions will be needed.
• Studied /usr/src/lib/libc to understand chain of userspace calls between C API and system call. Implemented proof-of-concept redirection library intercepting open(…).

• Continued to work on the file access interception library. It is working well enough to redirect file accesses from shell and basic tools: sh, cat, ls - more thorough testing will be needed.

• Modified bsd.port.mk to behave as follows:
  • All built dependencies of a port are installed to ${PORTBLDROOT} (=${PORTSDIR}/build on my test system).
  • Ports resolve dependencies to ${PORTBLDROOT} instead of to local system.
  • All port-building operations (configure, make, make install, etc.) are run in a chroot, in which ${LOCALBASE} is mapped to ${PORTBLDROOT}${LOCALBASE} and base system is mapped as-is.
• Added hooks to several Mk/Uses/* and ports to allow switch to chroot where appropriate (far from complete).
• Successfully compiled handful of ports (including C libraries, GNU build tools, perl modules, and python modules) under this modified system.
• Next stage:
  • Replace the chroot with an equivalent solution in userspace (filesystem namespace redirection), to eliminate requirement of superuser intervention.

• Switched bsd.port.mk from using a chroot for namespace manipulation to using the userspace namespace tool.
• Enabled namespace tool to read path mappings from environment, supplied by bsd.port.mk
• Re-ran port builds. Several problems which had not occurred when using chroot became introduced.
• Studied interactions between executables, namespace tool, dynamic linker, and standard library to understand the cases in which file accesses fail to be intercepted.
• Next stage:
  • Research, design, implement, and test appropriate solutions for the problems encountered with userspace file access interception.

• Filesystem namespace intercept tool now redirects paths to shared object files read by rtld.
• Fixed several problems with running programs under the namespace tool.
• Developed a plan for providing correct ldconfig behavior for programs run as part of port building.

• Next:
  • Implement ldconfig fix.
  • Create a port of freebsd-user-namespace to be used by ports framework, similarly to how ports-mgmt/pkg is used.
  • Set up automated testing of ports to assess progress and catch regressions.

• Ldconfig hints file is maintained at each dependency port installation, allowing build tools to run as expected.
• Fixed several more problems with running programs under the namespace tool:
  • Some ports contain /bin/sh scripts with no shebang. FreeBSD allows this; Namespace tool's execve implementation is changed accordingly.
  • Fixed a bug in execve implementation concerning argv[0]
  • Debugging output, when requested, now does not get written to wrong file after (v)fork and/or exec.
• Have ports use devel/bmake instead of /usr/bin/make, where possible, as the latter is statically linked and cannot handle ports Makefiles' dependencies on installed files.
• Confirmed that devel/llvm60 and all its dependencies build successfully. This selection of ports represents many build tools commonly used in building of other ports. To see for yourself: make -C /usr/ports/devel/llvm60 all-depends-list

• Created a port of freebsd-user-namespace: devel/userns.
• Integrated devel/userns into PORTS_SEPARATED_BUILD mode of ports framework.
• freebsd-user-namespace looks up addresses from /usr/lib/debug to live-patch ld-elf.so.1 where needed.
• Improved integration of devel/userns and devel/bmake into bsd.ports.mk such that these tools are created when required by PORTS_SEPARATED_BUILD mode.
• Confirmed that devel/llvm60 and all its dependencies build successfully, this time in a fresh FreeBSD 12.0-RELEASE installation with https://github.com/therontarigo/freebsd-ports -b separated as /usr/ports. This selection of ports represents many build tools commonly used in building of other ports: python, perl, lua, GNU, CMake, …

• Eliminated the need for any special setup to be able to use PORTS_SEPARATED_BUILD. The mode may be enabled by this one variable, at the command line or in /etc/make.conf.
• Created a lightweight framework for testing large numbers of ports under the separated build mode. It does the following:
  • Handles FLAVORS.
  • Saves success/failure/skipped status.
  • Skips ports which have failed dependencies.
  • Builds each port in its own pristine environment. (This allows concurrent building of unrelated ports)
  • Saves success/failure/skipped status.
  • Skips ports which have failed dependencies.
  • Builds each port in its own pristine environment. (This allows parallel (concurrent) port build to work)
• Attempted build of 2000 ports, selected at random, and their dependencies.
• Identified (and applied temporary fixes for) some ports causing highest number of skipped dependents.
• Retried the skipped ports.
• Developed a plan of a general fix for the most common cause of failure: custom target definitions in port Makefiles running build tools (also from ports) without the userns wrapper.

• 5613 ports tried
• 2958 (53%) succeeded
• 177 (3%) failed
• 2478 (44%) skipped due to failed dependencies
• Success rate: 94%

(Not updated)