Differences

This shows you the differences between two versions of the page.

--- summerofcode2019projects:portsseparatedbuild [2019/06/19 18:47] – TheronTarigo
+++ summerofcode2019projects:portsseparatedbuild [2019/08/26 16:57] (current) – TheronTarigo
@@ Line 1: / Line 1: @@
 ====== Separation of Ports Build Process from Local Installation ======
+%%GSoC%% 2019 project
   * Student: [[/TheronTarigo|TheronTarigo]]
   * Mentor:  [[/BakulShah|BakulShah]]
-===== Overview =====
+===== Introduction =====
-This project aims to provide the capability of the %%FreeBSD%% ports infrastructure to safely and cleanly build ports and all their dependencies without superuser privileges, jails, or touching the installed system in anyway, in the interest of improving the safety, reliability, and repeatability of ports building without the administrative and resource overhead of a separated build host or jail.
+The %%FreeBSD%% ports framework currently makes it possible, in a single command,
+to build a third-party software package and all of its dependencies.  However,
+that dependency installation mechanism has the limitation that dependencies
+must be fully installed onto the build system before building of ports that
+require them can proceed.
-This will bring the building capability of ports tree in line with what is already possible for %%FreeBSD%% base system: separate build vs. install workflow.
+In general, there is no way for the mainstream ports framework to complete all
+work within a temporary location without frequently interrupting the process to
+modify the build host to install these dependencies.
-Currently this level of separation can only be accomplished in practice through chroot or Jail.  This project will eliminate the need for cooperation of the root user since /usr/local will not need to be touched.  To enable this, file accesses will be redirected to a location controlled by the ports build process through use of a userspace library to intercept file accesses.
+Modifying the build host in the middle of a build is often undesirable and
+unsafe.  To avoid this, existing port-building tools such as Poudriere and
+Synth use jails to accomplish isolation.  However, due to their extensive scope
+of features, these tools are not designed for nor appropriate for integration
+into Mk/bsd.port.mk as a standard feature of the ports tree.
-This implementation takes the possibility of cross-building of ports into consideration as a long-term goal (far beyond end of %%GSoC%% due to the significant challenges and history of ports not taking this into consideration).
+The Ports Separated Build project enables building of ports in isolation by
+staging all tools, include files, and shared objects into a temporary location
+used exclusively by the ports build process, eliminating the need to install
+intermediate dependencies onto the build host.  It enables this through changes
+to the scripts and makefiles provided with the ports tree in /usr/ports/Mk/
+rather than through a separate tool.  It does not rely on Jails or other
+privileged features.
-Where possible, the need for the userspace file interception hack should be eliminated, by switching to use of build tools supporting sysroot (this is analogous to desirability of sysroot-capable compiler vs. user-mode emulation for cross-arch compilation).  The interception tool serves to cover otherwise uncooperative software.
+In this way, it is made to behave more like the %%FreeBSD%% src tree, in which
+"buildworld" is a self-contained and self-sufficient process requiring no
+special preparation of the build host.
-===== Technical Challenges =====
+Currently, this works for 97% of the ~4200 ports tested.  Remaining obstacles
+to a general solution working for 100% of ports are discussed later in this
+document.
-Running the entire ports framework in a chroot (as is trivially possible) defeats the purpose of the project, since it only moves the problem of isolating each port's build environment from a physical host to a virtual one: Ports framework itself, not just a blanket chroot created by system administrator, should configure the build-environment of each port.
+===== Code =====
-The commands needed for the ports infrastructure are logically separated as follows:
+The changes to the ports tree to enable this are published at:
-  * Ports-framework commands, such as dependency calculation, fetching(*), patching(*), and packaging
-    * (*) fetching and patching may require tools from ports in unusual cases
-  * Port-building commands, such as configure, make, (staged)install, tests
-However, the existing implementation of the ports framework does not make this a clear distinction, since no such distinction has been needed for the existing capabilities of the ports framework.  The distinction becomes important when the building processes of ports must occur in a specially prepared environment managed by the ports framework itself.  Places where setup of this environment is needed roughly follows existing usage of ${CONFIGURE_ENV} / ${MAKE_ENV} / etc.
-===== Current Status =====
+  * https://github.com/therontarigo/freebsd-ports/tree/separated
+  * Difference with respect to latest pull from upstream: https://github.com/freebsd/freebsd-ports/compare/b1282523e949c12df991cd746e053b4de8b6dfc9...therontarigo:separated
-==== Ports building scheme ====
+The only major *new* piece of software involved is a userspace library for
+manipulating an unsuspecting program's filesystem namespace, redirecting file
+paths as configured by an environment variable.  It is published in its own
+repository:
+  * https://github.com/therontarigo/freebsd-user-namespace
+It is available as a port: devel/userns (found in the freebsd-ports repository
+above)
+===== Working status =====
+It is possible to perform the following (without root):
+%%%%''%%$ git clone https://github.com/therontarigo/freebsd-ports -b separated --depth 1%%''%%%%
+%%%%''%%$ cd freebsd-ports%%''%%%%
+%%%%''%%$ make PORTS_SEPARATED_BUILD=1 PORTSDIR=$PWD PORTBLDBASE=$HOME/ports -C <category>/<portname> config-recursive package-recursive %%''%%%%
+It will be more telling of actual success if this is done on a system or jail
+that doesn't already have any ports installed to /usr/local (or instead use a
+different LOCALBASE) since otherwise there is a chance that something accessed
+the real /usr/local when looking for a dependency (only a controlled location,
+defaulting to ${PORTBLDBASE}/depsroot, is meant to be accessible for
+dependencies in PORTS_SEPARATED_BUILD mode).
+To the extent that building succeeds (the scheme is known to be compatible with
+some ports and not with others), the resulting packages are placed into
+$HOME/ports/packages/All.  They can then be installed (as superuser) onto the
+system.
+From the user's perspective, the most significant difference is that if this
+were done from the upstream ports framework, the process would be interrupted
+several times to install dependencies into the real /usr/local after requesting
+permission.
+===== Background and Motivations =====
+(this section will be of little interest to anyone already familiar with the
+history and established behavior of the Ports framework).
+  * Want: Given only a POSIX-compliant environment (of any architecture), without root or any other special privilege, a large collection of software is buildable from sources for any supported architecture without manual intervention and configuration.  Actual usability of software is expected to vary depending on target environment.
+  * Have: %%FreeBSD%% ports collection (as-is):
+    * Large collection of software buildable from sources: yes
+    * Without manual intervention: generally yes
+    * Without manual configuration: yes
+    * Given only a POSIX-compliant environment: NO
+    * Without root or any other special privilege: NO
+    * For another architecture (or otherwise binary incompatible system): generally, no
+Now, much work has gone into compatibility with BSD environment (not strictly
+POSIX) and to make use of BSD/%%FreeBSD%% extensions where they are useful.
+Therefore the POSIX-compatibility goal can be reasonably relaxed to a goal of
+requiring only %%FreeBSD%% base.  (A POSIX-compatible bootstrapper to set up the
+BSD tools would bridge this gap).
+Most ports require more than just a %%FreeBSD%%-base environment: they need some
+additional tools to be installed and configured.  Special privileges must be
+granted by the OS to accomplish this modification to the environment:
+Installation of packages (Ports tree default) or management of Jails (Poudriere
+and similar tools).
+POSIX-compliant environment (or %%FreeBSD%% base) is only a starting point; build
+systems (including ports framework itself) can create a modified environment
+when needed.  It is the kind of modification that needs to be made that
+determines whether special system privileges and features are required.
+Compare this limitation of the existing ports framework to the base system
+source tree:
+  * Tools executed: found in base system installation.
+  * Files read: /usr/src (can be changed)
+  * Files created: /usr/obj (can be changed)
+  * For another architecture: YES
+  * It is sufficiently self-contained that adding the capability to build the base system under a different host OS is a reasonable possibility.
+===== Implementation details =====
+==== Ports building scheme overview ====
   * Ports are built using PORTS_SEPARATED_BUILD=1 set (otherwise normal ports behavior is left alone as the default).
   * Port building is started from unprivileged user.  All compilation work is run as this user.
-    * Superuser is requested for configuring and switching into chroot (switches back to user after entering chroot).  This will be obviated by integration of userspace file redirection tool.
   * Ports built as dependencies are installed to a working directory, not system-wide.
   * Dependency resolution is patched to use this working directory.
   * All port-building commands (configure scripts, compilers, interpreters) run under file redirection to see dependency install area instead of local system.
-    * Current implementation of file redirection: chroot.
-==== Compatible ports ====
+==== Section outline ====
-  * Anything using standard configure/make procedure
+  * PORTBLDROOT
-  * Builds using Autoreconf; requires patch to Mk/Uses/autoreconf.mk
+  * %%UserNS%%
-  * Builds using Cmake; requires patch to Mk/Uses/cmake.mk
+  * %%PortEnv%%
-  * USES=perl; requires patch to Mk/Uses/perl5.mk
+  * bmake
-  * USES=python; requires patch to Mk/Uses/python.mk
+  * sub-make
+  * ldconfig
+  * pkg install scripts
+  * bootstrap
-==== Incompatible ports ====
+==== PORTBLDROOT ====
-  * Anything using custom targets in port's Makefile (requires patch for each port)
+Required dependency ports are installed to a temporary location rather than
+into the root of the build host.  The temporary location is specified by the
+PORTBLDROOT make variable.
+==== UserNS ====
+To address the need of ported software to find dependencies in expected
+locations under ${LOCALBASE} without performing the real installation to this
+location, a library providing a virtual filesystem namespace has been developed
+as the first phase of this project.
+"%%UserNS%%" intercepts an unsuspecting program's C library API calls to rewrite
+pathnames in file operations, giving the appearance to the application of a
+filesystem namespace other than that of the host system.
+It is included in the modified ports tree as devel/userns.
+The most comparable previous work is the Linux-only "fakechroot" library.  The
+"fake chroot" functionality is only a special case of %%UserNS%%'s capability,
+which allows creating a namespace as an aggregate of arbitrarily many locations
+in the real filesystem.
+%%UserNS%% is controlled through an environment variable.  Its basic usage within
+the ports framework is:
+''%%FILEPATHMAP=${LOCALBASE}%${PORTBLDROOT}${LOCALBASE}:/%/%%''
+This is a colon-separated list of ''%%virtual%real%%'' entries:
+  * ${LOCALBASE} is resolved to ${PORTBLDROOT}${LOCALBASE}
+  * All other file accesses are passed through as-is.
+Actual usage involves further remappings.
+==== PORTENV ====
+A new Make variable is introduced, PORTENV, to expand to the command-line
+prefix needed to run a command inside the virtual file namespace facilitated by
+%%UserNS%%.
+${PORTENV} defaults to ${SETENV} when separated-build mode is not used.
+==== bmake ====
+%%FreeBSD%%'s make, /usr/bin/make, is a statically linked executable and therefore
+cannot utilize %%UserNS%%.  This is not a problem for the ports framework itself
+but when ports internally require a Make utility for building, %%UserNS%% should be
+used so that Makefile file-existence checks work correctly.
+%%FreeBSD%%'s built-in make is imported from %%NetBSD%% Make.  The %%FreeBSD%% port
+devel/bmake is a dynamically-linked alternative, with the only functional
+difference being that it uses its own (%%NetBSD%%-style) makefile includes in
+/usr/local/share/mk instead of %%FreeBSD%%'s /usr/share/mk.  This is easily
+corrected by a wrapper script in Mk/Exec/make.  %%UserNS%% redirects /usr/bin/make
+to this wrapper so that it is always used instead by any port attempting to use
+"make".
+Many ports use devel/gmake (GNU Make) instead, which is also compatible with
+%%UserNS%%.
+==== sub-make ====
+Ports themselves do not know about ${PORTENV}.  Make does not provide a
+reliable mechanism for running all target commands with an alternate shell.
+Usage of ${SETENV} by ports' targets is inconsistent, and MAKE_ENV /
+CONFIGURE_ENV etc. do not capture all cases.
+Instead, any port-defined targets (typically pre-* post-* do-* ... *-configure
+*-build *-install etc.) are run in a Make subprocess which itself executes
+under the PORTENV environment.
+==== ldconfig ====
+One of the requirements for PORTENV is that it enables execution of programs
+installed into PORTBLDROOT.  This requires a ld-elf.so.hints file that is
+correct for that environment.
+%%FreeBSD%% uses /etc/rc.d/ldconfig to maintain the system-default
+/var/run/ld-elf.so.hints.
+A new script, ${PORTSDIR}/Mk/Scripts/ldconfig.sh provides similar
+functionality, processing the library directories under ${PORTBLDROOT} to
+maintain ${PORTBLDROOT}/var/run/ld-elf.so.hints.  PORTENV sets
+LD_ELF_HINTS_PATH to use this hints file.
+==== pkg install scripts ====
+Some packages contain post-install scripts which must be run for the package to
+be properly installed.  PORTENV is used to run these scripts when packages are
+installed to ${PORTBLDROOT} such that the operations of the scripts apply to
+files within ${PORTBLDROOT}.
+Use of /usr/sbin/service from post-install scripts is specially handled.  It is
+remapped by %%UserNS%% to ${PORTSDIR}/Mk/Exec/service, which supports only the
+services which are deemed relevant to proper installation of port-building
+tools.  Currently, the only service is "ldconfig", which executes
+${PORTSDIR}/Mk/Scripts/ldconfig.sh described above.
+==== bootstrap ====
+The ports tree is expected to install any tools it needs which do not come from
+the base system.  The official ports tree as-is installs ports-mgmt/pkg before
+building any port, since it is an essential part of the ports build system's
+ability to check dependencies and to install ports once they are build.
+This ability to bootstrap the pkg tool is extended to also build devel/bmake
+and devel/userns and prepare them within ${PORTBLDROOT}, since these are
+essential components of separated-build mode functionality.
+===== Causes of incompatibilities with the separated-build scheme =====
+  * Failure to use PORTENV.  If a port somehow executes a command outside of this environment, %%UserNS%% is not used at all and virtual file accesses will fail.  Generally this is always fixable by changes within Mk/ files.
+  * Breakage of PORTENV environment.  LD_PRELOAD and other variables must be set correctly for %%UserNS%% to function.  If a port's script attempts to execute programs after changing these variables, those programs may fail to run or fail to resolve file locations as expected.
+  * Incompatibility with %%UserNS%%.  %%UserNS%% relies on LD_PRELOAD to override %%LibC%% functions.  Its handling of paths works in many cases but remains buggy for special cases.  Therefore %%UserNS%% is not always a working substitute for actual installation of files into a location in which a port expects to find them.
+  * Usage of statically linked executables.  %%UserNS%% has no effect on these, so when a port uses such an executable to access files from the port's dependencies, it fails.
+  * Usage of full paths in C Preprocessor includes.  The base toolchain, including the preprocessor, is statically linked.  Command-line include and library paths are rewritten to use the temporary dependency location but absolute #include paths bypass this.  In practice there is not a good reason for ports to use these paths and they can be corrected to avoid doing this.
+    * If lookup of %%UserNS%%-provided paths from the compiler is absolutely required, there are a few possibilities:
+      * Including a dynamically-linked compiler toolchain in the basic %%FreeBSD%% install requires a change to %%FreeBSD%% base.
+      * A dynamically-linked toolchain can be built from ports (as is done with devel/bmake), however: Clang is a large piece of software.  Rebuilding it from ports is unacceptable for some users, and would be a major annoyance.
+      * Developing a Clang extension integrating %%UserNS%% file lookup into the compiler is a potential solution.
+      * Use one of the previous two solutions only for ports which are marked as requiring it.
+===== Needed work =====
+The current status of the project is that it is usable for some but not all
+ports.  Much work remains before it can be adopted officially into the ports
+tree:
+  * Verification of non-breakage of conventional ports-build process
+    * Before any changes can be merged into the official ports tree, they must be verified to not break or change the existing and well-established behavior.
+      * Wherever possible, modification of existing Mk/ files are either:
+        * Insertions of references to variables which default to being empty.
+        * Insertions of code blocks guarded by conditionals which default to not using enabling the added code.
+      * This is intended to eliminate chances of breakages when the separated-build mode is *not* used.  The intention is that it can coexist with the conventional system as an optional, sometimes-useful mechanism, available as a tool for ports that are already compatible, and as a starting point for future work on expanding its compatibility.
+  * %%UserNS%% bugs
+    * Relative paths are not translated.
+    * "/../" in pathnames prevent correct translation of paths.
+    * Symbolic links to virtual locations in the namespace do not work.
+      * Note that realpath(3) does work as expected within %%UserNS%%; symlinks and ".." are resolved correctly.  Utilizing realpath(3) before all virtual -> real path translations appears to provide correct behavior for basic file operations, but introduces a bug that symlinks become indistinguishable from normal files and directories, and the extra system interaction required creates a performance regression.
+    * %%UserNS%% works generally only for conventional POSIX API file operations provided with absolute paths.
+      * Luckily, tools used in the building of software use this kind of file access almost exclusively.
+      * Rather than bring %%UserNS%% up to full general correctness, it may be more sensible to rebase it on the global namespace emulation provided by libpreopen.  This potentially has the very desirable side-effect that all building work currently working under %%UserNS%% can be sandboxed with Capsicum.
+        * libpreopen: https://github.com/musec/libpreopen
+    * %%UserNS%% requires /usr/lib/debug/libexec/ld-elf.so.1.debug to exist so that the address of the open(...) function within ld-elf.so.1 is known, which is used to live-patch it to use the %%UserNS%%-provided open(...) call. This requires %%FreeBSD%% to be installed with debugging information included. One way to relax this limitation would be to maintain a table of ld-elf.so.1 file hashes to appropriate addresses for the versions of this file from all recent %%FreeBSD%% releases.
+  * Ports tree
+    * A large block of variable definitions was added to the top of Mk/bsd.port.mk.  This needs to be moved to its own file.  Most of these additions are processed more than once due to their misplacement within Mk/bsd.port.mk, but this can be more easily fixed once the additions are moved to a separate file.  Another run of ports building is needed to verify that this refactoring doesn't break the progress that has been made.
+    * ''%%make install%%''
+      * Presently this target is overridden to provide the means of preparing dependencies in the PORTBLDROOT location.  It should be restored to its original functionality of actually installing the built port onto the host system, requesting superuser privileges as needed.
+    * Concurrent building
+      * By using a different PORTBLDROOT location for each port, it is now possible to safely build multiple ports concurrently.  However, the core dependency-handling scripts and makefiles likely need to be rewritten to take advantage of this.
+    * Patching all Mk/Uses/*.mk
+      * By design of the "portenv" mechanism, port-defined targets are always executed within the prepared environment, while target commands of makefile-includes in Mk/ must explicitly use ${PORTENV} where needed. Some Mk/Uses/*.mk have been amended accordingly; several remain to be done.
+===== What could have been done differently =====
+Possible alternatives to %%UserNS%% approach:
+  * Build all build tools for the host machine, using temporary location as prefix (but then existing packages can't be used for these since they use default prefixes)
+    * This supposes there is only one location in the ports-building process used for installing build tools.  What if two unrelated ports' build tools conflict?  What if the presence of one build tool subtlely influences the behavior of another in a way which is difficult to debug?
+    * Depending on user needs, each build tool may need to be built twice: once for port-building system (alternative prefix) and once for direct installation onto system (default prefix).
+  * Rebuild and repackage any build tool (any port which needs to run to build another port) to be relocatable i.e. can be installed to and run from any location.
+    * This has additional benefit of easing the process of debugging conflicts between a piece of software and changes to the tools it requires.
+    * The resulting package is appropriate both for usage on the build machine and for final installation on the target machine.
+    * However:
+      * Adding this capability may be infeasible for some tools.
+      * Requires much work rewriting port Makefiles for these tools.
+      * Maintenance burden is increased if patches must be used.
+  * Use cross-sysroot for locating all dependency files (not tools)
+===== Future direction: cross-compilation support =====
+Although only tangential to the primary results of the project, one of the
+deliverables as a side-effect is the following collection of notes on adding a
+generally working cross-compilation capability to the ports framework.
+When cross compiling, there is an important distinction between two different
+kinds of build-time dependency:
+  * Library dependencies: provide files needed by the cross-toolchain to build the port.
+  * Tool dependencies: provide executables which need to be run on the build machine to do the work of building the port.
+%%FreeBSD%%'s ports tree did not initially record these distinctions in ports'
+build-time dependency information.
+%%NetBSD%%'s pkgsrc system, similar to and inspired by %%FreeBSD%%'s ports, has slowly
+been solving this problem by adding a new type of dependency, TOOL_DEPENDS.
+This is similar to BUILD_DEPENDS except that it identifies a port providing
+commands which must be executed on the build host.
+%%FreeBSD%% porting convention already provides a way to record this information,
+however in a way which might not be as obvious: the indication that a port
+depends on a program for building is an entry of the form:
+''%%BUILD_DEPENDS+=  progname:category/portname%%''
+Many ports already make use of this.  Unfortunately this convention is not
+always adhered to.  By enforcing it as a rule, a port's attempts to use a
+port-provided program at build-time without specifying it appropriately in
+BUILD_DEPENDS can be considered as a bug in the port and fixed accordingly.
+One way to enforce this is to remove ${LOCALBASE} from PATH and replace it with
+the path to a directory created by the ports framework containing only symlinks
+to the programs which have actually been registered correctly as dependencies.
+This needs to be done recursively so that run-depends of those programs are
+also satisfied.
+PORTBLDROOT from the ports-separated-build project will need to be split into
+two different kinds of root, each with a specific purpose:
+  * Executable tool root.  Contains the host-architecture executables required, and all support files needed to run them.
+  * Target system root.  Contains the library objects, headers, etc. for the target architecture.  An overlay of this root (containing only what is supplied by ported dependencies) on top of a %%FreeBSD%% installation root for the target architecture is used as the sysroot of the compiler toolchain for cross-building of ports.
+====== GSoC 2019 ======
+The original goals of the %%GSoC%% project, followed by weekly updates made during
+the project, are preserved below.
+===== Project Objectives =====
+This project aims to provide the capability of the %%FreeBSD%% ports infrastructure
+to safely and cleanly build ports and all their dependencies without superuser
+privileges, jails, or touching the installed system in any way, in the interest
+of improving the safety, reliability, and repeatability of ports building
+without the administrative and resource overhead of a separated build host or
+jail.
+This will bring the building capability of ports tree in line with what is
+already possible for %%FreeBSD%% base system: separate build vs. install workflow.
+Currently this level of separation can only be accomplished in practice through
+chroot or Jail.  This project will eliminate the need for cooperation of the
+root user since /usr/local will not need to be touched.  To enable this, file
+accesses will be redirected to a location controlled by the ports build process
+through use of a userspace library to intercept file accesses.
+This implementation takes the possibility of cross-building of ports into
+consideration as a long-term goal (far beyond end of %%GSoC%% due to the
+significant challenges and history of ports not taking this into
+consideration).
+Where possible, the need for the userspace file interception hack should be
+eliminated, by switching to use of build tools supporting sysroot (this is
+analogous to desirability of sysroot-capable compiler vs. user-mode emulation
+for cross-arch compilation).  The interception tool serves to cover otherwise
+uncooperative software.
+==== Goal #1 - working isolated build ====
+User (no special privileges, no ahead-of-time preparation by system admin) may
+do:
+''%%make -C /usr/ports/<category>/<port> package%%''
+-> The port and its dependencies are built and installed within a pristine
+environment (File access attempts to /usr/local never occur).
+-> The packaged port is identical (with exception of dates and hostnames) to
+the package from official %%FreeBSD%% repository, assuming the same SVN revision
+has been used.
+==== Goal #2 - efficiency & performance ====
+The building as done according to the first goal is done efficiently.
+==== Goal #3 - concurrency ====
+Independent ports are built concurrently.
+===== Project Deliverables =====
+  * File path redirection library, capable of emulating chroot behavior on any dynamically-linked executable (BSD replacement for "fakechroot") (Nominally completed for amd64 arch; bugs may be found)
+  * bsd.port.mk coordinates usage of temporary dependency location (Completed)
+  * Most ports are buildable without any interaction with real ${PREFIX} (In progress)
+  * Packages built by modified process are identical to packages build by existing ports framework
+===== Technical Challenges =====
+Running the entire ports framework in a chroot (as is trivially possible)
+defeats the purpose of the project, since it only moves the problem of
+isolating each port's build environment from a physical host to a virtual one:
+Ports framework itself, not just a blanket chroot created by system
+administrator, should configure the build-environment of each port.
 ===== Weekly updates =====
 ==== 06/03 ====
   * Familiarized myself with the port dependency mechanisms in /usr/ports/Mk/Scripts.
   * Hacked on bsd.port.mk to check assumptions and to get an idea of what additions will be needed.
-  * Studied /usr/src/lib/libc to understand chain of userspace calls between C API and system call. Implemented proof-of-concept redirection library intercepting open(...).
+  * Studied /usr/src/lib/libc to understand chain of userspace calls between C API and system call. Implemented proof-of-concept redirection library intercepting open(...).
 ==== 06/10 ====
-  * Continued to work on the file access interception library.  It is working well enough to redirect file accesses from shell and basic tools: sh, cat, ls - more thorough testing will be needed.
+  * Continued to work on the file access interception library.  It is working well enough to redirect file accesses from shell and basic tools: sh, cat, ls - more thorough testing will be needed.
 ==== 06/18 ====
   * Modified bsd.port.mk to behave as follows:
     * All built dependencies of a port are installed to ${PORTBLDROOT} (=${PORTSDIR}/build on my test system).
     * Ports resolve dependencies to ${PORTBLDROOT} instead of to local system.
     * All port-building operations (configure, make, make install, etc.) are run in a chroot, in which ${LOCALBASE} is mapped to ${PORTBLDROOT}${LOCALBASE} and base system is mapped as-is.
   * Added hooks to several Mk/Uses/* and ports to allow switch to chroot where appropriate (far from complete).
   * Successfully compiled handful of ports (including C libraries, GNU build tools, perl modules, and python modules) under this modified system.
+  * Next stage:
+    * Replace the chroot with an equivalent solution in userspace (filesystem namespace redirection), to eliminate requirement of superuser intervention.
+==== 06/24 ====
+  * Switched bsd.port.mk from using a chroot for namespace manipulation to using the userspace namespace tool.
+  * Enabled namespace tool to read path mappings from environment, supplied by bsd.port.mk
+  * Re-ran port builds.  Several problems which had not occurred when using chroot became introduced.
+  * Studied interactions between executables, namespace tool, dynamic linker, and standard library to understand the cases in which file accesses fail to be intercepted.
+  * Next stage:
+    * Research, design, implement, and test appropriate solutions for the problems encountered with userspace file access interception.
+==== 07/01 ====
+  * Filesystem namespace intercept tool now redirects paths to shared object files read by rtld.
+  * Fixed several problems with running programs under the namespace tool.
+  * Developed a plan for providing correct ldconfig behavior for programs run as part of port building.
+  * Next:
+    * Implement ldconfig fix.
+    * Create a port of freebsd-user-namespace to be used by ports framework, similarly to how ports-mgmt/pkg is used.
+    * Set up automated testing of ports to assess progress and catch regressions.
+==== 07/09 ====
+  * Ldconfig hints file is maintained at each dependency port installation, allowing build tools to run as expected.
+  * Fixed several more problems with running programs under the namespace tool:
+    * Some ports contain /bin/sh scripts with no shebang.  %%FreeBSD%% allows this; Namespace tool's execve implementation is changed accordingly.
+    * Fixed a bug in execve implementation concerning argv[0]
+    * Debugging output, when requested, now does not get written to wrong file after (v)fork and/or exec.
+  * Have ports use devel/bmake instead of /usr/bin/make, where possible, as the latter is statically linked and cannot handle ports Makefiles' dependencies on installed files.
+  * Confirmed that devel/llvm60 and all its dependencies build successfully. This selection of ports represents many build tools commonly used in building of other ports.  To see for yourself: make -C /usr/ports/devel/llvm60 all-depends-list
+==== 07/15 ====
+  * Created a port of freebsd-user-namespace: devel/userns.
+  * Integrated devel/userns into PORTS_SEPARATED_BUILD mode of ports framework.
+  * freebsd-user-namespace looks up addresses from /usr/lib/debug to live-patch ld-elf.so.1 where needed.
+  * Improved integration of devel/userns and devel/bmake into bsd.ports.mk such that these tools are created when required by PORTS_SEPARATED_BUILD mode.
+  * Confirmed that devel/llvm60 and all its dependencies build successfully, this time in a fresh %%FreeBSD%% 12.0-RELEASE installation with https://github.com/therontarigo/freebsd-ports -b separated as /usr/ports. This selection of ports represents many build tools commonly used in building of other ports: python, perl, lua, GNU, CMake, ...
+==== 07/22 ====
+  * Eliminated the need for any special setup to be able to use PORTS_SEPARATED_BUILD.  The mode may be enabled by this one variable, at the command line or in /etc/make.conf.
+  * Created a lightweight framework for testing large numbers of ports under the separated build mode.  It does the following:
+    * Handles FLAVORS.
+    * Saves success/failure/skipped status.
+    * Skips ports which have failed dependencies.
+    * Builds each port in its own pristine environment.  (This allows concurrent building of unrelated ports)
+    * Saves success/failure/skipped status.
+    * Skips ports which have failed dependencies.
+    * Builds each port in its own pristine environment.  (This allows parallel (concurrent) port build to work)
+  * Attempted build of 2000 ports, selected at random, and their dependencies.
+  * Identified (and applied temporary fixes for) some ports causing highest number of skipped dependents.
+  * Retried the skipped ports.
+  * Developed a plan of a general fix for the most common cause of failure: custom target definitions in port Makefiles running build tools (also from ports) without the userns wrapper.
+==== Latest summary of port building success ====
+  * 5613 ports tried
+  * 2958 (53%) succeeded
+  * 177 (3%) failed
+  * 2478 (44%) skipped due to failed dependencies
+  * Success rate: 94%
 ===== Timeline =====
+(Not updated)
 | Week | Task | Progress |
@@ Line 70: / Line 620: @@
 | 06/03 - 06/07 | Implement file path redirection library and test against common tools | File redirection library is working for basic file operations performed by common shell utilities. |
 | 06/10 - 06/14 | Modify bsd.port.mk (and related) to utilize install location and redirection lib | Working branch of freebsd-ports utilizes a virtual install location, and some ports are able to build using dependencies installed to this location.  See above. |
-| 06/17 - 06/21 | Demonstrate successful building of a port with dependencies |  |
+| 06/17 - 06/21 | Demonstrate successful building of a port with dependencies | Ports with dependencies function under chroot, not under userspace file tool. |
 | 06/24 - 06/28 | Document and present findings |  |
 | 07/01 - 07/12 | Identify ports using statically linked build tools.  Add option for building such tools to include file path redirection support. |  |