What do to about /usr/local in toolchains

The question about what to do about adding /usr/local/{include,lib} to various toolchains' search paths. A poll about what the project and its users desires can be found here. It shows there's much interest in changing the status quo.

Two main things are driving this effort. First, there's a desire to make things more consistent. /usr/local/lib is added to the dynamic library search path, but not to the static libraries. The system compiler doesn't add /usr/local to its paths, but the ports-built compilers add it, but in an inconsistent way to how the ports-built binutils. We're also inconsistent with other systems that add it.

Second, there's a desire for the programming environment to be consistent with other systems. There's a desire to have the following steps work like other systems:

1. pkg install foo
2. cc prog-using-libfoo.c -lfoo
3. ./a.out

On other systems, this just works. Either because all packages are installed in /usr, or because the system toolchain looks in /usr/local (or where ever foo winds up).

People have been talking about this issue in the project for literally decades. The time is right to stop talking and pick a path forward.

It should be noted that LOCALBASE and /usr/local will be used interchangeably in this document. They are the same thing and reviewers shouldn't nit-pick unless it actually matters. I'll note when they aren't actually the same thing.

The current arrangement of how the toolchains in the system is a bit of a dog's breakfast.

1. gcc and clang both have similar include search paths. /usr/include is in both of them by default. /usr/local/include is not.
2. ld and clang both have /lib and /usr/lib in their search paths. /usr/local/lib is not searched by default.
3. ld.so searches /lib and /usr/lib by default. /etc/rc.d scripts adds /usr/compat/lib, /usr/local/lib and /usr/local/lib/compat/pkg. These libraries are added after /usr/lib.
4. The gcc port adds ${LOCALBASE}/include before /usr/include.
5. The binutils port adds ${LOCALBASE}/lib after /usr/lib
6. Something, and I've not found where, is also adding libraries added by compiler parts.
7. LOCALBASE is /usr/local. All packages are installed relative to this root.

A number of problems exist with the current status quo.

1. Static libraries are handled differently than dynamic ones.
2. System compiler's include path is inconsistent with ports-built compiler's include path.
3. System library path in ld is inconsistent with the ports-built ld.
4. The search order for includes in the ports-built compilers is different than the search order for the libraries in the base system and the ports-built binutils.
5. Needing to add -I/usr/local/include and -L/usr/local/lib to compilation lines for programs not using the ports system is a major friction point for some users.
6. The ports system can redefine LOCALBASE to any path, which adds an additional layer of complexity for some external build systems.

In FreeBSD, by default, we include /usr/local/lib (not influenced by LOCALBASE) in the dynamic library search path. The default path is not set in the dynamic linker itself, but via the ld_config_paths series of variables. So dynamic libraries are found by default, but static libraries are not.

When building with a ports-built compiler, ${LOCALBASE}/include is searched before /usr/include. When building with the system compiler, /usr/local/include is not searched. This can lead different builds to fail when CC=cc (from the base) versus CC=gcc50 (eg, from ports).

Ports-built ld from the binutils port searches ${LOCALBASE}/lib for all libraries, while the system effectively searches just for dynamic libraries. The base system doesn't respect LOCALBASE setting for this. The ports-build ld only respects LOCALBASE at build time, not at run time.

There's a disconnect with the ports-built compilers' include path (which is before, so overrides the base system) and the base system and ports-build binutils library path (which is after, so does not override the base system). This is wrong and should be sorted out. There's a longer history of after, but that makes it hard to setup situations where ports override base system components.

Other systems don't require adding -I/usr/local/include for packages that are installed. Ours is an odd-man out in requiring it. Some do this by installing into /usr instead of /usr/local. Some do this by including the path in the compiler. We do the worst of both worlds from a convenience point of view. It's a purer environment, but that purity comes at the cost of much friction for independent software developers. In addition, when adding ports, often times this is an extra step required to integrate it into the ports tree.

The ports system allows LOCALBASE to be changed from its default of /usr/local to something else. This creates an additional layer of complexity for doing ports. It also poses challenges to integrating /usr/local into the base. LOCALBASE is only evaluated at build-time for ports. The base system would need to grow knowledge of LOCALBASE and people that define it may need to move it from a ports-specific config file to the global config file, or also added a copy to src.conf.

TDB