ToDo [FreeBSD WIKI]

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
([[/MarkLinimon|MarkLinimon]] note: this is [[/AlejandroPulver|AlejandroPulver]]'s list of his priorities.  As of 20110610, it has not been updated for a while, and parts are now obsoleted by current events.)

====== Ports TODO ======

This is a recompilation of all the ports/package system related tasks wished, planned, and in progress. It includes links to the respective original documents if already exist, or not if it was my idea or if I didn't know about its existence (in that case please let me know).

I ([[/AlejandroPulver|AlejandroPulver]]) plan to work on some of these as my time permits (going to start the first year of university), as part of [[/SummerOfCode|SummerOfCode]] if possible. If you have more ideas and/or want to work on the existing items, please let me know.

The sub-projects/items are organized so they are grouped under related upper level concepts/functionalities. Recently, the contents were organized in ordered //stages//, to represent a coherent schedule (step-by-step) where features can be implemented progressively and with less breakage.

If editing this page (a clarification never hurts), please try to keep it on topic (for example, requesting a new portupgrade option would better fit into [[/portupgrade]]), include references if any, and let me know about changes. For style and markup, corrections are welcome (this is my first wiki page).

**Table of Contents**

===== Stage 1: Current status =====

This step consists in gathering knowledge about the status of the current system compared to other contemporary ones, including references. It will later also hold the improvements made so far.

==== Comparison with other systems ====

:!: TODO: in progress, use footnotes. Also use colors and text like VCS cmp.

^ Feature/subject ^ pkgsrc ^ (%%FreeBSD%%) ports ^ (%%OpenBSD%%) ports ^ mports ^ Portage ^ dpkg/rpm + apt ^
| Used by OSes | %%NetBSD%%, %%DragonflyBSD%% | %%FreeBSD%% | %%OpenBSD%% | %%MidnightBSD%% | Gentoo, Sabayon | Ubuntu, Debian, Fedora, Redhat |
| Ports | ✔ | ✔ | ✔ | ✔ | ✔ | ✘ ((There are source packages, which contain both the full source code of the program and local patches, and are manually compiled and uploaded by the maintainer.)) |
| Packages | ✔ | ✔ | ✔ | ✔ | {o} ((A Gentoo derivate called Sabayon is developing a tool called [http://en.wikipedia.org/wiki/Sabayon_Linux#Features entropy], which uses ''%%binmerge%%'' for binary packages.)) | ✔ |
| Primary focus (src/bin) | ? | src | bin | ? | src | bin |
| Packages and ports integration | ? | {o} ((Some features are only supported by ports, like versioned dependencies. Also ports install directly, without generating packages.)) | ✔ ((Ports create the package and then uses the package tools to install.)) | ? | ? | ✘ |
| Many packages out of same port | ? | {o} ((Shared code in one port, called by a slave port which only defines the required options (requires a directory with a ''%%Makefile%%'' for each package).)) | ✔ | ? | ✘ | ? |
| Apps divided in many packages | ✘ | ✘ | ✘ | ✘ | ✘ | ✔ ((Packages are split in ''%%-doc%%'', ''%%-dev%%'', ''%%-common%%'', ''%%-data%%'', etc. The good point is that packages which don't depend on the architecture are built only once, runtime dependencies don't require headers, and documentation is optional. However to build anything you need all the ''%%-dev%%'' versions.)) |
| Self-contained packages | ? | ? | ? | ? | ? | ? |
| Building in a clean environment | ✔ | {o} (tinderbox/portbuild) | ✔ | ✔ (recently) | ? | ✔ |
| ''%%PREFIX%%'' clean | ✔ | ✔ | ✔ | ✔ | ✘ | ✘ |
| ''%%CFLAGS%%'' clean | ✔ | ✔ | ✔ | ✔ | ✔ | ✘ |
| Compilation options framework | ? | ? | ? | ? | ? | ? |
| Rerrolled distfiles support | ? | ? | ? | ? | ? | ? |
| Differential updates (deltas) | ? | ? | ? | ? | ? | ? |
| ''%%MASTER_SITE%%'' priority/sorting | ? | ? | ? | ? | ? | ? |
| Hack collection / porting scripts | ? | ? | ? | ? | ? | ? |
| Parallel building | ? | ? | ? | ? | ? | ? |
| Multiple versions of the same app | {o} (dirs) | {o} (dirs) | ? | {o} (dirs) | ✔ (''%%*.ebuild%%'') | {o} (branches) |
| Multiple coexisting versions of the same app (symlink/wrapper) | ? | ? | ? | ? | ? | ? |
| Virtual ports/packages | ? | ✘ | ? | ? | ✔ | ✔ |
| Config files updating tool | ? | ? | ? | ? | ? | ? |
| Startup scripts management tool | ? | ? | ? | ? | ? | ? |
| Automatic restart of daemons | ? | ? | ? | ? | ? | ? |
| Automatic packaging list | ? | ? | ? | ? | ? | ? |
| Automatic management of users/groups | ? | ? | ? | ? | ? | ? |
| Initial configuration and integration of apps | ? | ? | ? | ? | ? | ? |
| License framework (info/agreement) | ? | ? | ? | ? | ? | ? |

**References**

^ Symbol ^ Descrption ^
| ✔ | Fully supported |
| {o} | Partially supported, workaround used or WIP |
| ✘ | Not supported |

==== References ====

  * [[http://www.freebsd.org/cgi/query-pr-summary.cgi?responsible=portmgr|Infrastructure PRs]] usually contain patches, see also closed ones which were not committed.
  * [[http://en.wikipedia.org/wiki/Comparison_of_Linux_distributions#Features|Number of packages]] of Linux distributions (note that Debian and Ubuntu split most of them in components and offer many built with different options).
  * Conference paper about package management (with comparison and ideas): [[http://people.freebsd.org/~sat/pmp/papref.pdf|PDF]] [[http://people.freebsd.org/~sat/pmp/papref|TXT]].
  * [[http://www.dinoex.de/schulungen/package-en.html|Lightweight FreeBSD package cluster in a jail]] (:!: TODO: take ideas from here).
  * [[AndrewPantyukhin/Upak]] (effort to unify packages).
  * [[http://lwn.net/Articles/240399/|Package management in Gentoo Linux]].

===== Stage 2: Clean up infrastructure =====

==== Clean up USE and WITH variables ====

See [[http://www.freebsd.org/projects/ideas/#p-ports-cleanup-use|Project Ideas entry]].

==== Reorganization of ports/Mk ====

=== Separation of core/plugin files ===

Recently, ''%%bsd.port.mk%%'' was split into different files, which still provide the "core" functionality. These are mixed with "plugin" files (i.e. ''%%bsd.apache.mk%%''), still required for the system of course, but in an //upper// layer.

It would be good for purposes of organizations to move them into a sub-directory (like ''%%plugins%%''). This way modification policies to them could be more relaxed, being able to affect only a subset of ports, and maintained more efficiently. For example, ''%%bsd.mesalib.mk%%'' which resides in ''%%graphics/libGL%%'' could be moved there.

=== Internal API for bsd.*.mk files ===

Provide an API to add more targets to the lists (i.e. could be used by ''%%bsd.autotools.mk%%'' instead of overriding), build stages, extract methods, and the like. This is intended for ''%%bsd.*.mk%%'' files, but could solve for example a common case: a slave port being unable to add a target if already defined by its master.

It could also be used to add extra functionality which may want to be kept separate for matters of organization, but still being able to integrate well with ''%%bsd.port.mk%%'' without being part of it, or doing unclean hacks.

=== bsd.xxx.mk generator ===

If ''%%bsd.*.mk%%'' integration facilities aren't possible or enough, a set of macros (using an appropiate existing macro language) for maintaining the "plugin" mk files would be useful. This could help generating common make code such as autodetection for installed versions (like ''%%WANT_SDL%%'' and ''%%HAVE_SDL%%''), target integration, and most of the shared code.

Of course the processed files would be used instead of doing each time, but would ease maintaining.

===== Stage 3: Improving current system =====

==== Dependencies ====

=== BUILD and RUN dependencies ===

There are many ports that use pairs of ''%%BUILD_DEPENDS%%''/''%%RUN_DEPENDS%%'' referring to the same port, it may be convenient to add a new dependency type for this case (the name ''%%BOTH_DEPENDS%%'' is inspired from the new ''%%USE_BISON%%'' usage - ''%%build%%''/''%%run%%''/''%%both%%''). A better idea for the name is welcome (maybe ''%%BR_DEPENDS%%'').

Also many ports use ''%%RUN_DEPENDS:=${BUILD_DEPENDS}%%'', and unfortunately others use the incorrect version with ''%%=%%'' instead of ''%%:=%%'', causing the port to have things like ''%%devel/gmake%%'' in ''%%RUN_DEPENDS%%''.

:idea: //Implementation:// very few lines of make code in ''%%bsd.port.mk%%'', does not affect anything else.

=== Removing ${PORTSDIR} from dependencies ===

:!: [[/PavLucistnik|PavLucistnik]] has suggested that this is unneccessary.

Every dependency has ''%%${PORTSDIR}%%'' before the port directory. Removing this requirement would be the first step towards some kind of virtual dependencies (and more over abstract ports, see related item).

:idea: //Implementation:// very few lines of make code in ''%%bsd.port.mk%%'' to add ''%%${PORTSDIR}%%'' if not present (for backwards compatibility) when needed. %%OpenBSD%% has already done this.

=== File-less dependencies ===

Allow to omit it, or add new dependency variables which don't require it, as it's a common case for meta-ports, ports that can install different mutually exclusive files, and future possible virtual ports. This could be done the same way as with current versioned dependencies: passing to [[http://man.freebsd.org/cgi/man.cgi?query=pkg_info&apropos=0&sektion=0&manpath=FreeBSD+6.3-RELEASE+and+Ports&format=html|pkg_info(1)]]. See [[http://www.gentoo.org/proj/en/devrel/handbook/handbook.xml?part=2&chap=1#doc_chap5|Gentoo dependencies]], which also have similar (actually more powerful) fuctionality as in %%FreeBSD%% (see [[http://man.freebsd.org/cgi/man.cgi?query=pkg_glob&apropos=0&sektion=0&manpath=FreeBSD+6.3-RELEASE+and+Ports&format=html|pkg_glob(1)]]).

:idea: //Implementation:// some lines of make code in ''%%bsd.port.mk%%'' to check ''%%/var/db/pkg%%'' if ":" isn't present in dependency line.

=== Build time CONFLICTS ===

Detect ports that conflicts while port/package build only, but has no runtime problems. Using ''%%CONFLICTS%%'' is way to harsh.

  - a port that won't build while an old version of itself is installed.

  - a port that won't build while some include binaries are installied.

:idea: This would be solved if implementing support for multiple concurrent versions of the same application, with appropiate symlinks and/or fake environment. %%NetBSD%%'s approach of symlinking requested dependencies into the build root would also solve it.

=== Plugin and addon ports ===

:!: Rework (evaluate the real need in the existing ports, and think of a simple non-breaking implementation if needed)

If you have one port of an application and many ports for plugins for that application, the latter must depend on the former, that is unable to depend on its plugins (i.e. offering them through ''%%OPTIONS%%'') to avoid a circular dependency.

To solve this a variation of ''%%RUN_DEPENDS%%'' that installs but doesn't register a dependency could be made, only installing the requested ports after the current one. Even better, registering these dependencies separately as optional.

Otherwise more specific plugin variables could be made, even mentioning both the plugin port and the functionality it adds to the current port (which could be expanded to more than plugins, but optional run dependencies that don't need a build option to be supported).

It would also be useful to be able to mention related applications that can be used in conjunction with other, like Debian's //suggested// and //recommended// optional dependencies.

=== Explicit and implicit dependencies ===

Currently both implicit and explicit dependencies are registered (i.e. port A depends on B which depends on C, when installing A appears as requiring both B and C).

The problem is that most if not all ports rely on implicit dependencies to work (i.e. they don't list all actual explicit dependencies as such).

After implementing ''%%OPTIONS%%'' support in dependencies, some (only a part) of these problems could be solved (i.e. if really needed, A would depend on B built with C support).

The rest (most) of the issues would need basically changing almost every port. But the majority of these dependencies are to libraries and ''%%ldconfig%%'' could help a lot (most run only dependencies are listed as ''%%RUN_DEPENDS%%'').

At least explicit and implicit dependencies could be recorded separetely, either in different files or in form of a tree/graph so they can be managed in a better way, and these problems could be tracked progressively.

==== Fetch improvements ====

=== More protocols ===

See [[/Metalink]], which also mentions P2P.

Also implementing support for ''%%.md5%%'', ''%%.gpg%%'', and other checksums residing next to the distfile on the official site would be a good idea.

=== Fetching from dynamic URLs ===

Some websites provide the file from an URL that doesn't end with the file name, such as PHP scripts. In these cases, either a mirror of the file or defining ''%%FETCH_BEFORE_ARGS%%'' (to ''%%-o ${DISTNAME}%%'') is needed. A simple framework could be provided, for example, using the current grouping code to allow mapping a URL to a filename and adding the corresponding arguments to ''%%fetch%%'' (current manual approach doesn't work with more than one file).

=== External programs ===

Support external programs, like (simple) download managers using automatic mirror search for FTP/HTTP, and multiple sockets for the same file.

=== Calculate fastest MASTER_SITES ===

There is a port [[http://www.freshports.org/ports-mgmt/fastest_sites|ports-mgmt/fastest_sites]] that finds the fastest mirror for every mirror list in ''%%bsd.sites.mk%%'', with output suitable for inclusion in ''%%make.conf%%''.

There is no equivalent for individual ports which define all ''%%MASTER_SITES%%'', but it's not needed. What could be done is revisit each port to ensure they use ''%%bsd.sites.mk%%'' entries whenever possible, and add more if applicable.

==== Desktop ====

As this is a targetted audience (from what was recently discussed), these are very important, and not so difficult to implement.

=== Menu entries ===

Ports should install standard menu entries, both for graphical and console applications (maybe a shorter version of ''%%DESKTOP_ENTRIES%%'' could be provided, respecting the next item).

=== Synchronizing menus ===

A tool like Debian's [[http://www.handhelds.org/~nelson/menu/|menu]] (and [[http://www.debian.org/doc/packaging-manuals/menu-policy/ch2.html#s2.1|structure]]) should be provided to keep different desktop and window manager menus in sync.

See [[http://www.gentoo.org/proj/en/glep/glep-0016.html|Gentoo GLEP]].

=== Registering X session types ===

A variable should be provided to maintain a list of available desktops/window managers to be used by X display managers at the login screen (session type selection).

=== Running display managers with limited retries ===

The recommended way (from the Handbook) to set up display managers is with ''%%/etc/ttys%%''. I've seen some Linux distributions use an rc script to launch them, this was because it had a limited number of retries (possibly with delay), so avoiding locking if X fails to start (i.e. if ''%%xorg.conf%%'' is edited wrong, a failed driver update, etc). By personal experience, this turned useful when experimenting with my X configuration.

==== More automatizations ====

=== Managing entries in common files ===

For example, lines in ''%%/etc/shells%%'', PHP extensions list, X session types, and other //required// places (i.e. not ''%%rc.conf%%'', for that see post-install tool). It needs to support the basic methods of doing this (i.e. plain entries, .ini files, etc).

See current ports in shells category, others that append to ''%%/etc/sysctl.conf%%'', etc. for ideas.

This could be delegated to the post-installation tool.

=== Collect the pkg-message output ===

See [[http://www.freebsd.org/projects/ideas/#p-ports-collect-messages|Project Ideas entry]] and [[http://www.mail-archive.com/freebsd-ports@freebsd.org/msg13275.html|possible implementation]].

Instead of just printing the contents of all pkg-message files at the end, it could be made configurable to only indicate the installed packages that have one. This could also be left to the post-installation tool.

=== Provide facilities for constructing pkg-message on-the-fly ===

There are many ports that echo messages when built, sometimes depending on knobs/OPTIONS, but are often lost. Some macros to add text to ''%%pkg-message%%'' could be added to use there. And they could be used by the ports system to append information in some cases.

:idea: //Implementation:// some variables in ''%%bsd.port.mk%%'' to reproduce ''%%echo ... > <pkg-message-tmp>%%'', ''%%cat ... > <pkg-message-tmp>%%'' and even ''%%echo ... | fmt > <pkg-message-tmp>%%'' (i.e. ''%%PKGMSG_ECHO%%'', ''%%PKGMSG_CAT%%'' and ''%%PKGMSG_TEXT%%'').

==== More ports information ====

=== Co-maintainership / implicit approval ===

It may be a good idea to provide a table with permission delegation for maintainers to other submitters and committers (i.e. without approval), being able to specify the exact case (and also custom comments about preferences). This table should be somewhere in the ports tree, by maintainer and port (one with defaults, other with overrides). An example table could be:

^ Who ^ Clean up ^ Update ^ Integrate/normative ^ Fix ^ Security fix ^
| foo%%%%@bar.com | ✘ | ✔ | ✘ | ✔ | ✔ |
| joe%%%%@bar.com | ✔ | ✘ | ✘ | ✘ | ✔ |
| submitters | ✘ | ✘ | ✔ | ✘ | ✘ |
| committers | ✔ | ✘ | ✔ | ✔ | ✔ |

Of course the clear adventage of this comes when automating (i.e. integrating with GNATS bot).

See [[/MaintainerNotes]].

=== License ===

A ''%%LICENSE%%'' variable could be set in each port, as in Gentoo, and then have a "license pool" directory with all of them. IIRC in PC-BSD's PBI packages a license agreement is presented to the user in the setup, and this could benefit them too.

:idea: //Implementation:// add the variable and a directory to place licenses (subdirectory ''%%licenses%%'' under ports?).

==== COMPAT_43TTY ====

See [[http://www.freebsd.org/projects/ideas/#p-ports-comp43tty|Project Ideas entry]].

Most of this has already been done.

==== portbuilt/tinderbox chroot "patching" ====

:!: Pav says it doesn't worth, as it takes seconds, and the prioritized areas are others. Ask Joe?

Currently to build a packages these systems recreate the chroot environment from an archive. This is to ensure each port is built in a clean environment. The deletion/extraction of chroots takes time, and the changes that remain are minimal (as ports are deinstalled for checking correctness and the extra/missing files check).

So it could be turned into a clean environment a program like [[http://man.freebsd.org/cgi/man.cgi?query=rsync&apropos=0&sektion=0&manpath=FreeBSD+6.3-RELEASE+and+Ports&format=html|rsync(1)]]. It would require having an extracted chroot somewhere else, or a virtual FS mounted from the chroot archive. Also ''%%mount_unionfs%%'' (stable as of %%FreeBSD%% 7) could be used to keep the original chroot clean and save space (and easing other tasks as detecting new/changed/missing files). See also [[http://www.freshports.org/sysutils/fusefs-unionfs/|sysutils/fusefs-unionfs]] as a possible alternative.

==== Improving OPTIONS ====

See [[http://www.freebsd.org/projects/ideas/#p-ports-options|Project Ideas entry]].

In addition, I would suggest:

  - Introducing types (enumerations, paths, UID/GID, boolean, etc), forgetting the old WITH/WITHOUT concept. Maybe a prefix should be used to avoid a possible name collision.
  - The previous concept would remove "conflicts" or "mutually exclusive" (i.e. consider ''%%GUI={gtk,qt,motif,none}%%'', there is no place for user/port error, no need to re-run ''%%make config%%'', etc).
  - Support extended descriptions, to be even used for ports that want to show messages before building (when don't find anywhere else, that's strange).
  - Integrate with ''%%KNOBS%%'', use common names and allow "global" settings in /etc/make.conf.
  - Provide a "fake port" to configure the system itself (i.e. changing ''%%PREFIX%%''/''%%DESTDIR%%'', as it would be simply a //path// variable), eliminating the need of manualy tweaking /etc/make.conf (i.e. setting default Apache, Tcl/Tk, wxWidgets, etc). The implementation should also provide an API to be used internally for this case.
  - As a backwards feature similar to the graphical extension, allow it to behave as old knobs (i.e. either show text or nothing, as ''%%BATCH%%'' but still respect user settings). It would completely replace knobs so it should conform everyone (see [[http://www.freshports.org/editors/vim/|editors/vim]] for example) who currently has a good reason to avoid ''%%OPTIONS%%''. So it should have a configurable behavior, possibly asking people who don't like current ''%%BATCH%%'' mode (and the implementation in general).
  - A ''%%dialog%%'' with more features would be needed, or multiple calls otherwise (see ''%%dialog%%'' clones in ports).

===== Stage 4: Integration of ports and packages =====

Also the package database has to be optimized for speed, maybe with SQL (SQLite is file-based).

==== New policies ====

See %%MidnightBSD%%'s [[http://www.midnightbsd.org/documentation/mports/policy.html|mports policies]].

=== Consistency ===

:!: These could be written //after// implementing previously mentioned functionality.

A policy of common consistency between many aspects of the ports have to be determined. There are many ways to do the same thing, incompatible with each other, so that functionality should be provided by the ports/packages system and it should be flexible to support all cases.

A policy for file contents should also be developed (i.e. file width, length, sign to determine items, prompts in example commands, similar to the documentation project).

Note that these should //not// be excessively restrictive, but enough to allow ports/packages to work together, and avoid cases where there are many ways of doing something but there is no a "recommended" or existing method.

A file hierarchy complementary description for ports would be a good idea.

=== Undesired output ===

No port should ever call ''%%ECHO*%%''. Appropiate information places:

^ Purpose ^ Previous "wrong" use ^ Correct practice ^
| Manually fetching ''%%DISTFILES%%'' | Manually displaying instructions | Use ports framework |
| Pre-build information, tunables, etc | Manually displaying, possibly before ''%%sleep%%'' | Use new ''%%OPTIONS%%'' |
| Post-install information | Manually displaying information | Framework for ''%%pkg-message%%'', or ''%%UPDATING%%'' |
| Show ''%%pkg-message%%'' | ''%%${CAT} ${PKGMESSAGE}%%'' and similar | New framework should handle it |

There is no problem in build/install messages as they would form part of the corresponding log messages if implemented, and fine otherwise, but ports don't have to expect the user reads them.

=== Undesired functionality ===

No port should ever show ''%%pkg-message%%'' or run ''%%pkg-install%%'' or similar setup scripts, they should be handled by the system (ports/packages). This would allow better integration of ports and packages, allowing separating tasks between them (like only installing from packages).

Some ''%%pkg-message%%'' files have blank lines and/or headings/footers for cosmetic purposes. This should be left to the displayer, possibly configurable by the user.

This would be problematic with current ports as all of them already do the mentioned actions (if available), so they could be marked with a flag (like ''%%USE_PKGAUTO%%'') to let the ports system take care of ''%%pkg-message%%'' and ''%%pkg-install%%''. Something similar should be done to avoid duplicating commands in ''%%Makefile%%'' and ''%%pkg-plist%%'' (''%%post-install%%'' and ''%%@exec%%''/''%%@unexec%%'').

==== Integration of ports, packages and upgrade tools ====

There are some feature differences between ports and packages systems (mostly packages lag behind ports). It has to be decided exactly which tasks are done by each of them. The same happens with the integration of an upgrading utility with both ports and packages systems. Some of them should use shared functionality from the others.

=== Integration of ports with corresponding built packages ===

:?: By [[/ThierryThomas]].

One weakness of our packages system is the incoherences between the users' ports tree and the available packages:

  - The user updates his ports tree (cvsup or portsnap) with the index.

  - Then he tries to update from packages (pkg_add -r or portupgrade -PP).

Unfortunately, many available packages are outdated and wont be installed... so the user has to install them from ports.

There is an easy way to solve this problem:

  - When a build is terminated, portmgr copy the packages to the ftp  server (as they do currently).

  - Added step: they roll a tarball from the used portstree and copy it to  the same location.

Then we can provide a slightly modified version of portsnap that won't download the latest version of ports tree from http://portsnap.freebsd.org/ but a some days older version coherent with the available packages.

With this mechanism we could warranty to always use packages if they have been built.

:idea: //Implementation:// roll a tarball as mentioned, add date support to ''%%portsnap%%'', make a list of available "package sets" with portstree or a reference to it, accessible by web and relevant tools.

Update ([[/UlfLilleengen|UlfLilleengen]]):
Why can't we instead change the package building procedure to make sure the latest packages are always built? For instance, when a port is updated, a daemon watching the ports tree or a commit hook can trigger a port rebuild of the port in question, thus making sure that the package is always built from the latest port, and doesn't have to wait for a complete sweep over all the packages to complete.

=== Optionified packages ===

:?: By [[/TimKientzle|TimKientzle]]

Darwinports/macports has a very nice idea:  It names each package according to the version and options used to build it.  When you install a package, it will try to find a pre-built binary and if that fails, it will download and compile from scratch.  This allows the central repository to keep pre-built versions for the most popular packages (and it knows what those are because it can record the requests to the central repository) without limiting end users.  In the ideal case, the central repository would automatically prioritize package builds so that the central package cluster utilized its limited CPU resources as efficiently as possible.

=== Fake builds ===

In %%OpenBSD%% ports install in a "fake" environment, then are packaged and finally (if doing so) installed to the system. This avoids duplicating installation code in the ports and packages. Recently %%MidnightBSD%%'s mports switched to this method (it's important since it was based in %%FreeBSD%% 6.1 beta so changes could be imported).

See [[http://www.midnightbsd.org/docs/mports/index.html|mports]].

=== Symlinked builds ===

This is %%NetBSD%%'s approach, which consists in symlinkng required dependencies to the work directory and building from there (translating "-I/usr/local/include" and such by a compiler wrapper).

It also has the advantage of eliminating conflicts of base system and ports version of the same libraries, as only one of them is symlinked to the work directory.

=== Rewrite pkg tools ===

See Project Ideas: [[http://www.freebsd.org/projects/ideas/#p-ports-db|hashed .db]] and [[http://www.freebsd.org/projects/ideas/#p-ports-pkgtools|improvements]].

Also, ''%%pkg_create -b%%'' does not archive ''%%pkg-message%%''.

=== Write an upgrading tool ===

See [[http://www.freebsd.org/projects/ideas/#p-ports-upgrade|Project Ideas entry]].

=== Mixing data from ports and executables from packages ===

An intermediate approach from Debian (a package for data, docs, bin, etc) and Gentoo (everything is built) would be having ports for the data of large applications (the ones likely to have NO_PACKAGE because of the size) split from the binary parts where the package is generated. Then when installing the data is fetch from official mirrors and binary package downloaded from repository.

==== pkg_add extracting directly to destination ====

:?: By [[/TimKientzle|TimKientzle]]

The current %%FreeBSD%% pkg_add would be a lot faster if it extracted directly to the final location instead of going through an intermediate holding directory.  This is possible, but it's tricky.  (Which is why I never actually finished my pkg_add replacement.)

===== Stage 5: Next generation features =====

==== Parallelization ====

See [[http://www.freebsd.org/projects/ideas/#p-ports-parallel|Project Ideas entry]].

=== Package tools and locking ===

See previous link.

=== Multiple ports at the same time ===

See [[http://uyota.asablo.jp/blog/2007/07/26/1681765|ports+]].

IMHO it should also allow different concurrent ''%%make%%'' operations (the ''%%-jX%%'' is for illustration purposes but should be used internally, controlled from a variable):

<code>
% cd /usr/ports/mail/claws-mail && make -j2 install clean &
% cd /usr/ports/x11-wm/icewm && make -j3 install clean &
% cd /usr/ports/mail/claws-mail && make -j1 install clean &
% cd /usr/ports/mail/claws-mail-gtkhtml2_viewer && make -j4 install clean &
</code>

Applications that fail should be recorded so other ''%%make%%'' instances don't try again and again, possibly logging the output (see related item).

In addition, it should support for example fetching a port while another is building.

=== Multiple processes in one port (make -jN) ===

Needs testing which ports work and which don't, but some would present random failures so a test should be kept running on a "mostly working" tree for some time. The addition of ''%%-jX%%'' to ''%%MAKE_ARGS%%'' is mentioned but other systems are omitted (like ''%%scons%%'', ''%%jam%%'', ''%%cmake%%''), which are more likely to be free of these problems.

//Implementation:// some lines of make code in ''%%bsd.port.mk%%'' to check for user setting, available CPU and if port can be built with many jobs (whitelist/blacklist file, or per-port variable). Support for other build systems (''%%bsd.scons.mk%%'', upcoming ''%%bsd.cmake.mk%%'', etc). The rest is filling the lists.

We have two PR's to achieve 'make -jN' for building a port.  The simpler one is ports/122120 and the more complicated one is ports/113132.  The two may be combinable.

==== Cross-platform support ====

=== Cross-building ports ===

[[/RobertWatson|RobertWatson]] reported to have tried, and saw these cases:

  - Ports using ''%%configure%%'', where appropiate flags need to be passed (''%%--host%%'' and ''%%--target%%'').
  - Simpler ports that work by setting the compiler to the cross-compiler.

=== Binary compatibility (amd64/ia64 -> i386) ===

The recent ''%%DESTDIR%%'' functionality could be used for this (like a ''%%/compat/freebsd-i386%%'' chroot), the Linux emulation approach would work fine. For the ports side, dependencies would have to be built in the corresponding chroot or using packages, and //leaf// ports installed normally, but executed in the chroot. This would require kernel changes, otherwise i386 programs could simply be installed there, and the user could use [[http://man.freebsd.org/cgi/man.cgi?query=mount_nullfs&apropos=0&sektion=0&manpath=FreeBSD+6.3-RELEASE&format=html|mount_nullfs(8)]] to access the real root from there. In either case, individual ports wouldn't have to change.

==== More automatizations ====

=== Automatic dynamic pkg-plist ===

Some ports use static and others (less) dynamic packaging lists. A variable to track installation could be provided for such ports. We could even consider getting rid of pkg-plist and fetching them from the pointyhat cluster if needed (like for searching). Gentoo uses dynamic packaging lists.

See Linux framework in ''%%bsd.port.mk%%'' which has ''%%AUTOMATIC_PLIST%%'' (but it's a different case for native ports).

=== Automatic logging and error reporting facilities ===

Output from the ports (that's not ''%%==>%%'') could be logged internally, not requiring external tools. Even an additional "short log" could be created, including recent installs/removals/failures (but not all details) with corresponding dates. It has the following advantages:

  - Allow not showing the output (build, install, etc), as it is logged, for uninterested users.
  - Analyze recent unexpected failures with less effort.
  - Give a short summary of the error by running the corresponding ''%%portbuild%%'' error identifying script with the log.
  - Take note of failed ports and flag them so other parallel processes don't try again.
  - Provide more assisted bug report generation, collecting all the needed information (including commonly omitted items as installed packages, date of ports tree, etc).
  - Reduce ports@ and/or PRs traffic by redirecting the bug reports to a desired place, and even checking if the problem already exists and has been solved, otherwise adding a follow up.

:idea: //Implementation:// some lines of make code to support logging to a directory with a file per port, keeping list of failed ports (allowing "clearing" the error bit, manually or automatically), generating a report, etc; for the log use ''%%tee%%'', ''%%cat%%'', redirections, etc.

=== ports/UPDATING hints ===

A simple language could be integrated, to give the ports upgrading tools relevant information. For example, an entry like ''%%HINTS:%%'' could be added, which could instruct to:

  * Show the information after the port has been upgraded.
  * Disallow upgrading, showing the user the message and letting him/her take some needed action.
  * Rebuild a specific port before upgrading.
  * Rebuild all dependencies after upgrading (could replace PORTREVISION bumps in library dependencies).
  * Other information (for example, how to upgrade Gnome to a newer version).

See: there is a parser that only shows corresponding entries [[http://www.mail-archive.com/freebsd-ports@freebsd.org/msg10695.html|here]].

=== Transactional updates ===

When updating ports it's possible to end up with a broken system (in case one fails and others depend on it). As popular updating tools backup existing packages the chance is reduced, but in some scenarios a set of packages has to be updated at once.

In these cases ports could be built and installed in a separate environment, and only installed after being sure everything is fine there.

=== Create a post-installation management tool ===

This tool would implement the following functionality, some currently assigned to the ports/package system:

  - Running setup scripts (i.e. like ''%%post-install%%''), both system and user level (replacing ''%%make install-user%%'' for some ports). Some even have knobs/OPTIONS to choose doing this or not. By delegating to this tool, the user can choose to do it automatically or manually.
  - Assist in merging configuration files. By running [[http://man.freebsd.org/cgi/man.cgi?query=sdiff&apropos=0&sektion=0&manpath=FreeBSD+6.3-RELEASE&format=html|sdiff(1)]], used by mergemaster. Even a more complex script/program could be made to compare and try to integrate the 3 files (old default config, new default config, and current config). See [[http://gentoo-wiki.com/HOWTO_etc-update|Gentoo etc-update]].
  - Add corresponding lines to /etc/rc.conf (maybe commented), remove, etc. See [[http://gentoo-wiki.com/Rc-update|Gentoo rc-update]].
  - Keep track of which ports need configuration file updating, initial setup, have ''%%pkg-message%%'', etc.
  - Keep track of any ports leftovers, including configuration files and user placed files/dirs. Ports that have ''%%@unexec rm ... || echo "Please remove ..."%%'' could delegate that action to this tool (and others that don't should too).
  - Could replace manual steps from ''%%ports/UPDATING%%'', and together with "''%%UPDATING%%'' hints" could be completely automated.

=== Wrappers for existing specific package subsystems ===

There are many ports that only wrap another subsystem (usually assisted by a ''%%bsd.*.mk%%'' component), like:

  * Linux packages (:!: present before but removed, ask for the actual reasons; rpm-pkg2port is in progress)
  * Perl CPAN
  * PHP Pear/Pecl

In these cases most of the information present in the port could be obtained from these //subsystems// (like ''%%pkg-descr%%'', dependencies, etc) and avoid creating and updating these kinds of ports (only a list of the ones which are used would be needed, and in the worst case a patch for it). For the Linux compatibility we could use the package management system of the distribution installed in ''%%/compat/linux%%''.

See [[Upak/PMS_Links|this page]] for links to the mentioned subjects.

=== Self testing of packages ===

When a package is built, they could run some code to verify that the application actually works. This could be done both after building and before installing (first installing into a fake environment for testing).

Many programs include self tests (like ''%%make test%%''), and with others a simple execution to show it doesn't crash would be enough, but better if also proving to work as expected (i.e. for client/server applications, making a listen/connection on localhost).

Keep in mind that in many ports some self tests fail. In this case the package could be built/used anyways (checking if only the expected tests failed and no more), but recording in a text file which cases failed.

==== More ports information ====

=== Tags ===

An extensive system for further categorization of ports, offering similar alternatives, complementaries (AFAIK this is curerntly not in tags), and the like in a flexible way. Could be as a file with its own syntax, outside the port files. For example, about complementary programs, Debian packages mention other packages that can be used with it (under the names //suggested// and //recommened//).

The tags are managed by a program, it just has to be integrated with other tools (say, a tag-aware ''%%portsearch%%''). About packages that can be used in conjunction with others, they may have to be added manually (but with tags it could be easy).

See [[http://www.tobez.org/port-tags/|Ports Tags]], [[http://debtags.alioth.debian.org/|Debian Debtags]] and [[Ports/NewCategories]]

=== Architecture independent ===

Ports that do not depend on the architecture (i.e. docs, scripts, fonts, data, etc) should be marked as such (maybe ''%%ARCH_INDEPENDENT%%'', ''%%USE_NOARCH%%'', etc). Then an ''%%_${ARCH}%%'' suffix should be added to packages automatically (not changing ''%%PKGBASE%%'', just for packages), being ''%%_all%%'' or ''%%_noarch%%'' in these cases. This way these packages would only be built once, avoiding waste of resources (like scarce CPU time for other architectures, disk space, etc). More over, with suffixed packages it would be possible to use them for a compatibility chroot (i.e. ''%%/compat/freebsd-i386%%'' for amd64/ia64 systems).

:idea: //Implementation:// add variable, but extensive changes in package infrastructure (i.e. from CDs to FTP).

=== Ports statistics ===

Add an optional feature to collect statistics of ports usage (i.e. which ports are installed/removed and corresponding dates), then graphs could be generated and used for different purposes (like priorizing some specific PRs).

==== Abstract ports ====

This refers to ports that don't have a physical directory with a ''%%Makefile%%'', because it doesn't need it. This could save many inodes and make it more flexible their management.

Supporting them would need a new interface to the ports system (and changes to tools that directly call ''%%make%%''), for example ''%%make port=editors/vim-lite%%'' (in this case the physical port exists, but could be implemented with "option variations", not requiring a directory, see below). The list of virtual ports would have to be read by ''%%make%%''.

This could also be a good chance to find what features should be in the ports system by itself (check for outdated ports, etc), so developing external tools would require less work.

=== Virtual ===

These ports are for managing conflictive ports which provide the same functionality, for example ''%%devel/fam%%'' and ''%%devel/gamin%%'' (controlled by ''%%{WANT,WITH}_FAM_SYSTEM%%''), ''%%print/ghostscript-gnu%%'' and ''%%print/ghostscript-gpl%%'' (controlled by ''%%WITH_GHOSTSCRIPT_GNU%%'').

With virtual ports a dependency could be just to ''%%virtual/ghostscript%%'', and implement a consistent yet flexible framework for "prefering" and "requiring" one of each implementations.

These ports provide symlinks which change depending on which port is occupying the place, allowing to change it (for example ''%%mailserver%%'', which could be ''%%postfix%%'' or ''%%qmail%%'').

=== Option variations ===

These just refer to ports with specific options set, for example, the ''%%editors/vim-lite%%'' port only defines ''%%LITE%%'' and includes ''%%editors/vim%%''. The only reason for it is to create packages for it, and maybe provide users another way of ''%%cd /usr/ports/editors/vim && make -DLITE ...%%''. A simple file format could be created for this, like:

<code>
editors/vim-lite|editors/vim|LITE=yes
editors/vim6+ruby|editors/vim6|WITH_RUBY=yes
</code>

This could even be used for PHP 5 modules which currently use the same approach but are many more. And ''%%*-devel%%'' ports could be integrated (as many are) into the main one, removing the unneeded directory (which is often removed and brought back when needed for some ports). Same with ''%%*-doc%%'' files and similar when possible.

%%OpenBSD%% has different suffixed packages that are produced by the same port (see feature comparison).

=== Meta-ports ===

Currently meta-ports exist as physical, but could be replaced by a file listing its dependencies (which could even by be used to create "mapped" or "aliased" ports - 2 or more ports under the same name).

=== Versioned ports ===

Different versions of the same port could be put together with this concepts, without creating different directories, and still having separate ports.

If having many versioned ports, it may be needed to specify stricter versioned dependencies, or supporting //only// using the default ones. Gentoo does this, and it has both adventages and disadventages (as an adventage, if an update goes wrong in most cases one can easily use the old version, not requiring ''%%PORTEPOCH%%'' or version hacks).

=== Multiple concurrent versions ===

Allow multiple concurrent versions of the same application (different versions, or the same built with different options/patches). This could be done with symlinks/wrappers, having one as the //default// and others suffixed.

==== Interoperability with other package management systems ====

=== Sharing files ===

It would be very useful to be able to share architecture and system independent application files between different OSes and/or chroots. The most common of these are data files.

=== Correlation between applications ===

A list of ports/packages names of each package management system which refer to the same application. Useful for sharing information like website, description, license, updates, vulnerabilities, etc.

=== PC-BSD PBI integration ===

PC-BSD seems the most used %%FreeBSD%% desktop these days, and it's a shame their packages (PBI) aren't compatible with our ports. The key concepts of PBI packages are:

  * Applications are entirely self-contained, I.E. they have all libraries needed for operation included in the package.

  * Applications stay in one program directory, minus some sym-links to ''%%/usr/local/bin%%'' or a few other places.

  * Applications can be installed to different locations: - I.E. ''%%/Programs/Firefox2.0.0.12%%'' or ''%%/data/Firefox2.0.0.12%%''

  * The package is a binary, which can be executed directly, or double-clicked in a GUI: - ''%%# ./Firefox2.0.0.12.pbi%%'' or ''%%# ./Firefox2.0.0.12.pbi -text%%''

  * The PBI format includes a png preview icon which shows up in KDE.

  * The PBI format uses lzma compression, md5 hash-checking.

  * The PBI format includes kmenu / icon data, to place shortcuts on a users desktop. (Including a LOT of options for those shortcuts).

=== Leaving /usr/local as a default PREFIX ===

:?: By [[/TimKientzle|TimKientzle]]

It was a mistake for %%FreeBSD%% to use ''%%/usr/local%%'' as the package base. It means that if I install something from hand (untar, configure, make, make install), it gets mixed up with files being managed by the package system.  This seems to be a big headache for every package system I've used:  Competing package management.  In practice, every system has a mix of platform-native packages, CPAN modules, Python eggs, and hand-configured software all mixed together. You can't prevent this from happening, and most current package systems don't deal well with files that were added by other systems.  The only really practical way to avoid it is to have system packages install into a separate hierarchy so that ''%%/usr/local%%'' is left for the local sysadmin to use as they see fit.

==== User ports ====

=== Many ports trees ===

For users testing their own ports it would be useful a functionality similar to Gentoo's overlays or Debian's repositories. Also for using local patches.

=== Many prefixes/localbases ===

For user installations it would be useful to support mixing system and user ports through symlinks/unionfs, and relocating them. This would require tweaking ''%%PATH%%'', and ''%%CPPFLAGS%%''/''%%LDFLAGS%%'' if not using symlinks/unionfs.

==== Resource management ====

Ports should specify which resources they'll be using (net ports, space, user/group ID, etc), and in the future it should be possible to reassign them.

See conference paper link above (stage 1 references).

==== Self-contained packages ====

:?: By [[/TimKientzle|TimKientzle]]

I really like the idea of installing each package into a separate directory and collecting symlinks where a common base is required.  At one time, I managed my own %%FreeBSD%% systems using a home-grown package system that installed each package into a separate dir:
''%%/usr/mypkg/package/<name>-<version>/%%''
Each of these package dirs followed a standard layout:
''%%.../bin .../lib .../etc .../sbin%%''
A set of symlink dirs collected together pointers into these package dirs:

<code>
$ ls -l /usr/mypkg/package/bin
...   program -> /usr/mypkg/package/foobar-4.7/bin/program
</code>

In particular, note that the symlink dirs can be completely erased and rebuilt at any time.  Simply link every ''%%/usr/mypkg/package/*/bin/*%%'' to ''%%/usr/mypkg/bin/%%''.  Conflicts (more below) can be detected and resolved (by renaming, possibly) after packages are installed.  Removing a package is completely trivial.  The package dir also becomes a natural place for the management system to keep track of dependencies.

While people don't think of it as such, the %%MacOS%% application model also deserves study.  Each application is a directory (the Finder actually presents this dir as if it were a single file) with a standard internal structure for binaries, resource files, icons, and other configuration data.  Because each application is self-contained, the system actually doesn't care where the application resides.  In particular, applications can live in user directories or system directories without change.  This interacts badly with the Unix "PATH" model, of course, unless you want
to implement some form of smart symlink that can re-find the target whenever the target moves.  The key point for me, though, is the use of the "dir-per-application" approach, not the fact that packages can migrate after installation.)

==== Automatic conflict resolving ====

:?: By [[/TimKientzle|TimKientzle]]

Conflicts are an ugly problem.  Many systems require that packages tell which other packages they conflict with.  This is impossible to maintain in the long run.  Better to detect resource conflicts automatically.  For example, filenames can conflict and since every package has a manifest, we can detect that automatically.  If each package advertises port usage, for example, then the package system can also detect port conflicts automatically.  (Of course, port usage then needs to be a configuration option: If the package system tells you that you can't install Postfix because Sendmail is already using port 25, you need to have the option of either removing Sendmail or installing Postfix on a different port.)

==== Integration with the base system ====

Being able to integrate ports in the base system would be a very good thing, for example, for installing modified/development/experimental versions of applications (such as dtrace or base system projects). Also for updating, providing patched kernels, etc.

-- [[/AlejandroPulver|AlejandroPulver]] 2008-02-22 13:51:06 UTC

----

{{tag>Ports Todo}}