Python Ports Policy

Status: Defacto. If in doubt, check with the FreeBSD Python Team

This policy document is used:

• As a reference to a well-defined target-state
• To inform discussions and technical implementations
• To provide FreeBSD Python Users and Organisations with insight into the Python@ team goals
• To orient and focus resources towards common initiatives or goals.

FreeBSD currently supports Python (CPython) branches until they are end-of-life (EoL) as defined upstream.

• Python Release Support

Note: Upstream currently only provides bug fixes for the latest minor release (and Python 2.7 at the time of this writing). In order to reduce maintenance overheads and inconsistencies between ports, it is likely that the FreeBSD Python team will modify its branch support policy to only support branches that are also supported by upstream in the future.

Guideline: Prefix all ports unless Python version is entirely irrelevant.

• Port directory name MUST be lowercase(py-PORTNAME)

• Python ports MUST be prefixed with py-, named category/py-<PORTNAME>, unless there is a compelling case not to.
  • Compelling cases do NOT include:
    • PyPi name or PORTNAME already contains py, python, python-* or similar prefix or suffix
    • Package is not listed in PyPI
• Python packages that are complete end-user products NOT present in PyPI AND not likely to ever be, MAY omit a py- prefix, A. BUT …
  • Check with the FreeBSD Python Team

Guideline: Match port names to upstream names exactly.

IMPORTANT

Check if a package is registered in PyPI (CHEESESHOP), even when using alternate code sources.

• If the package is registered on PyPI
  • PORTNAME MUST exactly match the registered PyPI package name, including casing.
  • The PyPI package name is that which is displayed in the URL at https://pypi.org/<package-name>
  • Note: The displayed PyPI package name can be different than the package name specified in setup.py:name
• If the package is NOT registered on PyPI:
  • PORTNAME MUST exactly match setup.py:name value.
• Ports MUST NOT modify the name if it already contains py, python, python-* or similar prefix or suffix
• Ports MUST include all non-alpha numeric characters in the name (such as dashes (-))
  • This also applies if PyPi (CHEESESHOP) is NOT used for MASTER_SITES or DISTFILES. Example: USE_GITHUB=yes

Guideline: Prefix all ports unless Python version is entirely irrelevant.

• Ports MUST use PKGNAMEPREFIX, unless there is a compelling case not to.
• PKGNAMEPREFIX MUST be set to ${PYTHON_PKGNAMEPREFIX}

Guideline: Always include python category unless use of Python is entirely irrelevant.

• Ports MUST add python as a secondary category, unless there is a compelling reason not to.
• Ports SHOULD add multiple CATEGORIES as needed to aid discovery by users.

Guideline: Match upstream metadata as completely and closely as possible.

• Ports SHOULD match the upstream short description or summary where possible, modulo requirements and changes for portlint compliance
  • For Python ports, this almost always means what is defined in setup.py:description

Guideline: Use standard Python ecosystem infrastructure whenever possible.

• Ports SHOULD use MASTER_SITES=CHEESESHOP if a Python package is registered on PyPI, unless there is a compelling (temporary) case not to.

Examples where temporarily using an alternate MASTER_SITES is warranted:

• Some files are not correctly packaged or included at all in the PyPI sdist, such as licenses, test suite files or data. Submit an upstream issue or PR in these cases.
• A source distribution (sdist) has not been uploaded yet

In these cases, issues or pull request SHOULD be created upstream to rectify the problem, and MASTER_SITES switched to CHEESESHOP when the change is released in a later version.

This ensures upstreams packaging and deployment pipeline is well tested and standardised (Python standards), as it is heavily relied upon (setuptools, autoplist, documentation, discoverability, etc).

Guideline: Python ports are declarative (“this supports x,y,z”, not imperative (“install/use x,y,z”). Guideline: Match upstream metadata as completely and closely as possible.

• Python ports MUST declare USES=python :<version-spec> as completely and accurately as possible, without being incorrect, given existing syntax constraints.
  • :<version-spec> MUST match the Python versions declared supported by upstream exactly. If upstream declarations are incorrect or incomplete, fix them in the port and send the changes upstream.
• Python ports MUST NOT limit or restrict Python version support in :version-spec.
• Python ports SHOULD NOT leave :<version-spec> empty (Note: transitionary guideline. This will eventually be a MUST)

Guideline: Use all standard Python ecosystem infrastructure whenever possible.

• Packages following PEP-517 (have a pyproject.toml specifying a build backend that builds an installable wheel) SHALL use pep517, unless there is a compelling reason not to.

• Packages that use distutils (directly executes a setup.py, import distutils/setuptools), SHOULD use distutils, unless there is a compelling reason not to.

• Packages following PEP-517 (have a pyproject.toml specifying a build backend that builds an installable wheel) and built as such SHALL use autoplist.
• Packages that use distutils (have a setup.py, import distutils/setuptools), SHOULD use autoplist, unless there is a compelling reason not to.

Note: autoplist can be used in combination with PLIST_FILES, PLIST_DIRS and explicit entries in pkg-plist

• concurrent MUST be set when any files (pathnames/filenames) installed by a package are NOT Python-version specific, like /usr/local/bin/script.
  • This includes ports that support a single-version of Python, as supported versions can change at any time.
  • concurrent SHOULD be set, even in cases where no such filenames are installed, as this may change in future versions.

See Also: https://reviews.freebsd.org/D20927#454268

Guideline: Match upstream metadata as completely and closely as possible.

IMPORTANT:

Be careful when referencing dependencies only from requirements.txt files. Unless the setup.py parses those requirements.txt entries, only setup.py defined dependencies are canonical/actual.

Be careful, dependencies are often declared inconsistently, incompletely or in multiple places (Eg: requirements.txt, test_requirements, tox.ini)

• Ports MUST exactly match the dependencies declared by the upstream package.
  • Refer to setup.py:setup_requires for BUILD_DEPENDS
  • Refer to setup.py:install_requires for RUN_DEPENDS
  • Refer to setup.py:tests_require for TEST_DEPENDS

• Port maintainers SHOULD fix missing or incorrectly specified dependencies, in the port, AND upstream the changes.
• Use common sense for upstream version requirements that set maximum versions or pin very specific versions, eg: ==, <=, <
  • IMPORTANT: Common sense does not create an exception to the exactly matching dependencies requirement above.

• Ports MAY patch dependencies in the upstream sources (WRKSRC), if they need to be added, modified or removed.
  • When patching dependencies, changes MUST be verified to work correctly, ideally via the upstream test suite.
  • Example: Changing == (pinned) dependencies to >= or removing maximum version <X.Y version specifiers.

• setup.py:extras_require or otherwise OPTIONAL dependencies, SHOULD be reflected in equivalent port OPTIONS.
• OPTION name SHOULD be named after the extras_require 'key' name.
• Port OPTIONS for optional functionality, SHOULD be enabled by default (defined in OPTIONS_DEFAULT).

• If a package depends on another packages optional dependencies, the package MUST depend on those dependencies directly/explicitly.
  • Example: Package A depends on PackageB[optional] (defined in extras_require). Package A should depend explicitly on all dependencies listed in Package B's extras_require{ optional: ..., ...} list.

Guideline: Ports should be testable unless they cannot be.

• Python ports SHOULD include a do-test target if the package includes tests.
  • Example: @(cd ${TEST_WRKSRC} && ${SETENV} ${TEST_ENV} ${PYTHON_CMD} ${PYDISTUTILS_SETUP} test)
• Python ports SHOULD add TEST_DEPENDS referring to the package tests_require dependencies.

Guideline: Document downstream patches & changes that diverge from upstream.

• Python ports SHOULD document their patches
  • If the patch is standalone, place the comment in the header
  • If the patch is combined with others changes, place the comment inline, in each relevant hunk

Guideline: Upstream is a first priority.

• Python port maintainers SHOULD (strongly encouraged) aim to:
  • Craft port patches that are likely to be accepted upstream, even if they are committed locally first.
  • Create upstream issues for bug reports and enhancement requests. Eg:
    • Better documented requirements in install_requires, tests_require, setup_requires
    • Optional dependencies made a part of extras_require
    • Support for python setup.py test command (for regression testing)
    • Adding LICENSE to repository and sdist
  • Regularly test and report test suite failures of Python packages on FreeBSD

• Ports that install Python modules or packages (anything going into ${PYTHON_SITELIBDIR}), MUST use ${PYTHON_CMD} at build and installation time, not python, python2 or python3.
• Executable scripts being installed MUST use ${PYTHON_CMD} as shebang line. See: Mk/Uses/shebangfix.mk

• Ports depending on Python to build MUST use${PYTHON_CMD} for the build system/command invocation.
• Ports MUST NOT use version-unspecific Python invocations such as python, python2 or python3.

This allows ports to be built with different default versions of Python.

• Ports that install Python scripts, which use a package or module being provided by another Python port, MUST use a specific Python port version. The shebang line MUST use ${PYTHON_CMD}.
• If the Python module dependency can be installed for multiple Python versions in parallel, the port providing the script MUST use ${PYTHON_PKGNAMESUFFIX} and the script SHOULD be suffixed, too. If the port is installed for the default Python version, the script SHOULD be symlinked to the original name.

Example:

A port devel/foobar provides a script foofoo.py, which requires textproc/py-barbar at runtime

• devel/foobar MUST use USES=python,run
• devel/foobar MUST add textproc/py-barbar as RUN_DEPENDS
• foofoo.py MUST use ${PYTHON_CMD} as shebang
• devel/foobar MUST use PKGNAMESUFFIX=${PYTHON_PKGNAMESUFFIX}
• if devel/foobar and textproc/py-barbar support different Python versions:
  • foofoo.py MUST be installed as foofoo-${PYTHON_PKGNAMESUFFIX}.py
  • foofoo-${PYTHON_PKGNAMESUFFIX}.py SHOULD symlink to foofoo.py, if the Python version of the port matches the default Python version being specified by the system (DEFAULT_VERSION)

• Patches & Back-ports
  • An issue is created in the Python Bug Tracker
  • Links to issues are included with commits to ports
  • Patches are correctly commented

Description: Upstream issue summary or commit short log
Issue ID: URL to upstream Issue
Changeset: URL to upstream Commit
TODO: Comments on what needs to be done to remove local patch

• For Django packages
  • Should be added in the www/ category, with www as the primary CATEGORIES, and python as a secondary category
  • More specific categories can be added as additional/virtual categories if appropriate
• For Flask packages: TBD. Where specific policies are not defined, Python Ports Policy applies
• For zope packages: TBD. Where specific policies are not defined, Python Ports Policy applies
• For <framework> packages: TBD. Where specific policies are not defined, Python Ports Policy applies