Python
FreeBSD supports multiple, concurrently installable Python versions and environments for Python users and developers to work with. Every Python available in the FreeBSD ports tree can be built for any (and multiple) Python version using the FLAVORS and DEFAULT_VERSIONS framework mechanisms.
Binary packages available from the official FreeBSD Package repositories are built using a DEFAULT_VERSION of Python, which evolves over time.
Our Python Team works closely with upstream, aiming to resolve issues with an upstream-first approach and stays close-to-upstream with respect to documentation, guidance and recommendations.
FreeBSD is officially supported by, and provides multiple FreeBSD continuous integration workers for the Python project to ensure future versions of FreeBSD are supported, and so bugs and regressions are identified and resolved prior to release.
#freebsd-python
IRC channel (ircs://irc.libera.chat/#freebsd-python) (Libera Chat)
-
Goals
∞ - Ongoing
↻ - In Progress
☐ - Open
☐ Files that fail to compile (compileall) are still added to distutils (therefore setuptools, therefore pip) –record output, breaking plists
☐ (Un)Official poudriere repository with Python 3.x default (If you want to do this talk to
koobs@)
☐ Support USES=python:x.x,x.y,x.z (or x.x-x.z,!x.y) version specification
☐ Productivity: Tool to create Python port from a Python (PyPI) sdist (and its setup.py file, etc)
☐ Add a stage to Python framework to find and strip shared libraries in a consistent manner
Right now, either each port does it itself, or not at all (BLANKET)
Once done, remove blocks from ports that do this themselves
☐ Idea: Extend USES=python[:features] or USE_PYTHON=<feature>
☐ :tests - Adds TEST_DEPENDS to BUILD_DEPENDS, adds regression-test: ${PYTHON_CMD} ${PYDISTUTILS_SETUP} test
☐ :<nosetests|pytest|tox|etc> - Adds the right things to TEST_DEPENDS, then adds/runs regression-test: <correct invocation>
Q: How can we easily switch tests on/off for upstream packages that fix or break the test suite over time
Q: Can we unconditionally turn tests on, but not have packaging fail if the test fails? Fiddle with stderr? Test suite args?
☐ Remove Python 3 specific port duplicates, once pkg_install support is retired. Ports with the same ORIGIN can be installed for different Python versions in parallel.
☐ Remove devel/py-setuptoolsXX
☐ Remove devel/py3-dbus, devel/py3-gobject3, devel/py3-xdg, audio/py3-pylast, textproc/py3-libxml2, graphics/py3-cairo, multimedia/py3-gstreamer1
☐ Resolve
Distutils install_data command contains directories in –record <RECORDFILE> ☐ Don't package (plist) or install .pyc and .pyo files. Details:
CompiledPackages ☐ Support for granular dependencies (eg: >=1.0,<2.0)
☐ bsd.python.mk is PREFIX-safe
-
☐ All Python ports are local-patch free (upstream or remove all local patches)
☑
Issue #18178: Redefinition of malloc(3) family of functions at build time (Status: Upstreamed)
-
-
☐ Fix gtk20's /usr/local/bin/gtk-builder-convert dependency to Python by splitting it up into an own slave port?
☐ Integrate lang/pypy into ports framework (knobs etc)
☐ Python 3.x is the default Python language version
☐ Python version agnostic DOCSDIR and EXAMPLESDIR; they currently need to be overridden manually - may conflict with the PYTHON_CONCURRENT_INSTALL efforts
☐ Make it possible to add extra arguments to PYDISTUTILS_INSTALLARGS and friends without having to include bsd.port.pre.mk.
☐ create guideline for which ports python@ will accept to maintain. In particular, address 'leaf ports' and 'non-infrastructure'
☑ - Done
☑ Python ports framework complies with all relevant
PEP's
☑
PEP-394: The “python” Command on Unix-Like Systems
☑
PEP-449: Removal of the PyPI Mirror Auto Discovery and Naming Scheme -
r400105 ☑
PEP-3147: PYC Repository Directories (
we use different site-packages/ and support concurrent installation (which creates distinct directories)
☑ Deprecate INSTALLS_EGGINFO: Use –record output to determine if a Python package installs .egg-info metadata files or directories.
☑ –record <RECORD> can contain paths that include ../ or ./
☑ Split bsd.python.mk into a USES for consumer ports and an port-tree specific .mk file
☑ Reduce shared/static build complexity in lang/pythonXX ports using lang/python34 as template
☑ Remove easy_install specific bits in bsd.python.mk
☑ PYDISTUTILS_SETUP does not strip arguments properly, which causes setuptools and distutils to jump to wrong conclusions. This is a nuisance right now (manifest_maker warnings), but can become a serious issue.
☑ DATADIR, EXAMPLESDIR, binaries, scripts and files in ETCDIR need to be python prefix safe
☑ Use python setup.py install --root=<path> instead of setup.py easy_install in Mk/bsd.python.mk
☑ Ports limited to certain Python versions must not rely on ${PREFIX}/bin/python, but the proper python versions
20131218:
Commit r336850 - lang/python has been removed as implicit build dependency
-
☑ STAGE support for setuptools-based ports (USE_PYDISTUTILS=easy_install)
Mk/bsd.python.mk currently uses the easy_install setup.py target to install setuptools-based modules.
setuptools and distutils build and install targets support a –root option. The easy_install target and CLI command do not.
☑ 20131110: Workaround easy_install by installing in stage dir and recompiling
r333286 (Credit: nivit, wg)
-
-
-
☑ Zope
☑ (plone (import plone4))
☑ (turbogears2 (update to 2.1))
-
Other Interesting Things
PEP-446 – Make newly created file descriptors non-inheritable (Python 3.4, FreeBSD *_CLOEXEC Support)
PEP-427 – The Wheel Binary Package Format 1.0
The upstream Python packaging team has expressed interest in understanding the challenges faced by OS packagers with Python packaging tools. This information will be used to inform ongoing improvements and developments. Please document your experiences below:
StageDir with setuptools-based (easy_install) modules – Doesnt support –destdir or equivalent
Steps to switch the Python default version
A successful exp-run is mandatory
Set the new default version in
bsd.ports.default.mk
Uses/python.mk
Bump the PORTREVISION of following ports
devel/py-setuptools
lang/cython
math/py-numpy
Add UPDATING instructions
Legend:
pyMM, pythonMM, pythonM.M or MM = Old Python version, e.g. py37, python37, python3.7 or 37
pyNN, pythonNN, pythonN.N or NN = New Python version, e.g. py38, python38, python3.8 or 38
Current instructions are:
The default version of python3 and python was switched to N.N.
For ports users wanting to keep version M.M as default,
add DEFAULT_VERSIONS+= python=M.M python3=M.M to make.conf
Following procedures may ease the upgrade:
# sh
# for i in $(pkg query -g %n 'pyMM-*'); do pkg set -yn ${i}:pyNN-${i#pyMM-}; done
# pkg upgrade
For portmaster users:
# sh
# portmaster -o lang/pythonNN pythonMM
# REINSTALL="$(pkg info -qo "*pyMM*")"
# pkg delete -f "*pyMM*"
# portmaster $REINSTALL
# REBUILD=$(pkg query "%n:%dn" | grep py3 | grep -v pyNN | cut -d : -f 1 | sort -u)
# portmaster $REBUILD
# REBUILD2=$(pkg list | grep python-MM | xargs pkg which | awk '{print $6}' | sort -u)
# portmaster $REBUILD2
Please note that the instructions from above for portmaster are not perfect for more complex scenarios of installed ports (see TODO below).
TODO for the instructions in UPDATING
Following parts should be added with the next Python switchover:
Refine instructions for Portmaster
With more complex scenarios of installed ports the upgrade might not go smoothly. There have been reported issues if following ports were installed:
There are improved instructions that still needs to be tested thoroughly:
Improved UPDATING instructions for Portmaster
Add clean up instructions
The following is a draft based on a suggestion from ThierryThomas (thierry) to improve the user experience.
Final steps (for pre-built packages & portmaster):
If no longer required, Python M.M can be removed via
"pkg remove pythonMM" and the directory /usr/local/lib/pythonM.M can
then be deleted afterwards, if not empty.
Python Team