python

This is an old revision of the document!

Contact

Goals

∞ - Ongoing

↻ - In Progress

☐ - Open

  • ☐ Files that fail to compile (compileall) are still added to distutils (therefore setuptools, therefore pip) –record output, breaking plists
  • ☐ (Un)Official poudriere repository with Python 3.x default (If you want to do this talk to koobs@)
  • ☐ Support USES=python:x.x,x.y,x.z (or x.x-x.z,!x.y) version specification
  • ☐ Productivity: Tool to create Python port from a Python (PyPI) sdist (and its setup.py file, etc)
  • ☐ Add a stage to Python framework to find and strip shared libraries in a consistent manner
    • Right now, either each port does it itself, or not at all (BLANKET)
    • Once done, remove blocks from ports that do this themselves
  • ☐ Idea: Extend USES=python[:features] or USE_PYTHON=<feature>
    • ☐ :tests - Adds TEST_DEPENDS to BUILD_DEPENDS, adds regression-test: ${PYTHON_CMD} ${PYDISTUTILS_SETUP} test
    • ☐ :<nosetests|pytest|tox|etc> - Adds the right things to TEST_DEPENDS, then adds/runs regression-test: <correct invocation>
    • Q: How can we easily switch tests on/off for upstream packages that fix or break the test suite over time
    • Q: Can we unconditionally turn tests on, but not have packaging fail if the test fails? Fiddle with stderr? Test suite args?
  • ☐ Remove Python 3 specific port duplicates, once pkg_install support is retired. Ports with the same ORIGIN can be installed for different Python versions in parallel.
    • ☐ Remove devel/py-setuptoolsXX
    • ☐ Remove devel/py3-dbus, devel/py3-gobject3, devel/py3-xdg, audio/py3-pylast, textproc/py3-libxml2, graphics/py3-cairo, multimedia/py3-gstreamer1
  • ☐ Resolve Distutils install_data command contains directories in –record <RECORDFILE>
    • Observable with PYDISTUTILS_AUTOPLIST when data_files contains a directory-only entry
    • Original commit and issue #444589
    • Example: devel/buildbot-slave (setup.py)
    • Example: databases/pyspatialite (setup.py)
    • Example: graphics/py-pygraphviz (setup.py)
  • ☐ Don't package (plist) or install .pyc and .pyo files. Details: CompiledPackages
  • ☐ Support for granular dependencies (eg: >=1.0,<2.0)
    • ☐ Look at what adding pip support might look like (supports these version checks)
  • ☐ bsd.python.mk is PREFIX-safe
    • What specific things are currently not PREFIX-safe ?
  • ☐ Consistent Python ports prefix policy and remove PYTHON_PKGNAMESUFFIX (see this blog post for details)
  • ☐ All Python ports are local-patch free (upstream or remove all local patches)
  • ☐ Fix gtk20's /usr/local/bin/gtk-builder-convert dependency to Python by splitting it up into an own slave port?
  • ☐ Integrate lang/pypy into ports framework (knobs etc)
  • ☐ Python 3.x is the default Python language version
  • ☐ Python version agnostic DOCSDIR and EXAMPLESDIR; they currently need to be overridden manually - may conflict with the PYTHON_CONCURRENT_INSTALL efforts
  • ☐ Make it possible to add extra arguments to PYDISTUTILS_INSTALLARGS and friends without having to include bsd.port.pre.mk.
  • ☐ create guideline for which ports python@ will accept to maintain. In particular, address 'leaf ports' and 'non-infrastructure'

☑ - Done

  • ☑ Python ports framework complies with all relevant PEP's
    • ☑ PEP-394: The “python” Command on Unix-Like Systems
    • ☑ PEP-449: Removal of the PyPI Mirror Auto Discovery and Naming Scheme - r400105
    • ☑ PEP-3147: PYC Repository Directories (we use different site-packages/ and support concurrent installation (which creates distinct directories)
  • ☑ Deprecate INSTALLS_EGGINFO: Use –record output to determine if a Python package installs .egg-info metadata files or directories.
  • ☑ –record <RECORD> can contain paths that include ../ or ./
  • ☑ Split bsd.python.mk into a USES for consumer ports and an port-tree specific .mk file
  • ☑ Reduce shared/static build complexity in lang/pythonXX ports using lang/python34 as template
  • ☑ Remove easy_install specific bits in bsd.python.mk
  • ☑ PYDISTUTILS_SETUP does not strip arguments properly, which causes setuptools and distutils to jump to wrong conclusions. This is a nuisance right now (manifest_maker warnings), but can become a serious issue.
  • ☑ DATADIR, EXAMPLESDIR, binaries, scripts and files in ETCDIR need to be python prefix safe
  • ☑ Use python setup.py install --root=<path> instead of setup.py easy_install in Mk/bsd.python.mk
  • ☑ Ports limited to certain Python versions must not rely on ${PREFIX}/bin/python, but the proper python versions
    • 20131218: Commit r336850 - lang/python has been removed as implicit build dependency
  • ☑ Chase the distribute merge back into setuptools
  • ☑ STAGE support for setuptools-based ports (USE_PYDISTUTILS=easy_install)
    • Mk/bsd.python.mk currently uses the easy_install setup.py target to install setuptools-based modules.
    • setuptools and distutils build and install targets support a –root option. The easy_install target and CLI command do not.
    • ☑ 20131110: Workaround easy_install by installing in stage dir and recompiling r333286 (Credit: nivit, wg)
  • ☑ 20131029: freebsd-python: List of ports with "python 3" errors in poudriere
    • lang/python installation conflict
  • ☑ Retired devel/py-psyco (12 March 2012: Psyco is unmaintained and dead)
  • ☑ Automate pkg-plist generation (r328403: PYDISTUTILS_AUTOPLIST)
  • ☑ Zope
    • remove zope27
    • remove 28 29 210
    • update 211 (it's on it latest version already)
    • import 212 (do we really need this?)
    • import 213
  • ☑ (plone (import plone4))
  • ☑ (turbogears2 (update to 2.1))
  • ☑ (import lang/pypy port by David Naylor (ports/160265))
  • ☑ [EXP-RUN] [patch] lang/python: speed up upgrade-site-packages) ports/155970 r304961

Other Interesting Things

  • PEP-446 – Make newly created file descriptors non-inheritable (Python 3.4, FreeBSD *_CLOEXEC Support)
  • PEP-427 – The Wheel Binary Package Format 1.0

Tooling Challenges

The upstream Python packaging team has expressed interest in understanding the challenges faced by OS packagers with Python packaging tools. This information will be used to inform ongoing improvements and developments. Please document your experiences below:

  • StageDir with setuptools-based (easy_install) modules – Doesnt support –destdir or equivalent

Steps to switch the Python default version

  1. A successful exp-run is mandatory
  2. Set the new default version in
    • bsd.ports.default.mk
      • PYTHON3_DEFAULT
    • Uses/python.mk
      • _PYTHON_VERSIONS (retain order of preferred versions)
      • _PYTHON_PORTBRANCH
  3. Bump the PORTREVISION of following ports
    • devel/py-setuptools
    • lang/cython
    • math/py-numpy
  4. Add UPDATING instructions

Legend: 

pyMM, pythonMM, pythonM.M or MM = Old Python version, e.g. py37, python37, python3.7 or 37
pyNN, pythonNN, pythonN.N or NN = New Python version, e.g. py38, python38, python3.8 or 38

Current instructions are: 

The default version of python3 and python was switched to N.N.

For ports users wanting to keep version M.M as default,
add DEFAULT_VERSIONS+= python=M.M python3=M.M to make.conf

Following procedures may ease the upgrade:

# sh
# for i in $(pkg query -g %n 'pyMM-*'); do pkg set -yn ${i}:pyNN-${i#pyMM-}; done
# pkg upgrade

For portmaster users:
# sh
# portmaster -o lang/pythonNN pythonMM
# REINSTALL="$(pkg info -qo "*pyMM*")"
# pkg delete -f "*pyMM*"
# portmaster $REINSTALL
# REBUILD=$(pkg query "%n:%dn" | grep py3 | grep -v pyNN | cut -d : -f 1 | sort -u)
# portmaster $REBUILD
# REBUILD2=$(pkg list | grep python-MM | xargs pkg which | awk '{print $6}' | sort -u)
# portmaster $REBUILD2
Please note that the instructions from above for portmaster are not perfect for more complex scenarios of installed ports (see TODO below).

TODO for the instructions in UPDATING

Following parts should be added with the next Python switchover:

Refine instructions for Portmaster

With more complex scenarios of installed ports the upgrade might not go smoothly. There have been reported issues if following ports were installed:

  • databases/tdb
  • devel/gobject-introspection
  • devel/meson
  • devel/tevent

There are improved instructions that still needs to be tested thoroughly:

Improved UPDATING instructions for Portmaster

Add clean up instructions

The following is a draft based on a suggestion from ThierryThomas (thierry) to improve the user experience. 

Final steps (for pre-built packages & portmaster):

If no longer required, Python M.M can be removed via
"pkg remove pythonMM" and the directory /usr/local/lib/pythonM.M can
then be deleted afterwards, if not empty.

Python Team

,
  • python.1666221880.txt.gz
  • Last modified: 2022/10/19 23:24
  • by KubilayKocak