Show pageOld revisionsBacklinksBack to top This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong. ====== Python ====== %%FreeBSD%% supports multiple, concurrently installable Python versions and environments for Python users and developers to work with. Every Python available in the %%FreeBSD%% [[Ports|ports]] tree can be built for any (and multiple) Python version using the [[https://man.freebsd.org/cgi/man.cgi?query=ports&sektion=7&format=html#EXAMPLES|FLAVORS]] and [[Ports/DEFAULT_VERSIONS|DEFAULT_VERSIONS]] framework mechanisms. Binary packages available from the official %%FreeBSD%% Package repositories are built using a [[https://cgit.freebsd.org/ports/tree/Mk/bsd.default-versions.mk|DEFAULT_VERSION of Python]], which evolves over time. Our [[#team|Python Team]] works closely with upstream, aiming to resolve issues with an upstream-first approach and stays close-to-upstream with respect to documentation, guidance and recommendations. %%FreeBSD%% is officially supported by, and provides [[https://buildbot.python.org/all/#/workers|multiple FreeBSD continuous integration workers]] for the Python project to ensure future versions of %%FreeBSD%% are supported, and so bugs and regressions are identified and resolved prior to release. \\ ====== Contact ====== * #freebsd-python IRC channel (ircs:%%//%%irc.libera.chat/#freebsd-python) (Libera Chat) * [[https://lists.freebsd.org/mailman/listinfo/freebsd-python|freebsd-python mailing list]] ====== Goals ====== ===== ∞ - Ongoing ===== * Resolve [[https://bugs.freebsd.org/bugzilla/buglist.cgi?bug_status=New&bug_status=Open&bug_status=In%20Progress&email1=python%40FreeBSD.org&emailtype1=exact&f1=assigned_to&f2=assigned_to&j_top=OR&list_id=39597&o1=equals&o2=equals&order=bug_status%2Cpriority%2Cbug_severity&product=Base%20System&product=Community&product=Documentation&product=Ports%20%26%20Packages&product=Services&query_based_on=&query_format=advanced&v1=python%40FreeBSD.org&v2=freebsd-python%40FreeBSD.org|Open Python Issues]] * Keep Python Ports up-to-date: [[https://portscout.freebsd.org/python@freebsd.org.html|Portscout]] and [[https://repology.org/projects/?search=python%3A&maintainer=&category=&inrepo=freebsd¬inrepo=&repos=&families=&repos_newest=&families_newest=2-&outdated=on|Repology]] * Fullfill the [[Python/PortsWishlist|Python Port Wishlist]] * Improve [[Python/PortsPolicy|Python Ports Policy]] ===== ↻ - In Progress ===== ===== ☐ - Open ===== * ☐ Files that fail to compile (compileall) are still added to distutils (therefore setuptools, therefore pip) --record output, breaking plists * [[https://bugs.python.org/issue20397|Python issue 20397]] Needs patch * ☐ (Un)Official poudriere repository with Python 3.x default (If you want to do this talk to [[KubilayKocak|koobs@]]) * ☐ Support USES=python:x.x,x.y,x.z (or x.x-x.z,!x.y) version specification * ☐ Productivity: Tool to create Python port from a Python (%%PyPI%%) sdist (and its setup.py file, etc) * ☐ Add a stage to Python framework to find and strip shared libraries in a consistent manner * Right now, either each port does it itself, or not at all (BLANKET) * Once done, remove blocks from ports that do this themselves * ☐ Idea: Extend USES=python[:features] or USE_PYTHON=%%<%%feature> * ☐ :tests - Adds TEST_DEPENDS to BUILD_DEPENDS, adds regression-test: ${PYTHON_CMD} ${PYDISTUTILS_SETUP} test * ☐ :%%<%%nosetests|pytest|tox|etc> - Adds the right things to TEST_DEPENDS, then adds/runs regression-test: %%<%%correct invocation> * Q: How can we easily switch tests on/off for upstream packages that fix or break the test suite over time * Q: Can we unconditionally turn tests on, but not have packaging fail if the test fails? Fiddle with stderr? Test suite args? * ☐ Remove Python 3 specific port duplicates, once pkg_install support is retired. Ports with the same ORIGIN can be installed for different Python versions in parallel. * ☐ Remove devel/py-setuptoolsXX * ☐ Remove devel/py3-dbus, devel/py3-gobject3, devel/py3-xdg, audio/py3-pylast, textproc/py3-libxml2, graphics/py3-cairo, multimedia/py3-gstreamer1 * ☐ Resolve //Distutils [[http://hg.python.org/cpython/file/tip/Lib/distutils/command/install_data.py#l64|install_data]] command contains directories in --record %%<%%RECORDFILE>// * Observable with PYDISTUTILS_AUTOPLIST when //data_files// contains a directory-only entry * Original [[http://hg.python.org/cpython/rev/fbaf4e55263f|commit]] and [[https://bugs.python.org/issue444589|issue #444589]] * Example: devel/buildbot-slave ([[https://github.com/buildbot/buildbot/blob/master/slave/setup.py#L104|setup.py]]) * Example: databases/pyspatialite ([[https://github.com/lokkju/pyspatialite/blob/master/setup.py#L237|setup.py]]) * Example: graphics/py-pygraphviz ([[https://github.com/pygraphviz/pygraphviz/blob/master/setup.py#L46|setup.py]]) * ☐ Don't package (plist) or install .pyc and .pyo files. Details: [[Python/CompiledPackages|CompiledPackages]] * ☐ Support for granular dependencies (eg: >=1.0,<2.0) * ☐ Look at what adding pip support might look like (supports these version checks) * ☐ bsd.python.mk is PREFIX-safe * What specific things are currently **not** PREFIX-safe ? * ☐ Consistent Python ports prefix policy and remove PYTHON_PKGNAMESUFFIX (see [[http://www.sysfault.org/posts/freebsd-python-ports-pre-suffix.html|this blog post for details]]) * ☐ All Python ports are local-patch free (upstream or remove all local patches) * ☑ [[http://bugs.python.org/issue18178|Issue #18178]]: Redefinition of malloc(3) family of functions at build time (Status: Upstreamed) * ☐ [[http://svnweb.freebsd.org/ports/head/lang/python27/files/patch-setup.py|Disable ncurses detection patch]] (Status: needs upstreamable patch) * [[http://www.freebsd.org/cgi/query-pr.cgi?pr=ports/171246|ports/171246]]: Make python curses module work with unicode * ☐ [[http://svnweb.freebsd.org/ports/head/lang/python27/files/extra-patch-setup.py|Posix P1003.1b semaphore support patch]] (Status: needs issue created) * [[http://www.freebsd.org/cgi/query-pr.cgi?pr=ports/178301|ports/178301]]: lang/python2[67]: SEM option is inconsistent * ☐ Fix gtk20's /usr/local/bin/gtk-builder-convert dependency to Python by splitting it up into an own slave port? * ☐ Integrate lang/pypy into ports framework (knobs etc) * [[https://reviews.freebsd.org/D8069|Mk/Uses/python.mk: add support for PyPy]] patch review * ☐ Python 3.x is the default Python language version * ☐ Python version agnostic DOCSDIR and EXAMPLESDIR; they currently need to be overridden manually - may conflict with the PYTHON_CONCURRENT_INSTALL efforts * ☐ Make it possible to add extra arguments to PYDISTUTILS_INSTALLARGS and friends without having to include bsd.port.pre.mk. * ☐ create guideline for which ports python@ will accept to maintain. In particular, address 'leaf ports' and 'non-infrastructure' ===== ☑ - Done ===== * ☑ Python ports framework complies with all relevant [[http://www.python.org/dev/peps/|PEP's]] * ☑ [[http://www.python.org/dev/peps/pep-0394|PEP-394]]: The "python" Command on Unix-Like Systems * ☑ [[http://www.python.org/dev/peps/pep-0449|PEP-449]]: Removal of the %%PyPI%% Mirror Auto Discovery and Naming Scheme - [[https://svnweb.freebsd.org/ports?view=revision&revision=400105|r400105]] * ☑ [[http://www.python.org/dev/peps/pep-3147|PEP-3147]]: PYC Repository Directories (//we use different site-packages/ and support concurrent installation (which creates distinct directories//) * ☑ Deprecate INSTALLS_EGGINFO: Use --record output to determine if a Python package installs .egg-info metadata files or directories. * 20141002: [[https://svnweb.freebsd.org/ports?view=revision&revision=369800|Commit r369800]] * ☑ --record %%<%%RECORD> can contain paths that include ../ or ./ * 20140925: [[http://svnweb.freebsd.org/ports?view=revision&revision=369296|Commit r369296]] * ☑ Split bsd.python.mk into a USES for consumer ports and an port-tree specific .mk file * 20140809: [[http://svnweb.freebsd.org/ports?view=revision&revision=364450|Commit r364450]] * ☑ Reduce shared/static build complexity in lang/pythonXX ports using lang/python34 as template * 20140802: [[http://svnweb.freebsd.org/ports?view=revision&revision=363790|Commit r363790]] [[http://svnweb.freebsd.org/ports?view=revision&revision=363730|Commit r363730]] [[http://svnweb.freebsd.org/ports?view=revision&revision=363788|Commit r363788]] * ☑ Remove easy_install specific bits in bsd.python.mk * 20140708: [[http://svnweb.freebsd.org/ports/head/Mk/bsd.python.mk&revision=361263|Commit r361263]] * ☑ PYDISTUTILS_SETUP does not strip arguments properly, which causes setuptools and distutils to jump to wrong conclusions. This is a nuisance right now (manifest_maker warnings), but can become a serious issue. * 20140628: [[http://svnweb.freebsd.org/ports?view=revision&revision=359663|Commit r369663]] * ☑ DATADIR, EXAMPLESDIR, binaries, scripts and files in ETCDIR need to be python prefix safe * 20140607: [[http://svnweb.freebsd.org/ports?view=revision&revision=356921|Commit r356921]] * ☑ Use ''%%python setup.py install --root=<path>%%'' instead of ''%%setup.py easy_install%%'' in Mk/bsd.python.mk * 20131215: [[http://svnweb.freebsd.org/ports?view=revision&revision=336553|Commit r336553]] * ☑ Ports limited to certain Python versions must not rely on ''%%${PREFIX}/bin/python%%'', but the proper python versions * 20131218: [[http://svnweb.freebsd.org/ports?view=revision&revision=336850|Commit r336850]] - lang/python has been removed as implicit build dependency * ☑ Chase the [[http://pythonhosted.org/setuptools/merge.html|distribute merge back into setuptools]] * 20131102: [[http://lists.freebsd.org/pipermail/freebsd-python/2013-November/006119.html|[RFC devel/py-setuptools]] * 20131127: [[http://svnweb.freebsd.org/ports?view=revision&revision=335043|Commit r335043]] (Credit: wg) * ☑ STAGE support for setuptools-based ports (''%%USE_PYDISTUTILS=easy_install%%'') * Mk/bsd.python.mk currently uses the ''%%easy_install setup.py%%'' target to install setuptools-based modules. * setuptools and distutils ''%%build%%'' and ''%%install%%'' targets support a --root option. The ''%%easy_install%%'' target and CLI command do not. * ☑ 20131110: Workaround easy_install by installing in stage dir and recompiling [[https://svnweb.freebsd.org/ports?view=revision&revision=333286|r333286]] (Credit: nivit, wg) * ☑ 20131029: [[http://lists.freebsd.org/pipermail/freebsd-python/2013-October/006077.html|freebsd-python: List of ports with "python 3" errors in poudriere]] * lang/python installation conflict * ☑ Retired devel/py-psyco ([[http://psyco.sourceforge.net/|12 March 2012: Psyco is unmaintained and dead]]) * 20131015: [[https://svnweb.freebsd.org/ports?view=revision&revision=330399|Commit r330399]] * ☑ Automate pkg-plist generation ([[https://svnweb.freebsd.org/ports?view=revision&revision=328403|r328403: PYDISTUTILS_AUTOPLIST]]) * ☑ Zope * <del>remove zope27</del> * <del>remove 28 29 210</del> * <del>update 211</del> (it's on it latest version already) * <del>import 212</del> (do we really need this?) * <del>import 213</del> * ☑ (plone (import plone4%%))%% * ☑ (turbogears2 (update to 2.1%%))%% * ☑ [EXP-RUN] [patch] lang/python: speed up upgrade-site-packages) [[http://www.freebsd.org/cgi/query-pr.cgi?pr=155970|ports/155970]] [[https://svnweb.freebsd.org/ports?view=revision&revision=304962|r304961]] ====== Other Interesting Things ====== * [[http://www.python.org/dev/peps/pep-0446/|PEP-446]] -- Make newly created file descriptors non-inheritable (Python 3.4, %%FreeBSD%% *_CLOEXEC Support) * Supercedes [[http://www.python.org/dev/peps/pep-0433/|PEP-433]] * [[http://www.python.org/dev/peps/pep-0427/|PEP-427]] -- The Wheel Binary Package Format 1.0 ====== Tooling Challenges ====== The upstream Python packaging team has expressed interest in understanding the challenges faced by OS packagers with Python packaging tools. This information will be used to inform ongoing improvements and developments. Please document your experiences below: * [[Ports/StageDir|StageDir]] with setuptools-based (easy_install) modules -- Doesnt support --destdir or equivalent ====== Steps to switch the Python default version ====== - A successful exp-run is mandatory - Set the new default version in * bsd.ports.default.mk * PYTHON3_DEFAULT * Uses/python.mk * _PYTHON_VERSIONS (retain order of preferred versions) * _PYTHON_PORTBRANCH - Bump the PORTREVISION of following ports * devel/py-setuptools * lang/cython * math/py-numpy - Add UPDATING instructions Legend: <code> pyMM, pythonMM, pythonM.M or MM = Old Python version, e.g. py37, python37, python3.7 or 37 pyNN, pythonNN, pythonN.N or NN = New Python version, e.g. py38, python38, python3.8 or 38 </code> Current instructions are: <code> The default version of python3 and python was switched to N.N. For ports users wanting to keep version M.M as default, add DEFAULT_VERSIONS+= python=M.M python3=M.M to make.conf Following procedures may ease the upgrade: # sh # for i in $(pkg query -g %n 'pyMM-*'); do pkg set -yn ${i}:pyNN-${i#pyMM-}; done # pkg upgrade For portmaster users: # sh # portmaster -o lang/pythonNN pythonMM # REINSTALL="$(pkg info -qo "*pyMM*")" # pkg delete -f "*pyMM*" # portmaster $REINSTALL # REBUILD=$(pkg query "%n:%dn" | grep py3 | grep -v pyNN | cut -d : -f 1 | sort -u) # portmaster $REBUILD # REBUILD2=$(pkg list | grep python-MM | xargs pkg which | awk '{print $6}' | sort -u) # portmaster $REBUILD2 </code> <callout type="warning"> Please note that the instructions from above for portmaster are not perfect for more complex scenarios of installed ports (see TODO below). </callout> ===== TODO for the instructions in UPDATING ===== Following parts should be added with the next Python switchover: ==== Refine instructions for Portmaster ==== With more complex scenarios of installed ports the upgrade might not go smoothly. There have been reported issues if following ports were installed: * databases/tdb * devel/gobject-introspection * devel/meson * devel/tevent There are improved instructions that still needs to be tested thoroughly: [[https://lists.freebsd.org/pipermail/freebsd-ports/2021-May/121155.html|Improved UPDATING instructions for Portmaster]] ==== Add clean up instructions ==== The following is a draft based on a suggestion from ThierryThomas (thierry) to improve the user experience. <code> Final steps (for pre-built packages & portmaster): If no longer required, Python M.M can be removed via "pkg remove pythonMM" and the directory /usr/local/lib/pythonM.M can then be deleted afterwards, if not empty. </code> ====== Python Team ====== * You! [[#contact|Come and say hi!]] :) * DaniloBaio * SofianBrabez * FukangChen * BabakFarrokhi * WilliamGrzybowski * WenHeping * PoChuanHsieh * LiWenHsu * KaiKnoblich * KubilayKocak * StevenKreuzer * CharlieLi * RuslanMakhmatkhanov (rm) * MuhammadMoinurRahman * NicolaVitale (nivit@) * [[Python/CompiledPackages|CompiledPackages]] * [[Python/PEP-517|PEP-517]] * [[Python/PortsPolicy|PortsPolicy]] * [[Python/PortsWishlist|PortsWishlist]] * [[Python/PyPy|PyPy]] * [[Python/setuptools|setuptools]] ---- {{tag>Ports}} {{tag>Team}} {{tag>Stale}} python.txt Last modified: 2023/01/29 14:29by GrahamPerrin