python:compiledpackages

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
python:compiledpackages [2012/06/03 20:02] – typo MarkLinimonpython:compiledpackages [2023/02/15 21:56] (current) – typo, mention -B CharlieLi
Line 4: Line 4:
  
 CPython's different setup mechanisms as well as the CPython interpreter itself create byte-compiled copies CPython's different setup mechanisms as well as the CPython interpreter itself create byte-compiled copies
-(//.pyc// and/or //.pyo//) of a Python file at installation or run-time. The byte-compiled versions of the+(//.pyc// and sometimes //.pyo// prior to Python 3.5) of a Python file at installation or run-time. The byte-compiled versions of the
 Python files are used in the first place by the interpreter to speed up loading and importing the files, when Python files are used in the first place by the interpreter to speed up loading and importing the files, when
 Python code is to be executed by a program. Python code is to be executed by a program.
Line 10: Line 10:
 ===== Trouble in Userland ===== ===== Trouble in Userland =====
  
-Right now, both package installers, //pkg_install// and the upcoming //pkgng//, as well as the ports tree's+Right now, both package installers, the former //pkg_install// and the current //pkg//, as well as the ports tree's
 ''%%make install%%'' and ''%%make deinstall%%'' mechanisms install Python packages including the byte-compiled ''%%make install%%'' and ''%%make deinstall%%'' mechanisms install Python packages including the byte-compiled
 files. files.
Line 16: Line 16:
 At packaging and installation time, the files are picked up by the **//fake-pkg//** targets, a checksum for At packaging and installation time, the files are picked up by the **//fake-pkg//** targets, a checksum for
 them is created and written into ''%%${PKG_DBDIR}/${PKGNAME}/+CONTENTS%%'' (//pkg_install//) or the package checksum file them is created and written into ''%%${PKG_DBDIR}/${PKGNAME}/+CONTENTS%%'' (//pkg_install//) or the package checksum file
-(//pkgng//).+(//pkg//).
  
 When a byte-compiled version of the original //.py// file is created, CPython stores the file's **st_mtime** and a When a byte-compiled version of the original //.py// file is created, CPython stores the file's **st_mtime** and a
Line 28: Line 28:
 the checksum verification will also fail for every package that installs python modules. the checksum verification will also fail for every package that installs python modules.
  
-===== Avoiding the Trouble =====+===== Mitigation considerations =====
  
-To avoid those issues, the most simple solution would be to avoid creating a checksum for those files, since the CPython +On first glance, the most simple solution would be to avoid creating a checksum for those files, since the CPython 
-interpreter will not stop (re)compiling the files as long as+interpreter will not stop (re)compiling the files unless
  
-  * ''%%PYTHONDONTWRITEBYTECODE%%'' is set in the environment +  - ''%%PYTHONDONTWRITEBYTECODE%%'' is set in the environment or ''%%-B%%'' passed to ''%%${PYTHON_CMD}%%'' 
-  * CPython's distutils hooks set the --no-compile parameter (to avoid the initial creation of //.pyc/////.pyo// files) +  - PEP-517 or distutils install commands pass the ''%%--no-compile%%'' parameter (to avoid the initial creation of //.pyc/////.pyo// files) 
-  * the CPython package directories (''%%${PYTHON_SITELIBDIR}%%'' and friends) are read-only and hence inaccessible for write operations+  - the CPython package directories (''%%${PYTHON_SITELIBDIR}%%'' and friends) are read-only and hence inaccessible for write operations
  
-The %%FreeBSD%% port maintainers and packaging tools are not in the position to decide for users whether byte-ompiling files+2) and 3) work as long as ports are built as a non-root user. When root, the CPython package directories are accessible for write operations, so bytecode is written, causing filesystem violations. 1) is not feasible unless ''%%login.conf%%'' is changed or another way to set the environment variable for the root user is realised. 
 + 
 +The %%FreeBSD%% port maintainers and packaging tools are not in the position to decide for users whether byte-compiling files
 should be performed or not. Hence the three requirements from above should be supported but not enforced on users. should be performed or not. Hence the three requirements from above should be supported but not enforced on users.
  
-==== Solution for pkg_install ====+===== Current approach ===== 
 + 
 +This is a hybrid of not compiling bytecode during the port build process and avoiding package checksum mismatches (described in historical approaches). Using //pkg// triggers, bytecode files are compiled or removed after the pkg transactions complete, mimicking what Python's own packaging tools like pip do by default. Only operates on ''%%${PYTHON_SITELIBDIR}%%'' and follows [[https://peps.python.org/pep-3147/|PEP-3147]]. Implemented as [[https://reviews.freebsd.org/D34739|D34739]] 
 + 
 +This approach was chosen primarily due to the ''%%RECORD%%'' file from Python Packaging Authority's [[https://packaging.python.org/en/latest/specifications/recording-installed-packages/|recording installed packages]] standard not requiring bytecode to be included, and thus //devel/py-installer//, used during port build's install/stage, does not include them. (By contrast, the distutils install command's ''%%--record%%'' parameter does include bytecode.) While PEP-3147 specifies a uniform file and directory format for bytecode, it is not prudent to infer from that when such files are not included in ''%%RECORD%%''. Additionally, this provides the most flexibility for users to decide whether they want bytecode or not, especially when such an option in the //lang/python// ports is available. Implementations other than CPython do not compile or use bytecode, a CPython implementation detail, so packaging the same Python ports with those additional flavours would break with bytecode included in plists. 
 + 
 +==== Deterministic bytecode ==== 
 + 
 +By default, the **st_mtime** checking between bytecode and original sources is not deterministic and thus results in non-reproducible builds. [[https://peps.python.org/pep-0552/|PEP-552]] addresses this by providing the option of verifying a deterministic hash instead. Passing the appropriate flags for this option to any compile commands during the port build process would have a higher chance of using packaged bytecode rather than immediate invalidation. 
 + 
 +This approach was not chosen because the aforementioned pitfalls were not addressed. Further, no installation tool that %%PyPA%% supports supplies a parameter to pass ''%%invalidation_mode%%'' to ''%%compileall%%''. 
 + 
 +===== Historical approaches ===== 
 + 
 +//Retained for historical purposes and commentary.// 
 + 
 +==== Avoiding package checksum mismatches ==== 
 + 
 +=== Solution for pkg === 
 + 
 +In contrast to //pkg_install// //pkgng// picks up the ''%%${TMPPLIST}%%'' and creates the checksum internally, unreachable for any 
 +post-checksum operation as for //pkg_install//. Hence ''%%pkg register%%'' should be enhanced to receive a set of excludes (e.g. 
 +as regular expression) to ignore the checksum creation for files matching on it. 
 + 
 +**//TODO://** //Implement/describe the pkgng extension.// 
 + 
 +=== Solution for pkg_install ===
  
 To avoid checksum mismatches for //pkg_install// (//pkg_delete// and ''%%make deinstall%%''), ''%%${PKG_DBDIR}/${PKGNAME}/+CONTENTS%%'' must To avoid checksum mismatches for //pkg_install// (//pkg_delete// and ''%%make deinstall%%''), ''%%${PKG_DBDIR}/${PKGNAME}/+CONTENTS%%'' must
 not contain checksums for the byte-compiled files. Since changing //pkg_install// just for this case is not a viable option, the not contain checksums for the byte-compiled files. Since changing //pkg_install// just for this case is not a viable option, the
-necessary code might go into the port tree's Makefile Infrastructure.+necessary code might go into the port tree's Makefile infrastructure.
  
 By enhancing the **fake-pkg** target in ''%%bsd.port.mk%%'', we can strip the offending checksums quite easily, while at the same time By enhancing the **fake-pkg** target in ''%%bsd.port.mk%%'', we can strip the offending checksums quite easily, while at the same time
-we can avoid to mess around with ''%%${TMPPLIST}%%'', causing side effects for other ports.+we can avoid messing around with ''%%${TMPPLIST}%%'', causing side effects for other ports.
  
 <code> <code>
Line 61: Line 89:
 **//Note://** //[[http://people.freebsd.org/~mva/python_checksum.patch]]// will always contain the most recent version of the patch. **//Note://** //[[http://people.freebsd.org/~mva/python_checksum.patch]]// will always contain the most recent version of the patch.
  
-==== Solution for pkgng ==== +==== Avoid byte-compiling ====
- +
-In contrast to //pkg_install// //pkgng// picks up the ''%%${TMPPLIST}%%'' and creates the checksum internally, unreachable for any +
-post-checksum operation as for //pkg_install//. Hence ''%%pkg register%%'' should be enhanced to receive a set of excludes (e.g. +
-as regular expression) to ignore the checksum creation for files matching on it. +
- +
-**//TODO://** //Implement/describe the pkgng extension.// +
- +
-===== Avoid byte-compiling =====+
  
 If checksums are not created anymore, audits based on the packaging tools create a hole for the byte-compiled files, since If checksums are not created anymore, audits based on the packaging tools create a hole for the byte-compiled files, since
Line 76: Line 96:
 minimum of necessary files. minimum of necessary files.
  
-==== pkg_install ====+=== pkg === 
 + 
 +**//TODO://** //Is it necessary for pkgng to do something here?// 
 + 
 +=== pkg_install ===
  
 To create a sort of //frozen// package without any byte-compiled files, byte-compiling should be made optional and it should To create a sort of //frozen// package without any byte-compiled files, byte-compiling should be made optional and it should
Line 83: Line 107:
  
 At the moment ''%%PYDISTUTILS_INSTALLARGS%%'' enforces the //-c -O1// options, causing //.pyc// and //.pyo// files to be created At the moment ''%%PYDISTUTILS_INSTALLARGS%%'' enforces the //-c -O1// options, causing //.pyc// and //.pyo// files to be created
-without giving a user the choice to influence the behaviour. This however is a necessity, so that users do not have to deal+without giving a user the chance to influence the behaviour. This however is a necessity, so that users do not have to deal
 with potential security issues. with potential security issues.
  
Line 99: Line 123:
   * a new ''%%${PYDISTUTILS_BUILDARGS}%%'' is necessary to change ''%%distutils%%'''s ''%%--build-platlib%%'' and ''%%--build-purelib%%'' directories to the build directory   * a new ''%%${PYDISTUTILS_BUILDARGS}%%'' is necessary to change ''%%distutils%%'''s ''%%--build-platlib%%'' and ''%%--build-purelib%%'' directories to the build directory
  
-This change enables us to track which Python files are installed as Python packages and moudles (and hence+This change enables us to track which Python files are installed as Python packages and modules (and hence
 would be byte-compiled by default). would be byte-compiled by default).
  
Line 140: Line 164:
  
 **//Note://** //http://people.freebsd.org/~mva/pyc_compile.bsd.python.mk.patch will always contain the most recent version of the patch.// **//Note://** //http://people.freebsd.org/~mva/pyc_compile.bsd.python.mk.patch will always contain the most recent version of the patch.//
- 
-==== pkgng ==== 
- 
-**//TODO://** //Is it necessary for pkgng to do something here?// 
  
  • python/compiledpackages.1338753753.txt.gz
  • Last modified: 2012/06/03 20:02
  • by MarkLinimon