====== Ports and Compiled Python Packages ======

===== Preface =====

CPython's different setup mechanisms as well as the CPython interpreter itself create byte-compiled copies
(//.pyc// and sometimes //.pyo// prior to Python 3.5) of a Python file at installation or run-time. The byte-compiled versions of the
Python files are used in the first place by the interpreter to speed up loading and importing the files, when
Python code is to be executed by a program.

===== Trouble in Userland =====

Right now, both package installers, the former //pkg_install// and the current //pkg//, as well as the ports tree's
''%%make install%%'' and ''%%make deinstall%%'' mechanisms install Python packages including the byte-compiled
files.

At packaging and installation time, the files are picked up by the **//fake-pkg//** targets, a checksum for
them is created and written into ''%%${PKG_DBDIR}/${PKGNAME}/+CONTENTS%%'' (//pkg_install//) or the package checksum file
(//pkg//).

When a byte-compiled version of the original //.py// file is created, CPython stores the file's **st_mtime** and a
magic number within the byte-compiled file and recompiles those files on mismatches of the //.py// file's st_mtime or
the interpreter's magic number.

Recompiling those files leads to checksum mismatches for the package management tools, which might cause problems
on auditing secured environments or the behaviour of package build systems (which check the integrity).
If post-install hooks cause a recompile (since the st_mtime of a file as well as the installed Python interpreter's
magic number might be different on the target system, compared to the system on which the packages were generated),
the checksum verification will also fail for every package that installs python modules.

===== Mitigation considerations =====

On first glance, the most simple solution would be to avoid creating a checksum for those files, since the CPython
interpreter will not stop (re)compiling the files unless

  - ''%%PYTHONDONTWRITEBYTECODE%%'' is set in the environment or ''%%-B%%'' passed to ''%%${PYTHON_CMD}%%''
  - PEP-517 or distutils install commands pass the ''%%--no-compile%%'' parameter (to avoid the initial creation of //.pyc/////.pyo// files)
  - the CPython package directories (''%%${PYTHON_SITELIBDIR}%%'' and friends) are read-only and hence inaccessible for write operations

2) and 3) work as long as ports are built as a non-root user. When root, the CPython package directories are accessible for write operations, so bytecode is written, causing filesystem violations. 1) is not feasible unless ''%%login.conf%%'' is changed or another way to set the environment variable for the root user is realised.

The %%FreeBSD%% port maintainers and packaging tools are not in the position to decide for users whether byte-compiling files
should be performed or not. Hence the three requirements from above should be supported but not enforced on users.

===== Current approach =====

This is a hybrid of not compiling bytecode during the port build process and avoiding package checksum mismatches (described in historical approaches). Using //pkg// triggers, bytecode files are compiled or removed after the pkg transactions complete, mimicking what Python's own packaging tools like pip do by default. Only operates on ''%%${PYTHON_SITELIBDIR}%%'' and follows [[https://peps.python.org/pep-3147/|PEP-3147]]. Implemented as [[https://reviews.freebsd.org/D34739|D34739]]

This approach was chosen primarily due to the ''%%RECORD%%'' file from Python Packaging Authority's [[https://packaging.python.org/en/latest/specifications/recording-installed-packages/|recording installed packages]] standard not requiring bytecode to be included, and thus //devel/py-installer//, used during port build's install/stage, does not include them. (By contrast, the distutils install command's ''%%--record%%'' parameter does include bytecode.) While PEP-3147 specifies a uniform file and directory format for bytecode, it is not prudent to infer from that when such files are not included in ''%%RECORD%%''. Additionally, this provides the most flexibility for users to decide whether they want bytecode or not, especially when such an option in the //lang/python// ports is available. Implementations other than CPython do not compile or use bytecode, a CPython implementation detail, so packaging the same Python ports with those additional flavours would break with bytecode included in plists.

==== Deterministic bytecode ====

By default, the **st_mtime** checking between bytecode and original sources is not deterministic and thus results in non-reproducible builds. [[https://peps.python.org/pep-0552/|PEP-552]] addresses this by providing the option of verifying a deterministic hash instead. Passing the appropriate flags for this option to any compile commands during the port build process would have a higher chance of using packaged bytecode rather than immediate invalidation.

This approach was not chosen because the aforementioned pitfalls were not addressed. Further, no installation tool that %%PyPA%% supports supplies a parameter to pass ''%%invalidation_mode%%'' to ''%%compileall%%''.

===== Historical approaches =====

//Retained for historical purposes and commentary.//

==== Avoiding package checksum mismatches ====

=== Solution for pkg ===

In contrast to //pkg_install// //pkgng// picks up the ''%%${TMPPLIST}%%'' and creates the checksum internally, unreachable for any
post-checksum operation as for //pkg_install//. Hence ''%%pkg register%%'' should be enhanced to receive a set of excludes (e.g.
as regular expression) to ignore the checksum creation for files matching on it.

**//TODO://** //Implement/describe the pkgng extension.//

=== Solution for pkg_install ===

To avoid checksum mismatches for //pkg_install// (//pkg_delete// and ''%%make deinstall%%''), ''%%${PKG_DBDIR}/${PKGNAME}/+CONTENTS%%'' must
not contain checksums for the byte-compiled files. Since changing //pkg_install// just for this case is not a viable option, the
necessary code might go into the port tree's Makefile infrastructure.

By enhancing the **fake-pkg** target in ''%%bsd.port.mk%%'', we can strip the offending checksums quite easily, while at the same time
we can avoid messing around with ''%%${TMPPLIST}%%'', causing side effects for other ports.

<code>
 		${ECHO_MSG} "===>   Registering installation for ${PKGNAME}"; \
 		${MKDIR} ${PKG_DBDIR}/${PKGNAME}; \
 		${PKG_CMD} ${PKG_ARGS} -O ${PKGFILE} > ${PKG_DBDIR}/${PKGNAME}/+CONTENTS; \
+		${SED} -i -e '/\.py[co]$/{n;d;}' ${PKG_DBDIR}/${PKGNAME}/+CONTENTS; \
 		${CP} ${DESCR} ${PKG_DBDIR}/${PKGNAME}/+DESC; \
 		${ECHO_CMD} ${COMMENT:Q} > ${PKG_DBDIR}/${PKGNAME}/+COMMENT; \
 		if [ -f ${PKGINSTALL} ]; then \
</code>

**//Note://** //[[http://people.freebsd.org/~mva/python_checksum.patch]]// will always contain the most recent version of the patch.

==== Avoid byte-compiling ====

If checksums are not created anymore, audits based on the packaging tools create a hole for the byte-compiled files, since
the auditor cannot tell, if they were modified without keeping track of their **st_mtime** elsewhere, creating a possible
security issue. Additionally, avoiding byte-compiled files can save diskspace by reducing the installed package size to a
minimum of necessary files.

=== pkg ===

**//TODO://** //Is it necessary for pkgng to do something here?//

=== pkg_install ===

To create a sort of //frozen// package without any byte-compiled files, byte-compiling should be made optional and it should
be up to the user, whether byte-compiling is wanted or not. For the ports tree, it would be necessary to make compiling at
installation time optional, so ''%%distutils%%'' and ''%%easy_install%%'' are aware of a //--no-compile// flag.

At the moment ''%%PYDISTUTILS_INSTALLARGS%%'' enforces the //-c -O1// options, causing //.pyc// and //.pyo// files to be created
without giving a user the chance to influence the behaviour. This however is a necessity, so that users do not have to deal
with potential security issues.

To make byte-compiling optional, several prerequisites have to be met.

  * pkg-plist should not know anything about .pyc/.pyo files to avoid extensive post-operations to clean up everything
  * compile args shoud be made optional, allowing the user to turn them on and off
  * depending on the user setting, ''%%${TMPPLIST}%%'' needs to be populated automatically with the necessary entries for the byte-compiled files.

''%%distutils%%'' (and ''%%easy_install%%'', since it just utilizes ''%%distutils%%'') differentiate between C extensions and pure python
extensions and use different intermediate build directories. This needs to be aligned so that the last requirement (auto-populating)
can be met for mixed package installations.

  * the introduction of a ''%%${PYDISTUTILS_BUILDDIR}%%'' allows ports and the Makefile structure to set it as required
  * a new ''%%${PYDISTUTILS_BUILDARGS}%%'' is necessary to change ''%%distutils%%'''s ''%%--build-platlib%%'' and ''%%--build-purelib%%'' directories to the build directory

This change enables us to track which Python files are installed as Python packages and modules (and hence
would be byte-compiled by default).

<code>
-PYDISTUTILS_BUILDARGS?=
-PYDISTUTILS_INSTALLARGS?=	-c -O1 --prefix=${PREFIX}
+PYDISTUTILS_BUILDDIR?=		${WRKSRC}/build/lib
+PYDISTUTILS_BUILDARGS?=		--build-platlib ${PYDISTUTILS_BUILDDIR} --build-purelib ${PYDISTUTILS_BUILDDIR}
+.if !defined(WITHOUT_PYTHON_BYTECOMPILE)
+PYDISTUTILS_COMPILEARGS?=	-c -O1
+.else
+PYDISTUTILS_COMPILEARGS?=	--no-compile
+.endif
+PYDISTUTILS_INSTALLARGS?=	--prefix=${PREFIX}
</code>

To populate the ''%%${TMPPLIST}%%'' automatically, ''%%${PYDISTUTILS_BUILDDIR}%%'' now can be scanned for any //*.py// file.
For each file, a //.pyc// and //.pyo// entry can be added to ''%%${TMPPLIST}%%''.

<code>
.if defined(USE_PYDISTUTILS) && !defined(WITHOUT_PYTHON_BYTECOMPILE)
_RELDIR=	${PYTHONPREFIX_SITELIBDIR:S/^${PREFIX}\///}
add-plist-post:	add-plist-pyc
add-plist-pyc:
	@${TOUCH} ${TMPPLIST}.pyc_tmp
	@for i in `find ${PYDISTUTILS_BUILDDIR} -type f -name '*.py'`; do \
		PYC=`${ECHO_CMD} $$i | ${SED} "s|.py$$|.pyc|"`; \
		NEWC=`${ECHO_CMD} $${PYC} | ${SED} "s|${PYDISTUTILS_BUILDDIR}||"`; \
		NEWO=`${ECHO_CMD} $${NEWC} | ${SED} "s|.pyc$$|.pyo|"`; \
		${ECHO_CMD} "${_RELDIR}$${NEWC}" >> ${TMPPLIST}.pyc_tmp; \
		${ECHO_CMD} "${_RELDIR}$${NEWO}" >> ${TMPPLIST}.pyc_tmp; \
	done; \
	${CAT} ${TMPPLIST} >> ${TMPPLIST}.pyc_tmp; \
	${CAT} ${TMPPLIST}.pyc_tmp > ${TMPPLIST};

.endif
</code>

The last task to do is cleaning up **//all//** pkg-plist files in the ports tree from the //.pyc// and //.pyo// entries.

**//Note://** //http://people.freebsd.org/~mva/pyc_compile.bsd.python.mk.patch will always contain the most recent version of the patch.//