| Both sides previous revision Previous revision Next revision | Previous revision |
| python:pypy [2013/05/06 19:17] – DavidNaylor | python:pypy [2022/08/15 05:17] (current) – Add title section, remove uncessary horizontal rule KubilayKocak |
|---|
| ---- | ====== PyPy ====== |
| |
| The development for lang/pypy is done at [[http://github.com/DragonSA/pypy|GitHub]]. | ====== Building ====== |
| |
| ====== Packages ====== | Building %%PyPy%% can be done using the Ports Collection or directly from source. The former is recommended. |
| Pkgng packages for all supported %%FreeBSD%% versions will be provided for i386 and amd64 in due course. | |
| |
| ====== Building ====== | The %%PyPy%% build process has two phases: |
| Building [[/PyPy|PyPy]] can be done using the Ports Collection or directly from source. The former is recommended. | * The translation phase converts the language interpreter, written in RPython, into C code. This is done using a python interpreter (currently either cPython or %%PyPy%%). This is a single thread task. |
| | |
| The [[/PyPy|PyPy]] build process has two phases: | |
| * The translation phase converts the language interpreter, written in RPython, into C code. This is done using a python interpreter (currently either cPython or [[/PyPy|PyPy]]). This is a single thread task. | |
| * The compilation phase compiles the C code into a single executable. This is a highly parallel task. | * The compilation phase compiles the C code into a single executable. This is a highly parallel task. |
| |
| See below for detailed discussions on the various aspects of building [[/PyPy|PyPy]]. | See below for detailed discussions on the various aspects of building %%PyPy%%. |
| |
| To install [[/PyPy|PyPy]] from ports: | To install %%PyPy%% from ports: |
| |
| ''%%# make -C /usr/ports/lang/pypy install%%'' | ''%%# make -C /usr/ports/lang/pypy install%%'' |
| |
| ===== Instances ===== | |
| [[/PyPy|PyPy]] supports a diverse set of configurations, the port supports installing multiple instances of [[/PyPy|PyPy]] that is built using different sets of configurations. Each set of configurations is called an //instance//. Four predefined instances are provided by the port (two are broken) and additional instances can be specified (see the following section). | |
| |
| The default instance (called ''%%DEFAULT%%'') is always built. The remaining three instances can be included using the port's options. | |
| |
| ^ Name ^ Binary ^ Options ^ Description ^ | |
| | ''%%DEFAULT%%'' | ''%%pypy%%'' | ''%%--Ojit%%'' | Default [[/PyPy|PyPy]] with JIT | | |
| | ''%%SANDBOX%%'' | ''%%pypy-sandbox%%'' | ''%%--sandbox -Ojit%%'' | Sandboxed [[/PyPy|PyPy]], requires special supervisor to run | | |
| | ''%%CLI%%'' | ''%%pypy-cli%%'' | ''%%--backend=cli -O2%%'' | (BROKEN) [[/PyPy|PyPy]] that runs on a .NET virtual machine | | |
| | ''%%JVM%%'' | ''%%pypy-jvm%%'' | ''%%--backend=jvm -O2%%'' | (BROKEN) [[/PyPy|PyPy]] that runs on a Java virtual machine | | |
| |
| The ''%%jit%%'' optimisation level is converted to ''%%2%%'' for platforms that are not supported by the JIT. The predefined instances are defined at ''%%${PORTSDIR}/lang/pypy/files/bsd.pypy.inst.mk%%''. | |
| |
| ===== Custom Instances ===== | |
| Custom instances can be specified for the port to build and install. It is recommended that the custom instance specification is placed in ''%%/etc/make.conf%%'' as the full specification is required for each invocation of ''%%make%%'' (i.e. the port does not remember the specification between stages). | |
| |
| The list of instances to build is passed by the ''%%PYPY_INST%%'' environment variable. When this environment variable is defined all predefined instances (including ''%%DEFAULT%%'') are //not// automatically added. The predefined instances may be added manually. | |
| |
| The list of variables that defines an instance specification are: | |
| |
| ^ Variable ^ Description ^ | |
| | ''%%PYPY_${inst}_NAME%%'' | The binary name | | |
| | ''%%PYPY_${inst}_TRANSLATION_ARGS%%'' | The translation arguments | | |
| | ''%%PYPY_${inst}_OPT%%'' | The optimisation level | | |
| | ''%%PYPY_${inst}_OBJSPACE_ARGS%%'' | The object space arguments | | |
| |
| ''%%${inst%%''} is the instance name, as passed in ''%%PYPY_INST%%''. See [[http://doc.pypy.org/en/latest/config/|PyPy documentation]] for a list of supported parameters. | |
| |
| ==== Example ==== | |
| The following example defines a custom instance, with name ''%%CUSTOM%%'' and binary name ''%%pypy-custom%%'', and also includes the ''%%DEFAULT%%'' and ''%%SANDBOX%%'' predefined instances. ''%%DEFAULT%%'' is defined first as the first instance is used to compile the python files (.py -> .pyc) and needs to be a stand-alone instance (''%%SANDBOX%%'' requires a supervisor to work). | |
| |
| <code> | |
| # cat >> /etc/make.conf << _EOF | |
| PYPY_INST= DEFAULT SANDBOX CUSTOM | |
| PYPY_CUSTOM_NAME= pypy-custom | |
| PYPY_CUSTOM_TRANSLATE_ARGS= --gcrootfinder=shadowstack --gc=generation | |
| PYPY_CUSTOM_OPT= 0 | |
| PYPY_CUSTOM_OBJSPACE_ARGS= --no-objspace-usepycfiles --objspace=thunk | |
| _EOF | |
| # make -C /usr/ports/lang/pypy all install | |
| </code> | |
| |
| ===== Memory Requirements ===== | |
| Building [[/PyPy|PyPy]] requires a lot of memory. Most systems are not capable of building [[/PyPy|PyPy]], if this applies to your system please consider using a package as listed above. [[/PyPy|PyPy]] is also capable of building itself which, thanks to the JIT, is faster than using Python but also uses more memory. This behaviour is a port option, dependent on ''%%${LOCALBASE}/bin/pypy%%'' being present. | |
| |
| By default [[/PyPy|PyPy]] builds all instances in parallel which stacks the required memory (i.e. each build will use the maximum memory listed below). If your system has sufficient memory to build one [[/PyPy|PyPy]] instance but not multiple instances use the build flag ''%%-DDISABLE_MAKE_JOBS%%'' to build the [[/PyPy|PyPy]] instances in series. | |
| |
| The [[/PyPy|PyPy]] port checks if your system has sufficient memory and refuses to build if it detects insufficient memory. This behaviour can be overwritten with the build flag ''%%-DPYPY_IGNORE_MEMORY%%''. Each extra level of optimisation requires more memory when building [[/PyPy|PyPy]], with the JIT taking the most memory, so consider using a lower level of optimisation if memory is a limitation and disabling the memory checks however the primary reason to use [[/PyPy|PyPy]] is for the JIT and the associated performance improvements. | |
| |
| The following memory requirements were determined using ''%%sh -c "rm -f /tmp/stop; (while [ ! -f /tmp/stop ]; do ps -o rss,comm | grep -R '(python|pypy)'; sleep 1; done) | sort -n | tail -n 1"%%'' on an x86_64 system for building the DEFAULT instance: | |
| |
| ^ Translator ^ 32-bit ^ 64-bit ^ | |
| | python2.7 | 2.1GiB | 4.3GiB | | |
| | pypy2.0 | 2.4GiB | 5.5GiB | | |
| | System | 0.2GiB | 0.4GiB | | |
| |
| See ''%%${PORTSDIR}/lang/pypy/Makefile%%'' for more accurate figures. | |
| |
| ===== Compilation Time ===== | |
| [[/PyPy|PyPy]] takes a long time to build. The "translation" phase of building [[/PyPy|PyPy]] can take most of the time. If concurrency is disables, via ''%%-DDISABLE_MAKE_JOBS%%'', then the compilation phase can also take a signification amount of time. | |
| |
| Each instance built can be translated concurrently if concurrency is not disabled. | |
| |
| The following times are based on an Intel Core i7-2600 (4 physical cores + hyper-threading = 8 logical cores) for building the DEFAULT instance: | |
| |
| ^ Translator ^ Phase ^ 32-bit ^ 64-bit ^ | |
| | python2.7 | Translation | 83 mins | 74 mins | | |
| | ::: | Compilation | 6 mins | 6 mins | | |
| | ::: | //Total// | //89 mins// | //80 mins// | | |
| | pypy2.0 | Translation | 38 mins | 35 mins | | |
| | ::: | Compilation | 6 mins | 6 mins | | |
| | ::: | //Total// | //44 mins// | //41 mins// | | |
| |
| ===== Building without port ===== | ===== Building without port ===== |
| See [[http://pypy.org/download.html#building-from-source|PyPy instructions]] on building [[/PyPy|PyPy]] directly from source. | See [[http://pypy.org/download.html#building-from-source|PyPy instructions]] on building %%PyPy%% directly from source. |
| | |
| ====== Status ====== | |
| [[/PyPy|PyPy]] builds and runs under %%FreeBSD%% under most configurations. Some configurations are not supported upstream and others do not work under %%FreeBSD%%. Below is a list of what does not work under %%FreeBSD%% and/or upstream: | |
| * //Boehm GC:// The Boehm GC works upstream but does not work under %%FreeBSD%% | |
| * //Unit Tests:// All unit tests pass upstream, many tests do not pass under %%FreeBSD%% (and at least one test segfaults pypy). Many %%FreeBSD%% failures are fixed upstream so the stable release is not a fair reflection. | |
| * //ASMGCC GC:// The ASMGCC GC is not supported on non-Linux systems and only works with particular GCC compilers with particular compiler options (i.e. very temperamental). | |
| * //CLI backend:// The CLI (.NET) based [[/PyPy|PyPy]] is not supported upstream | |
| * //JVM backend:// The JVM (Java) based [[/PyPy|PyPy]] is not supported upstream | |
| |
| ====== TODO ====== | ====== TODO ====== |
| * Detect peak memory usage using ''%%time -l%%'' | * Properly check sandbox support and create sandbox VM-thing (does upstream support sandboxing?) |
| * Detect MAKE_JOBS_NUMBER and use min(MAKE_JOBS_NUMBER, COUNT(PYPY_INST%%))%% when detecting memory requirements. | * Add pypy-based numpy port??? |
| * Make a pypy-devel port (of nightlies, for fixing pypy on %%FreeBSD%%) | * Make a jitviewer port (depends on flask pygments simplejson Jinja2 Werkzeug) |
| * Make a jitviewer port (depends pn flask pygments simplejson Jinja2 Werkzeug) | |
| * Support valgrind | * Support valgrind |
| * Properly check sandbox support and create sandbox VM-thing | * Integrate %%PyPy%% into bsd.python.mk |
| * Integrate [[/PyPy|PyPy]] into bsd.python.mk | |
| * Upstream appropriate patches | |
| |