This is an old revision of the document!
The development for lang/pypy is done at GitHub.
Packages
Pkgng packages for all supported FreeBSD versions will be provided for i386 and amd64 in due course.
Building
Building PyPy can be done using the Ports Collection or directly from source. The former is recommended.
The PyPy build process has two phases:
- The translation phase converts the language interpreter, written in RPython, into C code. This is done using a python interpreter (currently either cPython or PyPy). This is a single thread task.
- The compilation phase compiles the C code into a single executable. This is a highly parallel task.
See below for detailed discussions on the various aspects of building PyPy.
To install PyPy from ports:
# make -C /usr/ports/lang/pypy install
Instances
PyPy supports a diverse set of configurations, the port supports installing multiple instances of PyPy that is built using different sets of configurations. Each set of configurations is called an instance. Four predefined instances are provided by the port (two are broken) and additional instances can be specified (see the following section).
The default instance (called DEFAULT) is always built. The remaining three instances can be included using the port's options.
| Name | Binary | Options | Description |
|---|---|---|---|
DEFAULT | pypy | --Ojit | Default PyPy with JIT |
SANDBOX | pypy-sandbox | --sandbox -Ojit | Sandboxed PyPy, requires special supervisor to run |
CLI | pypy-cli | --backend=cli -O2 | (BROKEN) PyPy that runs on a .NET virtual machine |
JVM | pypy-jvm | --backend=jvm -O2 | (BROKEN) PyPy that runs on a Java virtual machine |
The jit optimisation level is converted to 2 for platforms that are not supported by the JIT. The predefined instances are defined at ${PORTSDIR}/lang/pypy/files/bsd.pypy.inst.mk.
Custom Instances
Custom instances can be specified for the port to build and install. It is recommended that the custom instance specification is placed in /etc/make.conf as the full specification is required for each invocation of make (i.e. the port does not remember the specification between stages).
The list of instances to build is passed by the PYPY_INST environment variable. When this environment variable is defined all predefined instances (including DEFAULT) are not automatically added. The predefined instances may be added manually.
The list of variables that defines an instance specification are:
| Variable | Description |
|---|---|
PYPY_${inst}_NAME | The binary name |
PYPY_${inst}_TRANSLATION_ARGS | The translation arguments |
PYPY_${inst}_OPT | The optimisation level |
PYPY_${inst}_OBJSPACE_ARGS | The object space arguments |
${inst} is the instance name, as passed in PYPY_INST. See PyPy documentation for a list of supported parameters.
Example
The following example defines a custom instance, with name CUSTOM and binary name pypy-custom, and also includes the DEFAULT and SANDBOX predefined instances. DEFAULT is defined first as the first instance is used to compile the python files (.py → .pyc) and needs to be a stand-alone instance (SANDBOX requires a supervisor to work).
# cat >> /etc/make.conf << _EOF PYPY_INST= DEFAULT SANDBOX CUSTOM PYPY_CUSTOM_NAME= pypy-custom PYPY_CUSTOM_TRANSLATE_ARGS= --gcrootfinder=shadowstack --gc=generation PYPY_CUSTOM_OPT= 0 PYPY_CUSTOM_OBJSPACE_ARGS= --no-objspace-usepycfiles --objspace=thunk _EOF # make -C /usr/ports/lang/pypy all install
Memory Requirements
Building PyPy requires a lot of memory. Most systems are not capable of building PyPy, if this applies to your system please consider using a package as listed above. PyPy is also capable of building itself which, thanks to the JIT, is faster than using Python but also uses more memory. This behaviour is a port option, dependent on ${LOCALBASE}/bin/pypy being present.
By default PyPy builds all instances in parallel which stacks the required memory (i.e. each build will use the maximum memory listed below). If your system has sufficient memory to build one PyPy instance but not multiple instances use the build flag -DDISABLE_MAKE_JOBS to build the PyPy instances in series.
The PyPy port checks if your system has sufficient memory and refuses to build if it detects insufficient memory. This behaviour can be overwritten with the build flag -DPYPY_IGNORE_MEMORY. Each extra level of optimisation requires more memory when building PyPy, with the JIT taking the most memory, so consider using a lower level of optimisation if memory is a limitation and disabling the memory checks however the primary reason to use PyPy is for the JIT and the associated performance improvements.
The following memory requirements were determined using sh -c "rm -f /tmp/stop; (while [ ! -f /tmp/stop ]; do ps -o rss,comm | grep -R '(python|pypy)'; sleep 1; done) | sort -n | tail -n 1" on an x86_64 system for building the DEFAULT instance:
| Translator | 32-bit | 64-bit |
|---|---|---|
| python2.7 | 2.1GiB | 4.3GiB |
| pypy2.0 | 2.4GiB | 5.5GiB |
| System | 0.2GiB | 0.4GiB |
See ${PORTSDIR}/lang/pypy/Makefile for more accurate figures.
Compilation Time
PyPy takes a long time to build. The “translation” phase of building PyPy can take most of the time. If concurrency is disables, via -DDISABLE_MAKE_JOBS, then the compilation phase can also take a signification amount of time.
Each instance built can be translated concurrently if concurrency is not disabled.
The following times are based on an Intel Core i7-2600 (4 physical cores + hyper-threading = 8 logical cores) for building the DEFAULT instance:
| Translator | Phase | 32-bit | 64-bit |
| python2.7 | Translation | 83 mins | 74 mins |
| Compilation | 6 mins | 6 mins | |
| Total | 89 mins | 80 mins | |
| pypy2.0 | Translation | 38 mins | 90 mins |
| Compilation | 6 mins | ?? mins | |
| Total | 44 mins | ??? mins |
Building without port
See instructions provided by the PyPy project on building PyPy directly from source.
Status
PyPy builds and runs under FreeBSD under most configurations. Some configurations are not supported upstream and others do not work under FreeBSD. Below is a list of what does not work under FreeBSD and/or upstream:
- Boehm GC: The Boehm GC works upstream but does not work under FreeBSD
- Unit Tests: All unit tests pass upstream, many tests do not pass under FreeBSD (and at least on test segfaults pypy). Many FreeBSD failures are fixed upstream so the stable release is not a fair reflection.
- ASMGCC GC: The ASMGCC GC is not supported on non-Linux systems and only works with particular GCC compilers with particular compiler options (i.e. very temperamental).
- CLI backend: The CLI (.NET) based PyPy is not supported upstream
- JVM backend: The JVM (Java) based PyPy is not supported upstream
TODO
- Detect MAKE_JOBS_NUMBER and use min(MAKE_JOBS_NUMBER, COUNT(PYPY_INST)) when detecting memory requirements.
- Make a pypy-devel port (of nightlies, for fixing pypy on FreeBSD)
- Make a jitviewer port (depends pn flask pygments simplejson Jinja2 Werkzeug)
- Support valgrind
- Properly check sandbox support and create sandbox VM-thing
- Integrate PyPy into bsd.python.mk
- Upstream appropriate patches