Virtual Memory Compression

• Student: Paavo-Einari Kaipila
• Mentors: Edward Napierala and Mark Johnston

Virtual memory compression is a memory management technique implemented by multiple operating systems ranging from Windows to Darwin. Depending on the implementation, it can either reduce swapping to a physical storage medium or completely replace it. In the vast majority of systems, compression and decompression are generally known to be faster than disk IO. It has also been reported to offer energy savings.

• zram on Linux. Enabled by default on Android and Chromium OS.
• zswap on Linux.
• Darwin since 13.0. Enabled at least in macOS, perhaps on other Apple systems too.
• Windows since 10525

• Systems without writable storage
• Systems with extremely slow writable storage (eg. hard drive, eMMC)
• Systems with very limited write cycles (eg. eMMC, old solid state drives)
• Systems that do not need excessive amounts (more than physical memory) of swap and want to benefit from increased speed

The primary goal is to implement a compressed in-memory disk that can be utilized as a swap storage. The current plan is to integrate it to existing md(4) and a corresponding userland configuration tool mdconfig(8). Lz4 will be the default algorithm due to its good speed/compression ratio, while zstd with a configurable compression level is also available as an option. I will also investigate an idea that I’ve never seen implemented before: a compressed memory disk that by default uses a very fast compression algorithm (lz4) but changes to a more compact but slower one (zstd with an aggressive compression level) when under extreme memory pressure. I call this adaptive compression. The idea is to use a fast algorithm untill x % of RAM is utilized and to actively start recompressing existing data using a more compact algorithm when x + y % of RAM is utilized.

A secondary goal (only once the first one is considered ready and there's still time) is to implement a compressed cache that sits in front of a physical swap device. I am unsure how far I am able to go with this, since it involves some fairly complex parts, such as finding compressed page pairs (or groups) that fit nicely to a single physical block on the disk and are likely to be fetched at the same time.

* TBA

Development and testing will happen in a 13.0-CURRENT system that I already have inside a bhyve virtual machine. As the project progresses, testing will gradually move to physical hardware that I have at disposal. That includes two ThinkPads (X230 and T480) and two Raspberry Pis (2 and 3). Testing will consist of real-world applications, such as running Poudriere builds, databases, and serving web content.

• TBA