VirtualMemoryCompression [FreeBSD WIKI]

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Virtual Memory Compression ======
  * Student: [[/PaavoEinariKaipila|Paavo-Einari Kaipila]]
  * Mentors: [[/EdwardTomaszNapierala|Edward Napierala]] and [[/MahdiMokhtari|Mahdi Mokhtari]]
  * [[https://freebsd.slack.com/messages/CJPKXAHE1|A Slack channel]] for anyone who happens to be interested in the project. Register with your freebsd.org email.

===== Introduction =====

Virtual memory compression is a memory management technique implemented by multiple operating systems ranging from Windows to Darwin. Depending on the implementation, it can either reduce swapping to a physical storage medium or completely replace it. In the vast majority of systems, compression and decompression are generally known to be faster than disk IO. It has also been reported to offer energy savings.

==== Prior art ====
  * [[https://kernelnewbies.org/Linux_3.14#zram:_Memory_compression_mechanism_considered_stable|zram]] on Linux. Enabled by default on Android and Chromium OS.
  * [[https://lwn.net/Articles/537422/|zswap]] on Linux.
  * [[https://www.lifewire.com/understanding-compressed-memory-os-x-2260327|Darwin]] since 13.0. Enabled at least in macOS, perhaps on other Apple systems too.
  * [[https://www.howtogeek.com/319933/what-is-memory-compression-in-windows-10/|Windows]] since 10525

==== The target audience for the project ====
  * Systems without writable storage
  * Systems with extremely slow writable storage (eg. hard drive, eMMC)
  * Systems with very limited write cycles (eg. eMMC, old solid state drives)
  * Systems that do not need excessive amounts (more than physical memory) of swap and want to benefit from increased speed

===== Project Plan =====

==== Primary goal: “zram for FreeBSD” ====

The primary goal is to implement a compressed in-memory disk that can be utilized as a swap storage. The current plan is to integrate it to existing md(4) and the corresponding userland configuration tool mdconfig(8). Lz4 will be the default algorithm due to its good speed/compression ratio, while zstd with a configurable compression level is also available as an option. I will also investigate an idea that I’ve never seen implemented before: a compressed memory disk that by default uses a very fast compression algorithm (lz4) but changes to a more compact but slower one (zstd with an aggressive compression level) when under extreme memory pressure. I call this adaptive compression. The idea is to use a fast algorithm untill x % of RAM is utilized and to actively start recompressing existing data using a more compact algorithm when x + y % of RAM is utilized.

==== Secondary goal: “zswap for FreeBSD” ====

A secondary goal (only once the first one is considered ready and there's still time) is to implement a compressed cache that sits in front of a physical swap device. I am unsure how far I am able to go with this, since it involves some fairly complex parts, such as finding compressed page pairs (or groups) that fit nicely to a single physical block on the disk and are likely to be fetched at the same time.

===== Milestones =====

  * May 6 - 27
    * soft start, getting to know my mentors
    * getting familiar with DTrace as I believe it will be useful
    * a script for building the system, installing it into an image and launching a VM in one go ✔ done
  * May 27
    * Coding officially starts. From this day forward, I will spend about 7-8 hours per day on the project.
  * June 21 - 23
    * Finnish midsummer (juhannus, a national holiday). I’m taking few days off.
  * By mid-evaluations June 24 - 28
    * A compressed in-memory backend for md(4)
    * Required new options for mdconfig(8)
  * By final evaluations August 19 - 26
    * Testing, stabilization and benchmarking. Also on non-X86 systems.
    * trying adaptive compression
    * an rc script for configuring a compressed memory disk as a swap storage
    * a userland utility for creating temporary file systems with compressed in-memory storage, probably as part of mdmfs(8)
    * documentation
    * (possibly) a compressed cache in front of another storage device

===== Development and testing =====

Development and testing will happen in a 13.0-CURRENT system that I already have inside a bhyve virtual machine. As the project progresses, testing will gradually move to physical hardware that I have at disposal. That includes two [[/CategoryThinkpad|ThinkPad]]s (X230 and T480) and two Raspberry Pis (2 and 3). Testing will consist of real-world applications, such as running Poudriere builds, databases, and serving web content.

===== The Code =====
https://github.com/pavetheway91/freebsd