summerofcode2019projects:virtualmemorycompression

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision		 Previous revision
summerofcode2019projects:virtualmemorycompression [2019/05/09 02:24] – PaavoEinariKaipilasummerofcode2019projects:virtualmemorycompression [2019/06/23 20:37] (current) – PaavoEinariKaipila
Line 1: Line 1:
 ====== Virtual Memory Compression ====== ====== Virtual Memory Compression ======
-  * Student: [[/PaavoEinariKaipila|Paavo-Einari Kaipila]] (pkaipila@gmail.com) +  * Student: [[/PaavoEinariKaipila|Paavo-Einari Kaipila]] 
-  * Mentors:  Edward Napierala (trasz@freebsd.org) and Mark Johnston (markj@freebsd.org)+  * Mentors: [[/EdwardTomaszNapierala|Edward Napierala]] and [[/MahdiMokhtari|Mahdi Mokhtari]] 
 +  * [[https://freebsd.slack.com/messages/CJPKXAHE1|A Slack channel]] for anyone who happens to be interested in the project. Register with your freebsd.org email.
  
-===== Introuduction =====+===== Introduction =====
  
 Virtual memory compression is a memory management technique implemented by multiple operating systems ranging from Windows to Darwin. Depending on the implementation, it can either reduce swapping to a physical storage medium or completely replace it. In the vast majority of systems, compression and decompression are generally known to be faster than disk IO. It has also been reported to offer energy savings. Virtual memory compression is a memory management technique implemented by multiple operating systems ranging from Windows to Darwin. Depending on the implementation, it can either reduce swapping to a physical storage medium or completely replace it. In the vast majority of systems, compression and decompression are generally known to be faster than disk IO. It has also been reported to offer energy savings.
Line 21: Line 22:
 ===== Project Plan ===== ===== Project Plan =====
  
-This page gives one example of a project page format.  Include what makes sense to you.  Some students have included links to submitted changes in Phabricator, others to blog posts, etc.+==== Primary goal: “zram for FreeBSD” ====
  
-===== Deliverables =====+The primary goal is to implement a compressed in-memory disk that can be utilized as a swap storage. The current plan is to integrate it to existing md(4) and the corresponding userland configuration tool mdconfig(8). Lz4 will be the default algorithm due to its good speed/compression ratio, while zstd with a configurable compression level is also available as an option. I will also investigate an idea that I’ve never seen implemented before: a compressed memory disk that by default uses a very fast compression algorithm (lz4) but changes to a more compact but slower one (zstd with an aggressive compression level) when under extreme memory pressure. I call this adaptive compression. The idea is to use a fast algorithm untill x % of RAM is utilized and to actively start recompressing existing data using a more compact algorithm when x + y % of RAM is utilized.
  
-* deliverable 1 +==== Secondary goal: “zswap for FreeBSD” ==== 
-* deliverable 2 + 
-* ...+A secondary goal (only once the first one is considered ready and there's still time) is to implement a compressed cache that sits in front of a physical swap device. I am unsure how far I am able to go with this, since it involves some fairly complex parts, such as finding compressed page pairs (or groups) that fit nicely to a single physical block on the disk and are likely to be fetched at the same time.
  
 ===== Milestones ===== ===== Milestones =====
  
-* Dated list of milestones+  * May 6 - 27 
 +    * soft start, getting to know my mentors 
 +    * getting familiar with DTrace as I believe it will be useful 
 +    * a script for building the system, installing it into an image and launching a VM in one go ✔ done 
 +  * May 27 
 +    * Coding officially starts. From this day forward, I will spend about 7-8 hours per day on the project. 
 +  * June 21 - 23 
 +    * Finnish midsummer (juhannus, a national holiday). I’m taking few days off. 
 +  * By mid-evaluations June 24 - 28 
 +    * A compressed in-memory backend for md(4) 
 +    * Required new options for mdconfig(8) 
 +  * By final evaluations August 19 - 26 
 +    * Testing, stabilization and benchmarking. Also on non-X86 systems. 
 +    * trying adaptive compression 
 +    * an rc script for configuring a compressed memory disk as a swap storage 
 +    * a userland utility for creating temporary file systems with compressed in-memory storage, probably as part of mdmfs(8) 
 +    * documentation 
 +    * (possibly) a compressed cache in front of another storage device
  
-===== Test Plan =====+===== Development and testing =====
  
-A test plan.+Development and testing will happen in a 13.0-CURRENT system that I already have inside a bhyve virtual machine. As the project progresses, testing will gradually move to physical hardware that I have at disposal. That includes two [[/CategoryThinkpad|ThinkPad]]s (X230 and T480) and two Raspberry Pis (2 and 3). Testing will consist of real-world applications, such as running Poudriere builds, databases, and serving web content.
  
 ===== The Code ===== ===== The Code =====
- +https://github.com/pavetheway91/freebsd
-  * https://github.com/... +
- +
-===== Notes ===== +
- +
-===== Useful links =====+
  
  • summerofcode2019projects/virtualmemorycompression.1557368671.txt.gz
  • Last modified: 2019/05/09 02:24
  • by PaavoEinariKaipila