dirhashdynamicmemory

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
dirhashdynamicmemory [2008/10/07 19:51] – NickBarkasdirhashdynamicmemory [2009/07/07 10:17] (current) – NickBarkas
Line 11: Line 11:
 ===== Plan ===== ===== Plan =====
  
-  * Either change dirhash memory allocation to use buffer cache, or keep existing memory allocation model but register new callback function to free older dirhashes when vm_lowmem events occur. +  * Either change dirhash memory allocation to use buffer cache, or keep existing memory allocation model but register new callback function to free older dirhashes when vm_lowmem events occur. **Done** 
-  * Once code works reasonably well in my own tests, call for testing from the community to shake out more bugs if possible. +  * Once code works reasonably well in my own tests, call for testing from the community to shake out more bugs if possible. **Done** 
-  * Test for performance differences with dynamic dirhash vs. static under different work loads (e.g. huge maildirs, subversion fsfs repositories with a great many revisions, different amounts of memory in machine, etc.) +  * Test for performance differences with dynamic dirhash vs. static under different work loads (e.g. huge maildirs, subversion fsfs repositories with a great many revisions, different amounts of memory in machine, etc.). **Done** 
-  * Get code in a state suitable for being committed to -CURRENT +  * Get code in a state suitable for being committed to -CURRENT. **Done** 
-  * (Time permitting) Begin work on on-disk directory indexing in UFS2.+  * (Time permitting) Begin work on on-disk directory indexing in UFS2. **Not done**
  
 ===== Status ===== ===== Status =====
  
-I blog about this project [[http://moduli.net/soc/|here]]. The code I am working on can be found in the %%FreeBSD%% Perforce depot [[http://perforce.freebsd.org/depotTreeBrowser.cgi?FSPC=//depot/projects/soc2008/snb%2ddirhash&HIDEDEL=NO|here]].+I blogged about this project [[http://moduli.net/soc/|here]]. The code I am working on can be found in the %%FreeBSD%% Perforce depot [[http://perforce.freebsd.org/depotTreeBrowser.cgi?FSPC=//depot/projects/soc2008/snb%2ddirhash&HIDEDEL=NO|here]].
  
 (2008-6-19) I am working on implementing a function to be registered with EVENTHANDLER_REGISTER() for vm_lowmem events that will free dirhashes. I am trying to determine how to decide which hashes to free, and will need to find how many should be freed each time this event occurs. Determining the latter might be easiest through trial and error, watching how often vm_lowmem events pop up when the system is low on memory. (2008-6-19) I am working on implementing a function to be registered with EVENTHANDLER_REGISTER() for vm_lowmem events that will free dirhashes. I am trying to determine how to decide which hashes to free, and will need to find how many should be freed each time this event occurs. Determining the latter might be easiest through trial and error, watching how often vm_lowmem events pop up when the system is low on memory.
Line 29: Line 29:
 (2008-8-14) Updated patches again. Now there are two: one against HEAD {{dirhashdynamicmemory/dirhash_lowmem_head_2008-8-14.patch?linkonly|dirhash_lowmem_head_2008-8-14.patch}} and one that should work both on 7.0 and 7-STABLE {{dirhashdynamicmemory/dirhash_lowmem_7-stable_2008-8-14.patch?linkonly|dirhash_lowmem_7-stable_2008-8-14.patch}}. These fix a bug in the previous code where hashes older than 5 seconds would never be deleted and instead only one hash at the head of the TAILQ would be removed with each vm_lowmem event. Also, now the deletion code is more aggressive: all hashes older than 5 seconds (tunable via a sysctl, vfs.ufs.dirhash_reclaimage) will always be deleted when a vm_lowmem event occurs, and if that doesn't free up at least 10% of the memory currently being used by dirhash, more hashes will be destroyed from the head of the TAILQ until we do free up at least 10% of the memory initially used. Also, below I have posted some results from some benchmarking. (2008-8-14) Updated patches again. Now there are two: one against HEAD {{dirhashdynamicmemory/dirhash_lowmem_head_2008-8-14.patch?linkonly|dirhash_lowmem_head_2008-8-14.patch}} and one that should work both on 7.0 and 7-STABLE {{dirhashdynamicmemory/dirhash_lowmem_7-stable_2008-8-14.patch?linkonly|dirhash_lowmem_7-stable_2008-8-14.patch}}. These fix a bug in the previous code where hashes older than 5 seconds would never be deleted and instead only one hash at the head of the TAILQ would be removed with each vm_lowmem event. Also, now the deletion code is more aggressive: all hashes older than 5 seconds (tunable via a sysctl, vfs.ufs.dirhash_reclaimage) will always be deleted when a vm_lowmem event occurs, and if that doesn't free up at least 10% of the memory currently being used by dirhash, more hashes will be destroyed from the head of the TAILQ until we do free up at least 10% of the memory initially used. Also, below I have posted some results from some benchmarking.
  
-===== Benchmarks =====+(2008-10-8) Results about a whole bunch of new benchmarks are added below. Also, the dirhash code for 8-CURRENT has been updated in perforce to work with some new changes made to the dirhash code in SVN. New patch: {{dirhashdynamicmemory/dirhash_lowmem_head_2008-10-12.patch?linkonly|dirhash_lowmem_head_2008-10-12.patch}}
  
-==== With and without vm_lowmem handler ====+(2008-11-19) My [[http://univ-rav.u-strasbg.fr/avc/courseaccess?id=2814&type=pdf|slides]] and [[http://univ-rav.u-strasbg.fr/avc/courseaccess?id=2814&type=ogg|audio]] from my talk at %%EuroBSDCon%% 2008 are available online.
  
-Here is a comparison of the speed running a number of different tasks that are fairly filesystem intensive with and without my dirhash low memory event handling code compiled into the kernel. The system used for testing is a 1.83GHz Core Duo Macbook with 2GB of memory and a 5400RPM 2.5" SATA hard drive. It's running i386 8-CURRENT, and WITNESS and INVARIANTS related options in the kernel are disabled. The system was running multiuser with only a few extra processes like wpa_supplicant, sshd, and powerd running. The vfs.ufs.dirhash_maxmem sysctl was set to 100MB at all times for both kernels, and vfs.ufs.dirhash_reclaimage was 5 seconds when using the vm_lowmem handling kernel. Each test was run with a random amount of wait time from 1 to 5 minutes between each test, and the whole set of tests was repeated for a total of four runs for the set.+(2009-7-7) I committed the dirhash vm_lowmem handler to -CURRENT about a month ago, and it will be included in 8.0-RELEASE. Also I plan to commit a backport of these changes to 7-STABLE, probably around September. 
 + 
 +===== Benchmarks =====
  
-|  | mean run time - no vm_lowmem handler | std. deviation - no vm_lowmem handler | mean run time - with vm_lowmem handler | std. deviation - with vm_lowmem handler | +I measured the performance of the changes I made to the dirhash code using a Python script that does a number of different operations on large directories, hopefully in a fairly realistic way. The script can be found [[http://perforce.freebsd.org/fileLogView.cgi?FSPC=//depot/projects/soc2008/snb%2ddirhash/benchmark/dh%5fbench.py|here in perforce]]. It first untars 1000 randomly generated maildirs containing a total of one million tiny email messages. Then it begins ten threads which add new messages to these maildirs, while simultaneously kicking off a build of a %%FreeBSD%% kernel. The mail delivery threads continue running as the script then extracts a very large tarball containing the entire %%FreeBSD%% src subversion repository from a revision a few months old, checks out from the extracted repository and does a few other subversion operations on it, then removes the extracted repository. Next the script blocks until the mail delivery threads finish delivering all the messages specified in the script, and next kicks off ten threads that perform a number of IMAP operations on all of the maildirs. Once those are all finished, all the generated directories are removed. Times are measured for the initial maildir extraction, the kernel build, time to untar the svnroot, all svn operations, removing the svnroot while mail continue to be delivered, the mail creation threads, the imap threads, and removing all the remaining directories at the end.
-| postmark | 2378s | 16s | 2327s | 77s | +
-| portsnap extract | 312s | 55s | 315s | 57s | +
-| tar extract | 505s | 1,8s | 498s | 5,5s | +
-| svn checkout | 1084s | 131s | 1023s | 16,7s | +
-| rm -rf ports | 76s | 3,7s | 78s | 1,5s | +
-| make buildworld | 4373s | 2,3s | 4382s | 23s | +
-| rm -rf src | 115s | 6,5s | 115s | 9,1s | +
-| rm -rf svnroot | 20s | 5,5s | 18,7s | 4,3s |+
  
-postmark was run with 1 000 000 files, 500 subdirectories, and 10 000 transactions. The portsnap extract test just involved extracting an already fetched ports tree into a clean directory with portsnap extract. tar extraction was of a bzipped tarball containing a copy of the %%FreeBSD%% svn repository, svnmirror-base-r179637.tbz2, fetched from ftp://ftp.freebd.org/pub/FreeBSD/development/subversion. The svn checkout test did a subversion checkout from this newly extracted local svnroot of revision 178000 from HEAD, and make buildworld was run in the checked out src directory. The different rm tests just timed doing rm -rf on the extracted ports and svnroot directories, and the checked out src directory.+I ran the tests on both %%FreeBSD%% 7.0 and 8-CURRENT for the AMD64 architecture. A GENERIC kernel is used on 7.0, and on -CURRENT the only change to the GENERIC config is that INVARIANTS and WITNESS related options are disabled. Furthermore malloc debugging is disabled on the -CURRENT system in userland. On the 7.0 system the creation threads deliver a total of 25 000 new messages to maildirs, while on 8-CURRENT I increased this to 50 000 because delivering 25 000 messages would take less time than all the other tasks which run alongside the mail creation threads.
  
-The difference in run times was not significantly different with the two versions of code. During the postmark and portsnap tests no vm_lowmem events were seen, but a few of them did get fired and delete dirhashes in the later tests that used more memory. This did not seem to slow things down much though. Another test run was bonnie++ -r 2048 -n 1000 -s 0 -u root, and that also had little difference in the number of various operations per second for the different kernels.+The hardware used for all tests has a 1.8GHz Intel Core 2 Duo, 1GB memory, and a 3.5" 80GB 7200 RPM IDE hard drive. The maildirs are extracted, have messages added, and IMAP operations are done on a secondary hard drive, which is a 2.5" 40GB 5400RPM SATA drive. All tests are run with the system running multiuser, but not much besides sshd, openntpd, and postfix along with normal default %%FreeBSD%% daemons are running in the background.
  
-==== Different vfs.ufs.dirhash_reclaimage values ====+Each battery of tests is run 5 times for a number of different dirhash settings. Using a kernel without my patch, tests are run with the maximum dirhash memory usage at the default 2MB and 64MB. With the kernel containing the vm_lowmem handler patch, 64MB is used across the board but the dirhash_reclaimage parameter is tweaked. On 7.0, 1s, 3s, 5s, and 10s are all tried for reclaim age, and on 8-CURRENT all these values are tried in addition to 4s and 8s. I put the results from all the test runs into a Gnumeric spreadsheet here: {{dirhashdynamicmemory/dh_benchmark_results.gnumeric?linkonly|dh_benchmark_results.gnumeric}}. The graphs from this spreadsheet were exported as pngs and included below.
  
-On the same system I also ran a series of tests with various different values for the length of time a dirhash can be unused before a vm_lowmem event will result in it always being deleted. All times are in seconds:+==== FreeBSD 7.0 Results ====
  
-| reclaim age: | 0 seconds | 1 second | 2 seconds | 5 seconds | 10 seconds | 30 seconds | 60 seconds | +{{dirhashdynamicmemory/maildirs7.png}} 
-| tar extract mean time | 863,13 | 835,40 | 901,71 | 860,43 | 878,11 | 897,49 | 901,84 | +{{dirhashdynamicmemory/kernel7.png}} 
-| tar extract std. deviation | 24,4 | 3,67 | 10,32 | 8,11 | 11,80 | 4,50 | 6,84 | +{{dirhashdynamicmemory/svnroot7.png}} 
-| svn log mean time | 107,95 | 108,08 | 107,75 | 106,76 | 107,50 | 106,63 | 107,19 | +{{dirhashdynamicmemory/svn7.png}} 
-| svn log std. deviation | 0,94 | 0,98 | 1,98 | 1,79 | 0,71 | 1,09 | 1,30 | +{{dirhashdynamicmemory/rmsvn7.png}} 
-| rm mean time | 14,83 | 14,39 | 14,87 | 16,56 | 17,41 | 23,84 | 26,24 | +{{dirhashdynamicmemory/mail7.png}} 
-| rm std. deviation | 1,09 | 0,56 | 0,48 | 3,51 | 2,55 | 3,01 | 2,19 | +{{dirhashdynamicmemory/imap7.png}} 
-| bonnie++ mean time | 682,04 | 639,01 | 761,93 | 721,40 | 779,59 | 801,76 | 807,29 | +{{dirhashdynamicmemory/rmother7.png}}
-| bonnie++ std. deviation | 60,86 | 59,27 | 33,77 | 30,37 | 32,60 | 22,81 | 18,14 |+
  
-While running these tests I ran a second program that continuously allocated and freed memory at random intervals, in 100MB chunks. This forced vm_lowmem events to be raised rather frequently. The bonnie++ tests ran four at a time. Each bonnie++ was run with arguments "-n 500:0:0:500 -s 0 -r 2048" to do only file creation tests, and use ~500 000 files in 500 sub directories. The tar, svn, and rm tests ran one at a time, but they were started as soon as the four bonnie++ runs were kicked off in the background. The tar extraction test is the same as above using the %%FreeBSD%% subversion repository copy. svn log is the time to run svn log on the extracted repository. The rm time is just the time to rm -rf the extracted svnroot directory. This suite of tests was run four times for each of the different reclaim age values.+==== FreeBSD 8-CURRENT Results ====
  
-I was surprised to find that having the reclaim age very low led to the best performance. Reducing it all the way down to 0 was not optimal, though. I plan to next run a different series of tests using a range of reclaim age values again, but this time I'll try to simulate a more realistic work load. Instead of running a background process that randomly allocates and frees up huge chunks of memory, which probably isn't very realistic, I will run a series of tests that should stress dirhash usage while also using enough memory to raise vm_lowmem events on their own in a hopefully more natural way. I also plan to test different values for the minimum amount of memory to free on each vm_lowmem event (currently 10% is hard coded in).+{{dirhashdynamicmemory/maildirs8.png}} 
 +{{dirhashdynamicmemory/kernel8.png}} 
 +{{dirhashdynamicmemory/svnroot8.png}} 
 +{{dirhashdynamicmemory/svn8.png}} 
 +{{dirhashdynamicmemory/rmsvn8.png}} 
 +{{dirhashdynamicmemory/mail8.png}} 
 +{{dirhashdynamicmemory/imap8.png}} 
 +{{dirhashdynamicmemory/rmother8.png}}
  
  • dirhashdynamicmemory.1223409068.txt.gz
  • Last modified: 2008/10/07 19:51
  • by NickBarkas