zfstuningguide

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
zfstuningguide [2014/05/09 19:14] – fixed dangerous advice BrendanGreggzfstuningguide [2023/04/02 17:15] (current) – 2 GB minimum for ZFS, according to OpenZFS documentation. GrahamPerrin
Line 1: Line 1:
-----+See also: [[https://www.solaris-cookbook.eu/solaris/solaris-10-zfs-evil-tuning-guide/|Solaris: ZFS Evil Tuning Guide]], [[http://man.freebsd.org/loader.conf|loader.conf(5)]], [[http://man.freebsd.org/sysctl|sysctl(8)]].
  
-See also: [[http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide|Solaris: ZFS Evil Tuning Guide]], [[http://man.freebsd.org/loader.conf|loader.conf(5)]], [[http://man.freebsd.org/sysctl|sysctl(8)]].+===== ZFS Tuning Guide =====
  
-===== ZFS Tuning Guide ===== +[[https://openzfs.github.io/openzfs-docs/Project%20and%20Community/FAQ.html#performance-considerations|OpenZFS documentation recommends a minimum of 2GB of memory for ZFS]]; additional memory is //strongly recommended when the compression and deduplication features are enabled//.
-(Work in Progress)+
  
-To use ZFS, at least 1 GB of memory is recommended (for all architectures) but more is helpful as ZFS needs *lots* of memory.  Depending on your workload, it may be possible to use ZFS on systems with less memory, but it requires careful tuning to avoid panics from memory exhaustion in the kernel.+Depending on your workload, it may be possible to use ZFS on systems with less memory, but it requires careful tuning to avoid panics from memory exhaustion in the kernel.
  
 A 64-bit system is preferred due to its larger address space and better performance on 64-bit variables, which are used extensively by ZFS. 32-bit systems are supported though, with sufficient tuning. A 64-bit system is preferred due to its larger address space and better performance on 64-bit variables, which are used extensively by ZFS. 32-bit systems are supported though, with sufficient tuning.
Line 74: Line 73:
 ZFS has the ability to extend the ARC with one or more L2ARC devices, which provides the best benefit for random read workloads.  These L2ARC devices should be faster and/or lower latency than the storage pool.  Generally speaking this limits the useful choices to flash based devices.  In very large pools the ability to have devices faster than the pool may be problematic.  In smaller pools it may be tempting to use a spinning disk as a dedicated L2ARC device.  Generally this will result in lower pool performance (and definitely capacity) than if it was just placed in the pool.  There may be scenarios in lower memory systems where a single 15K SAS disk can improve the performance of a small pool of 5.4k or 7.2 drives, but this is not a typical case. ZFS has the ability to extend the ARC with one or more L2ARC devices, which provides the best benefit for random read workloads.  These L2ARC devices should be faster and/or lower latency than the storage pool.  Generally speaking this limits the useful choices to flash based devices.  In very large pools the ability to have devices faster than the pool may be problematic.  In smaller pools it may be tempting to use a spinning disk as a dedicated L2ARC device.  Generally this will result in lower pool performance (and definitely capacity) than if it was just placed in the pool.  There may be scenarios in lower memory systems where a single 15K SAS disk can improve the performance of a small pool of 5.4k or 7.2 drives, but this is not a typical case.
  
-By default the L2ARC does not attempt to cache streaming/sequential workloads, on the assumption that the combined throughput of your pool disks exceeds the throughput of the L2ARC devices, and therefore, this workload is best left for the pool disks to serve. This is usually the case. If you believe otherwise (number of L2ARC devices X their max throughput > number of pool disks X their max throughput), then this can be toggled with the following sysctl:+By default the L2ARC does not attempt to cache prefetched/streaming workloads, on the assumption that most data of this type is sequential and the combined throughput of your pool disks exceeds the throughput of the L2ARC devices, and therefore, this workload is best left for the pool disks to serve. This is usually the case. If you believe otherwise (number of L2ARC devices X their max throughput > number of pool disks X their max throughput, or you are not doing large amounts of sequential access), then this can be toggled with the following sysctl:
  
 <code> <code>
Line 80: Line 79:
 </code> </code>
  
-The default value of 1 does not allow caching of streaming and/or sequential workloads.  Switching it to 0 will allow streaming/sequential reads to be cached.  Note that this value is a run time sysctl that is read at pool import time, which means the default mechanism of setting sysctls in %%FreeBSD%% will not work. (Values set in /etc/sysctl.conf are set after ZFS pools are imported)+The default value of 1 does not allow caching of streaming and/or sequential workloads, and will not read from L2ARC when prefetching blocks.  Switching it to 0 will allow prefetched/streaming reads to be cached, and may significantly improve performance if you are storing many small files in a large directory hierarchy (since many metadata blocks are read via the prefetcher and would ordinarily always be read from pool disks).
  
 The default throttling of loading the L2ARC device is 8 Mbytes/sec, on the assumption that the L2ARC is warming up from a random read workload from spinning disks, for which 8 Mbytes/sec is usually more than the spinning disks can provide. For example, at a 4 Kbyte I/O size, this is 2048 random disk IOPS, which may take at least 20 pool disks to drive. Should the L2ARC throttling be increased from 8 Mbytes, it would make no difference in many configurations, which cannot provide more random IOPS. The downside of increasing the throttling is CPU consumption: the L2ARC periodically scans the ARC to find buffers to cache, based on the throttling size. If you increase the throttling but the pool disks cannot keep up, you burn CPU needlessly. In extreme cases of tuning, this can consume an entire CPU for the ARC scan. The default throttling of loading the L2ARC device is 8 Mbytes/sec, on the assumption that the L2ARC is warming up from a random read workload from spinning disks, for which 8 Mbytes/sec is usually more than the spinning disks can provide. For example, at a 4 Kbyte I/O size, this is 2048 random disk IOPS, which may take at least 20 pool disks to drive. Should the L2ARC throttling be increased from 8 Mbytes, it would make no difference in many configurations, which cannot provide more random IOPS. The downside of increasing the throttling is CPU consumption: the L2ARC periodically scans the ARC to find buffers to cache, based on the throttling size. If you increase the throttling but the pool disks cannot keep up, you burn CPU needlessly. In extreme cases of tuning, this can consume an entire CPU for the ARC scan.
  
-If you are using the L2ARC in its typical use case: say, fewer than 30 pool disks, and caching a random read workload for ~4 Kbyte I/O, then 8 Mbytes is usually sufficient. If you are not this typical use case: say, you are caching streaming workloads, or have several dozens of disks, then you may want to consider tuning the rate. Modern L2ARC devices (SSDs) can handle an order of magnitude higher than the default. It can be tuned by setting the following sysctls:+If you are using the L2ARC in its typical use case: say, fewer than 30 pool disks, and caching a random read workload for ~4 Kbyte I/O which is mostly being pulled from the pool disks, then 8 Mbytes is usually sufficient. If you are not this typical use case: say, you are caching streaming workloads, or have several dozens of disks, then you may want to consider tuning the rate. Modern L2ARC devices (SSDs) can handle an order of magnitude higher than the default. It can be tuned by setting the following sysctls:
  
 <code> <code>
Line 91: Line 90:
 </code> </code>
  
-The former value sets the runtime max that data will be loaded into L2ARC.  The latter can be used to accelerate the loading of a freshly booted system. Note that the same caveats apply about these sysctls and pool imports as the previous one. While you can improve the L2ARC warmup rate, keep an eye on increased CPU consumption due to scanning by the l2arc_feed_thread(). Eg, use DTrace to profile on-CPU thread names (see [[ZFSTuningGuide/DTrace/One-Liners|ZFSTuningGuide/DTrace/One-Liners]]).+The former value sets the runtime max that data will be loaded into L2ARC.  The latter can be used to accelerate the loading of a freshly booted system. Note that the same caveats apply about these sysctls and pool imports as the previous one. While you can improve the L2ARC warmup rate, keep an eye on increased CPU consumption due to scanning by the l2arc_feed_thread(). Eg, use DTrace to profile on-CPU thread names (see [[DTrace/One-Liners|DTrace One-Liners]]).
  
 The known caveats: The known caveats:
Line 109: Line 108:
 ==== Application Issues ==== ==== Application Issues ====
  
-ZFS is a copy-on-write filesystem. As such metadata from the top of the hierarchy is copied in order to maintain consistency in case of sudden failure, i.e. loss of power during a write operation. This obviates the need for an fsck-like requirement of ZFS filesystems at boot. However the downside to this is that applications which perform updates in place to large files, e.g. databases, will likely perform poorly in this application of the filesystem due to excessive I/O from copy-on-write (a fast SLOG device -- e.g. a SSD -- can help regarding the write performance of databases or any application which is doing synchronous writes (e.g. open with O_FSYNC) to the FS to make sure the data is on non-volatile storage when the write-call returns). Additionally, database applications, such as Oracle, maintain a large cache (called the SGA in Oracle) in memory will perform poorly due to double caching of data in the ARC and in the application's own cache. Reducing the ARC to a minimum can improve performance of applications which maintain their own cache. At http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide there are some generic recommendations for ZFS on Solaris which mostly apply to %%FreeBSD%% too.+ZFS is a copy-on-write filesystem. As such metadata from the top of the hierarchy is copied in order to maintain consistency in case of sudden failure, i.e. loss of power during a write operation. This obviates the need for an fsck-like requirement of ZFS filesystems at boot. However the downside to this is that applications which perform updates in place to large files, e.g. databases, will likely perform poorly in this application of the filesystem due to excessive I/O from copy-on-write (a fast SLOG device -- e.g. a SSD -- can help regarding the write performance of databases or any application which is doing synchronous writes (e.g. open with O_FSYNC) to the FS to make sure the data is on non-volatile storage when the write-call returns). Additionally, database applications, such as Oracle, maintain a large cache (called the SGA in Oracle) in memory will perform poorly due to double caching of data in the ARC and in the application's own cache. Reducing the ARC to a minimum can improve performance of applications which maintain their own cache. At [[https://archive.vn/ZWZxY|ZFS Best Practices Guide]] there are some generic recommendations for ZFS on Solaris which mostly apply to %%FreeBSD%% too.
  
 ===== General Tuning ===== ===== General Tuning =====
Line 143: Line 142:
 # Set TXG write limit to a lower threshold.  This helps "level out" # Set TXG write limit to a lower threshold.  This helps "level out"
 # the throughput rate (see "zpool iostat").  A value of 256MB works well # the throughput rate (see "zpool iostat").  A value of 256MB works well
-# for systems with 4 GB of RAM, while 1 GB works well for us w/ 8 GB on +# for systems with 4 GB of RAM, while 1 GB works well for us w/ 8 GB on
 # disks which have 64 MB cache. <<BR>> # disks which have 64 MB cache. <<BR>>
  
Line 174: Line 173:
 ===== NFS tuning ===== ===== NFS tuning =====
  
-The combination of ZFS and NFS stresses the ZIL to the point that performance falls significantly below expected levels. The best solution is to put the ZIL on a fast SSD (or a pair of SSDs in a mirror, for added redundancy). The next best solution is to disable ZIL with the following setting in ''%%loader.conf%%'' (up to ZFS version 15):+The combination of ZFS and NFS stresses the ZIL to the point that performance falls significantly below expected levels. The best solution is to put the ZIL on a fast SSD (or a pair of SSDs in a mirror, for added redundancy). You can now enable/disable ZIL on a per-dataset basis (as of ZFS version 28 / %%FreeBSD%% 8.3+). 
 + 
 +  * ''%% zfs set sync=disabled tank/dataset %%'' 
 + 
 +The next best solution is to disable ZIL with the following setting in ''%%loader.conf%%'' (up to ZFS version 15):
  
 <code> <code>
-vfs.zfs.zil_disable="1" +vfs.zfs.zil_disable="1"
 </code> </code>
  
-In latest ZFS (version 28) the vfs.zfs.zil_disable loader tunable was replaced with the "sync" dataset property. You can now enable/disable ZIL on a per-dataset basis.+the vfs.zfs.zil_disable loader tunable was replaced with the "sync" dataset property.
  
-  * ''%% zfs set sync=disabled tank/dataset %%'' +Disabling ZIL is not recommended where data consistency is required (such as database servers) but will not result in file system corruption. See [[https://archive.is/pV9cy|ZFS Evil Tuning Guide]], section "Disabling the ZIL (Don't)".
- +
-Disabling ZIL is not recommended where data consistency is required (such as database servers) but will not result in file system corruption. See http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Disabling_the_ZIL_.28Don.27t.29 .+
  
 ZFS is designed to be used with "raw" drives - i.e. not over already created hardware RAID volumes (this is sometimes called "JBOD" or "passthrough" mode when used with RAID controllers), but can benefit greatly from good and fast controllers. ZFS is designed to be used with "raw" drives - i.e. not over already created hardware RAID volumes (this is sometimes called "JBOD" or "passthrough" mode when used with RAID controllers), but can benefit greatly from good and fast controllers.
Line 198: Line 199:
 *zfs set primarycache=metadata tank/db *zfs set primarycache=metadata tank/db
 *zfs set atime=off tank/db *zfs set atime=off tank/db
-*zfs set recordsize=16k tank/db+*zfs set recordsize=16k tank/db/innodb 
 +*zfs set recordsize=128k tank/db/logs
 *zfs set zfs:zfs_nocacheflush = 1 *zfs set zfs:zfs_nocacheflush = 1
-*sysctl vfs.zfs.prefetch_disable=1+*zfs set sync=disabled tank/db 
 + 
 +Note: %%MySQL%% 5.6.6 and newer (and related %%MariaDB%% / Percona forks)  has ''%%innodb_file_per_table = on%%'' as default, so IBD files are not created under ''%%tank/db/innodb%%'' (defined by ''%%innodb_data_home_dir%%'' in your ''%%my.cnf%%''), they are created under //tank/db/%%<%%db_name>/// and you should use ''%%recordsize=16k%%'' on this dataset too or switch back to ''%%innodb_file_per_table = off%%'' 
 + 
 +References 
 +*[[https://web.archive.org/web/20170501084558/https://blogs.oracle.com/realneel/entry/mysql_innodb_zfs_best_practices|MySQL Innodb ZFS Best Practices (Oracle)]] 
 + 
 +===== Scrub and Resilver Performance ===== 
 +If you're getting horrible performance during a scrub or resilver, the following sysctls can be set: 
 + 
 +<code> 
 +vfs.zfs.scrub_delay=0 
 +vfs.zfs.top_maxinflight=128 
 +vfs.zfs.resilver_min_time_ms=5000 
 +vfs.zfs.resilver_delay=0 
 +</code> 
 + 
 +Setting those sysctls to those values increased my (Shawn Webb's) resilver performance from 7MB/s to 230MB/s. 
 + 
 +---- 
 + 
 +{{tag>HowTo}} 
 +{{tag>Stale}} 
 +{{tag>NeedsContent}} 
 +{{tag>Zfs}}
  
  • zfstuningguide.1399662878.txt.gz
  • Last modified: 2014/05/09 19:14
  • by BrendanGregg