zfstuningguide

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
zfstuningguide [2015/05/02 19:08] – Reflect v28 as default and deprecated FreeBSD versions as comments BernardSpilzfstuningguide [2023/04/02 17:15] (current) – 2 GB minimum for ZFS, according to OpenZFS documentation. GrahamPerrin
Line 1: Line 1:
-----+See also: [[https://www.solaris-cookbook.eu/solaris/solaris-10-zfs-evil-tuning-guide/|Solaris: ZFS Evil Tuning Guide]], [[http://man.freebsd.org/loader.conf|loader.conf(5)]], [[http://man.freebsd.org/sysctl|sysctl(8)]].
  
-See also: [[http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide|Solaris: ZFS Evil Tuning Guide]], [[http://man.freebsd.org/loader.conf|loader.conf(5)]], [[http://man.freebsd.org/sysctl|sysctl(8)]].+===== ZFS Tuning Guide =====
  
-===== ZFS Tuning Guide ===== +[[https://openzfs.github.io/openzfs-docs/Project%20and%20Community/FAQ.html#performance-considerations|OpenZFS documentation recommends a minimum of 2GB of memory for ZFS]]; additional memory is //strongly recommended when the compression and deduplication features are enabled//.
-(Work in Progress)+
  
-To use ZFS, at least 1 GB of memory is recommended (for all architectures) but more is helpful as ZFS needs *lots* of memory.  Depending on your workload, it may be possible to use ZFS on systems with less memory, but it requires careful tuning to avoid panics from memory exhaustion in the kernel.+Depending on your workload, it may be possible to use ZFS on systems with less memory, but it requires careful tuning to avoid panics from memory exhaustion in the kernel.
  
 A 64-bit system is preferred due to its larger address space and better performance on 64-bit variables, which are used extensively by ZFS. 32-bit systems are supported though, with sufficient tuning. A 64-bit system is preferred due to its larger address space and better performance on 64-bit variables, which are used extensively by ZFS. 32-bit systems are supported though, with sufficient tuning.
Line 74: Line 73:
 ZFS has the ability to extend the ARC with one or more L2ARC devices, which provides the best benefit for random read workloads.  These L2ARC devices should be faster and/or lower latency than the storage pool.  Generally speaking this limits the useful choices to flash based devices.  In very large pools the ability to have devices faster than the pool may be problematic.  In smaller pools it may be tempting to use a spinning disk as a dedicated L2ARC device.  Generally this will result in lower pool performance (and definitely capacity) than if it was just placed in the pool.  There may be scenarios in lower memory systems where a single 15K SAS disk can improve the performance of a small pool of 5.4k or 7.2 drives, but this is not a typical case. ZFS has the ability to extend the ARC with one or more L2ARC devices, which provides the best benefit for random read workloads.  These L2ARC devices should be faster and/or lower latency than the storage pool.  Generally speaking this limits the useful choices to flash based devices.  In very large pools the ability to have devices faster than the pool may be problematic.  In smaller pools it may be tempting to use a spinning disk as a dedicated L2ARC device.  Generally this will result in lower pool performance (and definitely capacity) than if it was just placed in the pool.  There may be scenarios in lower memory systems where a single 15K SAS disk can improve the performance of a small pool of 5.4k or 7.2 drives, but this is not a typical case.
  
-By default the L2ARC does not attempt to cache streaming/sequential workloads, on the assumption that the combined throughput of your pool disks exceeds the throughput of the L2ARC devices, and therefore, this workload is best left for the pool disks to serve. This is usually the case. If you believe otherwise (number of L2ARC devices X their max throughput > number of pool disks X their max throughput), then this can be toggled with the following sysctl:+By default the L2ARC does not attempt to cache prefetched/streaming workloads, on the assumption that most data of this type is sequential and the combined throughput of your pool disks exceeds the throughput of the L2ARC devices, and therefore, this workload is best left for the pool disks to serve. This is usually the case. If you believe otherwise (number of L2ARC devices X their max throughput > number of pool disks X their max throughput, or you are not doing large amounts of sequential access), then this can be toggled with the following sysctl:
  
 <code> <code>
Line 80: Line 79:
 </code> </code>
  
-The default value of 1 does not allow caching of streaming and/or sequential workloads.  Switching it to 0 will allow streaming/sequential reads to be cached.  Note that this value is a run time sysctl that is read at pool import time, which means the default mechanism of setting sysctls in %%FreeBSD%% will not work. (Values set in /etc/sysctl.conf are set after ZFS pools are imported)+The default value of 1 does not allow caching of streaming and/or sequential workloads, and will not read from L2ARC when prefetching blocks.  Switching it to 0 will allow prefetched/streaming reads to be cached, and may significantly improve performance if you are storing many small files in a large directory hierarchy (since many metadata blocks are read via the prefetcher and would ordinarily always be read from pool disks).
  
 The default throttling of loading the L2ARC device is 8 Mbytes/sec, on the assumption that the L2ARC is warming up from a random read workload from spinning disks, for which 8 Mbytes/sec is usually more than the spinning disks can provide. For example, at a 4 Kbyte I/O size, this is 2048 random disk IOPS, which may take at least 20 pool disks to drive. Should the L2ARC throttling be increased from 8 Mbytes, it would make no difference in many configurations, which cannot provide more random IOPS. The downside of increasing the throttling is CPU consumption: the L2ARC periodically scans the ARC to find buffers to cache, based on the throttling size. If you increase the throttling but the pool disks cannot keep up, you burn CPU needlessly. In extreme cases of tuning, this can consume an entire CPU for the ARC scan. The default throttling of loading the L2ARC device is 8 Mbytes/sec, on the assumption that the L2ARC is warming up from a random read workload from spinning disks, for which 8 Mbytes/sec is usually more than the spinning disks can provide. For example, at a 4 Kbyte I/O size, this is 2048 random disk IOPS, which may take at least 20 pool disks to drive. Should the L2ARC throttling be increased from 8 Mbytes, it would make no difference in many configurations, which cannot provide more random IOPS. The downside of increasing the throttling is CPU consumption: the L2ARC periodically scans the ARC to find buffers to cache, based on the throttling size. If you increase the throttling but the pool disks cannot keep up, you burn CPU needlessly. In extreme cases of tuning, this can consume an entire CPU for the ARC scan.
Line 109: Line 108:
 ==== Application Issues ==== ==== Application Issues ====
  
-ZFS is a copy-on-write filesystem. As such metadata from the top of the hierarchy is copied in order to maintain consistency in case of sudden failure, i.e. loss of power during a write operation. This obviates the need for an fsck-like requirement of ZFS filesystems at boot. However the downside to this is that applications which perform updates in place to large files, e.g. databases, will likely perform poorly in this application of the filesystem due to excessive I/O from copy-on-write (a fast SLOG device -- e.g. a SSD -- can help regarding the write performance of databases or any application which is doing synchronous writes (e.g. open with O_FSYNC) to the FS to make sure the data is on non-volatile storage when the write-call returns). Additionally, database applications, such as Oracle, maintain a large cache (called the SGA in Oracle) in memory will perform poorly due to double caching of data in the ARC and in the application's own cache. Reducing the ARC to a minimum can improve performance of applications which maintain their own cache. At http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide there are some generic recommendations for ZFS on Solaris which mostly apply to %%FreeBSD%% too.+ZFS is a copy-on-write filesystem. As such metadata from the top of the hierarchy is copied in order to maintain consistency in case of sudden failure, i.e. loss of power during a write operation. This obviates the need for an fsck-like requirement of ZFS filesystems at boot. However the downside to this is that applications which perform updates in place to large files, e.g. databases, will likely perform poorly in this application of the filesystem due to excessive I/O from copy-on-write (a fast SLOG device -- e.g. a SSD -- can help regarding the write performance of databases or any application which is doing synchronous writes (e.g. open with O_FSYNC) to the FS to make sure the data is on non-volatile storage when the write-call returns). Additionally, database applications, such as Oracle, maintain a large cache (called the SGA in Oracle) in memory will perform poorly due to double caching of data in the ARC and in the application's own cache. Reducing the ARC to a minimum can improve performance of applications which maintain their own cache. At [[https://archive.vn/ZWZxY|ZFS Best Practices Guide]] there are some generic recommendations for ZFS on Solaris which mostly apply to %%FreeBSD%% too.
  
 ===== General Tuning ===== ===== General Tuning =====
Line 143: Line 142:
 # Set TXG write limit to a lower threshold.  This helps "level out" # Set TXG write limit to a lower threshold.  This helps "level out"
 # the throughput rate (see "zpool iostat").  A value of 256MB works well # the throughput rate (see "zpool iostat").  A value of 256MB works well
-# for systems with 4 GB of RAM, while 1 GB works well for us w/ 8 GB on +# for systems with 4 GB of RAM, while 1 GB works well for us w/ 8 GB on
 # disks which have 64 MB cache. <<BR>> # disks which have 64 MB cache. <<BR>>
  
Line 181: Line 180:
  
 <code> <code>
-vfs.zfs.zil_disable="1" +vfs.zfs.zil_disable="1"
 </code> </code>
  
 the vfs.zfs.zil_disable loader tunable was replaced with the "sync" dataset property. the vfs.zfs.zil_disable loader tunable was replaced with the "sync" dataset property.
  
-Disabling ZIL is not recommended where data consistency is required (such as database servers) but will not result in file system corruption. See http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Disabling_the_ZIL_.28Don.27t.29 .+Disabling ZIL is not recommended where data consistency is required (such as database servers) but will not result in file system corruption. See [[https://archive.is/pV9cy|ZFS Evil Tuning Guide]], section "Disabling the ZIL (Don't)".
  
 ZFS is designed to be used with "raw" drives - i.e. not over already created hardware RAID volumes (this is sometimes called "JBOD" or "passthrough" mode when used with RAID controllers), but can benefit greatly from good and fast controllers. ZFS is designed to be used with "raw" drives - i.e. not over already created hardware RAID volumes (this is sometimes called "JBOD" or "passthrough" mode when used with RAID controllers), but can benefit greatly from good and fast controllers.
Line 200: Line 199:
 *zfs set primarycache=metadata tank/db *zfs set primarycache=metadata tank/db
 *zfs set atime=off tank/db *zfs set atime=off tank/db
-*zfs set recordsize=16k tank/db+*zfs set recordsize=16k tank/db/innodb 
 +*zfs set recordsize=128k tank/db/logs
 *zfs set zfs:zfs_nocacheflush = 1 *zfs set zfs:zfs_nocacheflush = 1
-*sysctl vfs.zfs.prefetch_disable=1+*zfs set sync=disabled tank/db 
 + 
 +Note: %%MySQL%% 5.6.6 and newer (and related %%MariaDB%% / Percona forks)  has ''%%innodb_file_per_table = on%%'' as default, so IBD files are not created under ''%%tank/db/innodb%%'' (defined by ''%%innodb_data_home_dir%%'' in your ''%%my.cnf%%''), they are created under //tank/db/%%<%%db_name>/// and you should use ''%%recordsize=16k%%'' on this dataset too or switch back to ''%%innodb_file_per_table = off%%'' 
 + 
 +References 
 +*[[https://web.archive.org/web/20170501084558/https://blogs.oracle.com/realneel/entry/mysql_innodb_zfs_best_practices|MySQL Innodb ZFS Best Practices (Oracle)]]
  
 ===== Scrub and Resilver Performance ===== ===== Scrub and Resilver Performance =====
Line 215: Line 220:
  
 Setting those sysctls to those values increased my (Shawn Webb's) resilver performance from 7MB/s to 230MB/s. Setting those sysctls to those values increased my (Shawn Webb's) resilver performance from 7MB/s to 230MB/s.
 +
 +----
 +
 +{{tag>HowTo}}
 +{{tag>Stale}}
 +{{tag>NeedsContent}}
 +{{tag>Zfs}}
  
  • zfstuningguide.1430593687.txt.gz
  • Last modified: 2015/05/02 19:08
  • by BernardSpil