Please first read the following. Please direct your ZFS reports to freebsd-fs@FreeBSD.org.

The best first step is to capture stack traces of all threads in one of the following ways:

• procstat -kk -a in userland if it is still responsive and you are able to save the output somehow
• alltrace in ddb
• thread apply all bt in kgdb

If you see thread(s) with zio_wait call in their stacks and you also see thread(s) with zio_done call in their stacks, then this is very likely a true ZFS deadlock. Please report.

Similarly, if you see thread(s) with zio_wait call in their stacks and you also see thread(s) with zio_interrupt call in their stacks, then this is very likely a true ZFS deadlock. Please report.

If you do not see any threads with zio_wait call, but you see threads with the following calls (or similar):

• zfs_freebsd_read
• zfs_freebsd_write
• dmu_buf_hold_array
• arc_read
• buf_hash_find
• dmu_read_uio
• dmu_write_uio
• zfs_zget

then this is very likely a true ZFS deadlock. Please report.

If neither of the above is true. That is, you do see zio_wait and you don't see either of zio_done or zio_interrupt, then the problem is most likely with the storage layer:

• storage adapter/controller driver
• storage adapter/controller firmware
• storage adapter/controller hardware

Consider reporting this problem. Please be realistic about the problem. Do not expect a resolution in ZFS code.

Some notes:

• it's better to somehow verify that the threads are stuck where you see them as opposed to doing a lot of work and thus appearing in those places with high probability
• camcontrol tags disk -v can be used to check number of queued commands for controllers that work via CAM
• when reporting a problem please always include full information about thread stacks, don't cherry pick; the output can be large, upload it somewhere and post a link

If you are into deep debugging some very interesting/useful information can be seen in vdev_t structures associated with each leaf vdev of a pool.

vdev_queue = {
        vq_deadline_tree = {avl_root = 0xfffffe0338dbb248, avl_compar =
0xffffffff816855b0 <vdev_queue_deadline_compare>,
avl_offset = 584, avl_numnodes = 116, avl_size = 896},
        vq_read_tree = {avl_root = 0xfffffe019d0b65b0, avl_compar =
0xffffffff81685600 <vdev_queue_offset_compare>, avl_offset = 560, avl_numnodes =
8, avl_size = 896},
        vq_write_tree = { avl_root = 0xfffffe03e3d19230, avl_compar =
0xffffffff81685600 <vdev_queue_offset_compare>, avl_offset = 560, avl_numnodes =
108, avl_size = 896},
        vq_pending_tree = {avl_root = 0xfffffe025e32c230, avl_compar =
0xffffffff81685600 <vdev_queue_offset_compare>, avl_offset = 560, avl_numnodes =
10, avl_size = 896},

avl_numnodes provides a number of requests (zio-s) in a given queue. vq_deadline_tree is a queue of incoming requests, vq_read_tree and vq_write_tree are sub-queues for read and write requests correspondingly. vq_pending_tree is a queue of requests that have been issued to the underlying storage layer, ZFS is waiting for these requests to be completed.