===== Debugging of suspected ZFS deadlocks =====

**Please first read the following.**
**Please direct your ZFS reports to freebsd-fs@%%FreeBSD%%.org.**

----

The best first step is to capture stack traces of __all__ threads in one of the following ways:
  * ''%%procstat -kk -a%%'' in userland if it is still responsive and you are able to save the output somehow
  * ''%%alltrace%%'' in ddb
  * ''%%thread apply all bt%%'' in kgdb

----

If you see thread(s) with //zio_wait// call in their stacks __and__ you also see thread(s) with //zio_done// call in their stacks, then this is very likely a true ZFS deadlock.  Please report.

----

Similarly, if you see thread(s) with //zio_wait// call in their stacks __and__ you also see thread(s) with //zio_interrupt// __call__ in their stacks, then this is very likely a true ZFS deadlock.  Please report.

----

If you __do not__ see any threads with //zio_wait// call, but you see threads with the following calls (or similar):
  * //zfs_freebsd_read//
  * //zfs_freebsd_write//
  * //dmu_buf_hold_array//
  * //arc_read//
  * //buf_hash_find//
  * //dmu_read_uio//
  * //dmu_write_uio//
  * //zfs_zget//
then this is very likely a true ZFS deadlock.  Please report.

----

If neither of the above is true.  That is, you do see //zio_wait// and you don't see either of //zio_done// or //zio_interrupt//, then the problem is most likely with the storage layer:
  * storage adapter/controller driver
  * storage adapter/controller firmware
  * storage adapter/controller hardware
Consider reporting this problem. Please be realistic about the problem. Do not expect a resolution in ZFS code.

----

Some notes:
  * it's better to somehow verify that the threads are stuck where you see them as opposed to doing a lot of work and thus appearing in those places with high probability
  * ''%%camcontrol tags%%'' //disk// ''%%-v%%'' can be used to check number of queued commands for controllers that work via CAM
  * __when reporting a problem please always include full information about thread stacks, don't cherry pick; the output can be large, upload it somewhere and post a link__

If you are into deep debugging some very interesting/useful information can be seen in vdev_t structures associated with each __leaf__ vdev of a pool.

<code>
vdev_queue = {
        vq_deadline_tree = {avl_root = 0xfffffe0338dbb248, avl_compar =
0xffffffff816855b0 <vdev_queue_deadline_compare>,
avl_offset = 584, avl_numnodes = 116, avl_size = 896},
        vq_read_tree = {avl_root = 0xfffffe019d0b65b0, avl_compar =
0xffffffff81685600 <vdev_queue_offset_compare>, avl_offset = 560, avl_numnodes =
8, avl_size = 896},
        vq_write_tree = { avl_root = 0xfffffe03e3d19230, avl_compar =
0xffffffff81685600 <vdev_queue_offset_compare>, avl_offset = 560, avl_numnodes =
108, avl_size = 896},
        vq_pending_tree = {avl_root = 0xfffffe025e32c230, avl_compar =
0xffffffff81685600 <vdev_queue_offset_compare>, avl_offset = 560, avl_numnodes =
10, avl_size = 896},
</code>

**avl_numnodes** provides a number of requests (zio-s) in a given queue.
**vq_deadline_tree** is a queue of incoming requests, **vq_read_tree** and **vq_write_tree** are sub-queues for read and write requests correspondingly.
**vq_pending_tree** is a queue of requests that have been issued to the underlying storage layer, ZFS is waiting for these requests to be completed.

----

{{tag>Zfs}}