The best first step is to capture stack traces of all threads in one of the following ways:

• procstat -kk -a in userland if it is still responsive and you are able to save the output somehow
• alltrace in ddb
• thread apply all bt in kgdb

If you see thread(s) with zio_wait call in their stacks and you also see thread(s) with zio_done call in their stacks, then this is very likely a true ZFS deadlock. Please report.

Similarly, if you see thread(s) with zio_wait call in their stacks and you also see thread(s) with zio_interrupt call in their stacks, then this is very likely a true ZFS deadlock. Please report.

If you do not see any threads with zio_wait call, but you see threads with the following calls (or similar):

• zfs_freebsd_read
• zfs_freebsd_write
• dmu_buf_hold_array
• arc_read
• buf_hash_find
• dmu_read_uio
• dmu_write_uio
• zfs_zget

then this is very likely a true ZFS deadlock. Please report.

If neither of the above is true. That is, you do see zio_wait and you don't see either of zio_done or zio_interrupt, then the problem is most likely with the storage layer:

• storage adapter/controller driver
• storage adapter/controller firmware
• storage adapter/controller hardware

Consider reporting this problem. Please be realistic about the problem. Do not expect a resolution in ZFS code.

Some notes:

• it's better to somehow verify that the threads are stuck where you see them as opposed to doing a lot of work and thus appearing in those places with high probability
• camcontrol tags disk -v can be used to check number of queued commands for controllers that work via CAM
• when reporting a problem please always include full information about thread stacks, don't cherry pick; the output can be large, upload it somewhere and post a link