Compression/decompression/etc potentially takes a lot of CPU.

See a complaint here: http://thread.gmane.org/gmane.os.freebsd.stable/85097/focus=85100

One quick fix is to use PUSER priority as done in GELI driver. More investigation is needed.

Low memory conditions result in ZFS ARC deadlocks.

There are some memory allocations in ARC that are done while holding some locks. Buffer hash lock is the worst offender. Given the structure of the code (e.g. sometimes lock acquisition and memory allocation are couple of functions apart) it is quite hard to move all allocations outside of the lock scope.

Some possible remedies (but no real cure yet):

• arc_lowmem hook should never block to avoid the following deadlocks between ARC reclaim thread and:
  • a thread doing a memory allocation in ARC and ending up in arc_lowmem
  • pagedaemon calling arc_lowmem while there are other threads waiting on it in ARC memory allocations (via VM_WAIT)
• arc_evict_needed should check for KVA shortage even on amd64

ZFS usage patterns fragment KVA a lot. It seems to make sense to use a dedicated submap for ZFS buffer allocations. The submap should be sized based on maximum ARC size. To be determined if only data buffers should be allocated from the submap or all of the buffers.

This requires extending UMA API.

… or at least to woken up earlier from VM_WAIT.

In other words, try to emulate KM_PUSHPAGE.

Not sure if this is a good idea overall, but may have some merit.

Currently if an allocation of contiguous physical (multi-page) memory fails, then there is only a single invocation of vm_lowmem hook despiate the amount of laundering that is performed to free up some pages. There should be probably be some special side-channel to communicate the condition/effort to ARC. Perhaps ARC should go as far as shrinking to its minimum size.

Some questions first:

• should ARC be driven by pagedaemon (its thresholds) or by something else?
• at what threshold should ARC reclaiming start?
  • same as pagedaemon pageout?
  • a higher threshold?
  • a lower threshold?
  • why?
• how much memory should ARC give up in one pass (triggered by pageout)?