andriygapon:avfzfsfreebsdport

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
andriygapon:avfzfsfreebsdport [2012/10/15 17:23] – AndriyGaponandriygapon:avfzfsfreebsdport [2018/03/31 08:28] (current) – add to CategoryZfs. MarkLinimon
Line 7: Line 7:
  
 In %%FreeBSD%% only a vnode lock prevents vnode reclaiming. In %%FreeBSD%% only a vnode lock prevents vnode reclaiming.
-A reference does not help e.g. during forceful unmounting. +Usually a reference (v_usecount) prevents reclaiming of a vnode, but %%FreeBSD%% VFS allows an active vnode to be reclaimed. 
-That seems to be the only practical case today, but in theory VFS allows an active vnode to be reclaimed.+A reference would not help e.g. during forceful unmounting. 
 +That seems to be the only practical case today, but, again, in theory VFS allows an active vnode to be reclaimed.
  
 This problem is most visible in zfs_zget where we first get a znode under "object hold" lock. This problem is most visible in zfs_zget where we first get a znode under "object hold" lock.
Line 14: Line 15:
 Then we vref the node, drop all locks and return. Then we vref the node, drop all locks and return.
 As described above that does not provide any firm guarantee that either the vnode or the znode would stay valid after zfs_zget returns. As described above that does not provide any firm guarantee that either the vnode or the znode would stay valid after zfs_zget returns.
 +
 +Graceful reclaim is done using the following algorithm:
 +  * acquire the vnode lock
 +  * acquire the interlock
 +  * check that reference count is zero\\  ...
 +  * set VI_DOOMED flag\\  ...
 +  * release the interlock\\  ...
 +  * call VOP_RECLAIM
 +
 +Note that checking of the reference count and setting of VI_DOOMED flag are done atomically with respect to the interlock.\\ 
 +zfs_zget increments the reference count and checks VI_DOOMED. The reference is dropped and znode lookup is retried if VI_DOOMED is set.
 +Because of this lockstep the graceful reclaim and zfs_zget can not race.
 +The znode is under protection of the object hold mutex in both cases.
  
 ==== Locking ==== ==== Locking ====
Line 74: Line 88:
   - the blocked threads may be in a vnode operations themselves and thus may hold the locks on the vnodes   - the blocked threads may be in a vnode operations themselves and thus may hold the locks on the vnodes
 This leads to a deadlock. This leads to a deadlock.
 +
 +=== Some thoughts on solutions ===
 +
 +== znode reference count ==
 +
 +Add a reference count to znode, so it could linger after its vnode is reclaimed if it is referenced internally.
 +
 +== acquire z_teardown_lock in vop_reclaim ==
 +
 +Hold z_teardown_lock as a writer across zfs_freebsd_reclaim (and zfs_zinactive).
 +Thus no vnode operation and no vfs operation like zfs_vget and zfs_root could run while any vnode is being reclaimed and its znode is being destroyed.
 +Thus, no potential for a znode be destroyed while it is in use (within an area covered by ZFS_ENTER).
 +May have a performance hit though.
 +
 +== vnode lock in zfs_zget plus some trick ==
 +
 +We could acquire the vnode lock in zfs_zget by extending its signature with additional flags parameter.
 +zil_commit would be a special case and would signal to not acquire the lock.
 +In this case we would fully rely on z_teardown_lock to protect the znode after returning.
  
 ==== ZFS rulebook ==== ==== ZFS rulebook ====
Line 79: Line 112:
 Some porting rules that I've just made up: Some porting rules that I've just made up:
   * z_vnode field should be cleared only after z_sa_hdl is cleared (''%%zfs_znode_dmu_fini%%'');\\  we should not allow a znode with invalid z_vnode to be visible   * z_vnode field should be cleared only after z_sa_hdl is cleared (''%%zfs_znode_dmu_fini%%'');\\  we should not allow a znode with invalid z_vnode to be visible
 +
 +==== Unmount ideas scratchpad ====
 +
 +  * vflush can't be called with z_teardown_lock and/or z_teardown_inactive_lock exclusively held, because they are not recursive and vfs_root, vop_inactive and vop_reclaim may be called, additionally vop_write/vop_putpages could be called as well.
 +  * vflush can't be called after "gutting" the znodes in zfsvfs_teardown, because in this case vfs_root == zfs_root called by vflush would fail; this is because zfs_zget would attempt to create a new znode/vnode pair, but would fail to insert the vnode into mount queue because of MNTK_NOINSMNTQ.
 +  * practically, we should not allow any zfs_zget operations to run concurrently with forceful reclaiming of a vnode
 +  * since no locks can be held around vflush, then there are the following options:
 +    * use a flag like z_unmounted, but then a backdoor to zfs_root is needed
 +    * acquire exclusive z_teardown_lock in vop_reclaim iff v_usecount > 0 has been seen
 +    * make and use a variant of vflush that does not call vfs_root — the root node is provided as parameter
 +
 +----
 +
 +{{tag>Zfs}}
  
  • andriygapon/avfzfsfreebsdport.1350321803.txt.gz
  • Last modified: 2012/10/15 17:23
  • by AndriyGapon