konstantinbelousov:vm6

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
konstantinbelousov:vm6 [2010/04/14 21:13] – KonstantinBelousovkonstantinbelousov:vm6 [2022/06/08 00:51] (current) – Add Category: Stale (lastmod 2011) KubilayKocak
Line 17: Line 17:
 When adding pages for the file region that is extended by write() call, I need to truncate file to larger size. Doing this using VOP_SETATTR() appears to be prohibitely costly, at least on UFS, because ffs_truncate() allocates the block for last byte and writes it. When adding pages for the file region that is extended by write() call, I need to truncate file to larger size. Doing this using VOP_SETATTR() appears to be prohibitely costly, at least on UFS, because ffs_truncate() allocates the block for last byte and writes it.
  
-I introduced new VOP_EXTEND() that extends the file for new size. Default implementation does VOP_GETATTR/VOP_SETTATTR. On UFS, it only sets new size in inode, without allocating block for last byte. When extending the file that ended in fragment, the whole block or neccessary number of fragments at the previous end is reallocated, to keep the invariant that fragment is allowed only at direct blocks at end of file. Current fsck would treat such extended files as filesystem inconsistency, and return the size to pre-extend value.+When file is modified, suid bit shall be dropped. Doing it by VOP_GETATTR/VOP_SETATTR is inefficient. VOP_EXTEND is guaranteed to be called at least once for each write, and filesystem may use the opportunity to clean the bits. ffs_extend() handle this. 
 + 
 +I introduced new VOP_EXTEND() that extends the file for new size. Default implementation does VOP_GETATTR/VOP_SETTATTR. On UFS, custom implementation is provided, that sets new size in inode, without allocating block for last byte. When extending the file that ended in fragment, the whole block or neccessary number of fragments at the previous end is reallocated, to keep the invariant that fragment is allowed only at direct blocks at end of file. Current fsck would treat such extended files as filesystem inconsistency, and return the size to pre-extend value.
  
 Pressure on writers is another critical feature, both for performance and for responsivness of the system. I tried to do it simple, by counting number of pages dirtyfied by vnode_pager_write(), and doing the cleaning pass similar to the pageout launder if count of the pages becomes too big. Pressure on writers is another critical feature, both for performance and for responsivness of the system. I tried to do it simple, by counting number of pages dirtyfied by vnode_pager_write(), and doing the cleaning pass similar to the pageout launder if count of the pages becomes too big.
Line 42: Line 44:
 sysbench: http://people.freebsd.org/~pho/sysbench.png sysbench: http://people.freebsd.org/~pho/sysbench.png
  
-Known issues and parts not done:+==== Known issues and parts not done: ====
  
-1. vm_page_queue_lock contention.+=== vm_page_queue_lock contention === 
 +Jeff' concern is that the patch increases contention on vm_page_queue_lock, that is already highly contented. See sysbench graph above.
  
-Jeff' concern is that the patch increases contention on +=== VOP_REALLOCBLKS === 
-vm_page_queue_lock, that is already highly contented. See sysbench +Currently, the behaviour of the system when appending to the file is the same as if the file is created, then extended, mmaped into address space and then mapped area is filled. It is known that this caused severe file fragmentation, since buf clustering code does not see adjusted writes. Something similar for VOP_REALLOCBLKS ought to be implemented for page flushes.
-graph above.+
  
-2. VOP_REALLOCBLKS.+On the other hand, we have measured the current situation, and it is not much worse that for unpatched kernel. Patch contains the sequential write detection code. 
 +vm6 branch contains the UFS fragmentation measurement tool, see tools/tools/ufs/fragc. We run the following load:
  
-Currently, the behaviour of the system when appending to the file is +<code> 
-the same as if the file is created, then extended, mmaped into address +> I create some 200 files of various size in 1k increments. I then have 
-space and then mapped area is filled. It is known that this caused +> 60 threads that copy the files to a new location, in parallel. The 
-severe file fragmentation, since buf clustering code does not see +> files in the new location are then deleted. This is repeated 50 times. 
-adjusted writes. Something similar for VOP_REALLOCBLKS ought to be +> Finally the original files are copied to a new location and the 
-implemented for page flushes. +> originals are deleted. 
- +</code>
-2. Filesystem full handling. +
- +
-Similar to the item #2, the mode of operation makes it very hard to +
-inform write(2) callers on the free space exhaustion. +
- +
-This could be fixed by tracking the space (not) claimed by VOP_EXTEND +
-on the per-fs basis. +
- +
-3. Unset setuid/setgid bits after write. +
- +
-When file is modified, suid bit shall be dropped. Doing it by +
-VOP_GETATTR/VOP_SETATTR is inefficient. Either VOP_EXTEND should be +
-modified to also drop setuid/gid bit, and then called unconditionally +
-on write. Or another VOP could be added.+
  
-5. Swap-backed md deadlock.+The measurement of the fragmentation with use of fragc shown that both unpatched and patched kernel give fragmentation <0.2%, with some variations +/- 40% of absolute value on each side.
  
-Swap-backed md is deadlock-prone. When a page is not present in the +=== Filesystem full handling === 
-swap object, vm_page_grab() is called. On the low free pages +Similar to the item #2, the mode of operation makes it very hard to inform write(2) callers on the free space exhaustion.
-condition, it VM_WAITs. On the other hand, pageout daemon may need to +
-flush a page belonging to the md swap object, causing +
-producer/consumer deadlock. Page greatly increases the chances of +
-hitting the issue.+
  
-Some mitigation is presented in the patch, cleaning the dirtyfied +This could be fixed by tracking the space (not) claimed by VOP_EXTEND on the per-fs basis.
-pages immediately on low memory condition.+
  
-6. Rangelocks implementation is not optimal.+=== Swap-backed md deadlock === 
 +Swap-backed md is deadlock-prone. When a page is not present in the swap object, vm_page_grab() is called. On the low free pages condition, it VM_WAITs. On the other hand, pageout daemon may need to flush a page belonging to the md swap object, causing producer/consumer deadlock. Page greatly increases the chances of hitting the issue. 
 +Some mitigation is presented in the patch, cleaning the dirtyfied pages immediately on low memory condition.
  
-Rangelocks are implemented by msleep and mutex, suffering from the +=== Rangelocks implementation is not optimal === 
-same double-wait queue inefficiency as old sx and lockmgr. +Rangelocks are implemented by msleep and mutex, suffering from the same double-wait queue inefficiency as old sx and lockmgr.
-7. fsync(2) and O_SYNC efficiency.+
  
-fsync(2) and O_SYNC are implemented by vm_object_page_clean() for now. +=== fsync(2) and O_SYNC efficiency === 
-Jeff has the plan to make walking the list of object dirty pages and +fsync(2) and O_SYNC are implemented by vm_object_page_clean() for now. vm_object_page_clean() was rewritten recently. 
-flushing them much more efficient.+=== Possible opt-in for vmio from fs === 
 +I think that new i/o path should be explicitely allowed by mnt_kern_flags bit.
  
-8. Possible opt-in for vmio from fs.+----
  
-I think that new i/o path should be explicitely allowed by +{{tag>Stale}}
-mnt_kern_flags bit.+
  
  • konstantinbelousov/vm6.1271279638.txt.gz
  • Last modified: 2010/04/14 21:13
  • by KonstantinBelousov