Thursday June 8th 15:30 to 16:30 in DMS 1130

The goal of this session is to come up with strategies for hardening UFS/FFS.

If you would like to participate, contact the working group chairs below and CC devsummit@FreeBSD.org. You will be then added to this page. Please include a list of things you want to talk about or the areas you are interested in. This helps us in planning the session and to bring people together with common interests.

It is possible to bring in people who cannot attend in person via video conference or chat tools. Notes during the session will be published later on for the whole community to see what we discussed.

Some possible ideas: In the event of unrecoverable write errors, downgrade the filesystem to read-only rather than panic'ing. In the event of loss of the underlying media, forcibly unmount the filesystem rather than panic'ing. Review all the panic's in the filesystem and replace them with one of the two above approaches. This is not an exhaustive list and if you feel there is something missing that you want to talk about, contact one of the session chairs and we will include your topic here. Note that the numbering of the topics does not represent an ordering or importance indication of any kind, but rather a reference to the second table with the “topic of interest” column.

Note: General presentations about work you have done that does not require further discussions should be submitted for the FreeBSD Developer Summit track at BSDcan (see general developer summit page).

In order to attend you need register for the developer summit as well as by email for the session and be confirmed by the working group organizer. Follow the guidelines described on the main page or what you received by email. For questions or if in doubt ask the session chairs.

Please do NOT add yourself here. Your name will appear automatically once you received the confirmation email. You need to put your name on the general developer summit attendees list though.

Prior to the meeting, several folks that will not be able to attend submitted the following comments for consideration by the group:

Conrad Meyer comments:

I suspect we should consider, if we haven't already, actively hostile filesystems. What happens when a user picks a USB stick off the ground and plugs it in? There are similar attack vectors in cloud/jail space — what happens if root, in a jail, creates an md device with malicious contents and mounts it? In any event, we don't want mounting such filesystems to lead to code execution in kernel context.

Konstantin Belousov comments:

My immediate thought, after I reviewed a list above, is that there are quite common issues which lead to crash and which cannot be handled in the way you proposed. I myself saw it several times at my previous $JOB, with the hardware raid controllers (LSI brand).

Problem is that the failure mode, unfortunately, is not the binary drive working/it is failed. I saw several times a situation where controller started returning wrong block (i.e. asked for cg block, buffer is filled with the user data), and several times I saw when controller returned zero block. The later more often occurs with USB-attached disks under load.

I never saw a situation when a controller returned complete garbage instead of _some_ real block content or zeroes, but of course this cannot be excluded as well.

The result of any such malfunction is the kernel crash due to UFS trusting the content of the metadata blocks. I am not sure that it is possible to code around the problems to make the kernel survive, but even limited sanity checks for each metadata element obtained from the disk would cause expensive slowdown, I believe.

Kirk responds: As at least a partial solution, we could calculate checksums for superblocks, cylinder groups, and inodes (stored in those structures) which we could check each time they are read and updated each time they are written. For block pointers, we could check that they are in range for the filesystem.

Kostik responds: Indeed this is only a partial solution. E.g. substitutions of one cg block by another would go unnocited with catastrophic consequences. Same for the inode blocks.

It might be more feasible to run UFS over some crypto-strong authentification layer to verify that blocks are pristine, eg. geli.

Kirk responds: Cylinder group blocks contain their number (cg_x) which could also be checked along with its checksum to ensure we got the one that we expected. As you note, inode blocks have the same substitution problem (and with higher probability than cylinder group blocks). We could solve that in the same way if we added the inode number to each inode. To save space, we could encode the inode number in the checksum (e.g., calculate the checksum, then XOR in the inode number when writing and do the reverse when doing read verification).

Edward Tomasz Napierala comments:

I'm not sure I'll be able to attend, but the patch below might be relevant. It does three things: It adds a method for GEOM to notify the filesystem that the device went away, it adds a filesystem-independent vfs_orphan() function to prevent all future access to the vnodes (without unmounting it; doing that would make the future writes succeed, but silently go to the wrong filesystem), and makes UFS and msdosfs use it.

https://people.freebsd.org/~trasz/old/callback-6.diff

• Hardening does not mean turning UFS into ZFS: data checksumming, redundant data blocks
• Can run on lower-resource machines (implemented when 4 MB was a lot of memory)
• Now see KASSERTs and panics in the code. KASSERTs can be compiled out; not so much panics
• Thought was that corrupted data was worse than crashes; panic when finding an inconsistency that could result in corruption.
• Current thought: downgrade to read-only if disk is still readable (EIO); otherwise forcibly unmount if disk is permanently gone (ENODEV)
• Problem: existing downgrade code needs to flush out dirty data
• Forcible downgrade (never implemented): throw away dirty data; e.g. disk is failing/failed.
• Comments: Netflix would much prefer downgrade or removal from service
  • Comment: others would like mount option
• Comment: different behavior on virtual machines (may go away and then recover)
• Comments on wiki:
  • Conrad Meyer: malicious filesystems
  • kostic: disk returning wrong data; example, wrong cg block; could checksum and check cg number
  • Edward: vfs_orphan
• Re: malicious: hard to deal with, use ZFS or guineau-pig computer
  • mention of user-level fs for checking
• How to handle ENXIO/ENODEV: can't just throw away writes because of soft update (e.g. rollback; need to reread, may not be in buffer cache)
  • Could maybe lock dirty buffers in buffer cache but not retry
• I/O errors: may have been retried at lower level, or maybe device just lied by returning wrong block
• Useful to have data on which panics actually happen (e.g. from large company)
  • After session: Netflix and Backtrace I/O offered panic logs going back several years
• MAXPHYS: now 128K; 1 MB is good for many systems, not embedded; maybe tunable/boot time computed, need to check device drivers
  • geom has MAXIOSIZE; Netflix uses 8 MB on some
  • side issue, reading large logs from drive (128 MB); Linux can supposedly do in one read.
• questions on why some people prefer panics: on VMs, won't reboot until ready, then will work (e.g. filer rebooting)