| Both sides previous revision Previous revision Next revision | Previous revision |
| ext2fs [2017/10/31 16:01] – ext4 write support added to current. PedroGiffuni | ext2fs [2023/07/13 14:56] (current) – mention i_version support PedroGiffuni |
|---|
| ====== Ext2/3/4 Filesystem ====== | ====== Ext2/3/4 Filesystem ====== |
| |
| The Extended Filesystem was [[http://e2fsprogs.sourceforge.net/ext2intro.html|designed]], with much inspiration on UFS, to replace the minix filesystem used in early versions of linux. The filesystem is known to be very fast and continues to be very popular despite the availability of other interesting filesystems (JFS, reiserfs, XFS, etc). | The Extended Filesystem is popular in distributions of the linux kernel where it as created. Most BSD variants have at least some support for it and there are userland (fuse) drivers as well. This page serves to keep some notes related to %%FreeBSD%%'s kernel implementation that supports the original ext2 and sufficient features to support newer versions of the popular filesystem. |
| |
| In general Ext2fs is very similar to UFS but lacks support for fragments and as a consequence attempting to use bigger blocksizes penalizes the wasted space and also makes fragmentation a problem. The linux developers have also opted for supporting the faster async mode and compensate the eventual risks with more robust filechecking tools (fsck). The ext2 developers have tried to counter the weaknesses of the filesystem by adding new features in ext3 (journalling) and ext4 (extents) but the basic design seems to be exhausted and its designers are apparently considering more recent designs like btrfs. For freebsd supporting ext2fs and it variants is still important for interoperability and as a general experimentation tool. | The Extended Filesystem was [[http://e2fsprogs.sourceforge.net/ext2intro.html|designed]], with much inspiration on UFS, to replace the MINIX filesystem used in early versions of Linux. The filesystem is known to be very fast and continues to be very popular despite the availability of other interesting filesystems (JFS, reiserfs, XFS, etc). |
| |
| The initial %%FreeBSD%% ext2fs implementation was based on the [[http://www.cs.utah.edu/flux/lites/html/ext2fs.html|BSD Lites version]] by Godmar Back. The initial approach was to reuse the BSD interfaces and the similarities between UFS and ext2 and merge the specific block management routines from linux. | In general Ext2fs is very similar to UFS but lacks support for fragments and as a consequence attempting to use bigger blocksizes penalizes the wasted space and also makes fragmentation a problem. The Linux developers have also opted for supporting the faster async mode and compensate the eventual risks with more robust filechecking tools (fsck). The ext2 developers have tried to counter the weaknesses of the filesystem by adding new features in ext3 (journalling) and ext4 (extents) but the basic design seems to be exhausted and its designers are apparently considering more recent designs like btrfs. For freebsd supporting ext2fs and it variants is still important for interoperability and as a general experimentation tool. |
| | |
| | The initial %%FreeBSD%% ext2fs implementation was based on the [[http://www.cs.utah.edu/flux/lites/html/ext2fs.html|BSD Lites version]] by Godmar Back. The initial approach was to reuse the BSD interfaces and the similarities between UFS and ext2 and merge the specific block management routines from Linux. |
| %%NetBSD%% did a complete [[http://www.feyrer.de/NetBSD/ext2fs-rev1.html|reimplementation]] from UFS sources. | %%NetBSD%% did a complete [[http://www.feyrer.de/NetBSD/ext2fs-rev1.html|reimplementation]] from UFS sources. |
| |
| As the rest of the %%FreeBSD%% kernel code, we follow style(9). We do try to keep the code in sync where possible with UFS/FFS to ensure new UFS fixes and features can be applied to Ext2. When implementing new features it is important to give a thought on the layout: the preference should always be to avoid invasive changes in the files that are shared to some extent with UFS and add new functionality in new files. | As the rest of the %%FreeBSD%% kernel code, we follow style(9). We do try to keep the code in sync where possible with UFS/FFS to ensure new UFS fixes and features can be applied to Ext2. When implementing new features it is important to give a thought on the layout: the preference should always be to avoid invasive changes in the files that are shared to some extent with UFS and add new functionality in new files. |
| |
| The linux driver is, of course, copyleft, so code cannot be generally copied from one implementation to the other. Luckily our implementation is completely different and much more similar to UFS but developers are encouraged to read the existing (public) documentation before starting to work on new features. | The Linux driver is, of course, copyleft, so code cannot be generally copied from one implementation to the other. Luckily our implementation is completely different and much more similar to UFS but developers are encouraged to read the existing (public) documentation before starting to work on new features. |
| |
| ===== Main developments ===== | ===== Main developments ===== |
| |
| During a long time %%NetBSD%%'s and %%FreeBSD%%'s implementation were maintained independently and while they basically worked, development was stuck. The main difference between both implementations was basically the cleaner license in %%NetBSD%% implementation and the relatively good performance of the %%FreeBSD%% implementation. In both cases, running the BSD implementations in async mode (the default in linux) is considerably slower in the BSDs compared to the linux implementation. | During a long time, %%NetBSD%%'s and %%FreeBSD%%'s implementations were maintained independently and while they basically worked, development was stuck. The main difference between both implementations was basically the cleaner license in %%NetBSD%% implementation and the relatively good performance of the %%FreeBSD%% implementation. In both cases, running the BSD implementations in async mode (the default in Linux) is considerably slower in the BSDs compared to the Linux implementation. |
| |
| On year 2009, there was a Google SOC project: | On year 2009, there was a Google SOC project: |
| [[https://github.com/vmatyushin/netbsd-ext3-htree|HTree directory indexing for Ext3]] | [[https://github.com/vmatyushin/netbsd-ext3-htree|HTree directory indexing for Ext3]] |
| |
| ZhengLiu did a port of Vyacheslav Matyushin's HTree implementation. the HTree code in Linux requires a lot of workarounds for the possibility of a hash collision which were not considered for the %%FreeBSD%% port. | ZhengLiu did a port of Vyacheslav Matyushin's HTree implementation for %%NetBSD%%'s %%GSoC%%. The HTree code in Linux requires a lot of workarounds for the possibility of a hash collision which were not considered for the %%FreeBSD%% port. |
| The code was unstable for a while but finally got fixed on SVN [[https://svnweb.freebsd.org/base?view=revision&revision=294504|r294504]]. | The code was unstable for a while but finally got fixed on SVN [[https://svnweb.freebsd.org/base?view=revision&revision=294504|r294504]]. |
| |
| Recent additions include support for nanosecond/birthtime timestamps (2012), the benefit of a generic SEEK_DATA/SEEK_HOLE lseek() extension (2013), the support for sparse files from ext4 (DamjanJovanovic 2016) and the read-only support for Extended Attributes (FedorUporov 2017). | More recent additions include support for nanosecond/birthtime timestamps (2012), the benefit of a generic SEEK_DATA/SEEK_HOLE lseek() extension (2013), the support for sparse files from ext4 (DamjanJovanovic 2016) and support for Extended Attributes, Ext4 write support, uninit_gb, flex_bg and checksumming (FedorUporov 2017). |
| |
| ===== Features ===== | ===== Features ===== |
| In %%FreeBSD%% ext2, ext3 and ext4 are not different filesystems: ext2 is the base filesystem and some features from ext3 and ext4 are supported. All features in %%FreeBSD%%'s implementation follow UFS semantics and this can sometimes impose important differences. | In %%FreeBSD%% ext2, ext3 and ext4 are not different filesystems: ext2 is the base filesystem and some features from ext3 and ext4 are supported. All features in %%FreeBSD%%'s implementation follow UFS semantics and this can sometimes impose important differences. |
| |
| - %%FreeBSD%% supports most ext4 features in read-only mode: it is especially important to avoid the 64 bit feature. | * %%FreeBSD%% 13-current supports Big Endian Platforms, including %%PowerPC%%, through byteswapping. |
| |
| - %%FreeBSD%% 12+ includes initial support for writing in ext4 filesystems. | * %%FreeBSD%% 12.1 and later have support for DTrace to enable debugging in production. |
| |
| - %%FreeBSD%% 10.1+ uses by default reallocblk. This comes from UFS and helps preventing defragmentation issues. Linux doesn't have an exact equivalent but Ext4 does delayed allocation which is similar in concept. | * All supported %%FreeBSD%% versions support ext2/3 read and write with most of the features, excepting journalling. |
| |
| - %%FreeBSD%% 9.x+ fully supports the async mode, which is the default on linux. Unlike linux, the default is to use the sync mode which will make your filesystem more reliable at the cost of some performance. | * %%FreeBSD%% 12+ includes initial support for writing ext4 filesystems (thanks to Fedor Uporov). |
| |
| NOTE: If you are trying to mount Ext4 partitions remember to specifically mount the filesystem **read-only**. | * %%FreeBSD%% 10.1+ uses by default reallocblk. This comes from UFS and helps preventing defragmentation issues. Linux doesn't have an exact equivalent but Ext4 does delayed allocation which is similar in concept. In freeBSD 12 the feature was disabled by default as it was unstable. |
| |
| ===== Known issues ===== | * %%FreeBSD%% 9.x+ fully supports the async mode, which is the default on Linux. Unlike Linux, the default is to use the sync mode which will make your filesystem more reliable at the cost of some performance. |
| |
| - We don't support the Ext4 64bit incompat feature. We handle fine big disks but starting from ext2fsprogs v1.43, linux distributions are now activating the 64 bit feature even when partitions are smaller than 16T. The feature also causes problems for some linux bootloaders and utilities. Disabling it with tune2fs is the only alternative for now. | * The ports tree holds several userland (fuse) implementations. |
| |
| - We don't support byte swapping for Big Endian platforms. Not many people have such machines and UFS doesn't do it either so there has been no urgency. | NOTE: If you are trying to mount Ext4 partitions, and we don't yet support writing on your particular release, remember to specifically mount the filesystem **read-only**. |
| |
| - We lack support for ACLs: these are not very commonly used but are very useful. Once we have EA support, we can consider some translation scheme. | ===== Known issues ===== |
| |
| - We lack support for journalling which is inconvenient but this perhaps not really a problem if you are OK with running in sync mode, which is usually safe and the default on %%FreeBSD%%. Furthermore, at least in the linux implementation, journalling is known to reduce performance. | * We lack support for journalling which is inconvenient but is not necessarily be a problem if you are OK with running in sync mode, which is usually safe and the default on %%FreeBSD%%. Furthermore, at least in the Linux implementation, journalling is known to reduce performance. |
| |
| - We don't support the SSD trim command: the code to merge this feature from UFS is relatively simple but while on Linux this is a mount option, in %%FreeBSD%%'s UFS this is set in tunefs. For convenience we always choose to keep the same semantics as in UFS, so it's unsupported (for now). | * We don't support the SSD trim command: the code to merge this feature from UFS is relatively simple but while on Linux this is a mount option, in %%FreeBSD%%'s UFS this is set in tunefs. For convenience we always choose to keep the same semantics as in UFS, so it's unsupported (for now). |
| | |
| | * Starting from ext2fsprogs v1.43, Linux distributions are now activating the 64 bit feature even when partitions are smaller than 16T. This feature causes problems for some Linux bootloaders and utilities. The driver in %%FreeBSD%% 12 has support for the feature but we don't expect to merge it to previous %%FreeBSD%% versions. |
| |
| ===== Future Projects ===== | ===== Future Projects ===== |
| Depending on developer's interest there are some possibilities for future development: | Depending on developer's interest there are some possibilities for future development: |
| |
| - ACLs and write support for Extended Attributes. | * Journalling is a characteristic feature of ext3+. There has been at least one [[https://github.com/bitwize/src/commits/bitwize_ext3fs/sys/ufs/ext2fs|attempt]] to do a from-scratch implementation for %%NetBSD%%. Haiku also supports journalling in their [[https://git.haiku-os.org/haiku/tree/src/add-ons/kernel/file_systems/ext2|implementation]]. |
| |
| - Ext4fs RW support. Some Windows drivers support basic writing in ext4: there may be some features that we can't implement but adding write support and implementing more ext4 features remains possible. | * Trim support should be doable, based on the UFS implementation. |
| |
| - On Ext4, linux has recently [[http://lwn.net/Articles/544520/|widened]] the dirindex hash space from 31 to 63 bits. We have had experimental [[http://people.freebsd.org/~pfg/patches/ext2fs/ext2-dirhash64.patch|patch]] for this but it is unclear how much it is useful since they didn't update the on-disk format. | * Linux's Ext4 has support for [[http://blog.quarkslab.com/a-glimpse-of-ext4-filesystem-level-encryption.html|encryption]]. |
| |
| - Ext4 has support for [[http://blog.quarkslab.com/a-glimpse-of-ext4-filesystem-level-encryption.html|encryption]]. | * Performance testing. Benchmarks against UFS and the original Linux ext2 (or the one in %%FreeBSD%% 8.x) are always interesting. We would like to keep the performance as similar as possible to the Linux version but at this time it is not clear how the implementation compares with other implementations. |
| |
| - Journalling is a characteristic feature of ext3+. | * Some options never picked up much interest in Linux but may be interesting for hacking pleasure. One such option is [[http://e2compr.sourceforge.net/|e2compr]] which brings basic compression to ext2fs. |
| Haiku did an adaptation of their own journalling [[https://www.haiku-os.org/blog/jvff/2010-08-05_ext3_journal_implementation|layer]]. Unfortunately using %%FreeBSD%%'s gjournal, which could have been a clean option, is incompatible with the format used by linux. | |
| There has been at least one [[https://github.com/bitwize/src/commits/bitwize_ext3fs/sys/ufs/ext2fs|attempt]] to do a from-scratch implementation for %%NetBSD%%. | |
| |
| - Performance testing. Benchmarks against UFS and the original linux ext2 (or the one in %%FreeBSD%% 8.x) are always interesting. For research purposes we would like to keep the performance as similar as possible to the linux version. | * Perhaps see what would be missing to make it an installation option. |
| |
| - Some options never picked up much interest in linux but may be interesting for hacking pleasure. One such option is [[http://e2compr.sourceforge.net/|e2compr]] which brings basic compression to ext2fs. | * Support for [[https://lwn.net/Articles/905931/|i_version]]: The e2di_version field exits in our implementation but is unused, we could support an ioctl for linux binary applications. |
| | |
| - Filesystem virtualization: Running ext2fs as ZFS' ZVOLS is an interesting possibility for linux applications running in jails or bhyve. While it is not clear if such approach brings any specific advantage, it may be useful for some applications to use a real linux filesystem format. | |
| |
| ===== Documentation ===== | ===== Documentation ===== |
| |
| The most update information for the Ext4 implementation is available through the kernel project itself: | The most update information for the Ext4 implementation is available through the kernel project itself: [[https://ext4.wiki.kernel.org/index.php/Ext4_Disk_Layout|Ext4 (and Ext2/Ext3) Wiki]] |
| | |
| [[https://ext4.wiki.kernel.org/index.php/Ext4_Disk_Layout|Ext4 (and Ext2/Ext3) Wiki]] | |
| | |
| Some classic documentation for the linux implementation: | |
| | |
| [[https://www.kernel.org/doc/ols/2007/ols2007v2-pages-21-34.pdf|The new ext4 filesystem: current status and future plans]] | |
| | |
| [[http://ext2.sourceforge.net/2005-ols/2005-ext3-paper.pdf|State of the Art: Where we are with the Ext3 filesystem]] | |
| | |
| [[http://ext2.sourceforge.net/2005-ols/2005-ols-ext3-presentation.pdf|State of the Art: Where we are with the Ext3 filesystem (presentation)]] | |
| |
| [[http://kernel.org/doc/ols/2002/ols2002-pages-425-438.pdf|A Directory Index for Ext2]] | Some documentation for the Linux implementation: |
| |
| Patent: | * [[https://www.kernel.org/doc/ols/2010/ols2010-pages-121-132.pdf|Space Maps in Ext4]] |
| | * [[https://www.kernel.org/doc/ols/2007/ols2007v2-pages-21-34.pdf|The new ext4 filesystem: current status and future plans]] |
| | * [[http://ext2.sourceforge.net/2005-ols/2005-ext3-paper.pdf|State of the Art: Where we are with the Ext3 filesystem]] |
| | * [[http://ext2.sourceforge.net/2005-ols/2005-ols-ext3-presentation.pdf|State of the Art: Where we are with the Ext3 filesystem (presentation)]] |
| | * [[http://kernel.org/doc/ols/2002/ols2002-pages-425-438.pdf|A Directory Index for Ext2]] |
| |
| [[https://www.google.com/patents/US20100057755|US Patent 20100057755 A1]], File system with flexible inode structures. | Patent: [[https://www.google.com/patents/US20100057755|US Patent 20100057755 A1]], File system with flexible inode structures. |
| (Redhat has extended its patent [[https://www.redhat.com/en/about/patent-promise|promise]] to all open-source licenses). | (Red Hat has extended its patent [[https://www.redhat.com/en/about/patent-promise|promise]] to all open-source licenses). |
| |
| For %%FreeBSD%%'s specific ext2fs driver the general documentation for UFS is useful. | For %%FreeBSD%%'s specific ext2fs driver the general documentation for UFS is useful. |