HAST - Highly Available Storage

HAST allows to transparently store data on two physically separated machines connected over the TCP/IP network. Those two machines together will be called a cluster and each machine is one cluster node. HAST works in Primary-Secondary (Master-Backup, Master-Slave) configuration, which means that only one of the cluster nodes can be active at any given time. Active node will be called Primary node. This is the node that will be able to handle I/O requests to HAST-managed devices. Currently HAST is limited to two cluster nodes in total.

HAST operates on block level - it provides disk-like devices in /dev/hast/ directory for use by file systems and/or applications. Working on block level makes it transparent for file systems and applications. There in no difference between using HAST-provided device and raw disk, partition, etc. All of them are just regular GEOM providers in FreeBSD.

HAST can be compared to a RAID1 (mirror) where one of the components is local disk (on primary node) and second component is disk on remote machine (secondary node). Every write, delete or flush operation (BIO_WRITE, BIO_DELETE, BIO_FLUSH) is send to local disk and to remote disk over TCP connection (if secondary node is available). Every read operation (BIO_READ) is served from local disk, unless local disk isn't up-to-date or an I/O error occurs, then read operation is send to secondary node (if it is, of course, available).

It is very important to reduce synchronization time after node's outage. Synchronizing all data when connection fails for few minutes isn't very optimal. To provide fast synchronization HAST manages on-disk bitmap of dirty extents. Every extent is represented by one bit in bitmap, so this is the smallest block that can be marked as dirty by HAST. By default extent size is 2MB. Eventhough write will be confirmed to the file system above only when both nodes confirm it, it is still possible that the nodes can get out of sync, for example when write will succeed on primary node, but will never be send to secondary node. This is why extent has to be marked as dirty before writing the data. Of course it will be very slow if HAST would:

1. Mark extent as dirty and write metadata.
2. Write data.
3. Mark extent as clean and write metadata.

It would mean that each write operation will be turned into three write operations. To avoid this, HAST keeps fixed number of extents marked as dirty all the time. By default those are 64 most recently touched extents. This of course means that when nodes connect they have to synchronize by default 128MB of data, which don't really need to be synchronized, but on local network it is very quick operation and it is definiatelly worth it. Note that extent size has to be choosen carefully. If the extent size is too small, there will be a lot of metadata updates, which will degrade overall performance. If the extent size is too big, synchronization time can be much longer, which will also degrade performance before synchronization completes.

HAST is not responsible for selecting node's role (primary or secondary). Node's role has to be configured by an administrator of other software like heartbeat or ucarp using hastctl(8) utility.

When we work as primary and secondary is missing we will increase localcnt in our metadata. When secondary is connected and synced we make localcnt be equal to remotecnt, which means nodes are more or less in sync.

Split-brain condition is when both nodes are not able to communicate and are both configured as primary nodes. In turn, they can both make incompatible changes to the data and we have to detect that. Under split-brain condition we will increase our localcnt on first write and remote node will increase its localcnt on first write. When we connect we can see that primary's localcnt is greater than our remotecnt (primary was modified while we weren't watching) and our localcnt is greater than primary's remotecnt (we were modified while primary wasn't watching).

There are many possible combinations which are all gathered below. Don't pay too much attention to exact numbers, the more important is to compare them. We compare secondary's local with primary's remote and secondary's remote with primary's local. Note that every case where primary's localcnt is smaller than secondary's remotecnt and where secondary's localcnt is smaller than primary's remotecnt should be impossible in practise. We will perform full synchronization then. Those cases are marked with an asterisk. Regular synchronization means that only extents marked as dirty are synchronized (regular synchronization).