====== Switching a remote server from Linux to FreeBSD ======

How to switch a remote server from Linux to %%FreeBSD%% through ssh in a few easy steps. In this particular case, we want to retain the data, stored on ZFS in a two-way mirror pool **zdata**. The pool's providers are LUKS encrypted partitions, **sda3** and **sdb3** respectively, so our current set up is like this:

  * sda1 - the bios boot partition, 1M
  * sda2 - system root, mdadm md0 device
  * sda3 - LUKS provider for the zdata pool

**sdb** mirrors **sda**. This process assumes that a single disc can hold our **zdata** data twice and still have room for **zroot** and the bootloader partition. So for example, this was done with 2x3TB HDDs and zdata pool holding some 400GB so there was plenty of room for transfer.

===== 1. Disable any usage of the sdb disk =====

First let's drop **sdb** from the **zdata** pool and from **md0** device, because we need to completely rework the **sdb** disk.

<code>
# zpool offline zdata /dev/mapper/sdb3_crypto
# zpool detach zdata /dev/mapper/sdb3_crypto
# zpool labelclear /dev/mapper/sdb3_crypto
# mdadm /dev/md0 --fail /dev/sdb2 --remove /dev/sdb2
</code>

===== 2. Repartition sdb for the new, temporary layout =====

With **sdb** unused now, we repartition it for the new, temporary, layout using parted, or any of the partitioning tools of linux. With parted:

<code>
parted -s /dev/sdb mklabel gpt
parted -s /dev/sdb mkpart primary 40 512k
parted -s /dev/sdb mkpart primary 1Mib 10GiB
parted -s /dev/sdb mkpart primary 10GiB 1010GiB
parted -s /dev/sdb mkpart primary 1010GiB 100%
</code>

So we created the bootloader partition, sdb1 (future ada1p1); the zroot partition sdb2 (future ada1p2); the zdata partition sdb3 (future ada1p3) and the ztransfer partition sdb4. The idea is to move data from current zdata pool into ztransfer on sdb (ada1). Reboot and reinstall %%FreeBSD%% using ada0p2 + ada1p2 as providers for zroot, ada0p3 and ada1p3 as providers for GELI for zdata, move the data from ztransfer to new zdata, kill ztransfer, expand ada0p3 and ada1p3, expand the GELI providers, expand the zdata pool.

So far the server has been online and zdata serving without interruption even in degraded state. Before we continue with %%FreeBSD%% installation, we need to actually save the data from the old **zdata** mirrored pool into temporary **ztransfer**. Doing this **after** we've removed sdb from the pool allows disk-to-disk transfer which is fastest, otherwise ZFS would load balance **sda** and **sdb** at reading from it, while writing to **ztransfer** on **sdb** basically halving the available bandwidth, as **sdb** will be both read from and written to.

<code>
zfs snapshot -r zdata@transfer
zfs send -Rv zdata@transfer | zfs receive ztransfer
</code>

After this is done, the data has been copied over to the temporary **ztransfer** pool and we can proceed with reinstalling %%FreeBSD%% which will result in removal our existing root and zdata.

===== 3. Reboot and install FreeBSD =====

Reboot into %%FreeBSD%% installation / rescue environment and perform a regular install through ssh with bsdinstall, or by unpacking the tarballs. We partition the ex **sda**, now **ada0**:

<code>
gpart destroy -F ada0
gpart create -s gpt ada0
gpart add -t freebsd-boot -s 512k -b 40 ada0
gpart add -t freebsd-zfs -s 10G -a 1M ada0
gpart add -t freebsd-zfs -s 1000G ada0
</code>

Note that we only create three partitions on **ada0**. **p1** will be for the gpt bootloader, **p2** will hold our **zroot** pool and **p3** will hold our **zdata** pool. We currently create it to match the size of **sdb3**, which is temporarily 1T, but that is not strictly required as ZFS allows providers of different sizes in the mirror vdev, so technically **ada0p3** can be full remaining disk size.

<code>
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
zpool create zroot mirror ada0p2 ada1p2
geli init -l 256 -s 4096 ada0p3
geli init -l 256 -s 4096 ada1p3
geli attach ada0p3
geli attach ada1p3
zpool create zdata mirror ada0p3.eli ada1p3.eli
</code>

So now we have the **new zdata** pool, already in a mirror vdev that can receive data back from the temporary **ztransfer**. At this point we need to run ''%% bsdinstall %%'' or use whatever method to install %%FreeBSD%% onto the prepared **zroot** and **zdata**.

**Note:** With %%FreeBSD%% 11 it is possible to boot from encrypted zroot, so having two pools may not be needed.

===== 4. Reboot into installed FreeBSD and continue operations =====

After installing %%FreeBSD%% we can reboot into it and configure the server for its purpose. At this point the zdata pool only needs to receive data back from the temporary **ztransfer** pool, so let's get it back:

<code>
zfs send -Rv ztransfer@transfer | zfs receive zdata
</code>

When that is complete the server can resume operation. All that remains to be done is remove the temporary **ztransfer** pool and expand zdata to use full remaining disk space:

<code>
# ...
</code>

Total server downtime is time required to copy zdata over to ztransfer and back, plus time needed to partition and install %%FreeBSD%% and configure it for the wanted operation. One day, when ZFS allows creating degraded mirror pools, this operation can be reduced down to time required to reboot and reinstall %%FreeBSD%%, since in that case **sdb** can be partitioned and prepared with new (degraded) zdata, data copied over while the server is operational, without the requirement to use temporary **ztransfer** pool.