Migrate system from i386 (32-bit mode) to amd64 (64-bit mode)

If an amd64 hardware has been initially installed in 32-bit mode (i.e. as a plain i386 machine), it might later be desired to turn it into a full 64-bit machine.

The recommended way to do this is to back up all personal and important data, and reinstall using an amd64 installation medium.

However, it's also possible to migrate the machine through a rebuild from source code.

Much of this description has been inspired by Peter Wemm's mailing list article.

• some amd64 hardware, currently running in i386 mode (obviously ;.)
• a swap partition/volume of at least 450 MiB which can be used as a temporary root filesystem (miniroot)
• enough RAM so the machine could at least install a precompiled system without swap
• a recent /usr/src filesystem of whatever branch has been chosen
• a running i386 system matching whatever is installed under /usr/src (just to be sure everything is consistent)
• a safe backup of all important data, in case something goes wrong with the migration
• console access, so you can use the loader prompt and single user mode (ssh access is not sufficient!)

• If necessary, define your kernel config file in /etc/make.conf, create and edit it (otherwise, GENERIC will be used).
• Build amd64 world and kernel using:
  • make buildworld TARGET=amd64 TARGET_ARCH=amd64
  • make buildkernel TARGET=amd64 TARGET_ARCH=amd64
    This is supposed to pass without any issues. If not: resolve issues, and start over.
• Turn your swap into a miniroot:
  • swapinfo -h — make sure no swap is in use (if the swap is not free, reboot here)
  • swapoff /dev/ad4s1b (or whatever your swap device is named — replace that name in the steps below if it is different)
  • edit /etc/fstab to comment out the swap line
  • newfs -U -n /dev/ad4s1b
  • mount /dev/ad4s1b /mnt
  • cd /usr/src && make installworld TARGET=amd64 TARGET_ARCH=amd64 DESTDIR=/mnt
  • file /mnt/bin/echo — make sure this displays as ELF 64-bit LSB executable
  • cd /usr/src/etc && make distribution TARGET=amd64 TARGET_ARCH=amd64 DESTDIR=/mnt
  • cp /etc/fstab /mnt/etc/fstab

This completes your miniroot in the swap volume.

• Prepare the /usr/obj tree for later installation:
  • The 64-bit (cross) build should be in /usr/obj/amd64.amd64 now, so remove any leftover 32-bit stuff: rm -rf /usr/obj/usr/src.
  • cd /usr/obj && ln -s amd64.amd64/* . — this should get symlinks for lib32 and usr (the original file location must still be retained by now)
• Copy the 64-bit kernel into a bootable place, like
  • cp /usr/obj/amd64.amd64/usr/src/sys/$YOURKERNEL/kernel /boot/kernel.amd64
  • If any further KLDs are needed that have not been statically compiled into the kernel, copy them over to some place (presumably under /boot), too.
• Reboot, and stop the system at the loader prompt (in the loader menu, press “2” for FreeBSD 9, or press “6” for FreeBSD 8)
• unload to get rid of the (automatically loaded) 32-bit kernel
• load /boot/kernel.amd64, possibly followed by loading any essential 64-bit KLDs here
• boot -as to boot into single-user mode, with the loader asking for the location of the root filesystem (rather than figuring it out from /etc/fstab)
• At the loader prompt asking for the root filesytem, enter ufs:, followed by the name of the swap volume/partition; e.g. ufs:/dev/ad4s1b
• Press Enter to get the single-user shell
• Mount (at least) the /, /var and /usr filesystems under /mnt; examine /etc/fstab to know which resources to mount
• Bootstrap the system libraries and utilities from the miniroot:
  • chflags -R noschg /mnt/*bin /mnt/lib* /mnt/usr/*bin /mnt/usr/lib*
  • cd / && find *bin lib* usr/*bin usr/lib* | cpio -dumpv /mnt
  • mount -t devfs devfs /mnt/dev
• chroot /mnt so you can pretend living in the final target filesystem space
• cd /usr/src && make installkernel installworld — this should work without any issues
• exit so you're back in the miniroot environment
• reboot
• Boot into single-user mode (in the loader menu, press “6” then Enter for FreeBSD 9, or press “4” for FreeBSD 8)
• fsck -p (just make sure all filesystems are allright)
• mount -a -t ufs
• Edit /etc/fstab to re-enable the swap
• To give any existing (32-bit) ports a chance to work, do the following:
  • mkdir /usr/local/lib32
  • cd /usr/local/lib && find . -type f \( -name "*.so*" -o -name "*.a" \) | cpio -dumpv ../lib32
  • add the following line to /etc/rc.conf:
    • ldconfig32_paths="/usr/lib32 /usr/local/lib32"
• Turn /usr/obj into its canonical form (optional):
  • cd /usr/obj
  • rm * — just to remove the lib32 and usr symlinks (it will complain and not remove the amd64.amd64 directory)
  • mv amd64.amd64/* . && rmdir amd64.amd64
• Remove the temporary kernel (optional): rm /boot/kernel.amd64
• Exit the single-user shell, bringing the system into multiuser

Now the basic migration has been done.

Some 32-bit ports might not work, despite of the hackery to save their shared libs in a separate place. It's probably best to reinstall all ports. portupgrade -af doesn't work as it clobbers its own package database in the course of this operation (when reinstalling ruby and/or ruby-bdb).