ColinPercival is coordinating an effort to speed up the FreeBSD boot process and would like to see FreeBSD become far more competitive in 14.0. Some improvements have already been made since 13.0.

An Amazon EC2 c5.xlarge instance is being used as a reference platform and measuring the time between the instance entering “running” state and when it is possible to SSH into the instance.

Boot time consists of four consecutive stages:

1. System: BIOS/UEFI, before running any FreeBSD code – we can't do anything about this.
2. FreeBSD: boot loader.
3. FreeBSD: kernel, before init starts running.
4. FreeBSD: userland boot process.

The boot loader and kernel can be profiled using the TSLOG framework to generate a flamechart. To do this:

1. Build a kernel with 'options TSLOG'.
2. Check out the freebsd-boot-profiling repository
3. Run sh mkflame.sh > tslog.svg

Some statistics collected in August 2021¹⁾:

• FreeBSD 13.0 takes 19.00 seconds to get from “running” to “port TCP/22 closed” and another 5.13 seconds to get to “port TCP/22 open”, for a total of 24.13 seconds.
• Debian 10 takes a total of 10.35 seconds.
• Clear Linux 34640 takes a total of 1.23 seconds.

Performance is measured using the ec2-boot-bench utility.

{{boottime/tslog-11.1.svg?0x150 Flamegraph of FreeBSD 11.1-RELEASE boot time (28.208 seconds)}}

{{boottime/tslog-14.0.svg?0x150 Flamegraph of FreeBSD 14.0-CURRENT boot boot time (17.216 seconds)}}

Loader (~260 ms):

• Reading the kernel from disk takes ~150 ms. This would be faster with a smaller and/or a compressed kernel, but tooling work would be needed for kernel updates. This sometimes takes longer the first time an instance boots, possibly due to the underlying disk being “lazily” initialized from the backing EC2 snapshot.
• The loader spends ~45 ms printing status information to the console.
• The loader spends ~65 ms on other stuff (reading in the lua scripts, loading fonts, etc).

Kernel (~ 2400 ms):

• SYSINIT vm_mem takes ~25 ms on the benchmark system (8 GB of RAM), increasing increases proportionally to RAM; potentially a concern on larger systems.
• SYSINIT cpu_mp takes ~30 ms on the benchmark system (4 vCPUs), increasing proportionally to the number of APs; potentially a concern on larger systems.
• Mounting the root filesystem spends ~40 ms waiting for g_event quiescence. Disk/partition tasting perhaps? Needs investigation.
• SYSINIT cpu has a 1000 ms DELAY for TSC frequency calibration.
• SYSINIT apic_setup_io has a 1000 ms DELAY for local APIC clock frequency calibration.
• Probing and attaching devices takes ~150 ms.
• The kernel spends ~95 ms printing information to the console.
• The kernel spends ~60 ms on other stuff.

Userland (~ 8450 ms):

• /etc/rc.d/random takes 100 ms per entropy file being fed into the kernel; this pause is found in randomdev_write. This time is not present the first time an instance boots because new instances don't have any cached entropy.
• /etc/rc.d/netif takes ~4400 ms, including 2100 ms running rtsol(8) and two 1-second sleep(1)s.
• /etc/rc.d/rtsold takes ~2100 ms.
• /etc/rc.d/devd takes ~200 ms, mostly running many devmatches.
• Everything else combined takes ~850 ms.

In addition to the above, there are some performance issues which don't affect the benchmark system but should be addressed to help other platforms.

On Colin's 13.0-RELEASE laptop:

• hammer_time spends ~650 ms of DELAYs resetting the PS/2 keyboard controller (“early console”).
• DEVICE_ATTACH acpi_ec has ~800 ms of DELAYs.
• DEVICE_ATTACH em has ~120 ms of DELAYs.
• DEVICE_ATTACH atkbd spends ~450 ms of DELAYs resetting the PS/2 keyboard controller (again).

Boot performance improvements which have been prompted by this effort:

• 2017-09-07 (r323290, by markj): Speed up vm_page_array initialization. Reduces SYSINIT vm_mem time by ~ 100 ms on the benchmark system. (Scales with RAM size.)
• 2017-11-04 (r325383, by delphij): hpt* driver fixes. Reduces DEVICE_PROBE hpt* time by ~ 330 ms on the benchmark system.
• 2018-05-07 (r333335, by imp): Put the CPU starting on one line. Reduces SYSINIT start_aps time by ~ 120 ms on the benchmark system. (Scales with number of APs.)
• 2018-08-07 (r337411, by cperciva): Replace a pair of 8-bit writes to VGA memory by a 16-bit write. Reduces time spent in vt(4) by ~ 1300 ms on the benchmark system.
• 2018-08-25 (r338316, by cperciva): Cache the most recently drawn characters on terminal and avoid redrawing. Reduces time spent in vt(4) by ~ 500 ms on the benchmark system.
• 2018-08-26 (r338321, by cperciva): Disable atkbd0 and atkdbc0 in EC2 AMIs. This speeds up the boot process by ~ 2500 ms in EC2.
• 2021-06-21 (524260db7683, by cperciva): Tell gptboot to skip its 3 second wait. This speeds up the boot process by 3000 ms in EC2.
• 2021-09-08 (a8b89dff6ac0, by cperciva): Disable acpi_timer_test by default. Reduces time spent in DEVICE_PROBE acpi_timer by ~ 140 ms on the benchmark system.
• 2021-09-08 (b4cb3fe0e39a, by tsoome): Keep root file system open to preserve bcache segment between file accesses. Reduces time spent in loader by ~ 1600 ms on the benchmark system.
• 2021-09-16 (b43d7aa09b3c, by cperciva): Switch from BIOS to UEFI boot by default when creating AMIs. This speeds up the boot process by ~ 5000 ms in EC2.
• 2021-09-28 (7457840230c5, by cperciva): Set twiddle globaldiv to 16 by default. This speeds up the boot process by ~ 50 ms in EC2.
• 2021-09-30 (ce73f768b764, by cperciva): Don't free bcache for DEVT_DISK devs. This speeds up the boot process by ~ 40 ms in EC2.
• 2021-10-01 (several patches, by imp): Remove unnecessary DELAYs, a pause, and an extra reset in the nvme driver, and speed up polling. This speeds up the boot process by ~ 330 ms in EC2.
• 2021-10-03 (04b9b7c507c5, by cperciva): Track unconsumed readahead in loader. This speeds up the boot process by ~ 120 ms in EC2.
• 2021-10-03 (248682a58915, by cperciva): Allow readahead up to 256 kB I/Os in loader. This speeds up the boot process by ~ 80 ms in EC2.