Differences

This shows you the differences between two versions of the page.

--- boottime [2022/01/12 20:46] – boot time -= 1997 ms ColinPercival
+++ boottime [2023/07/25 13:34] (current) – Consistent with <https://github.com/cperciva/freebsd-boot-profiling/pull/3> for USAGE. GrahamPerrin
@@ Line 1: / Line 1: @@
-===== Introduction =====
+===== Boot Performance =====
 ColinPercival is coordinating an effort to speed up the %%FreeBSD%% boot process and would like to see %%FreeBSD%% become far more competitive in 14.0. Some improvements have already been made since 13.0.
@@ Line 9: / Line 9: @@
 Boot time consists of four consecutive stages:
-  - System: BIOS/UEFI, before running any %%FreeBSD%% code -- we can't do anything about this.
+  - System: BIOS/UEFI, before running any %%FreeBSD%% code – we can't do anything about this.
   - %%FreeBSD%%: boot loader.
   - %%FreeBSD%%: kernel, before init starts running.
@@ Line 15: / Line 15: @@
 ===== Boot Profiling =====
-The boot loader and kernel can be profiled using the TSLOG framework to generate a flamechart.  To do this:
-  - Build a kernel with 'options TSLOG'.
+For visual analysis of the four stages: use [[https://man.freebsd.org/cgi/man.cgi?query=tslog&sektion=4&manpath=freebsd-release|tslog(4)]] buffer dumps to generate flame graphs.
-  - Check out the [[https://github.com/cperciva/freebsd-boot-profiling|freebsd-boot-profiling repository]]
-  - Run ''%%sh mkflame.sh > tslog.svg%%''
+For example, run the following command to extend the GENERIC kernel configuration file for %%FreeBSD%%/amd64:
+''%%printf '%s\n' '' 'options         TSLOG                   # <https://wiki.freebsd.org/BootTime#tslog>' >> /usr/src/sys/amd64/conf/GENERIC%%''
+– then build, install and use the GENERIC kernel.
+Checkout the [[https://github.com/cperciva/freebsd-boot-profiling|freebsd-boot-profiling repository]], ''%%cd%%'' to the checkout, then run:
+''%%sh mkflame.sh > tslog.svg%%''
 ===== Current Performance =====
@@ Line 37: / Line 44: @@
 ==== How it's going ====
-{{boottime/tslog-14.0.svg?linkonly|{{boottime/tslog-14.0.svg?0x150 Flamegraph of FreeBSD 14.0-CURRENT boot boot time (9.401 seconds)}}}}
+{{boottime/tslog-14.0.svg?linkonly|{{boottime/tslog-14.0.svg?0x150 Flamegraph of FreeBSD 14.0-CURRENT boot boot time (8.090 seconds)}}}}
-(As of early December 2021.)
+(As of late March 2022.)
 ===== Known Performance Issues =====
-Loader (~260 ms):
+Loader (~260 ms, more on first boot):
   * Reading the kernel from disk takes ~150 ms. This would be faster with a smaller and/or a compressed kernel, but tooling work would be needed for kernel updates.  This sometimes takes longer the first time an instance boots, possibly due to the underlying disk being "lazily" initialized from the backing EC2 snapshot.
+    * The first time an EC2 instance boots, this I/O takes longer – about 500 ms is spent on loader I/O when the "disk cache" is cold.
   * The loader spends ~45 ms printing status information to the console.
   * The loader spends ~65 ms on other stuff (reading in the lua scripts, loading fonts, etc).
@@ Line 55: / Line 63: @@
   * The kernel spends ~60 ms on other stuff.
-Userland (~ 3500 ms):
+Userland (~ 3500 ms, more on first boot):
   * /etc/rc.d/netif takes ~1300 ms, including a 1-second sleep(1).
+    * The first time an EC2 instance boots, dhclient takes ~2200 ms for unknown reasons.
   * /etc/rc.d/rtsold takes ~1050 ms, including a 1-second sleep(1).
   * /etc/rc.d/devd takes ~240 ms, mostly running many devmatches.
@@ Line 93: / Line 102: @@
   * 2021-11-25 ([[https://cgit.freebsd.org/src/commit/?id=e29711da2352dae50c575ab884399a6147e9444d|e29711da2352]], by cperciva): Turn off random delay before IPv6 Router Solicitation by default.  This speeds up the boot process with IPv6 enabled by an average of 500 ms.
   * 2021-11-25 ([[https://cgit.freebsd.org/src/commit/?id=81075203a057c9ba3467a7f90ea1e9469d7f7723|81075203a057]], by cperciva): Turn off IPv6 Duplicate Address Detection in EC2.  This speeds up the boot process by 2000 ms in EC2.
+  * 2021-12-20 ([[https://cgit.freebsd.org/src/commit/?id=c1381f07f61a66979f1569995f37f2a0413c0413|c1381f07f61a]], by andrew): Skip unnecessary arm64 cache invalidation when caches are coherent.  This speeds up the boot process on amd64 EC2 instances (but not the reference c5.xlarge platform) by 300 ms.
   * 2021-12-28 ([[https://cgit.freebsd.org/src/commit/?id=33812d60b9601d7ef8a7222c063696bfd6a77f41|33812d60b960]], by cperciva): Check for root device before waiting in vfs_mountroot.  This has no effect on the boot process on EC2 c5.xlarge instances but saves 100 ms on some other systems.
   * 2022-01-12 ([[https://cgit.freebsd.org/src/commit/?id=c2705ceaeb09d8579661097fd358ffb5defb5624|c2705ceaeb09]], by cperciva): Calibrate clock frequencies statistically.  This speeds up the boot process by about 1997 ms on x86 systems.
+  * 2022-01-14 ([[https://cgit.freebsd.org/src/commit/?id=4a432614f68cf35879dbb4ebef089f5b8db95334|4a432614f68c]], by cperciva): Use 0x40000010 CPUID leaf for all VM types.  This speeds up the boot process by 100 ms in EC2.
 ----