This is an old revision of the document!

The following highly informal text is a readme file written for co-workers to help them maintain a custom low-level bootloader we use for Atmel ARM chips. It started as a list of hard-coded memory addresses and evolved into an informal description of the boot process. It was written in the context of FreeBSD 8.2, and a few details may be out of date for -current, but it's mostly the same.

I'm initially posting this as-is, but fully intended to clean it up. Really, that's my intention.

This document describes where things live in memory during various parts of the boot process, in case the layout ever needs to be changed (like when implementing a system with different memory sizes/ranges). This is all written in regard the Atmel at91rm9200 SoC, and should apply equally well to other Atmel at91 series. The boot2 bootloader is pretty much at91-specific, but the kernel init sequences described after the table of kernel flavors are the same for all ARM chips.

The boot2 binary is limited to about 12K in size because the Atmel rom code which loads that file to the lower portion of the 16K internal sram uses the upper portition of sram as stack and buffer space. The rm9200 manual says the rom code uses the upper 4K. Only the rom code has this 4K stack limitation, our boot2 uses a stack in sdram, and can use large amounts of stack or bss space.

The boot process is:

Atmel chip loads the boot2 code from either spi flash or i2c eeprom or the serial port into the 16K on-chip sram at physical address 0x00000000.

boot2's linker config script places the bss segment at an arbitrary address located in sdram, and the boot2 code must be careful not to access any variable in bss until after the sdram setup code runs.

The boot2 entry code in sys/boot/arm/libat91/arm_init.S sets the stack pointer to the top of sram, then calls _init() in libat91/at91rm9200_lowlevel.c.

The code in _init() sets up the processor clocks and sdram controller. After sdram is configured it zeroes out boot2's bss data area. It also initializes the dbgu serial port, so printf in boot2 works after _init().

When _init() returns to arm_init.S, the stack is relocated into sdram at the address specified in the SVC_STACK_USE constant near the top of the source file. It then calls main() in boot2.c.

At this point the boot2 code is running in a memory environment somewhat similar to normal: the stack will grow downwards towards bss and the stack size is the difference between the stack pointer and the end of bss (at this writing it's about 4MB of stack space).

The boot2 code loads the kernel from some storage or network source into physical ram. It uses the values in the elf headers of the file as the load address and entry point address. Because a FreeBSD kernel is typically linked to run at virtual address 0xC0000000 the loader masks out the high order part of the addresses and plugs in the high order part of the address where ram is physically located. For sdram on the rm9200 that means changing 0xCxxxxxxx to 0x2xxxxxxx. The virtual and physical addresses for the kernel linking are found in src/sys/arm/at91/std.tsc4370.

Once the kernel is loaded into ram, boot2 jumps to the entry point address specified in the elf header to launch the kernel (after again translating virtual to physical address). It passes four parameters (in registers r0-r3): the TSC_BOOTINFO magic number, a pointer to a tsc_bootinfo structure, and two unused parms are passed as zero.

The entry point on the kernel side depends on which flavor of kernel is being used. The available choices would make Baskin-Robins proud:

The .bin versions have the elf headers stripped off, and a bootloader would launch such a kernel by jumping to offset zero (the load address).

For a kernel without trampoline code, the entry point is in locore.S. This is described below.

The trampoline entry point is _startC() in sys/arm/elf_trampoline.c. This is specified in sys/conf/Makefile.arm which actually writes a little asm source file named tmphack.S which jumps to the _startC() routine. Tricky.

The stack on entry to _startC() is still the one from boot2. The first thing _startC() does is move the stack to be about 1MB above the end of the loaded kernel. At this point physical addressing is still in effect; the trampoline code is linked to run at the physical address specified in std.tsc4370, not at the virtual address the kernel is linked for.

The trampoline code decompresses the kernel to the ram that immediately follows the end of the loaded kernel, then copies the uncompressed kernel to the physical address specified in the decompressed elf headers. It sets up rudimentary page tables immediately following the kernel (I think this is unecessary, the kernel will do its own mmu setup), and then sets up the initial kernel entry stack to be right after these page tables.

Custom code we've added (not in stock FreeBSD) copies the tsc_bootinfo struct to the new stack so that it will remain accessible to the kernel during the handoff. (It turns out this may be a bit dodgy – the struct passed by boot2 lives in boot's bss, so it's essentially sitting in the middle of sdram somewhere, and could have been wiped out by uncompressing the kernel.)

The trampoline jumps to the entry point listed in the kernel's elf header.

The kernel entry point listed in the elf header is the label _start in the file sys/arm/locore.S. When entered directly from a bootloader the MMU is typically disabled and the pc is the physical address of the entry point. When entered from the trampoline the MMU is enabled and the pc is the virtual address. The code in locore.S is designed to work right either way.

The entry code crafts simple startup page tables, enables the MMU (but not caches, MD code does that later), and transitions the pc from physical to virtual addresses. The page tables are placed at the location specified by STARTUP_PAGETABLE_ADDR, which comes from sys/arm/at91/std.tsc4370. These have a short lifetime, the machine-dependent initarm() routine is expected to create and install more complete page tables.

The stack pointer is set to a small (2K) stack in bss, and the initarm() function is called, passing the same values in r0-r3 that were present on entry to _start. Each type of arm Soc has its own initarm() routine which does a lot of machine-specific and board-specific setup. It must preserve any incoming parms from the bootloader. Some early parts of kernel init are called from within initarm(), including cninit() which is the point at which printf() begins to work. The initarm() routine returns a pointer to what will become the supervisor mode stack for the kernel.

When initarm() returns, the code in locore.S installs the stack pointer it returned. At this point there is no longer any dependence on magical fixed addresses in memory that were compiled-in or came from a config file.

Next the code in locore.S calls mi_startup(), which is essentially like main() for the kernel. It should not return.

Summary of tweakable/configurable magical memory locations and their values at the time this was written:

sys/boot/arm/at91/libat91/arm_init.S SVC_STACK_USE (stack) 0x23F00000

sys/boot/arm/at91/linker.cfg bss 0x23B00000

arm/at91/std.tsc4370 STARTUP_PAGETABLE_ADDR 0x23FF0000 used in locore.S