• Student: astarasikov@FreeBSD.org
• Mentor: gavin@freebsd.org

Here I will describe the progress of porting FreeBSD to run on top of Android Emulator. Android Emulator runs on a lot of platforms and is easy to set up. It can be therefore used as a unified platform for hacking FreeBSD on ARM.

What's Android Emulator?

It's a fork of Qemu emulator with a new emulated SoC and a board called Goldfish. A somewhat nicer UI is also included.

I plan to look into how linux drivers for the Goldfish board are implemented and write the corresponding analogues for the FreeBSD kernel.

• 19 May: Started coding, pushed a prototype of a kernel that boots and outputs to UART
• 23 June: Mid-term submission
• 11 August: 'Pencils down' date
• 18 August: hard deadline

PDEV is a virtual device available in the Android Emulator. Writing and reading its MMIO registers allows to get the list of available emulated devices such as framebuffer, NAND, MMC along with their MMIO address ranges and IRQ numbers. I am currenly not sure how one patches OpenFirmware Tree at runtime, so all the PDEV does is printing the list of devices to dmesg. On the other hand, it is unlikely that the MMIO ranges or IRQs change since they are hardcoded in the Emulator source code.

It turned out that the Emulator boots from an emulated NAND chip and MMC is only used for user data, so writing NAND driver is more important now if we want to build distributable images which can be run without additional setup.

Framebuffer driver should work, I have verified that it initializes and the screen turns white because the buffer is set with 0xff at the end of driver initialization. I was unable to test the virtual console (SYSCONS) though - if we put framebuffer on the FDT device buss, it will call “sc_attach_unit” early and this will result in a null pointer dereference inside SYSCONS code since no virtual console is yet installed. If we put it onto “sysbus” instead, the driver will get probed later, probably after rootfs is mounted. Since I do not have a working MTD driver yet, I have not tested it.

Overall, the project turned out to be easier than planned since Android Emulator's Goldfish board is very similiar to QEMU's VERSATILE which is supproted by FreeBSD, although as I predicted, more time was spent debugging various initialization issues rather than writing code.

I believe it would be good to cleanup cruft from such drivers such as SYSCONS (which leads to font output being copy-pasted into every framebuffer driver and many drivers consisting 70% of NO-OP function stubs) and identify other areas where it would be useful to provide a higher level of abstraction for SoC drivers. Therefore, if I have a booting userland on emulator by the end of first week of July, I plan to port FreeBSD onto a real phone, a Google Nexus 5. I believe certain areas which have been recently improved in Linux such as clock management, runtime PM, complex display ↔ GPU mappings, may need more attention in the FreeBSD kernel.

The code is based off the 'HEAD' version of FreeBSD cloned around the start of June. You can check it out via

svn co https://socsvn.freebsd.org/socsvn/soc2014/astarasikov

http://benno.id.au/blog/2007/11/29/android-qemu - Description of the emulated board (Goldfish). Oh, and the up-to-date emulator does not contain the mentioned bugs :)

https://android.googlesource.com/platform/external/qemu.git/+/master/docs/GOLDFISH-VIRTUAL-HARDWARE.TXT - documentation for the emulated hardware from Android Emulator

Android Emulator supports ARMv5 (ARM926E) and ARM1136 cores, but not the coprocessor register instruction for domain control for these SoCs. Since we want to run on a vanilla emulator, let's just rely on the fact that the emulator also supports emulating ARM Cortex-A8 and only build FreeBSD for ARMv7

The latest version of Android Emulator is identical to QEMU in that it loads the kernel binary to 0x10000 (yes, 64K, not 1M) and jumps there. So let's just do what the versatilepb board does - link FreeBSD to 0x100000 (1M) and set up a trampoline. It should theoretically be possible to save some memory by linking to 0x10000 but I assume the kernel initially maps memory by 1M sections and that will fail unless additional patching is done.

Oh well, need to update with a script to build from an external directory

#!/usr/local/bin/bash
export TARGET=arm
export TARGET_ARCH=armv6
export DESTDIR=/root/handhelds/arm
export KERNCONF=GOLDFISH
pushd .
cd /usr/src/
make -j10 buildkernel KERNCONF=GOLDFISH
popd

Apprarently the same way as Versatile PB. Though, maybe this should be done by an LD script in kernel instead.

#!/bin/sh
KERN=/usr/obj/arm.armv6/usr/src/sys/GOLDFISH/kernel.bin
rm ./header
rm ./flash
/usr/bin/printf "\0\0\240\343" > header
/usr/bin/printf "\0\020\240\343" >> header
/usr/bin/printf "\0\040\240\343" >> header
/usr/bin/printf "\0\060\240\343" >> header
/usr/bin/printf "\001\366\240\343" >> header
dd of=flash bs=1M count=4 if=/dev/zero
dd of=flash bs=1 conv=notrunc if=header
dd of=flash bs=64k oseek=15 conv=notrunc if=$KERN

Create an Android Virtual device (install the SDK, run 'android'). Be sure to select an ARMv7A target (basically, installing an SDK for Android 4.X will make ARMv7A the default target). Name your AVD (Android Virtual Device) 'FreeBSD' (though I assume it will work even if you call it 'Windows 8')

Launch it from console specifying the avd name and the kernel file name ('flash' in this case)

emulator -shell-serial stdio -show-kernel -shell -avd FreeBSD  -kernel flash -qemu

In this section I describe the approach I used for debugging the initialization process of the FreeBSD kernel in Android Emulator (modified QEMU) on ARM.

I was using Mac OS X 10.9.4 as a Host OS, and I was using FreeBSD 10.0 in a VirtualBox VM for cross-compiling the kernel. I have configured the VM to bridge the virtual network adapter with my WiFi connection. My workflow was the following: I logged into the VM by SSH, launched the compilation, and then pulled the compiled kernel and debugging symbols to a directory in a Host OS. Then, I launched the Android Emulator in the Host OS.

In the subsequent sections, the following conventions are used: VM IP Address → 192.168.1.99 Working directory in the Host OS → /Users/alexander/Documents/handhelds/bsd/ VM RootFS mounted to Host OS → /Users/alexander/Documents/handhelds/bsd/mnt/

I have chosen the following approach to debugging: use QEMU's GDB server and a GDB from a cross-compiler toolchain. I confess I was a bit lazy and decided not to build GDB in FreeBSD, but used the Linaro toolchain (on both Linux and Mac OS X).

I also mounted the FreeBSD root file system from the VM via the sshfs. The primary reason to do so was to allow GDB to find sources (note that we need to use substitute-path to convert absolute paths to our local mount path). Besided, it allows to use a GUI text editor which in the Host OS, which may be considered useful by some people.

I used the following script (which I named fbsd.gdb) for the GDB and I invoked GDB as arm-none-eabi-gdb -x fbsd.gdb.

sshfs 192.168.1.99:/ mnt/

target remote tcp::1234
add-symbol-file ./kernel.debug 0xc0100000
set substitute-path / /Users/alexander/Documents/handhelds/bsd/mnt/

scp 192.168.1.99:/home/alexander/handhelds/flash . || exit -1
scp 192.168.1.99:/usr/obj/arm.arm/root/handhelds/bsd_head/astarasikov/head/sys/GOLDFISH/kernel.debug . || exit -1
killall -9 emulator || true
emulator -shell-serial stdio -show-kernel -shell -avd FreeBSD  -kernel flash -qemu -s -S

One problem that I've faced is no matter how many options I enable (DDB, KDB, GDB, VERBOSE_SYSINIT), all loaded modules and init functions were printed as hex addresses instead of symbol name strings.

JFYI, I have pushed the relevant changes to https://github.com/astarasikov/freebsd/tree/android_goldfish_arm_master

Of course, it would be possible to use GDB to symbolize each address, but that would be boring. So, I added a couple hacks to the kernel:

In kern/kern_module.c at function module_register_init, I have added a debugging print to echo the module name.

printf(" module name='%s' ", data->name);

In kern/init_main.c, I have replaced the DB_STGY_PROC with DB_STGY_ANY to make the kernel try to resolve the function name within all symbols.

In case the symbol is not resolved, I used the function db_printsym. Unlike symbol_name (which used db_search_symbol), this function performs a fuzzy lookup, so if the function pointer is not exactly at its entry point, it will still print the function name and the offset unlike db_search_symbol which will fail and result in printing the HEX address as is.

--- kern/init_main.c	(revision 270039)
+++ kern/init_main.c	(working copy)
@@ -263,14 +263,24 @@
 			const char *func, *data;
 
 			func = symbol_name((vm_offset_t)(*sipp)->func,
-			    DB_STGY_PROC);
+			    DB_STGY_ANY);
 			data = symbol_name((vm_offset_t)(*sipp)->udata,
 			    DB_STGY_ANY);
+
 			if (func != NULL && data != NULL)
 				printf("   %s(&%s)... ", func, data);
 			else if (func != NULL)
 				printf("   %s(%p)... ", func, (*sipp)->udata);
 			else
+			{
+			    printf("   ");
+				db_printsym((vm_offset_t)(*sipp)->func, DB_STGY_ANY);
+				printf("(&");
+				db_printsym((vm_offset_t)(*sipp)->udata, DB_STGY_ANY);
+				printf(")... ");
+			}
+
+			if (0)
 #endif
 				printf("   %p(%p)... ", (*sipp)->func,
 				    (*sipp)->udata);

So, hex address like 0xc010a27c(0) becomes something like “some_cool_function+0x20”

It is known that Android Emulator is not always based on the most recent version of Qemu. Knowing that VERSATILEPB board is supported by FreeBSD and works in upstream Qemu, I decided to add basic Goldfish board support to it and see what changes. I have pushed the branch named v1.7.0_with_goldfish to github.

https://github.com/astarasikov/qemu/tree/v1.7.0_with_goldfish

I have observed some difference in behaviour. Android Emulator crashes with vm_fault before launching init, while QEMU launches init (PID 1) and sends signals 12 (Invalid Syscall) and 11 (SIGSEGV) to it repeatedly.

I was not able to figure out why rootfs is not getting mounted. It looks like there are multiple reasons. For example, if one runs Android Emulator on a linux host, it may fail with invalid access to “0xffff000c”, while OS X version may die silently.

Just while writing this page I figured out that VERSATILEPB images I build do not boot as well. That may indicate that the cross-compiler is broken or something like that. Maybe a simple rebuild with correct options will just fix all issues?