Automated Kernel Crash Reporting System

This project aims to develop an information system for automated kernel crash reports for the FreeBSD Operating System. When a FreeBSD system has a crash and responsible is a kernel panic, a crash reporter program will collect some necessary data from the crashed system and deliver them automatically in the form of a report to a Central Collector machine. The Central Collector machine will receive these reports, store them in a database, recognize which crash reports refer to previously reported bugs, and provide two web pages, one for the Community and one for the Developers of FreeBSD, each with different features.

The approach of solving the problem is the division of the project in two parts. The first part is related with any FreeBSD system in the world that may crash (Client Side part). The second part is related with a central machine that collects the crash reports from the crashed systems (Server Side part).

In this part, the main goals are to locate, collect, form, and trasfer the debugging information from the crashed FreeBSD system to the Central Collector machine which is responsible for storing the crash reports.

The introduction of three new variables in the /etc/rc.conf file is necessary. These variables give the freedom to the System Administrator to configure the behavior of the crash reporting system. These three variables are called crashreport_enable, crashreport_program and crashreport_email and their default values are specified in the /etc/defaults/rc.conf file.

*Variable name: crashreport_enable *Possible values: “YES”, “NO” *Default value: “NO” *Description: Enable or disable the automated reporting of kernel crashes

*Variable name: crashreport_program *Possible values: “/absolute/path/to/program” *Default value: “/usr/sbin/crashreport” *Description: The path of the crashreport program

*Variable name: crashreport_email *Possible values: “sysadmin@freebsd.org”, “ ” *Default value: “ ” *Description: The email of the System Administrator. It will be used only for contact by the Central Collector machine.

In order to collect the debugging information from the crashed FreeBSD system and send them to the Central Collector machine, we have to create the crash reporter program. From now on, we will call it crashreport. The crashreport program is a (Bourne) Shell script. Its default location is in /usr/sbin/crashreport. It can be invoked either automatically or manually.

The automated behavior refers to the automatic reporting of the kernel crash during the first multi-user boot after the crash. In every boot, the /etc/rc.d/savecore shell script executes the /sbin/savecore program that checks for a crash dump in the specified dumpdev device. If /sbin/savecore finds a crash dump, it extracts the crash dump to the dumpdir directory. If the type of the crash dump was a full memory dump or a minidump, the dumpdir directory will be filled from two files. The vmcore.X and info.X. If the type of the crash dump was a textdump, the dumpdir directory will be filled from a tarred file called textdump.tar.X and from a file called info.X. The tarred file textdump.tar.X contains a bunch of .txt files (up to 5) that provide valuable debugging information. Subsequently, /etc/rc.d/savecore checks the rc variable crashinfo_enable. If this variable is set to “YES”, then /etc/rc.d/savecore executes the /usr/sbin/crashinfo shell script that analyzes the extracted core dump (so it works only with full memory dumps and minidumps) and generates a text file in the dumpdir directory. This file is called core.txt.X and contains valuable debugging information. Likewise, /etc/rc.d/savecore checks the rc variable crashreport_enable. If this variable is set to “YES”, then /etc/rc.d/savecore executes the /usr/sbin/crashreport. After the execution of the usr/sbin/crashreport the booting process continues normally.

The manual behavior refers to the invocation of the /usr/sbin/crashreport explicitly from the System Administrator. This behavior is useful in the case that the System Administrator did not have enabled the variable crashreport_enable for various reasons when the system had a crash and now he/she wants to report the kernel crash. Also, it is useful for sending more debugging information when asked from a FreeBSD developer. Only the superuser will be able to execute successfully the crashreport program.

The /usr/sbin/crashreport is a Bourne Shell script that provides three options. The options are as follows:

-d dumpdir Specifiy an alternate core dump directory that contains the debugging information. The default dump directory is /var/crash.

-f filename Specify the name of the file that contains the debugging information. Normally, it is either core.txt.X or textdump.tar.X

-e email Specify the email address of the System Administrator that it will be used for contact from the Central Collector machine. Normally, the email address is specified in the variable crashreport_email of the /etc/rc.conf file.

If none of the above options is specified, crashreport is capable of finding the right information on his own. That is, it can be invoked from the superuser without any options, and it will locate, collect and send the report without any further user interaction. It is important to mention that only the Superuser (root - uid 0) will be able to execute crashreport successfully. One technical reason is that the generated files core.txt.X and textdump.tar.X have read and write access only for root. Also, it is better only the System Administrator of the machine to be able to report a kernel crash and not every individual user that may have access to the machine.

In the first version of crashreport (known as kcrashreporter), we wanted to instruct crashreport to generate the needed debugging information on his own. This is not an issue anymore because the script /usr/sbin/crashinfo does this job pretty well. Also as we mentioned above, it is invoked from the rc.d savecore script when savecore finds and extracts successfully a new core dump. So if the crash dump type is a full memory dump or a minidump, crashinfo will generate a core.txt.X file that will contain valuable debugging information. In the case of a textdump, the usr/sbin/crashinfo script does not take place but the file textdump.tar.X contains the debugging information that we need.

An essential functionality of crashreport is the method that sends the report to the Central Collector machine. It uses the SSH protocol to transfer securely the report from the client to the server. Specifically, it uses the scp tool of OpenSSH with public key authentication. The configuration options can be divided in 2 parts, one for the client and one for the server. For the server machine we did the following:

• Register a DynDNS domain (akcrs.dyndns.org) that points to the (dynamic) IP address of the server machine (currently my Desktop computer). This domain will be used from the scp tool as the hostname field and as the portal to the Community and Developers web pages (akcrs.dyndns.org). *Create a new user (named reporter). This user will be used from the scp tool to transfer the report and store it in his home directory. *As the user reporter, we generate with ssh-keygen a pair of keys using the RSA algorithm. This is the user pair of keys. The public key is placed in the file authorized_keys of his .ssh directory (~/.ssh). The private key is distributed with crashreport. *We check the public key of the /etc/ssh/ssh_host_ecdsa_key.pub and distribute it with crashreport in order to register the fingerprint as known hosts in the Client side.

For the client, everything about the method for sending the report is integrated inside the crashreport script. That is, the host public key, the user private key and the scp command.

For a better understanding of the anatomy of crashreport, view the source code here.

In this part, the main goals are to receive the crash reports, check them for validity and security, recognize reports that refer to the same issue as previously logged reports, store them in a database, contact the submitters when needed, and provide one dynamic website with different features for the community and the developers.

The above goals define the functionalities of the Central Collector machine. The Central Collector machine will be a machine running a release version of FreeBSD with a DBMS installed for the manipulation of the database and a Web Server software in order to provide the website.

In the duration of development of the project, the Central Collector machine will be my desktop computer running FreeBSD 9.0-RELEASE with a stable ADSL connection. The role of the DBMS will be played from PostgreSQL and the Apache HTTP Server will serve the website. I hope that after the development of the project, the system will be accepted from the FreeBSD project and the actual machine will run the same software with a stable internet connection 24 hours a day and 7 days a week.

About the database, we will use a database management system and specifically the ORDBMS PostgreSQL. We choose Postgres because it is released under the PostgreSQL license, a license similar to the BSD license, it is a free and open source software, and it has high scores in various benchmarks. The schema of the database will be somewhat of the following form:

Bugs(ID, State, Votes, Notes)
Submitters(ID, Email, Password)
Reports(ID, Bug_ID, Submitters_ID, DATE, Backtrace, SystemRelatedInfo)

There are three tables, the table Reports, the table Bugs and the table Submitters. The table Reports stores all the crash reports that arrive in the system, while the table Bugs stores the unique crash reports. That is, if we have in Reports more than one report that is similar with other, then they share the same Bug_ID and there is only one tuple for this issue in Bugs. The table Submitters stores the email address along with a generated password and a unique ID for every submitter that send a report, either automatically or manually.

The primary key of Reports is the column named ID and the columns Bug_ID and Submitters_ID are foreign keys that references the columns ID of Bugs and ID of Submitters respectively. The column Date stores the date and time that the report was received. The column SystemRelatedInfo will probably span in more than one columns, each of them describing a characteristic of the panicked system that sent the report.

The primary key of Bugs is the column named ID. The column State defines the current state of the bug. This column can take only one of the following values:

*Open: This is the initial state of a bug. This state declares that there is at least one report that refers to it and the bug has not been reviewed from a FreeBSD developer yet. *Analyzed: This state declares that the bug is being reviewed from a FreeBSD developer but no solution exists yet. In this state, a FreeBSD developer can ask the submitter for further debugging information. *Feedback: This state declares that the bug has been solved and the submitter has been given a patch or other fix. The bug report remains in this state until the submitter acknowledges that the solution works. *Closed: This state declares that the bug has been solved and the submitter has confirmed the solution. *Suspended: This state declares that the bug has been analyzed from a developer and marked as impossible to be fixed any time soon for various reasons.

The bug states that described above have many similarities with the states of Problem Reports in the GNATS bug tracking system that is currently used from the FreeBSD project.

The column Votes is an integer that shows the importance of a bug based on the number of reports that arrive to the system and point to a particular bug. For example, if we have ten different reports that refer to the same bug, then the Votes column of this bug will have the value 10. The presence of the column Votes is for performance reasons, for easier manipulation of the data and logical design. Instead of counting the entries of a specific Bug_ID in the Reports table every time that we want to query the database for the most frequent bug report or something similar, we simply check the Votes column. Finally, the column Notes can be used only from the FreeBSD developers in order to inform the submitters of a particular bug. In this column, a FreeBSD developer may write valid kgdb commands that he/she wants to be executed from the submitter.

About the two Web pages, I prefer the Apache Web Server as the software that will be responsible to serve them. The two Web pages refer to two different groups. The first Web page will be accessed by everyone from everywhere and it will show the list of received reports along with some information about them. That is, the backtrace (not full backtrace in the case that exists), the Operating System and its release (for example, FreeBSD 9.0-STABLE), and the type of hardware platform of the panicked system (for example, amd64). This page will be called Community Web page. The second Web page will be accessible only by giving a valid username and password (only FreeBSD developers will have one) and when you are logged in, you have the ability to view all the data in the database and update some of them. This page will be called Developers Web page.

I would like to make a pause and mention two things about the privacy on the Community web page. Because the backtrace can reveal some sensitive information about the user that made the report, we will only make visible the function names of every function call and the file that they are located. That is, without the arguments that potentially contain private information. I think that with this way, the user can still view useful information about the crash (which function is responsible) without giving away further information.

Finally, about the handler. The handler will be a program written in PHP in conjunction with SQL that can be described as the core of the Central Collector machine. Handler will be an internal program that performs the following tasks:

*Receive the crash reports *Check the crash reports for security issues *Store the reports in the database *Match same reports *Provide a mechanism for votes *Sends emails to the submitters, either for feedback or informing them about the status of their report(s)

About the reception of the crash reports, we define the way that the handler will understand that a new crash report arrived in the system. This is related with the used protocol and method. A crash report is contained in a single ASCII file.

About the check of the reports, this action exists primarily for security reasons. Firstly, the system checks if an email address is bundled with the crash report and sends a confirmation email to the submitter to the address listed. The confirmation email will include a link that should be followed in a period of time in order to consider the report valid. If the submitter follow the link in this period, then the report becomes valid. If the email address of the submitter is not listed in the database, the system will add the email address along with a generated password and a unique id to the Submitters table. After the confirmation of the report, the system will examine the report for any malicious code and it will split the single text file in multiple parts ready to be stored in the database. Finally, the system will inform the submitter via a second email about his/her new account in the case that the system just registered it and send a link of his/her report. The method of the confirmation email will reduce the chances of storing multiple fake reports sent from an automated script.

Now that we are pretty sure that the received report is clean and valid, the handler actually stores the report in the database. In order to store the report, the system needs to define a Bug_ID for the report. About the matching of the same reports and the mechanism of votes, when a crash report arrive in the machine, the handler is responsible to recognize if there is any similar report already in the database. If yes, the handler match them under the same Bug_ID and increase the Votes field by one. Otherwise, it assigns a new Bug_ID and the value 1 in Votes.

In this point, I would like to explain in more details the process of how we will actually decide if a new bug report is the same as a previously logged bug report because it is a crucial of the system. Firstly, the whole process is based on the following assumptions:

1. Every backtrace produced from a kernel panic, contains a function call to panic().
2. Function names that contain the word trap can be found only before the call of panic().
3. Function names that contain the word syscall can be found only in the first frames of a backtrace.

The above assumptions have been derived from examining the backtraces of different kernel panics reported to the GNATS database. Driven by those assumptions and the comments of developers in every PR, we were led to the following conclusion:

• The frames that play the leading role of identifying a bug, lie between the first function that contains the word 'trap' (if it exists, otherwise panic()), and the last function that contains the word 'syscall'. (count from bottom to top)

For example, consider the following backtrace:

#0 kdb_backtrace
#1 panic
#2 rw_destroy
#3 padlock_detach
#4 device_detach
#5 devclass_driver_deleted
#6 devclass_delete_driver
#7 driver_module_handler
#8 module_unload
#9 linker_file_unload
#10 kern_kldunload
#11 amd64_syscall
#12 Xfast_syscall

From this backtrace, the important frames are from #2 to #10. The most important frame is the frame #2. The frames #0, #1 and #11, #12 are not vital in identifying the bug.

Also, consider the following backtrace:

#0 doadump
#1 boot
#2 panic
#3 trap_fatal
#4 trap
#5 calltrap
#6 rum_setup_tx_desc
#7 rum_start
#8 if_transmit
#9 ieee80211_start
#10 if_transmit
#11 ether_output_frame
#12 ether_output
#13 bpfwrite
#14 devfs_write_f
#15 dofilewrite
#16 kern_writev
#17 write
#18 syscall
#19 Xfast_syscall
#20 ??

From this backtrace, the important frames are from #6 to #17. The most important frame is the frame #6. The frames #0, #1, #2, #3, #4, #5 and #18, #19, #20 are not vital in identifying.

Based on the above things, we decided that the most significant factors for matching two reports are: *the Panic Message *the Function before the call of the panic() function or the call of a function that contains the word trap. Also, a frame in the bottom of the stack is the less significant frame, while the frame in the top of the stack is the most significant frame.

Based on all these things, we concluded that in order to say that 2 reports refer to the same bug, they should have at least the panic message and the function before a trap function or panic(), exactly the same.

In order to deal with different function names from release to release and from architecture to architecture, we have to perform a more flexible approach than comparing two strings character by character and return true if they are exactly the same, otherwise false.

For this reason, we add the notion of percentage in every factor. If two reports are exactly the same, then the matching returns a 100% similarity. For the two most significant factors, we choose that the panic message costs 25% and the backtrace costs 75%. For the backtrace, the most important frame (the top one before panic() or 'trap') costs 25% and all the others a 50%. For the remaining frames, we allocate symmetrically this 50% percent based on the place in the stack.

For every match, we calculate the similarity and if it is above 50% then these reports refer to the same bug.

For an implementation of the matching reports process that described above, check this demo that is written in PHP.

About the interaction of the system with the submitters, we can say that this will be achieved through emails using the SMTP protocol. The handler will send a confirmation email to the submitter of every report, send an email informing that the report was submitted successfully and a link to the reports web page in the Community Web page. Also, it will send an email when asked from a FreeBSD developer asking for more debugging information.

Our toolbox contains (Bourne) Shell programming for the construction of the crashreport program, the FreeBSD Operating System (a RELEASE version) for the Operating System of the Central Collector machine, the Apache Web Server as the Web Server that will serve the two web pages, the use of the ORDBMS PostgreSQL for the management of our database, the use of the PHP programming language with the SQL programming language for the construction of functionalities of the Central Collector machine.

An information system for automated and manual kernel crash reports for the FreeBSD project.

May 21 - June 03 (Week 01, 02) *Implementation of the crash reporter program (client-side)

June 04 - June 17 (Week 03, 04) *Implementation of the rc.d script (server-side)

June 18 - July 01 (Week 05, 06) *Connecting the client-side with the server-side *XML crash reports

July 02 - July 15 (Week 07, 08) *Implementation of the receive phase

July 16 - July 22 (Week 09) *Implementation of the check phase

July 23 - July 29 (Week 10) *Implementation of the store phase

July 30 - August 05 (Week 11) *Implementation of the recognize phase *Reorganization of server-side code

August 06 - August 12 (Week 12) *Construction of the dynamic website that will present the data

August 13 - August 19 (Week 13) *Updates for the website *Documentation *Testing the whole system

The recommended way to test the Client Side part is to create a virtual machine (ex. using VirtualBox) and install a FreeBSD version of your choice. Then, you can download the source in /usr/src from the socsvn repository and build the system. In the end, you will have a virtual machine running FreeBSD 10.0-CURRENT (at the moment a 2 months old revision) with this project's added code. The following is a guide that describes the steps after creating your FreeBSD virtual machine.

# Once installation is complete, obtain the Ports Collection
portsnap fetch
portsnap extract
portsnap update
 
# Install Subversion from the Ports Collection
cd /usr/ports/devel/subversion
make -DBATCH
make install
make clean
 
# Check out the project's client side code into /usr/src (your Working Copy)
svn checkout https://socsvn.freebsd.org/socsvn/soc2012/tzabal/client-side/akcrs-head/ /usr/src
 
# The following is from: http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/makeworld.html
cd /usr/src
make buildworld
make buildkernel
make installkernel
shutdown -r now
# Boot in single-user mode
mount -u /
mount -a -t ufs
adjkerntz -i
mergemaster -p
cd /usr/src
make installworld
mergemaster
reboot
 
# Periodically update your working copy
cd /usr/src
svn update
 
# In the case of minor changes, install manually
# Example of having only the usr.sbin/crashreport updated
cd usr.sbin/crashreport
make install

http://svnweb.freebsd.org/socsvn/soc2012/tzabal/

1. FreeBSD Developers' Handbook, Chapter 10 Kernel Debugging