Kernel Bug Report Triaging Guide

Author: RobertWatson

Kernel bug reports fall into several classes:

• Not a bug
• Kernel panic
• Diagnostic warnings
• Hang
• Broken functionality
• Feature request

Sometimes kernel bug reports come from a user or documentation failure – perhaps the feature doesn't in fact work the way the user thinks, perhaps because they documentation is wrong, or perhaps because the user is confused. It makes sense, when looking at a kernel bug report, to keep an open mind about what the source of the problem is, and confirm that the feature is intended to work the way the user expects.

A kernel panic occurs when it detects that a run-time invariant has failed – that is, a check has been added to the kernel source to look for unexpected situations that reflect serious bugs and the kernel has halted.

A panic report can be a very productive type of bug report, as it includes information about what a problem is, and where it was detected. Some other types of bug reports, such as hangs or broken functionality reports, can be converted to panic reports by asking the user to enable INVARIANTS or other run-time diagnostics that test for more types of errors, for example leading to a panic instead of a hang.

Common sources of panics are:

• Page fault, in which an invalid pointer is accessed by the kernel (NULL pointer, freed memory, corrupted pointer).
• Double fault, when a page fault is improperly handled
• A failed kernel assertion, typically as a result of a “if (x) panic(message)” or “KASSERT(!x, message)|.
• Memory scrubbing detects use-after-free

When diagnosing a panic, it is useful to have:

• General information about what the machine is being used for, and what lead to the panic (if known)
• The panic message
• If this is a page fault, there will be a trap frame including addresses
• Kernel stack trace of the running thread at the time of panic, ideally with source code locations
• Other useful information such as a list of threads, locks held, etc.
• Values of local variables in the stack frames around the panic

This information is often extracted using some combination of DDB, GDB, live debugging, and kernel dumps. A minimal dump report should include at least the panic message and stack trace, which may be sufficient to determine what general subsystems are involved, and possibly entirely diagnose the bug. Once it's possible to point the finger at a particular subsystem, it's easier to find a developer who might be able to help diagnose it.

As with kernel panics, these bugs are reported as a result of a developer anticipating a possible problem and generating debugging output that does not halt the machine (unlike a panic) when that situation occurs. A common source of non-panicking diagnostics is WITNESS, which detects possible deadlock situations and outputs to the console. Sometimes diagnostic warnings can be converted to panics or a drop into the debugger by reconfiguring the source of the warnings, which while more disruptive for the user, can lead to more debugging output. For example, WITNESS can be caused to drop into the debugger by setting the debug.witness.kdb sysctl to 1, and then used to gather stack and lock trace information before either panicking or continuing the kernel.

Hangs occur when the kernel gets into a state where it is unable to continue processing, but a kernel invariant has not detected the case and therefore not panicked the system. Hangs of various types are possible, and each requires different consideration:

• The completely hard hang. This might occur if the kernel gets into a loop on one CPU, and then other CPUs end up waiting for that CPU possibly with interrupts disabled. Check for this by hitting the caps lock key to see if the light turns on and off, or pinging the machine. A serial or console break won't get into the debugger, but if the box has a hardware watchdog or NMI button, that may be able to get you to the debugger.
• Deadlock. This occurs due to a cycle in locks or other locking or resource bug occurs, and while it is similar to a completely hard hang, there's a good chance of getting into the debugger using a console/serial break. Deadlocks happen when all threads that could be doing work end up waiting indefinitely for something that will never happen.
• Livelock. Like a deadlock, but rather than the system halting because no work can be done, the system is busy doing work but not making any progress. Likewise, a break will often get into the kernel.
• Other kinds of hangs that remind users of kernel hangs, but aren't. For example, if X11 locks up, the system itself may operate fine (i.e., SSH in) but no progress on the video console. It could be a kernel bug … but it could be a bug in an application. Other similar problems involve network outages, where a system appears to stop responding but actually it's the path to the system.

When the kernel hangs, the goal is to get into the kernel debugger and figure out what it's doing. As with panics, this involves stack traces, lock information, kernel thread information, etc. From this one can identify a potentially responsible subsystem and find a developer to help.

This is a broad class of failures where the system doesn't behave as documented – perhaps an unexpected error, a feature doesn't work, etc. There are no general rules here, except perhaps initially to step through the user's configuration and make sure it's not a user error. The good news is that often it's easy to identify the responsible subsystem, or at least a subsystem involved in the problem, so find a developer who might be able to help.

Often reports of broken functionality are actually feature requests, whether the reporter means that or not. No general rules here, except that our bug reporting system isn't a great place to put feature requests as they tend to get forgotten about. Usually the way to get a feature implemented is to snag a developer who does want to implement it or implement it yourself. There are various strategies to do the former. “Submit patches” is an unhelpful reply that may be taken poorly, so a more polite reply is suggested.