kernelbugtriage

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision		 Previous revision
kernelbugtriage [2008/01/24 16:07] – Initial musings. RobertWatsonkernelbugtriage [2022/07/13 01:23] (current) – Add Categories: NeedsContent, Stale. Remove Author in intro, we have wiki history KubilayKocak
Line 1: Line 1:
 ====== Kernel Bug Report Triaging Guide ====== ====== Kernel Bug Report Triaging Guide ======
-Author: RobertWatson 
  
 Kernel bug reports fall into several classes: Kernel bug reports fall into several classes:
Line 9: Line 8:
   * Hang   * Hang
   * Broken functionality   * Broken functionality
 +  * Performance regression/problem
   * Feature request   * Feature request
  
 ===== Not a bug ===== ===== Not a bug =====
  
-Sometimes kernel bug reports come from a user or documentation failure -- perhaps the feature doesn't in fact work the way the user thinks, perhaps because they documentation is wrong, or perhaps because the user is confused.  It makes sense, when looking at a kernel bug report, to keep an open mind about what the source of the problem is, and confirm that the feature is intended to work the way the user expects.+Sometimes kernel bug reports come from a user or documentation failure -- perhaps the feature doesn't in fact work the way the user thinks, perhaps because the documentation is wrong, or perhaps because the user is confused.  It makes sense, when looking at a kernel bug report, to keep an open mind about what the source of the problem is, and confirm that the feature is intended to work the way the user expects.
  
 ===== Kernel panic ===== ===== Kernel panic =====
Line 30: Line 30:
 When diagnosing a panic, it is useful to have: When diagnosing a panic, it is useful to have:
  
 +  * Machine architecture and general configuration
   * General information about what the machine is being used for, and what lead to the panic (if known)   * General information about what the machine is being used for, and what lead to the panic (if known)
   * The panic message   * The panic message
Line 42: Line 43:
  
 As with kernel panics, these bugs are reported as a result of a developer anticipating a possible problem and generating debugging output that does not halt the machine (unlike a panic) when that situation occurs.  A common source of non-panicking diagnostics is WITNESS, which detects possible deadlock situations and outputs to the console.  Sometimes diagnostic warnings can be converted to panics or a drop into the debugger by reconfiguring the source of the warnings, which while more disruptive for the user, can lead to more debugging output.  For example, WITNESS can be caused to drop into the debugger by setting the debug.witness.kdb sysctl to 1, and then used to gather stack and lock trace information before either panicking or continuing the kernel. As with kernel panics, these bugs are reported as a result of a developer anticipating a possible problem and generating debugging output that does not halt the machine (unlike a panic) when that situation occurs.  A common source of non-panicking diagnostics is WITNESS, which detects possible deadlock situations and outputs to the console.  Sometimes diagnostic warnings can be converted to panics or a drop into the debugger by reconfiguring the source of the warnings, which while more disruptive for the user, can lead to more debugging output.  For example, WITNESS can be caused to drop into the debugger by setting the debug.witness.kdb sysctl to 1, and then used to gather stack and lock trace information before either panicking or continuing the kernel.
 +
 +Anyone running -CURRENT is expected to be able to identify and be able to deal with diagnostic warnings, but we shouldn't necesarily expect this from users of -STABLE branches.
  
 ===== Hang ===== ===== Hang =====
Line 56: Line 59:
 ===== Broken functionality ===== ===== Broken functionality =====
  
-This is a broad class of failures where the system doesn't behave as documented -- perhaps an unexpected error, a feature doesn't work, etc.  There are no general rules here, except perhaps initially to step through the user's configuration and make sure it's not a user error.  The good news is that often it's easy to identify the responsible subsystem, or at least a subsystem involved in the problem, so find a developer who might be able to help.+This is a broad class of failures where the system doesn't behave as documented -- perhaps an unexpected error, a feature doesn't work, etc.  There are no general rules here, except perhaps initially to step through the user's configuration and make sure it's not a user error.  The good news is that often it's easy to identify the responsible subsystem, or at least a subsystem involved in the problem, and then find a developer who might be able to help. 
 + 
 +===== Performance regression/problem ===== 
 + 
 +This class of reports has to do with something either performing worse over time (%%MySQL%% got twice as many transactions/sec yesterday than today), or something that performs worse than expected (Linux gets twice as many transactions/sec).  These reports can be hard to deal with because understanding performance problems is very difficult.  There are a few questions you should ask yourself (and possibly also the bug reporter) before beginning work on a performance problem report: 
 + 
 +  * If a comparison between two systems is being made, is it an apples-to-apples comparison? 
 + 
 +Are you comparing %%FreeBSD%% 6 and %%FreeBSD%% 7 on the same hardware, or different hardware? 
 + 
 +Are you using the same version of %%MySQL%% on Linux and %%FreeBSD%%?  If so, are you using the same database storage format? 
 + 
 +Could you downgrade your version of Firefox on %%FreeBSD%% 7 to match what's on %%FreeBSD%% 6 and see if behaves the same way? 
 + 
 +  * Can we narrow down what has "changed" between "before" and "after" in a before/after scenario? 
 + 
 +Sometimes obvious changes in a configuration will have no effect on performance, and sometimes very subtle variables will have a big impact.  For example, they upgraded %%MySQL%% versions and also replaced a failed RAID controller with a seemingly-identical one from the same vendor.  Could they try downgrading the %%MySQL%% version and see if they still see the performance drop? 
 + 
 +With a bit of Q&A, you might discover that they changed the controller and that it has a different firmware minor revision that fixed a bug that lead to improved performance but greater risk of data loss... 
 + 
 +On a similar note, people often upgrade their OS and applications at the same time, and then blame either the OS or the applications for a performance loss.  Perhaps it was the other one... 
 + 
 +  * Can the performance change be captured in a simple test scenario without reproducing the entire setup? 
 + 
 +Often times performance problems are experienced in real-world installs involving complex configuration or confidential business data.  If the reporter can reproduce the problem using a more simple benchmark setup and help us reproduce it the same way, we're much more likely to be able to help them with performance.  The %%FreeBSD%% Project has centralized testing resources that can then be used to analyze and optimize performance using the benchmark, which is a very effective way to make progress. 
 + 
 +  * Is the benchmark a bad or misapplied benchmark? 
 + 
 +Often we receive reports of performance problems using a microbenchmark tool that don't actually correspond to real-world performance problems.  A few classic examples of this to illustrate the point: 
 + 
 +  - bsdtar reading data from a hard disk and writing to /dev/null more slowly than gnutar.  It turns out gnutar optimizes writing tarballs to /dev/null by not calling the write() system call on the data, undoubtly to help gnutar developers measure the internal performance of gnutar rather than the performance of /dev/null.  As a result, bsdtar "benchmarks" much more slowly than gnutar, but might well perform fine in practice with a real workload -- for example, writing to a tarball in a file system. 
 + 
 +  - There is significant variation in the performance of a disk based on where on the disk you are writing.  The outside of a disk spins at a higher speed, and more data can be fit on it contigously, so it performs better.  If someone installs two versions of %%FreeBSD%% on the same hard disk using two different disk partitions, one will inevitably be closer to the edge of the disk than the other.  If swap or a file system is part of the performance measurement, than the version closer to the edge may perform better because the hardware it's running on is simply faster, even though it appears to the end-user as though there's no configuration difference. 
 + 
 +  * Is the user pointing the finger in the wrong place? 
 + 
 +Some more examples: 
 + 
 +  - The user says "the network stack is too slow!" but really they mean "%%FreeBSD%%'s if_em driver is negotiating 100mbps instead of 1000mbps to my switch". 
 + 
 +  - The user says "the file system is too slow!" but really they mean "%%FreeBSD%%'s ATA driver is negotiating a slower UDMA mode than Linux". 
 + 
 +  - The user says "ftp is really slow!" but the actually mean "The resolver library isn't handling a case right, so name lookups take one extra timeout".
  
 ===== Feature request ===== ===== Feature request =====
  
 Often reports of broken functionality are actually feature requests, whether the reporter means that or not.  No general rules here, except that our bug reporting system isn't a great place to put feature requests as they tend to get forgotten about.  Usually the way to get a feature implemented is to snag a developer who does want to implement it or implement it yourself.  There are various strategies to do the former.  "Submit patches" is an unhelpful reply that may be taken poorly, so a more polite reply is suggested. Often reports of broken functionality are actually feature requests, whether the reporter means that or not.  No general rules here, except that our bug reporting system isn't a great place to put feature requests as they tend to get forgotten about.  Usually the way to get a feature implemented is to snag a developer who does want to implement it or implement it yourself.  There are various strategies to do the former.  "Submit patches" is an unhelpful reply that may be taken poorly, so a more polite reply is suggested.
 +
 +(We should probably have a PR classification for 'feature request').
 +
 +===== References =====
 +
 +[[https://docs.freebsd.org/en/books/developers-handbook/kerneldebug/|FreeBSD Developer Handbook: Kernel Debugging Guide]]
 +
 +----
 +
 +{{tag>HowTo}}
 +{{tag>Stale}}
 +{{tag>NeedsContent}}
  
  • kernelbugtriage.1201190852.txt.gz
  • Last modified: 2008/01/24 16:07
  • by RobertWatson