====== Debug with CI.FreeBSD.org ======

===== Mail =====

**TL;DR:**
  * "Failure" and "Unstable" - please check your changes (or other's change in the same run).
  * "Still *" - your change has not been tested, please watch the future tests.

==== *-build jobs ====

<code>
"Failure" - the build is suspected being broken by the following changes
"Still Failing" - the build has not been fixed by the following changes and
                  this is a notification to note that these changes have
                  not been fully tested by the CI system
</code>

==== *-test jobs ====

<code>
"Unstable" - some tests are suspected being broken by the following changes
"Still Unstable" - the failing test cases have not been fixed by these
               	following changes and this is a notification to note that
               	these changes have not been fully tested by the CI system
</code>

===== Reproduce =====

Get the commit hash in the parentheses of the first line of the notification mail.

<code>
FreeBSD-main-amd64-test - Build #21603 (becd9908beb8f1b47ddc6628cb005185a26ec85c) - Still Unstable
</code>

And from the body to see the failing case:

<code>
The failed test cases:

1 tests failed.
FAILED:  sys.netpfil.pf.killstate.multilabel
</code>

Get the VM image (replace the commit hash in the URL):
  * https://artifact.ci.freebsd.org/snapshot/head/becd9908beb8f1b47ddc6628cb005185a26ec85c/amd64/amd64/disk-test.img.zst

Or use base.txz, kernel.txz, etc. to install by yourself
  * https://artifact.ci.freebsd.org/snapshot/head/becd9908beb8f1b47ddc6628cb005185a26ec85c/amd64/amd64

Run tests manually (replace commit hash in the URL):

**note**: you may need to comment out ''%%test_suites.FreeBSD.ci = 'true'%%'' in ''%%/etc/kyua/kyua.conf%%''

<code>
sh /usr/share/examples/bhyve/vmrun.sh -m 4096m -c 2 -d disk-test.img test
sysctl -w kern.corefile=/tmp/%N.core (will be added to /etc/sysctl.conf)
cd /usr/tests/lib/libarchive
kyua debug functional_test:test_write_filter_zstd
</code>

===== Debugging hints =====

An assorted collection of potentially helpful hints for debugging failing test cases.

  * Most tests can be executed outside kyua: ''%%cd /usr/tests/sys/netpfil/pf && sudo ./names names%%'' atf-sh(1) tests will issue a warning about this, but will execute as expected.

  * atf-sh(1) tests are shell scripts, so adding ''%%set -x%%'' to the test body can be helpful.

  * atf-sh(1) tests typically fail at an ''%%atf_test%%'' call, which will be logged as 'Executing command [ ... ]'. There may be multiple ''%%atf_test%%'' calls in a single test, so note carefully which one is actually failing. 

===== Console Output =====

Some useful tips when reading console output

  * On a failed (not completed) test run, search for ''%%panic:%%'' (include ':') for the panic point
  * For -test job, search for ''%%+ fetch https://artifact%%'' for the test vm disk image download URL.