Debug with CI.FreeBSD.org

TL;DR:

• “Failure” and “Unstable” - please check your changes (or other's change in the same run).
• “Still *” - your change has not been tested, please watch the future tests.

"Failure" - the build is suspected being broken by the following changes
"Still Failing" - the build has not been fixed by the following changes and
                  this is a notification to note that these changes have
                  not been fully tested by the CI system

"Unstable" - some tests are suspected being broken by the following changes
"Still Unstable" - the failing test cases have not been fixed by these
               	following changes and this is a notification to note that
               	these changes have not been fully tested by the CI system

Get the commit hash in the parentheses of the first line of the notification mail.

FreeBSD-main-amd64-test - Build #21603 (becd9908beb8f1b47ddc6628cb005185a26ec85c) - Still Unstable

And from the body to see the failing case:

The failed test cases:

1 tests failed.
FAILED:  sys.netpfil.pf.killstate.multilabel

Get the VM image (replace the commit hash in the URL):

• https://artifact.ci.freebsd.org/snapshot/head/becd9908beb8f1b47ddc6628cb005185a26ec85c/amd64/amd64/disk-test.img.zst

Or use base.txz, kernel.txz, etc. to install by yourself

• https://artifact.ci.freebsd.org/snapshot/head/becd9908beb8f1b47ddc6628cb005185a26ec85c/amd64/amd64

Run tests manually (replace commit hash in the URL):

note: you may need to comment out test_suites.FreeBSD.ci = 'true' in /etc/kyua/kyua.conf

sh /usr/share/examples/bhyve/vmrun.sh -m 4096m -c 2 -d disk-test.img test
sysctl -w kern.corefile=/tmp/%N.core (will be added to /etc/sysctl.conf)
cd /usr/tests/lib/libarchive
kyua debug functional_test:test_write_filter_zstd

An assorted collection of potentially helpful hints for debugging failing test cases.

• Most tests can be executed outside kyua: cd /usr/tests/sys/netpfil/pf && sudo ./names names atf-sh(1) tests will issue a warning about this, but will execute as expected.

• atf-sh(1) tests are shell scripts, so adding set -x to the test body can be helpful.

• atf-sh(1) tests typically fail at an atf_test call, which will be logged as 'Executing command [ … ]'. There may be multiple atf_test calls in a single test, so note carefully which one is actually failing.

Some useful tips when reading console output

• On a failed (not completed) test run, search for panic: (include ':') for the panic point
• For -test job, search for + fetch https://artifact for the test vm disk image download URL.