Differences

This shows you the differences between two versions of the page.

--- summerofcode2017:smoketestingofbaseutilities [2017/07/09 17:39] – Add progress report for week 5 ; Update differential status ShivanshRai
+++ summerofcode2017:smoketestingofbaseutilities [2021/04/26 04:41] (current) – Fix MailTo macros. JethroNederhof
@@ Line 13: / Line 13: @@
 ====== Overview ======
 Smoke testing is a set of light tests which are done for checking basic functionalities of a software to ascertain if the crucial functions work correctly.\\
-%%FreeBSD%% currently has a set of [[https://wiki.freebsd.org/TestSuite|tests]] which are run using the ''%%kyua%%'' framework. These tests need to be first installed individually before they can be used for testing. This proves to be problematic in cases when direct testing of some newly installed or updated utility has to be performed. It makes testing changes to utilities and libraries difficult as one would like to perform tests (e.g. to ensure a proper build environment) before proceeding for installation.
+%%FreeBSD%% currently has a set of [[/TestSuite|tests]] which are run using the ''%%kyua%%'' framework. These tests need to be first installed individually before they can be used for testing. This proves to be problematic in cases when direct testing of some newly installed or updated utility has to be performed. It makes testing changes to utilities and libraries difficult as one would like to perform tests (e.g. to ensure a proper build environment) before proceeding for installation.
 ====== Project Description ======
@@ Line 26: / Line 26: @@
   * Study the current problems faced as a developer or tester when writing new test cases, and the problems faced as a release engineer or tester when executing existing test cases.
-===== Test Plan =====
+\\
+====== Initially Proposed Test Plan ======
 The file [[https://github.com/shivrai/smoketestsuite/blob/master/baseutils/ls/tests/functional_test.c|functional_test.c]] is a simple test file which checks whether the ''%%ls%%'' utility is properly linked by running trivial commands. The options supported by ''%%ls%%'' are stored in ''%%short_options[]%%'' and ''%%long_options[]%%'' present in [[https://github.com/shivrai/smoketestsuite/blob/master/baseutils/ls/tests/functional_test.h|functional_test.h]].
-The smoke tests can be run using the following commands -
-<code>
->> make
->> make test
-</code>
 It should be noted that finally the tests will be automated with appropriately passed options. The above commands will not have to be run for testing individual programs.
@@ Line 48: / Line 42: @@
 An automation script will be written which will populate ''%%short_options[]%%'' and ''%%long_options[]%%'' by following the above mentioned approaches and will generate relevant test files.
 The initial version of the generated test scripts will only test trivial functionalities.\\
-| ★ More details regarding automated generation of test scripts available [[https://lists.freebsd.org/pipermail/soc-status/2017-July/001079.html|here]] (More updates coming up!). Comments and suggestions are most welcome! |
 Although the test plan does not check the entire state of application under test, it suffices for the initial purposes of smoke testing.
+\\
+<callout type="warning">
+**Revised test plan**
+The test plan for the project has been completely revamped (a brief overview available [[https://lists.freebsd.org/pipermail/soc-status/2017-July/001079.html|here]]). Comments and suggestions are most welcome!
+</callout>
+\\
 ====== Expected Results ======
-  * Set of test programs to verify basic functionality of all the base utilities in %%FreeBSD%%.
+  * An automated tooling to generate test scripts which confirm minimal set of functionalities of a utility to check whether it is properly linked.
-  * An automated tooling to generate test scripts which confirm minimal set of functionalities of a utility to check whether it is properly linked.
+  * Set of test programs generated by the tool to verify basic functionality of all the base utilities in %%FreeBSD%%.
-(To be updated)
 ====== Milestones ======
@@ Line 64: / Line 65: @@
 |  **Start of Coding**  |||  ★  |||
 | 1 | May 30 | Start of coding. | ✔ |
-|  **Add kyua based tests**  |||  ★  |||
+|  **Add atf-sh tests**  |||  ★  |||
-| 1 | May 30 - June 5 | Start adding kyua based tests to utilities without test coverage. | ✔ |
+| 1 | May 30 - June 5 | Start adding atf-sh tests to utilities without test coverage.  | ✔ |
 | 2 | June 6 - June 12 | ::: | ::: |
-| 3 | June 13 - June 19 | [[https://github.com/shivrai/smoketestsuite/compare/c75665...d70e8e|Update]] smoke testing tool. | ✔ |
+| 3 | June 13 - June 19 | [[https://github.com/shivrai/smoketestsuite/compare/c75665...d70e8e|Update]] smoke testing tool.  | ✔ |
-| ::: | ::: | Fix a bug in the implementation of ln(1). | ::: |
+| ::: | ::: | Fix a bug in the implementation of ln(1).  | ::: |
-| 4 | June 20 - June 26 | Add functionality to generate trivial tests. | ✔ |
+| 4 | June 20 - June 26 | Add functionality to generate trivial tests.  | ✔ |
 |  **First evaluations**  |||  ★  |||
+|  **Start generating testcases to demonstrate tool behavior in different cases.**  |||  ★  |||
+| 5 | June 27 - July 3 | Demonstrate negative testcases for ln(1). [[https://github.com/shivrai/smoketestsuite/compare/4969ae...1a7e26|Updates]]  | ✔ |
+| ::: | ::: | Support for utilities with no supported options.  | ::: |
+| ::: | ::: | Collation of common usage message.  | ::: |
+| 6 | July 4 - July 10 | Demonstrate the tests which the tool can generate using date(1), ln(1) and stdbuf(1). [[https://github.com/shivrai/smoketestsuite/compare/1a7e26e...2b3e2a5|Updates]]  | ✔ |
+| 7 | June 11 - July 17 | Add functionality for reading annotations. [[https://github.com/shivrai/smoketestsuite/compare/2bf499...12d88b|Updates]]  | ✔ |
+|  **Second evaluations**  |||  ★  |||
+|  **Bug fixes and code cleanup**  |||  ★  |||
+| 8 | June 18 - July 24 | Add functionality of dynamically generating annotations. [[https://github.com/shivrai/smoketestsuite/compare/a7c38c...12d88b|Updates]]  | ✔ |
+| 9 | June 25 - Aug 1 | Miscellaneous fixes. [[https://github.com/shivrai/smoketestsuite/compare/da707e...efc325|Updates]]  | ✔ |
+| 10 | Aug 2 - Aug 8 | Updates to scripts. [[https://github.com/shivrai/smoketestsuite/compare/3daa63...478d74|Updates]]  | ✔ |
+| 11 | Aug 9 - Aug 15 | ::: | ::: |
+| 12 | Aug 16 - Aug 22 | Add functionality to stop test generation during runtime. [[https://github.com/shivrai/smoketestsuite/compare/afe9f9...e052cc|Updates]]  | ✔ |
+| 13 | Aug 23 - Aug 29 | ::: | ::: |
+|  **Update documentation**  |||  ★  |||
+|  **Final evaluations**  |||  ★  |||
-Table design adapted from [[https://wiki.freebsd.org/SummerOfCode2016/NonBSMtoBSMConversionTools#Milestones|NonBSMtoBSMConversionTools#Milestones]]
+Table design adapted from [[SummerOfCode2016/NonBSMtoBSMConversionTools#Milestones|NonBSMtoBSMConversionTools#Milestones]]
 ====== Deliverables ======
 ===== Deliverables for first evaluation =====
-  * Add ''%%kyua%%'' based test programs for a subset of base utilities which are currently without any test coverage. The maximum number of utilities possible in the duration will be covered.
+  * Add atf-sh test programs for a subset of base utilities which are currently without any test coverage. The maximum number of utilities possible in the duration will be covered.
   * Prepare a basic design of the automation tool. Cover generation of a limited number of test scripts.
-(To be updated)
+===== Deliverables for second evaluation =====
+  * Add functionality to enable users in defining which testcases to ignore while generating tests.
+  * Generate tests for all section 1 utilities and try to identify [[https://github.com/shivansh/smoketestsuite/issues|cases where the tool behaves unexpectedly]]. [[https://github.com/shivansh/smoketestsuite/issues?q=is%3Aissue+is%3Aclosed|Fix]] such cases.
+===== Deliverables for third evaluation =====
+  * Finish adding functionality to halt test generation during runtime.
+  * Clean up the code and prepare it for merge into the src tree.
+  * Add a [[https://github.com/shivansh/smoketestsuite/#automation-tool|diagram briefly depicting the architecture]].
 ====== Weekly Reports ======
+**Note:** While visiting links in the following reports to **//[[https://shivrai.github.io]]//**, they should be renamed to **//[[https://shivansh.github.io]]//**.
 ^  Week  ^  Report  ^
@@ Line 89: / Line 118: @@
 | 4 | [[https://lists.freebsd.org/pipermail/soc-status/2017-July/001079.html|2017-July/001079]] |
 | 5 | [[https://lists.freebsd.org/pipermail/soc-status/2017-July/001086.html|2017-July/001086]] |
+| 6 | [[https://lists.freebsd.org/pipermail/soc-status/2017-July/001089.html|2017-July/001089]] |
+| 7 | [[https://lists.freebsd.org/pipermail/soc-status/2017-July/001094.html|2017-July/001094]] |
+| 8 | [[https://lists.freebsd.org/pipermail/soc-status/2017-July/001098.html|2017-July/001098]] |
+| The test generation tool was ready at this point ||
+| 12 | [[https://lists.freebsd.org/pipermail/soc-status/2017-September/001107.html|Final Report]] |
 ====== Bugs reported ======
@@ Line 96: / Line 130: @@
 ====== The Code ======
-[[https://github.com/shivrai/smoketestsuite|github/smoketestsuite]] - Contains a rough implementation sketch of an in-progress smoke-testing tool.
-===== Differentials =====
+<callout type="info">
+[[https://github.com/shivansh/smoketestsuite|Github]]
+Contains the implementation of test generation tool. Also contains the implementation of the initially-proposed test plan.
+[[https://github.com/shivansh/smoketestsuite/#automation-tool|Architecture overview]]
+</callout>
+===== Differentials Submitted =====
 ^  Week  ^  Differential  ^  Status  ^
@@ Line 107: / Line 148: @@
 | 3 | [[https://reviews.freebsd.org/D11159|D11159]] | [[https://github.com/freebsd/freebsd/commit/612b309ec4b56659f0f044f7b04378caee9862c0|Merged]] ✔ |
 | ::: | [[https://reviews.freebsd.org/D11167|D11167]] | Merged [[https://github.com/freebsd/freebsd/commit/eab4b876ee6d2c4142d263a2e77cd4f008e05781|[1 ]] [[https://github.com/freebsd/freebsd/commit/0c9cbe6b24a7198ddc2a0dd196f30ad84c30b585|[2 ]] ✔ |
-| 4 | [[https://reviews.freebsd.org/D11315|D11315]] | [[https://github.com/freebsd/freebsd/commit/b19d13f1dc59db41237950efb6c4210828af3cde|Merged]] ✔ |
+| 4 | [[https://reviews.freebsd.org/D11315|D11315]] | Open |
-| ::: | [[https://reviews.freebsd.org/D11318|D11318]] | Open |
+| ::: | [[https://reviews.freebsd.org/D11318|D11318]] | [[https://github.com/freebsd/freebsd/commit/b19d13f1dc59db41237950efb6c4210828af3cde|Merged]] ✔ |
 | 5 | [[https://reviews.freebsd.org/D11529|D11529]] | [[https://github.com/freebsd/freebsd/commit/455feac4e250f416df84eb8f73d2c05e560cb334|Merged]] ✔ |
+| 10 | [[https://reviews.freebsd.org/D12036|D12036]] | [[https://github.com/freebsd/freebsd/commit/1ee19e88f3cb597232552013b2a57558529a8580|Merged]] ✔ |
+| The differential submitted on week 10 lays the basic structure of the generated tests |||
+| 12 | [[https://reviews.freebsd.org/D12249|D12249]] | Open |
 ----