Date: Sat, 01 Jul 2017 13:37:21 +0000 From: Shivansh Rai <shivansh@freebsd.org> To: "soc-status@freebsd.org" <soc-status@freebsd.org> Cc: Alan Somers <asomers@freebsd.org>, Brooks Davis <brooks@freebsd.org>, "Ngie Cooper (yaneurabeya)" <yaneurabeya@gmail.com> Subject: [GSOC17] Smoke testing of all base utilities - Week 4 Message-ID: <CAF%2Bp1Hv2_ny8kTfGr5GQkL-qnZpcReL0%2Ba0xSsn3WLCmUFxNfQ@mail.gmail.com>
index | next in thread | raw e-mail
Hello all, A little (too much) late (~4 days) for the progress report of week 4, my deepest apologies. This is the progress made in the last week, finishing the milestones which I promised in the previous report. A very brief gist of how the smoke testing tool works as of now - If a "known option" is encountered while looking in the hashmap containing pre-defined option definitions, a testcase to check the validity of the result of that option is produced. *NOTE:* Pre-defined option definitions try to relate option names with the most relevant keywords which might occur in their descriptions (for e.g. '-h' might contain "help", '-v' might contain "version" etc.) In case no such option was encountered, testcases to verify the correct usage message for all of the valid (accepted by the utility under test) options are produced when they are used incorrectly (since we couldn't guess their correct usage). The code is fairly readable (I guess, refactoring in progress), and it might give a better idea. For those wondering as this is extremely trivial, this mail from @brooks [1] will provide a very good context and the big picture. The state of the tool now is a small step forward as to what is finally expected. The tool [2] is ready for basic experimentation. It can be run via the following commands from <project_root>/tool - ``` cd <project_root>/tool make && make run ``` The expected behavior for this is that all the options accepted by ln(1) will fail (since we cannot guess (yet) how they are supposed to be used by just reading the man-page) and produce a usage message. An atf-sh test file `ln_test.sh` will be produced which will contain test-cases for all these options which basically runs them and check the validity of the generated usage message. Since ln(1) already has a test now, you can copy this test file to <local_freebsd_src>/bin/ln/tests and do the regular steps to run a kyua test - (BEWARE! Your source tree should have some form of version controlling to revert it back to the original state.) ``` cd <local_freebsd_src>/bin/ln/tests make install cd /usr/tests/bin/ln kyua test ``` This should produce 10 successful testcases. I also prepared a differential [4] for a test of getfacl(1) utility (I am yet to update the requested changes, will be doing them soon). *---(End of progress report, meta information ahead!)---* While I was writing the basic functionality today, I had an idea as to how to work upon generating sane option identifiers. Currently they are very simple (like "version", "help"). But since the tool will be going through (mostly) all the utilities' man pages in the base system, some sort of relation can be mapped between option names and the most frequent words which "happen to occur" in their descriptions based on the history of previously visited option definitions. This way, our comparison will not be limited to a simple `string.find()` for a single word while looking in a utility's option description, and we can check for the presence of these frequently occurring words in option descriptions and "try to guess" an appropriate usage for that specific option. A priority queue (heap?) can be a way to go, but a lot of thinking has to be put in for several other aspects. It all seems a bit far-fetched at the moment when the thought of a sane running time complexity surfaces, but hopefully some way can be figured out. Ideas and suggestions are most welcome! The above will be worked out before sending the next report, and also the description of how the tool works [5] will be updated shortly. Updates to the tool since last report: https://github.com/shivrai/smoketestsuite/compare/c2b5c7...4969ae [1]: https://shivrai.github.io/assets/tmp/smoke_testing_mail.pdf [2]: https://github.com/shivrai/smoketestsuite/tree/master/tool [4]: https://reviews.freebsd.org/D11315 [5]: https://shivrai.github.io/assets/tmp/GSoC17Automation.pdf PS. On a side note, I wasn't willing to merge the progress report for weeks 4 and 5 together, hence will be sending the next report for week 5 (current week) in a few days' duration. Thanks! With best regards, Shivansh Raihome | help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAF%2Bp1Hv2_ny8kTfGr5GQkL-qnZpcReL0%2Ba0xSsn3WLCmUFxNfQ>