Date: Wed, 06 Sep 2017 14:51:41 +0000 From: Shivansh Rai <shivansh@freebsd.org> To: "soc-status@freebsd.org" <soc-status@freebsd.org>, Alan Somers <asomers@freebsd.org>, Brooks Davis <brooks@freebsd.org>, "Ngie Cooper (yaneurabeya)" <yaneurabeya@gmail.com> Subject: [GSOC17] Smoke testing of all base utilities - Final Report Message-ID: <CAF%2Bp1HvGezYrrgvq=XfyGxsqXgiPwwHtnAOFf5PrcEPjttkOkQ@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
Hello all,
This is the final progress report for all the tasks that have been
accomplished for the project - Smoke testing of all base utilities.
Targets met in the timeline
-
The automation tool [1] can now generate tests which can check whether a
utility is properly linked. A basic structure of testcases in a generate=
d
test is as follows -
- invalid_usage : Verifies the behavior of a utility when an invalid
usage is performed for a supported option.
- ($option)_flag: Verifies the behavior of a utility when a valid
usage is performed for a supported option - $option.
- no_arguments : Verifies the behavior of a utility when it is used
without any supported options.
-
The generated tests are maintained at a separate location [2] and
revised with updated tests when the tool is updated.
-
The final differential [3] which lays the basic layout of the
auto-generated tests will be soon merged. Once it is approved, similar
batches of tests will be sent for review.
I have prepared a small diagram [4] which shows how different components of
the tool fit in with the testcase generation component. I am planning to
add more details to it.
Current and future scope for work
-
I am currently working on trying to resolve an interesting issue which I
have recently noticed while generating the tests. The description of the
issue is as follows -
The functionality to stop generation of tests during runtime was
successfully integrated recently [5] (after a failed previous attempt
summarized here -
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D221882). The
requirement of this functionality arose out of the fact that there are
utilities with options which prompt the user for input, and halt at this
point until some action is taken. However, currently the tool does not
concern itself with any form of =E2=80=9Cexternal=E2=80=9D intervention =
while generating
tests, so we put a timeout of 1 second for each testcase to complete (It=
is
assumed that 1 second is sufficient to successfully process the groff
script and generate the test ; a thread is spawned for individually
handling testcase generation for each utility). It so happens that on fi=
rst
encounter of such a utility (let=E2=80=99s say it is =E2=80=98u1=E2=80=
=99), all the future
instances seem to take more than 1 second to complete and are not genera=
ted
by the tool. Interestingly, if the groff script of =E2=80=98u1=E2=80=99 =
is removed (the
tool loads groff scripts from the directory src/groff before generating
tests), then everything seems to work fine until another utility like =
=E2=80=98u1=E2=80=99
is encountered.
This is an example trace demonstrating the above scenario -
... (Previous log) ...
Generating test for: ypcat(1) ...Successful
Generating test for: printenv(1) ...Successful
Generating test for: uniq(1) ...Failed!
Generating test for: ktrace(1) ...Failed!
Generating test for: revoke(1) ...Failed!
The utility for which first failure was encountered while test
generation is uniq(1) ; the test generation for ktrace(1) and revoke(1)
also fails, which should not happen. Let=E2=80=99s remove the groff scri=
pt for
uniq(1) from `src/groff` and try again -
... (Previous log) ...
Generating test for: ypcat(1) ...Successful
Generating test for: printenv(1) ...Successful
Generating test for: ktrace(1) ...Successful
Generating test for: revoke(1) ...Successful
As it can be observed, the utilities which failed to produce tests
earlier (ktrace(1), revoke(1)) are now successful.
I am suspecting this behavior is due to having an improper thread
cleanup routine for the failing instances (there is no cleanup being don=
e
yet apart from a call to timed_join() of boost::thread). Although I have
not been able to reproduce this behavior on my Linux machine. Looking in=
to
this currently! (interestingly we are studying about pthreads currently =
in
the OS course :P )
-
I am yet to finish the test for getfacl(1) that I started in D11315 [6]
(my deepest apologies for being so much late in this even after a remind=
er
from asomers@, am overloaded with deadlines at the moment :/ But will
complete it by this weekend, and this deadline is final and not subjecte=
d
to change! ).
-
The tool currently handles processing of groff scripts for section 1
utilities. However, brooks@ pointed out that the way in which things are
currently done might not work for section 8 utilities. So, the next task
will be to expand the tool to cover all the new cases.
The wiki page [7] is fairly up to date now, and I will continue to add
updates to it.
Finally, I would like to thank my mentors for all their help and prompt
responses (and awesome code reviews!) during the entire duration, and hope
to stay in touch with them. I would also like to thank the FreeBSD
community for giving me this great opportunity to learn and hope to be an
integral part of it in times to come.
[1]: https://github.com/shivansh/smoketestsuite/
[2]: https://github.com/shivansh/smoketests/
[3]: https://reviews.freebsd.org/D12036
[4]: https://github.com/shivansh/smoketestsuite/#automation-tool
[5]:
https://github.com/shivansh/smoketestsuite/commit/267540d534fab4a3bea183518=
d64d0b23cee19d7
[6]: https://reviews.freebsd.org/D11315
[7]: https://wiki.freebsd.org/SummerOfCode2017/SmokeTestingOfBaseUtilities
Thank you.
With best regards,
Shivansh Rai
=E2=80=8B
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAF%2Bp1HvGezYrrgvq=XfyGxsqXgiPwwHtnAOFf5PrcEPjttkOkQ>