Date: Sun, 23 Jul 2017 13:30:34 +0000 From: Shivansh Rai <shivansh@freebsd.org> To: "soc-status@freebsd.org" <soc-status@freebsd.org> Cc: Alan Somers <asomers@freebsd.org>, Brooks Davis <brooks@freebsd.org>, "Ngie Cooper (yaneurabeya)" <yaneurabeya@gmail.com> Subject: [GSOC17] Smoke testing of all base utilities - Week 7 Message-ID: <CAF%2Bp1HsjYXXF2sVxg-SpB0eiFa=U2HUxFRvyrQhH=athCBKfxQ@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
Hello all,
My apologies for a late progress report (~3 days), I was involved in
shifting my belongings to a new residence as final year is about to start.
A brief summary of the updates made to the tooling in the past week is
available here [1].
The tool can now generate annotations based on failed testcases during a
test runtime. The annotation file contains the name of testcases which we
don=E2=80=99t want to be added in the tests generated in future runs. An ex=
ample
for these kind of testcases are the ones produced previously in date(1) -
since most testcases were time/timezone dependent, one would prefer not to
include them in the generated test script. Hence, the annotation file for
date [2] contains all those testcase names. The newly generated test for
date(1) only has the successful testcases [3].
@brooks suggested that maintaining annotations in a separate file has an
advantage of not disrupting the existing test scripts.
For each generated test script in =E2=80=9Cgenerated_tests/*=E2=80=9D, we m=
aintain a
corresponding annotation file in =E2=80=9Cannotations/=E2=80=9C (in case th=
ere are failing
testcases). For e.g. the annotation file for =E2=80=9Cgenerated_tests/date_=
test.sh=E2=80=9D
is =E2=80=9Cannotations/date_test.annot=E2=80=9D.
A brief pseudocode for generating annotations is as follows -
for test_script in generated_tests/*
run $test_script via `kyua test`
# The above step is yet to be implemented as a
# few things have to be kept in mind such as:
# - placing the test_script in a correct destination
# - if the makefile exists, updating it; if not, then generating it
# - appropriately updating "etc/mtree/BSD.tests.dist"
extract failing testcases via `kyua report`
generate annotations for $test_script based on failed testcases
if annotation file exists
append newly found annotations
end if
I was unable to find a good interface for extracting exit statuses of
individual testcases from a test run (need to search more), hence extracted
the same from the output of kyua report. The current method will be updated
if I come across a cleaner approach.
Refactoring of the codebase is in progress alongside as I read C++11
guidelines and good practices.
On a different note, my PR for adding the atf_check_not_equal function to
atf-sh was merged [4]. Once the new version of atf(7) is released,
corresponding updates will be made which led to the creation of this PR [5]=
.
The next task is to populate the annotation files pertaining to different
architectures and test environments (will be automated). If a particular
testcase fails in any case, we add a corresponding annotation as the
generated tests should be architecture/environment independent.
Also, the first step in the above pseudocode will be implemented.
[1]: https://github.com/shivrai/smoketestsuite/compare/2bf499...12d88b
[2]:
https://github.com/shivrai/smoketestsuite/blob/master/tool/annotations/date=
_test.annot
[3]:
https://github.com/shivrai/smoketestsuite/blob/master/tool/generated_tests/=
date_test.sh
[4]:
https://github.com/jmmv/atf/commit/0824235a3bcb45f16e1dfae7dce347e2d8b10d26
[5]: https://reviews.freebsd.org/D11084?id=3D29301#inline-65206
Thank you.
With best regards,
Shivansh Rai
=E2=80=8B
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAF%2Bp1HsjYXXF2sVxg-SpB0eiFa=U2HUxFRvyrQhH=athCBKfxQ>