From owner-soc-status@freebsd.org Sun Jul 23 13:36:10 2017 Return-Path: Delivered-To: soc-status@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9C2FDD7E8A7 for ; Sun, 23 Jul 2017 13:36:10 +0000 (UTC) (envelope-from shivanshrai84@gmail.com) Received: from mail-qt0-f173.google.com (mail-qt0-f173.google.com [209.85.216.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6044C209B; Sun, 23 Jul 2017 13:36:09 +0000 (UTC) (envelope-from shivanshrai84@gmail.com) Received: by mail-qt0-f173.google.com with SMTP id r14so25650567qte.4; Sun, 23 Jul 2017 06:36:09 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to:cc; bh=MxLtcUKyznrvzyk/b+Kfinbp9IvK2cX0yj90kpNBp2M=; b=eJKsiXDxEAmReMgQvecqunu1ZIukUkBn9URG7fxTM5oNiZ55OsnPumRLyw19+4WYQq MXvnz+KwD2wo6oK0Pzh7xDc/RbRED7VhbtYoS14gsNDKs15ba6wIcKLBo9dg4QL2iaaL ud9sj2paTqqfmBVqvgCIdmAjSwL6It5nITMmmr2tkJaX7aqTa5FSel6daqfvBl4c+oUa 7hIJPJRGmthOiIqc06BQ9/lrizfGwyz/SxmHcdwBATsDurxJiiEWJCmpfnKokXVzolS0 lYZJCPryL0lrgtsf2cgiQTn3YIsrhu3lptA5Y2i1jl/wuo5p8RvOYjh5sj415IoMpWO0 TRww== X-Gm-Message-State: AIVw111UGglHQNd20/Xsm7Zxd3BSvTk8yGohKh/0VC6PG6/NkvXg1aW1 XqY3wDpHniFXZ1x3ZwY= X-Received: by 10.200.41.172 with SMTP id 41mr17520229qts.85.1500816646000; Sun, 23 Jul 2017 06:30:46 -0700 (PDT) Received: from mail-qk0-f173.google.com (mail-qk0-f173.google.com. [209.85.220.173]) by smtp.gmail.com with ESMTPSA id p52sm6923553qtc.74.2017.07.23.06.30.45 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 23 Jul 2017 06:30:45 -0700 (PDT) Received: by mail-qk0-f173.google.com with SMTP id d145so33757265qkc.2; Sun, 23 Jul 2017 06:30:45 -0700 (PDT) X-Received: by 10.233.235.132 with SMTP id b126mr17560813qkg.262.1500816645184; Sun, 23 Jul 2017 06:30:45 -0700 (PDT) MIME-Version: 1.0 From: Shivansh Rai Date: Sun, 23 Jul 2017 13:30:34 +0000 X-Gmail-Original-Message-ID: Message-ID: Subject: [GSOC17] Smoke testing of all base utilities - Week 7 To: "soc-status@freebsd.org" Cc: Alan Somers , Brooks Davis , "Ngie Cooper (yaneurabeya)" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: soc-status@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Summer of Code Status Reports and Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Jul 2017 13:36:10 -0000 Hello all, My apologies for a late progress report (~3 days), I was involved in shifting my belongings to a new residence as final year is about to start. A brief summary of the updates made to the tooling in the past week is available here [1]. The tool can now generate annotations based on failed testcases during a test runtime. The annotation file contains the name of testcases which we don=E2=80=99t want to be added in the tests generated in future runs. An ex= ample for these kind of testcases are the ones produced previously in date(1) - since most testcases were time/timezone dependent, one would prefer not to include them in the generated test script. Hence, the annotation file for date [2] contains all those testcase names. The newly generated test for date(1) only has the successful testcases [3]. @brooks suggested that maintaining annotations in a separate file has an advantage of not disrupting the existing test scripts. For each generated test script in =E2=80=9Cgenerated_tests/*=E2=80=9D, we m= aintain a corresponding annotation file in =E2=80=9Cannotations/=E2=80=9C (in case th= ere are failing testcases). For e.g. the annotation file for =E2=80=9Cgenerated_tests/date_= test.sh=E2=80=9D is =E2=80=9Cannotations/date_test.annot=E2=80=9D. A brief pseudocode for generating annotations is as follows - for test_script in generated_tests/* run $test_script via `kyua test` # The above step is yet to be implemented as a # few things have to be kept in mind such as: # - placing the test_script in a correct destination # - if the makefile exists, updating it; if not, then generating it # - appropriately updating "etc/mtree/BSD.tests.dist" extract failing testcases via `kyua report` generate annotations for $test_script based on failed testcases if annotation file exists append newly found annotations end if I was unable to find a good interface for extracting exit statuses of individual testcases from a test run (need to search more), hence extracted the same from the output of kyua report. The current method will be updated if I come across a cleaner approach. Refactoring of the codebase is in progress alongside as I read C++11 guidelines and good practices. On a different note, my PR for adding the atf_check_not_equal function to atf-sh was merged [4]. Once the new version of atf(7) is released, corresponding updates will be made which led to the creation of this PR [5]= . The next task is to populate the annotation files pertaining to different architectures and test environments (will be automated). If a particular testcase fails in any case, we add a corresponding annotation as the generated tests should be architecture/environment independent. Also, the first step in the above pseudocode will be implemented. [1]: https://github.com/shivrai/smoketestsuite/compare/2bf499...12d88b [2]: https://github.com/shivrai/smoketestsuite/blob/master/tool/annotations/date= _test.annot [3]: https://github.com/shivrai/smoketestsuite/blob/master/tool/generated_tests/= date_test.sh [4]: https://github.com/jmmv/atf/commit/0824235a3bcb45f16e1dfae7dce347e2d8b10d26 [5]: https://reviews.freebsd.org/D11084?id=3D29301#inline-65206 Thank you. With best regards, Shivansh Rai =E2=80=8B