From owner-soc-status@freebsd.org Thu Jul 13 17:38:53 2017 Return-Path: Delivered-To: soc-status@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 615C0DA9A8D for ; Thu, 13 Jul 2017 17:38:53 +0000 (UTC) (envelope-from shivanshrai84@gmail.com) Received: from mail-qk0-f172.google.com (mail-qk0-f172.google.com [209.85.220.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2533F8145D; Thu, 13 Jul 2017 17:38:52 +0000 (UTC) (envelope-from shivanshrai84@gmail.com) Received: by mail-qk0-f172.google.com with SMTP id 16so56657756qkg.2; Thu, 13 Jul 2017 10:38:52 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to:cc; bh=rVlGGbFf48EJSbrhxXCDzxeggOJaJ/NbKWyiUqE39YY=; b=n9u9l4PEZeVTVfda50kDOe3MJpr8vyk+6rnSb7J+4wi5pt4jawiAGrYM7bJdiDm3He OlraAMyjZxfg8hdhz6vAyS2SsQtliYlk5Mo9XHVxlLj/UjJP0h1+1u++l+ZAO6ereSo9 HopOGACS+MsmX48Cn40+0Hrvjyn8T7Rk/e2aEAoeggbydw6ClciXNAFiyBz4X0UFXVfE 0geT+g/hR0wiwhHi/nUJt9LXlfCFDkv/XCTsBs3s8z+Tx6NxyFnwYOqvyBxsSxOE27wY zJvlD/1sDKSNzDR6NaESuC/AAeS3wkQ7kA2Tj+usbnhe704+S1Yj2iQtoRMLuoawktLI 557g== X-Gm-Message-State: AIVw112NniZBq610bHwqZnXeKTx6qoJQ5KhroFSleQeKKLG4oC30sjp3 o45m1ihNBOhoI3UuNR0= X-Received: by 10.55.10.11 with SMTP id 11mr5846399qkk.251.1499967525809; Thu, 13 Jul 2017 10:38:45 -0700 (PDT) Received: from mail-qt0-f171.google.com (mail-qt0-f171.google.com. [209.85.216.171]) by smtp.gmail.com with ESMTPSA id l67sm873772qkh.34.2017.07.13.10.38.45 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 13 Jul 2017 10:38:45 -0700 (PDT) Received: by mail-qt0-f171.google.com with SMTP id r30so43966668qtc.0; Thu, 13 Jul 2017 10:38:45 -0700 (PDT) X-Received: by 10.200.46.184 with SMTP id h53mr6899351qta.96.1499967525235; Thu, 13 Jul 2017 10:38:45 -0700 (PDT) MIME-Version: 1.0 From: Shivansh Rai Date: Thu, 13 Jul 2017 17:38:34 +0000 X-Gmail-Original-Message-ID: Message-ID: Subject: [GSOC17] Smoke testing of all base utilities - Week 6 To: "soc-status@freebsd.org" Cc: Alan Somers , Brooks Davis , "Ngie Cooper (yaneurabeya)" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: soc-status@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Summer of Code Status Reports and Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Jul 2017 17:38:53 -0000 Hello all, Sending an early progress report to cover up the lag in the weekly schedule= . Previous 4 days were spent on polishing some corner cases in the automation tool. A summary of updates can be viewed here [1]. To demonstrate the tests which the tool can generate, the utilities date(1), ln(1) and stdbuf(1) were used. These tests are available here [2]. The following table summarizes the type of test-cases produced for these 3 utilities. +----------------+-----------------+-----------------+ | Test | (+ve) testcases | (-ve) testcases | +----------------+-----------------+-----------------+ | date_test.sh | 5 | 2 | +----------------+-----------------+-----------------+ | ln_test.sh | 0 | 11 | +----------------+-----------------+-----------------+ | stdbuf_test.sh | 1 | 0 | +----------------+-----------------+-----------------+ Some key-points worth noting : 1.) date_test.sh is time/timezone dependent, hence all the test-cases apart from invalid_usage will fail. The aim for using this utility is to demonstrate that the tool can generate positive test-cases along with negative ones. It also demonstrates the need to sanity check the tests. 2.) ln_test.sh demonstrates that if the utility produces a common usage message for every invalid usage, then the tool can define a variable containing this message and avoid duplication while generating the test-cases. Also, all the negative tests are collated under a single testcase named "invalid_usage". 3.) The groff script for stdbuf(1) is slightly different than others, hence the tool couldn=E2=80=99t generate any negative test-case in stdbuf_test.sh= as it couldn't pick up the supported options. This scenario will be taken into account next. The aim for using this utility is to demonstrate that the tool will always produce atleast one positive/negative testcase. Some updates were also made to the documentation [3]. The project wiki will be updated next. The plan for this week is to start working on generating tests for all the utilities which currently don=E2=80=99t have any test coverage. The automat= ion tool will be worked upon and improved during the process. As always, suggestions are most welcome! [1]: https://github.com/shivrai/smoketestsuite/compare/1a7e26e...2b3e2a5 [2]: https://github.com/shivrai/smoketestsuite/tree/master/tool/generated_tests [3]: https://github.com/shivrai/smoketestsuite/blob/master/README.md Thank you. With best regards, Shivansh Rai =E2=80=8B