From owner-soc-status@freebsd.org Wed Sep 6 17:46:39 2017 Return-Path: Delivered-To: soc-status@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DC4C4E0E51F for ; Wed, 6 Sep 2017 17:46:39 +0000 (UTC) (envelope-from shivanshrai84@gmail.com) Received: from mail-vk0-f41.google.com (mail-vk0-f41.google.com [209.85.213.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 35F7074AD2; Wed, 6 Sep 2017 17:46:38 +0000 (UTC) (envelope-from shivanshrai84@gmail.com) Received: by mail-vk0-f41.google.com with SMTP id t10so13048168vke.0; Wed, 06 Sep 2017 10:46:38 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=x2oZH3EE+7Z15vmIlvJ4rneHRG3xY2OYhZzz2jwHRDg=; b=PoBLwaqm78a0hxDCZjAu5srgaC0ZBf4mWas6PtebagwO2kjGf0G8E2Ye923HjW3vpm 9nXbLdvlrqW6cuwLLyEmLd3BJnKx/usbOReRolARfBNfo6iSuTWkbZf+XSx+L8spBPkH oNMWYcPH3OrlN7yFlWFtqp1KR6Z6OtNV8MAqMuVCd6ubnUzqkRGDiinaXyhuN5pACSz5 gX9aRU07Zrq8bB2tE+76GQFvCpoRn1H73XRJNJMddcVABM54yxOXzngpMwjKSC2jyQz7 d8reEbuOHWWwrC7WCy8YGc4g2ExBXHu/rSjGSVTJ1sdi0ojQzVDrOIsw2B8tApvwHAt/ ngiw== X-Gm-Message-State: AHPjjUhmMUQyXDGHx1msepTY9ANTGCo1jT8i97bR15UJC2z8XvAZ7kKh q480ixhT00BvJ573 X-Received: by 10.31.141.14 with SMTP id p14mr324413vkd.53.1504709514204; Wed, 06 Sep 2017 07:51:54 -0700 (PDT) Received: from mail-vk0-f47.google.com (mail-vk0-f47.google.com. [209.85.213.47]) by smtp.gmail.com with ESMTPSA id d21sm4022vke.20.2017.09.06.07.51.52 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 06 Sep 2017 07:51:53 -0700 (PDT) Received: by mail-vk0-f47.google.com with SMTP id m142so4746232vkf.2; Wed, 06 Sep 2017 07:51:52 -0700 (PDT) X-Google-Smtp-Source: ADKCNb7F1Pk5xVrFRXy5HCVUqJTK+zhJljFAA37GtqjU1pRnZfK9114iCg00FhfS9R61fbgnXLa3ZMkFL6w8vSktaAk= X-Received: by 10.31.198.67 with SMTP id w64mr1477225vkf.180.1504709512652; Wed, 06 Sep 2017 07:51:52 -0700 (PDT) MIME-Version: 1.0 From: Shivansh Rai Date: Wed, 06 Sep 2017 14:51:41 +0000 X-Gmail-Original-Message-ID: Message-ID: Subject: [GSOC17] Smoke testing of all base utilities - Final Report To: "soc-status@freebsd.org" , Alan Somers , Brooks Davis , "Ngie Cooper (yaneurabeya)" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: soc-status@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Summer of Code Status Reports and Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Sep 2017 17:46:40 -0000 Hello all, This is the final progress report for all the tasks that have been accomplished for the project - Smoke testing of all base utilities. Targets met in the timeline - The automation tool [1] can now generate tests which can check whether a utility is properly linked. A basic structure of testcases in a generate= d test is as follows - - invalid_usage : Verifies the behavior of a utility when an invalid usage is performed for a supported option. - ($option)_flag: Verifies the behavior of a utility when a valid usage is performed for a supported option - $option. - no_arguments : Verifies the behavior of a utility when it is used without any supported options. - The generated tests are maintained at a separate location [2] and revised with updated tests when the tool is updated. - The final differential [3] which lays the basic layout of the auto-generated tests will be soon merged. Once it is approved, similar batches of tests will be sent for review. I have prepared a small diagram [4] which shows how different components of the tool fit in with the testcase generation component. I am planning to add more details to it. Current and future scope for work - I am currently working on trying to resolve an interesting issue which I have recently noticed while generating the tests. The description of the issue is as follows - The functionality to stop generation of tests during runtime was successfully integrated recently [5] (after a failed previous attempt summarized here - https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D221882). The requirement of this functionality arose out of the fact that there are utilities with options which prompt the user for input, and halt at this point until some action is taken. However, currently the tool does not concern itself with any form of =E2=80=9Cexternal=E2=80=9D intervention = while generating tests, so we put a timeout of 1 second for each testcase to complete (It= is assumed that 1 second is sufficient to successfully process the groff script and generate the test ; a thread is spawned for individually handling testcase generation for each utility). It so happens that on fi= rst encounter of such a utility (let=E2=80=99s say it is =E2=80=98u1=E2=80= =99), all the future instances seem to take more than 1 second to complete and are not genera= ted by the tool. Interestingly, if the groff script of =E2=80=98u1=E2=80=99 = is removed (the tool loads groff scripts from the directory src/groff before generating tests), then everything seems to work fine until another utility like = =E2=80=98u1=E2=80=99 is encountered. This is an example trace demonstrating the above scenario - ... (Previous log) ... Generating test for: ypcat(1) ...Successful Generating test for: printenv(1) ...Successful Generating test for: uniq(1) ...Failed! Generating test for: ktrace(1) ...Failed! Generating test for: revoke(1) ...Failed! The utility for which first failure was encountered while test generation is uniq(1) ; the test generation for ktrace(1) and revoke(1) also fails, which should not happen. Let=E2=80=99s remove the groff scri= pt for uniq(1) from `src/groff` and try again - ... (Previous log) ... Generating test for: ypcat(1) ...Successful Generating test for: printenv(1) ...Successful Generating test for: ktrace(1) ...Successful Generating test for: revoke(1) ...Successful As it can be observed, the utilities which failed to produce tests earlier (ktrace(1), revoke(1)) are now successful. I am suspecting this behavior is due to having an improper thread cleanup routine for the failing instances (there is no cleanup being don= e yet apart from a call to timed_join() of boost::thread). Although I have not been able to reproduce this behavior on my Linux machine. Looking in= to this currently! (interestingly we are studying about pthreads currently = in the OS course :P ) - I am yet to finish the test for getfacl(1) that I started in D11315 [6] (my deepest apologies for being so much late in this even after a remind= er from asomers@, am overloaded with deadlines at the moment :/ But will complete it by this weekend, and this deadline is final and not subjecte= d to change! ). - The tool currently handles processing of groff scripts for section 1 utilities. However, brooks@ pointed out that the way in which things are currently done might not work for section 8 utilities. So, the next task will be to expand the tool to cover all the new cases. The wiki page [7] is fairly up to date now, and I will continue to add updates to it. Finally, I would like to thank my mentors for all their help and prompt responses (and awesome code reviews!) during the entire duration, and hope to stay in touch with them. I would also like to thank the FreeBSD community for giving me this great opportunity to learn and hope to be an integral part of it in times to come. [1]: https://github.com/shivansh/smoketestsuite/ [2]: https://github.com/shivansh/smoketests/ [3]: https://reviews.freebsd.org/D12036 [4]: https://github.com/shivansh/smoketestsuite/#automation-tool [5]: https://github.com/shivansh/smoketestsuite/commit/267540d534fab4a3bea183518= d64d0b23cee19d7 [6]: https://reviews.freebsd.org/D11315 [7]: https://wiki.freebsd.org/SummerOfCode2017/SmokeTestingOfBaseUtilities Thank you. With best regards, Shivansh Rai =E2=80=8B From owner-soc-status@freebsd.org Thu Sep 7 00:26:53 2017 Return-Path: Delivered-To: soc-status@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 26A50E1F86C for ; Thu, 7 Sep 2017 00:26:53 +0000 (UTC) (envelope-from paggas1@yandex.com) Received: from forward104o.mail.yandex.net (forward104o.mail.yandex.net [IPv6:2a02:6b8:0:1a2d::607]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "forwards.mail.yandex.net", Issuer "Yandex CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9EC1E6B3C0 for ; Thu, 7 Sep 2017 00:26:52 +0000 (UTC) (envelope-from paggas1@yandex.com) Received: from mxback8g.mail.yandex.net (mxback8g.mail.yandex.net [IPv6:2a02:6b8:0:1472:2741:0:8b7:169]) by forward104o.mail.yandex.net (Yandex) with ESMTP id 40B222C86237 for ; Thu, 7 Sep 2017 03:26:49 +0300 (MSK) Received: from smtp3o.mail.yandex.net (smtp3o.mail.yandex.net [2a02:6b8:0:1a2d::27]) by mxback8g.mail.yandex.net (nwsmtp/Yandex) with ESMTP id mU5CTtl6QG-QnZ4oCGC; Thu, 07 Sep 2017 03:26:49 +0300 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.com; s=mail; t=1504744009; bh=ZCbnx8q2PoA7OLJXMpl5fAOYuKZf2ywDolz/utfQru0=; h=To:From:Subject:Message-ID:Date; b=SXlEsr2uDUsAWVaMt5lScNj2N+QE6QCeU/Qrt4n01mEh7CSgfTFboINJcVlc/2AdZ KzB+7Vlzl4sr5h2acfws3Hdby6KNorG3VwkkjTqjsJu2an46aQWoJq2a0RhIgriK/5 EKiMfPzLqR/UWykIwDO+7sDqjaIlce5Y+7soxNpM= Received: by smtp3o.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id r4OmJN8hyh-QmDG91Qe; Thu, 07 Sep 2017 03:26:48 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client certificate not present) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.com; s=mail; t=1504744008; bh=ZCbnx8q2PoA7OLJXMpl5fAOYuKZf2ywDolz/utfQru0=; h=To:From:Subject:Message-ID:Date; b=Yr3+5ZsxRYmC+SrDMkEN0qOqf4LXIS74pEOOvkDXFnyGHjrKGaJyiaLuV7nxLrdNe YUiIq9tThhrdtAkDOkUZV8ErK/yRBj39FGl3ELuwo2OmXoNwsjGqgwsnAlhArug753 daQrMD7Kk3IfkhOsPT8RSSt9+IOMqqibm+C7V0LE= Authentication-Results: smtp3o.mail.yandex.net; dkim=pass header.i=@yandex.com To: soc-status@freebsd.org From: Panagiotes Mousikides Subject: GSoC status: PF testing, weeks 8-14 Message-ID: <39f22794-934d-02e0-8a77-f8d459bc8567@yandex.com> Date: Thu, 7 Sep 2017 00:26:47 +0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.2.1 MIME-Version: 1.0 Content-Language: en-US Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: soc-status@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Summer of Code Status Reports and Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Sep 2017 00:26:53 -0000 Hello everybody! Communication has not been so good for July and August, but here is what has happened since the last update on 2017-07-09. The pfctl tests have now been committed to FreeBSD and are part of the FreeBSD test suite. If you ever install from source and run the tests, our tests will now be part of that! A lot of work has also been put into the kernel pf tests. The approach chosen was to spin up virtual machines and run the tests there. For those of you that read the last status update, the approach followed was no. 2, using tap(4). The tests create a disk image and install a complete FreeBSD instance on it. The image is then cloned for each test for the number of virtual machines needed, with custom configuration for networking as needed. A test script is run that runs various commands on the virtual machines, according to the needs of the test. An important tool used for the tests is scapy, which exists as a binary package in FreeBSD. This tool, written in Python, is perfectly suited for generating and analyzing network traffic, and has been invaluable in implementing these tests. Have a look at our previous status updates at the soc-status@freebsd.org mailing list, as well as our FreeBSD Wiki page at https://wiki.freebsd.org/SummerOfCode2017/PfTesting, with links to the GitHub repository as well as the review patches. Best regards, Panagiotes