From owner-soc-status@freebsd.org Sat Jul 1 16:33:15 2017 Return-Path: Delivered-To: soc-status@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D2CEDD8DCD9 for ; Sat, 1 Jul 2017 16:33:15 +0000 (UTC) (envelope-from shivanshrai84@gmail.com) Received: from mail-qt0-f177.google.com (mail-qt0-f177.google.com [209.85.216.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9635B6712F; Sat, 1 Jul 2017 16:33:15 +0000 (UTC) (envelope-from shivanshrai84@gmail.com) Received: by mail-qt0-f177.google.com with SMTP id b40so10718412qtb.2; Sat, 01 Jul 2017 09:33:15 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to:cc; bh=oPbpdYNd+Y2PxCaLXrccZHiyCRtou10a3DbYtxy9HGE=; b=lBgdl1Nyjdni8f37+3P0KmDfm9JFcb7yY845S7GQ67K3Zuhc/+91v5QdBAGg5sCRTP qVQyiFbHTHlpvhRTHJBIX+WnwIJAoTN99I7k5omeYkw7ySoePFZcoDDVZeNWjbS2xzjv PuIzI3ursbEA88VBN9F3bWoZZBcVUoq0zWizA0/KDOrIcOt7W1JT566jRTV36Jt3zU5u UtVODIyBr7oIEOZop36gmNkGYnf3N2VClVkTjkMhvaWcS7xxUlCTe7VdVxD99oieP4B2 trKPZ7aotcY9xWTP1fJzb4c0o0YrBxca7ZRBFtjS3fsSivV8HW9fMhDns99rLSAAItz3 c8tQ== X-Gm-Message-State: AIVw111+hs4V5lm8ranWJ/eEr56H0Ny9tzXTruRgXgT0IbLvpXo78qEq rvwr4V7HhnrblUfV X-Received: by 10.237.51.167 with SMTP id v36mr13591566qtd.197.1498916252045; Sat, 01 Jul 2017 06:37:32 -0700 (PDT) Received: from mail-qk0-f181.google.com (mail-qk0-f181.google.com. [209.85.220.181]) by smtp.gmail.com with ESMTPSA id o93sm8548586qte.41.2017.07.01.06.37.31 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 01 Jul 2017 06:37:31 -0700 (PDT) Received: by mail-qk0-f181.google.com with SMTP id v143so35318522qkb.0; Sat, 01 Jul 2017 06:37:31 -0700 (PDT) X-Received: by 10.55.79.80 with SMTP id d77mr29315291qkb.155.1498916251443; Sat, 01 Jul 2017 06:37:31 -0700 (PDT) MIME-Version: 1.0 From: Shivansh Rai Date: Sat, 01 Jul 2017 13:37:21 +0000 X-Gmail-Original-Message-ID: Message-ID: Subject: [GSOC17] Smoke testing of all base utilities - Week 4 To: "soc-status@freebsd.org" Cc: Alan Somers , Brooks Davis , "Ngie Cooper (yaneurabeya)" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: soc-status@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Summer of Code Status Reports and Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 01 Jul 2017 16:33:15 -0000 Hello all, A little (too much) late (~4 days) for the progress report of week 4, my deepest apologies. This is the progress made in the last week, finishing the milestones which I promised in the previous report. A very brief gist of how the smoke testing tool works as of now - If a "known option" is encountered while looking in the hashmap containing pre-defined option definitions, a testcase to check the validity of the result of that option is produced. *NOTE:* Pre-defined option definitions try to relate option names with the most relevant keywords which might occur in their descriptions (for e.g. '-h' might contain "help", '-v' might contain "version" etc.) In case no such option was encountered, testcases to verify the correct usage message for all of the valid (accepted by the utility under test) options are produced when they are used incorrectly (since we couldn't guess their correct usage). The code is fairly readable (I guess, refactoring in progress), and it might give a better idea. For those wondering as this is extremely trivial, this mail from @brooks [1] will provide a very good context and the big picture. The state of the tool now is a small step forward as to what is finally expected. The tool [2] is ready for basic experimentation. It can be run via the following commands from /tool - ``` cd /tool make && make run ``` The expected behavior for this is that all the options accepted by ln(1) will fail (since we cannot guess (yet) how they are supposed to be used by just reading the man-page) and produce a usage message. An atf-sh test file `ln_test.sh` will be produced which will contain test-cases for all these options which basically runs them and check the validity of the generated usage message. Since ln(1) already has a test now, you can copy this test file to /bin/ln/tests and do the regular steps to run a kyua test - (BEWARE! Your source tree should have some form of version controlling to revert it back to the original state.) ``` cd /bin/ln/tests make install cd /usr/tests/bin/ln kyua test ``` This should produce 10 successful testcases. I also prepared a differential [4] for a test of getfacl(1) utility (I am yet to update the requested changes, will be doing them soon). *---(End of progress report, meta information ahead!)---* While I was writing the basic functionality today, I had an idea as to how to work upon generating sane option identifiers. Currently they are very simple (like "version", "help"). But since the tool will be going through (mostly) all the utilities' man pages in the base system, some sort of relation can be mapped between option names and the most frequent words which "happen to occur" in their descriptions based on the history of previously visited option definitions. This way, our comparison will not be limited to a simple `string.find()` for a single word while looking in a utility's option description, and we can check for the presence of these frequently occurring words in option descriptions and "try to guess" an appropriate usage for that specific option. A priority queue (heap?) can be a way to go, but a lot of thinking has to be put in for several other aspects. It all seems a bit far-fetched at the moment when the thought of a sane running time complexity surfaces, but hopefully some way can be figured out. Ideas and suggestions are most welcome! The above will be worked out before sending the next report, and also the description of how the tool works [5] will be updated shortly. Updates to the tool since last report: https://github.com/shivrai/smoketestsuite/compare/c2b5c7...4969ae [1]: https://shivrai.github.io/assets/tmp/smoke_testing_mail.pdf [2]: https://github.com/shivrai/smoketestsuite/tree/master/tool [4]: https://reviews.freebsd.org/D11315 [5]: https://shivrai.github.io/assets/tmp/GSoC17Automation.pdf PS. On a side note, I wasn't willing to merge the progress report for weeks 4 and 5 together, hence will be sending the next report for week 5 (current week) in a few days' duration. Thanks! With best regards, Shivansh Rai