From owner-soc-status@freebsd.org Sun May 20 22:30:04 2018 Return-Path: Delivered-To: soc-status@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 77CD6EF2083 for ; Sun, 20 May 2018 22:30:04 +0000 (UTC) (envelope-from aniket.ezio41@gmail.com) Received: from mail-wm0-f54.google.com (mail-wm0-f54.google.com [74.125.82.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DAC5374143; Sun, 20 May 2018 22:30:03 +0000 (UTC) (envelope-from aniket.ezio41@gmail.com) Received: by mail-wm0-f54.google.com with SMTP id w194-v6so22154362wmf.2; Sun, 20 May 2018 15:30:03 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to:cc; bh=bQToKDr0bHhwSaqNYPdeDp2GS9EJJ02zsZ43Nkf9aqg=; b=FLUWHzdhPSrsY6EqHxQzfW3mkw8NVE97SIYeTJ0cW3ziXQbcg3xnG3w8SgOPQ7Ao6y wGCaBYTKRW3ewHJ7EHnnFCMZ2gwaZreSCJ6b+d1TCOb5nDJaRI8IicpZi98I8EAGjPAt F6jG+oaX1hSRveWVjcaF2A8SYxqLS5zT1GY0XshI1Euw4UV1ZegpayHCfm9gagMd6jvf wov+uHCY1DVZ1DoEqvVYjoMsw/yZM5g0QqMLq1VadfkHOscjpqJvwQ1mM+1WeQAKfeRo +J6mOBunaxC0Z0faR6A9HA/ff+OQ9ZcrEaBoIhmZeXP51652a7HgjMbdjrPCa5AM41C3 pK0A== X-Gm-Message-State: ALKqPweCbHg+U3PMhyMC4TSE6GcizIQlePvbMsVM/WTxFzqEvzJSe0ug COD/KxUA/Z2N4Fc+IYfzN05M7a8U X-Google-Smtp-Source: AB8JxZqT3Jv9EwBiXcSBEK71oeTfxz9L4jWASkunzjoe+XO++t/Wzjm3xsk+Xa18sijAg17/E0V65Q== X-Received: by 2002:a50:a722:: with SMTP id h31-v6mr20885150edc.288.1526854895259; Sun, 20 May 2018 15:21:35 -0700 (PDT) Received: from mail-wm0-f42.google.com (mail-wm0-f42.google.com. [74.125.82.42]) by smtp.gmail.com with ESMTPSA id b4-v6sm6355463edr.29.2018.05.20.15.21.34 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 20 May 2018 15:21:35 -0700 (PDT) Received: by mail-wm0-f42.google.com with SMTP id t11-v6so22312867wmt.0; Sun, 20 May 2018 15:21:34 -0700 (PDT) X-Received: by 2002:a1c:5403:: with SMTP id i3-v6mr8860674wmb.37.1526854894428; Sun, 20 May 2018 15:21:34 -0700 (PDT) MIME-Version: 1.0 Received: by 10.223.151.38 with HTTP; Sun, 20 May 2018 15:21:34 -0700 (PDT) From: Aniket Pandey Date: Mon, 21 May 2018 03:51:34 +0530 X-Gmail-Original-Message-ID: Message-ID: Subject: [GSoC-18] Regression Test-Suite for Audit Framework [Week-1] To: soc-status@freebsd.org Cc: asomers@freebsd.org, robert.watson@cl.cam.ac.uk, gnn@neville-neil.com, gavin@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.26 X-BeenThere: soc-status@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Summer of Code Status Reports and Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 20 May 2018 22:30:04 -0000 Hello All, This project aims to develop a set of atf-c(3) test-cases for FreeBSD's audit subsystem such that it will be easier to automate the testing of Audit framework along with the entire operating system. The automation will be done by Kyua(1). I've been working on this project from late January and have made a considerable amount of progress since then. I'll list out the important highlights, breakthroughs and issues I've encountered along the way. * Initial attempt was to create a set of source files which would contain basic implementation of all common and recurring system calls in the audit records. Then I would develop an automation script which would check the presence of the system call along with the arguments, attributes, return status and possibly some other information. However, this approach was deprecated since FreeBSD already had Kyua(1) which could do the automation for me and had some really nice utility functions available. * On recommendation of Kristof and Alan, I switched to creating atf-c(3) test-cases for all system calls in various scenarios. However, considering the vast number of OpenBSM supported system calls and for the scope of the projectc, I planned on testing "success" and "failure" modes of each system call to ensure greater coverage. * Each test case would be independent, would launch auditd(8) if needed, would set local mode auditing using the defined preselection-parameters, check auditpipe(4) if it emitted the system call, and then conclude the tests. Detailed implementation details can be found in the Project Wiki. * Test-Suite organisation is to have one test-program per audit class, and have all its audit events within. (e.g file-create.c, file-attribute-access.c etc) Milestones: --------------- So far, I've been able to create 306 ATF test cases spanning over 6 test-programs for 62 system calls of ("fc", "fw", "fr", "fd", "fm", "fa" audit classes) and 5875 SLOC. The tests are currently passing and there general output can be seen here: https://pastebin.com/XHfAP6nH Bugs/Issues encountered: ---------------------------------- * Initially we had trouble with getting auditd(8) to properly start and basically it was creating a race condition within the test-cases. To resolve this, we had to poll(2) for "audit startup" and once it was confirmed, proceed with normal syscall testing. * After introducing the audit startup check, we encountered another problem with one of the libbsm(3) APIs, au_read_rec(3) which for some reason, returned corrupted records when checking for successful audit startup as a lot of other events were being generated too. To resolve this, we had to explicitly set the audit mask for local mode auditing as "ad" (for audit startup) * On trying to run the tests in FreeBSD-12 CURRENT when the auditd(8) is not already running, we encountered a reproducible kernel panic ( https://pastebin.com/u3DqGTS1), which was fixed by Alan in D15381 . * Some system calls like "lgetfh(2)" and "chflagsat(2)" were not being audited while syscalls like "getfh(2)" were audited as "nfs_getfh(2)". I've created a differential revision D15286 , which introduces the initial set of tests (24) for audit system. It is currently under review. I'll keep adding more test cases in the next week and hope to finish the remaining File-I/O audit events and complete tests for socket API. Bugs Reported -------------------- 1) https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=228374 2) https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=226713 Differential Revisions ---------------------------- 1) https://reviews.freebsd.org/D15286 2) https://reviews.freebsd.org/D15381 (@asomers) ProjectWiki: https://wiki.freebsd.org/SummerOfCode2018Projects/RegressionTestSuiteForAuditFramework TestSuite Repo: https://github.com/aniketp/AuditTestSuite Socsvn branch: https://svnweb.freebsd.org/socsvn/soc2018/aniketp/head/ Thank You, With Best Regards, Aniket Pandey