From owner-freebsd-testing@FreeBSD.ORG Mon Mar 3 20:32:14 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DE7ED988; Mon, 3 Mar 2014 20:32:13 +0000 (UTC) Received: from mail-we0-x231.google.com (mail-we0-x231.google.com [IPv6:2a00:1450:400c:c03::231]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 54175A37; Mon, 3 Mar 2014 20:32:13 +0000 (UTC) Received: by mail-we0-f177.google.com with SMTP id u57so1948625wes.8 for ; Mon, 03 Mar 2014 12:32:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=20cQWSOhMXTtT65Sy3UQWjRCLpEmMzu7hXJG3XRgzYQ=; b=otIbAhw2h56OY2/xmzKZCSNhcO0aWgmHQP71zByj95No9bUo0kr0EvNg1cXwkAcIYA jFmil2zti4DAH8SIRwWso4bMYeFeBJua65FFhh+jn7dxpA3p3vZ+9v7yUQwTeLeCiOMh YuAdhfwhnLFCN2cpCSWIqg7mZlu0sF97D+5yK1qCKf4tUjuGo/n+yR3EaXnVJ7LXt6Ru 4/GkZZejEUPCwPfI3BWrNhM4dlpCAj3dLEzBN1OTJHwYId9VfXsCUYQskMLlp0Eg9Tnh Fu1A15Lg8oXI2+ZbnCDT1S8y+PbdyFGL11qTE3GCHG35KIdlWP+bBGd7OuyNLKzB/JFS b1mA== MIME-Version: 1.0 X-Received: by 10.194.202.230 with SMTP id kl6mr21721600wjc.9.1393878717515; Mon, 03 Mar 2014 12:31:57 -0800 (PST) Sender: asomers@gmail.com Received: by 10.194.168.197 with HTTP; Mon, 3 Mar 2014 12:31:57 -0800 (PST) In-Reply-To: References: <20140303165226.GK85115@kiwi> Date: Mon, 3 Mar 2014 13:31:57 -0700 X-Google-Sender-Auth: 3z0Qm8WkXaE0SQMXE6AkuL0m6sA Message-ID: Subject: Re: atf-c for kernel function unit testing From: Alan Somers To: Julio Merino Content-Type: text/plain; charset=ISO-8859-1 Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Mar 2014 20:32:14 -0000 On Mon, Mar 3, 2014 at 10:32 AM, Julio Merino wrote: > On Mon, Mar 3, 2014 at 11:52 AM, R. Tyler Croy wrote: >> >> I'm wondering if anybody's explored using atf-c for adding unit tests to kernel >> level functions or driver code. > > We do this extensively in NetBSD: we have a mechanism (rump) to build > unmodified kernel-level code as user-space binaries and we later use > atf to unit-test the code in controlled tiny environments. (You'd > think about this as super-lightweight virtualization of the kernel.) > > Achieving this for any non-trivial kernel-level code is hard however, > but I believe that this approach has huge benefits. For example: we > are able to unit-test all file system and networking code in > user-space without any chances of crashing the system; there is also > the benefit of being able to control the test environment fully > without relying on any state of the running kernel. > > For more details see: > > http://en.wikipedia.org/wiki/Anykernel > http://wiki.netbsd.org/rumpkernel/ > > If the code you are interested in testing can easily be built without > ties to the kernel, you can relatively easily do the above. (In fact, > we used to do this in NetBSD for some tiny areas of the code before we > got rump.) > > At some point I also pondered the idea to write an interface and a > kernel module to allow writing unit tests within the kernel. This > might be interesting for some special cases (like testing live > hardware maybe) but it's a very fragile setup that should not be > pursued if other safer solutions suffice. Rump is definitely the ideal method. Lacking that, FreeBSD's options are to build kernel code in userspace, or to run unit tests in the kernel. Doing the latter is not currently possible with ATF. There are some places where it's currently done ad-hoc, such as in xnb(4), but it's not pretty. Compiling kernel code in userspace isn't always possible. When I wrote xnb(4), almost all of its unit tests depended on mbuf(9), and I found that there was no way to compile mbuf.h in userspace. ZFS is a rare exception. Almost all of the ZFS code is compiled in userspace as part of libzpool. That's how ztest works. My suggestion is to write new drivers to be compilable in userspace, rely on functional tests for existing drivers, and find somebody willing to port rump from NetBSD. -Alan