From owner-freebsd-fs@FreeBSD.ORG  Tue Oct 12 15:27:00 2010
Return-Path: <owner-freebsd-fs@FreeBSD.ORG>
Delivered-To: freebsd-fs@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id A7EA01065740
	for <freebsd-fs@freebsd.org>; Tue, 12 Oct 2010 15:27:00 +0000 (UTC)
	(envelope-from avg@icyb.net.ua)
Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140])
	by mx1.freebsd.org (Postfix) with ESMTP id ECE7F8FC14
	for <freebsd-fs@freebsd.org>; Tue, 12 Oct 2010 15:26:59 +0000 (UTC)
Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua
	[212.40.38.101])
	by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id SAA21523;
	Tue, 12 Oct 2010 18:26:55 +0300 (EEST)
	(envelope-from avg@icyb.net.ua)
Message-ID: <4CB47E3F.3050002@icyb.net.ua>
Date: Tue, 12 Oct 2010 18:26:55 +0300
From: Andriy Gapon <avg@icyb.net.ua>
User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US;
	rv:1.9.2.9) Gecko/20100920 Lightning/1.0b2 Thunderbird/3.1.4
MIME-Version: 1.0
To: Jeremy Chadwick <freebsd@jdc.parodius.com>
References: <20101011151508.GA10917@icarus.home.lan>
	<4CB32C75.2060000@icyb.net.ua>
	<20101011183707.GA13925@icarus.home.lan>
	<4CB3870F.7070107@icyb.net.ua>
	<20101012100709.GA29861@icarus.home.lan>
	<4CB4429C.9040109@icyb.net.ua>
	<20101012130245.GA32584@icarus.home.lan>
	<4CB46CE9.20905@icyb.net.ua>
	<20101012143559.GA34396@icarus.home.lan>
	<4CB47355.1050109@icyb.net.ua>
	<20101012151852.GA35014@icarus.home.lan>
In-Reply-To: <20101012151852.GA35014@icarus.home.lan>
X-Enigmail-Version: 1.1.2
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Cc: freebsd-fs@freebsd.org
Subject: Re: Locked up processes after upgrade to ZFS v15
X-BeenThere: freebsd-fs@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Filesystems <freebsd-fs.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-fs>,
	<mailto:freebsd-fs-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-fs>
List-Post: <mailto:freebsd-fs@freebsd.org>
List-Help: <mailto:freebsd-fs-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-fs>,
	<mailto:freebsd-fs-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 12 Oct 2010 15:27:00 -0000

on 12/10/2010 18:18 Jeremy Chadwick said the following:
> Got it -- just finished and is currently running/working.  I also
> installed ports/sysutils/DTraceToolkit and shells/ksh93 "just in case".
> 
> testbox# dtrace -l | head
>    ID   PROVIDER            MODULE                          FUNCTION NAME
>     1     dtrace                                                     BEGIN
>     2     dtrace                                                     END
>     3     dtrace                                                     ERROR
>     4   dtmalloc                                                 fbt malloc
>     5   dtmalloc                                                 fbt free
>     6   dtmalloc                                              cyclic malloc
>     7   dtmalloc                                              cyclic free
>     8   dtmalloc                                          zones_data malloc
>     9   dtmalloc                                          zones_data free
> 
> I can provide you root-level access to the box as well as serial console
> if you'd prefer to do the debugging yourself, otherwise step me through
> what's needed and I'll be happy to act as remote hands.

Great!  Let's start now :)
I would like you to run the following script with "dtrace -s <script name>" in one
terminal while running sendfile patched regression test (with TEST_EXTRA=100) in
another.  After sendfile program finishes, please ^C the DTrace script.
Please show me complete output that you'll get from the DTrace script.
Thanks!

fbt::vm_fault:entry
/execname == "sendfile"/
{
        self->vm_fault = 1;
}

fbt::vm_fault:return
/execname == "sendfile"/
{
        self->vm_fault = 0;
}

fbt::zfs_freebsd_read:entry
/self->vm_fault/
{
        self->zfs_read = 1;
}

fbt::zfs_freebsd_read:return
/self->vm_fault/
{
        self->zfs_read = 0;
}

fbt::vm_page_lookup:return
/self->zfs_read && arg1 != 0/
{
        @stacks[stack()] = count();
        printf("\n");
        printf("valid = 0x%02x\n", ((vm_page_t)arg1)->valid);
        printf("flags = 0x%04x\n", ((vm_page_t)arg1)->flags);
        printf("oflags = 0x%04x\n", ((vm_page_t)arg1)->oflags);
        printf("pindex = %u\n", ((vm_page_t)arg1)->pindex);
        printf("object = %p\n", ((vm_page_t)arg1)->object);
}

> This should prove educational in another way, as I use Solaris regularly
> at work but have never sat down to tinker with DTrace (just basic mdb
> and adb).  Fun fun fun.

Life has curious ways :)

-- 
Andriy Gapon