From owner-freebsd-fs@FreeBSD.ORG Tue Oct 12 16:10:41 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A6C611065694 for ; Tue, 12 Oct 2010 16:10:41 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id CD8AE8FC12 for ; Tue, 12 Oct 2010 16:10:40 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id TAA22173; Tue, 12 Oct 2010 19:10:37 +0300 (EEST) (envelope-from avg@icyb.net.ua) Message-ID: <4CB4887C.3050309@icyb.net.ua> Date: Tue, 12 Oct 2010 19:10:36 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.2.9) Gecko/20100920 Lightning/1.0b2 Thunderbird/3.1.4 MIME-Version: 1.0 To: Jeremy Chadwick References: <20101011183707.GA13925@icarus.home.lan> <4CB3870F.7070107@icyb.net.ua> <20101012100709.GA29861@icarus.home.lan> <4CB4429C.9040109@icyb.net.ua> <20101012130245.GA32584@icarus.home.lan> <4CB46CE9.20905@icyb.net.ua> <20101012143559.GA34396@icarus.home.lan> <4CB47355.1050109@icyb.net.ua> <20101012151852.GA35014@icarus.home.lan> <4CB47E3F.3050002@icyb.net.ua> <20101012154543.GA35378@icarus.home.lan> In-Reply-To: <20101012154543.GA35378@icarus.home.lan> X-Enigmail-Version: 1.1.2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: Locked up processes after upgrade to ZFS v15 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Oct 2010 16:10:41 -0000 on 12/10/2010 18:45 Jeremy Chadwick said the following: > Okay, I realised what I did wrong with the original incarnation of > your modified sendfile stuff -- the code defaults to using /tmp, which > idiotically I forgot to change to a ZFS filesystem (/tmp isn't ZFS > on the testbox). Now that I changed it to /home, I can reproduce the > problem. Excellent! OK, that explains your original results. Thanks for pinpointing the issue. > Secondly: the testbox is running kernel/world source from October 8th. > I *have not* applied your kernel patch at this point. Just a FYI. OK, can you now retest with the patch? I plan on committing it in the next couple of hours. > So here's what I get. Note that the sendfile process appears locked up > in zfsmrb state. > > > Terminal #1 (sendfile) > ------------------------ > testbox# ./sendfile > 1..11 > ok 1 > ok 2 > ok 3 > ok 4 > ok 5 > ok 6 > ok 7 > ok 8 > ok 9 > ok 10 > ok 11 > mmap test > ^C > > > Terminal #2 (DTrace script + ps output) > ----------------------------------------- > testbox# ./zfs_sendfile.d > dtrace: script './zfs_sendfile.d' matched 5 probes > CPU ID FUNCTION:NAME > 1 22458 vm_page_lookup:return > valid = 0x01 > flags = 0x0000 > oflags = 0x0001 > pindex = 4 > object = c614e550 > > ^C > > > 0xc457b43d > kernel`VOP_READ_APV+0x7a > kernel`vnode_pager_generic_getpages+0x329 > kernel`vop_stdgetpages+0x29 > kernel`VOP_GETPAGES_APV+0x83 > kernel`vnode_pager_getpages+0x19a > kernel`vm_fault+0x1139 > kernel`trap_pfault+0x173 > kernel`trap+0x2cb > kernel`0xc07d29bc > 1 Excellent! Behold the power of DTrace :-) > testbox# ps -axl | grep sendfile > 0 1318 1132 0 52 0 3324 1024 zfsmrb DL+ u0 0:00.01 ./sendfile > 0 1333 1170 0 44 0 3444 1200 - R+ 0 0:00.00 grep sendfile > testbox# procstat -k -k 1318 > PID TID COMM TDNAME KSTACK > 1318 100126 sendfile - mi_switch+0x11b sleepq_switch+0xc1 sleepq_wait+0x39 _sleep+0x282 vm_page_sleep+0xd5 zfs_freebsd_read+0x2f3 VOP_READ_APV+0x7a vnode_pager_generic_getpages+0x329 vop_stdgetpages+0x29 VOP_GETPAGES_APV+0x83 vnode_pager_getpages+0x19a vm_fault+0x1139 trap_pfault+0x173 trap+0x2cb calltrap+0x6 > -- Andriy Gapon