From owner-freebsd-current@FreeBSD.ORG Tue Aug 17 13:56:42 2010 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0C8BE106566C for ; Tue, 17 Aug 2010 13:56:42 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id BD18A8FC15 for ; Tue, 17 Aug 2010 13:56:41 +0000 (UTC) Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id 3F76346BA0; Tue, 17 Aug 2010 09:56:41 -0400 (EDT) Received: from jhbbsd.localnet (smtp.hudson-trading.com [209.249.190.9]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 3A6DE8A04E; Tue, 17 Aug 2010 09:56:40 -0400 (EDT) From: John Baldwin To: freebsd-current@freebsd.org Date: Tue, 17 Aug 2010 09:54:09 -0400 User-Agent: KMail/1.13.5 (FreeBSD/7.3-CBSD-20100217; KDE/4.4.5; amd64; ; ) References: <20100816185456.GU2396@deviant.kiev.zoral.com.ua> In-Reply-To: <20100816185456.GU2396@deviant.kiev.zoral.com.ua> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Message-Id: <201008170954.09041.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.0.1 (bigwig.baldwin.cx); Tue, 17 Aug 2010 09:56:40 -0400 (EDT) X-Virus-Scanned: clamav-milter 0.95.1 at bigwig.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-2.6 required=4.2 tests=AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on bigwig.baldwin.cx Cc: Kostik Belousov , pluknet Subject: Re: LOR on nfs: vfs_vnops.c:301 kern_descrip.c:1580 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Aug 2010 13:56:42 -0000 On Monday, August 16, 2010 2:54:56 pm Kostik Belousov wrote: > On Mon, Aug 16, 2010 at 09:07:24PM +0400, pluknet wrote: > > On 16 August 2010 21:05, pluknet wrote: > > > Hi. > > > > > > Seeing on mostly idle, recently updated current, while closing a file. > > > Presumably never reported on ML. > > > > > > lock order reversal: > > > 1st 0xffffff00198199f8 nfs (nfs) @ /usr/src/sys/kern/vfs_vnops.c:301 > > > 2nd 0xffffff000234a048 filedesc structure (filedesc structure) @ > > > /usr/src/sys/kern/kern_descrip.c:1580 > > > KDB: stack backtrace: > > > db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > > > _witness_debugger() at _witness_debugger+0x2e > > > witness_checkorder() at witness_checkorder+0x807 > > > _sx_xlock() at _sx_xlock+0x55 > > > fdinit() at fdinit+0x5b > > > fdcopy() at fdcopy+0x2a > > > fork1() at fork1+0x836 > > > kproc_create() at kproc_create+0x63 > > > nfs_nfsiodnew() at nfs_nfsiodnew+0xd7 > > > nfs_asyncio() at nfs_asyncio+0xa6 > > > nfs_strategy() at nfs_strategy+0x83 > > > bufstrategy() at bufstrategy+0x43 > > > nfs_writebp() at nfs_writebp+0xcf > > > nfs_flush() at nfs_flush+0x1dc > > > nfs_close() at nfs_close+0x213 > > > vn_close() at vn_close+0x10e > > > vn_closefile() at vn_closefile+0x5a > > > _fdrop() at _fdrop+0x23 > > > closef() at closef+0x5b > > > kern_close() at kern_close+0x110 > > > syscallenter() at syscallenter+0x1aa > > > syscall() at syscall+0x4c > > > Xfast_syscall() at Xfast_syscall+0xe2 > > > --- syscall (6, FreeBSD ELF64, close), rip = 0x80089830c, rsp = > > > 0x7fffffffea88, rbp = 0 --- > > > > > > > > > > Mostly the same (different 2nd lock path). > > > > lock order reversal: > > 1st 0xffffff00198199f8 nfs (nfs) @ /usr/src/sys/kern/vfs_vnops.c:301 > > 2nd 0xffffffff80ca47e0 proctree (proctree) @ /usr/src/sys/kern/kern_fork.c:335 > > KDB: stack backtrace: > > db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > > _witness_debugger() at _witness_debugger+0x2e > > witness_checkorder() at witness_checkorder+0x807 > > _sx_slock() at _sx_slock+0x55 > > fork1() at fork1+0x190 > > kproc_create() at kproc_create+0x63 > > nfs_nfsiodnew() at nfs_nfsiodnew+0xd7 > > nfs_asyncio() at nfs_asyncio+0xa6 > > nfs_strategy() at nfs_strategy+0x83 > > bufstrategy() at bufstrategy+0x43 > > nfs_writebp() at nfs_writebp+0xcf > > nfs_flush() at nfs_flush+0x1dc > > nfs_close() at nfs_close+0x213 > > vn_close() at vn_close+0x10e > > vn_closefile() at vn_closefile+0x5a > > _fdrop() at _fdrop+0x23 > > closef() at closef+0x5b > > kern_close() at kern_close+0x110 > > syscallenter() at syscallenter+0x1aa > > syscall() at syscall+0x4c > > Xfast_syscall() at Xfast_syscall+0xe2 > > --- syscall (6, FreeBSD ELF64, close), rip = 0x80089830c, rsp = > > 0x7fffffffea88, rbp = 0 --- > > > Both LORs are valid. The fork performed deep inside the VFS call stack > is obviously problematic. As a workaround, you may fix the number of > nfsiods. > > Proper fix might consist of creating a shepherd thread which only task > is to act on the requests on creating new nfsiods. > > Would you try to implement this ? I will provide the assistance, if needed. Or queue a task to create new nfsiod threads instead of invoking kproc_create() directly. That can be cheaper than creating a dedicated shepherd thread. -- John Baldwin