From owner-freebsd-current@FreeBSD.ORG Mon Aug 16 18:55:01 2010 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 76AEC106566C for ; Mon, 16 Aug 2010 18:55:01 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id 0D5A58FC20 for ; Mon, 16 Aug 2010 18:55:00 +0000 (UTC) Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id o7GIsu0g009772 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 16 Aug 2010 21:54:56 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4) with ESMTP id o7GIsugv020619; Mon, 16 Aug 2010 21:54:56 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4/Submit) id o7GIsuKn020618; Mon, 16 Aug 2010 21:54:56 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 16 Aug 2010 21:54:56 +0300 From: Kostik Belousov To: pluknet Message-ID: <20100816185456.GU2396@deviant.kiev.zoral.com.ua> References: Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="X1pyOwMoA9G3DWfK" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-2.2 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_50, DNS_FROM_OPENWHOIS autolearn=no version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: FreeBSD Current Subject: Re: LOR on nfs: vfs_vnops.c:301 kern_descrip.c:1580 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Aug 2010 18:55:01 -0000 --X1pyOwMoA9G3DWfK Content-Type: text/plain; charset=koi8-r Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Aug 16, 2010 at 09:07:24PM +0400, pluknet wrote: > On 16 August 2010 21:05, pluknet wrote: > > Hi. > > > > Seeing on mostly idle, recently updated current, while closing a file. > > Presumably never reported on ML. > > > > lock order reversal: > > =9A1st 0xffffff00198199f8 nfs (nfs) @ /usr/src/sys/kern/vfs_vnops.c:301 > > =9A2nd 0xffffff000234a048 filedesc structure (filedesc structure) @ > > /usr/src/sys/kern/kern_descrip.c:1580 > > KDB: stack backtrace: > > db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > > _witness_debugger() at _witness_debugger+0x2e > > witness_checkorder() at witness_checkorder+0x807 > > _sx_xlock() at _sx_xlock+0x55 > > fdinit() at fdinit+0x5b > > fdcopy() at fdcopy+0x2a > > fork1() at fork1+0x836 > > kproc_create() at kproc_create+0x63 > > nfs_nfsiodnew() at nfs_nfsiodnew+0xd7 > > nfs_asyncio() at nfs_asyncio+0xa6 > > nfs_strategy() at nfs_strategy+0x83 > > bufstrategy() at bufstrategy+0x43 > > nfs_writebp() at nfs_writebp+0xcf > > nfs_flush() at nfs_flush+0x1dc > > nfs_close() at nfs_close+0x213 > > vn_close() at vn_close+0x10e > > vn_closefile() at vn_closefile+0x5a > > _fdrop() at _fdrop+0x23 > > closef() at closef+0x5b > > kern_close() at kern_close+0x110 > > syscallenter() at syscallenter+0x1aa > > syscall() at syscall+0x4c > > Xfast_syscall() at Xfast_syscall+0xe2 > > --- syscall (6, FreeBSD ELF64, close), rip =3D 0x80089830c, rsp =3D > > 0x7fffffffea88, rbp =3D 0 --- > > > > >=20 > Mostly the same (different 2nd lock path). >=20 > lock order reversal: > 1st 0xffffff00198199f8 nfs (nfs) @ /usr/src/sys/kern/vfs_vnops.c:301 > 2nd 0xffffffff80ca47e0 proctree (proctree) @ /usr/src/sys/kern/kern_fork= .c:335 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > _witness_debugger() at _witness_debugger+0x2e > witness_checkorder() at witness_checkorder+0x807 > _sx_slock() at _sx_slock+0x55 > fork1() at fork1+0x190 > kproc_create() at kproc_create+0x63 > nfs_nfsiodnew() at nfs_nfsiodnew+0xd7 > nfs_asyncio() at nfs_asyncio+0xa6 > nfs_strategy() at nfs_strategy+0x83 > bufstrategy() at bufstrategy+0x43 > nfs_writebp() at nfs_writebp+0xcf > nfs_flush() at nfs_flush+0x1dc > nfs_close() at nfs_close+0x213 > vn_close() at vn_close+0x10e > vn_closefile() at vn_closefile+0x5a > _fdrop() at _fdrop+0x23 > closef() at closef+0x5b > kern_close() at kern_close+0x110 > syscallenter() at syscallenter+0x1aa > syscall() at syscall+0x4c > Xfast_syscall() at Xfast_syscall+0xe2 > --- syscall (6, FreeBSD ELF64, close), rip =3D 0x80089830c, rsp =3D > 0x7fffffffea88, rbp =3D 0 --- >=20 Both LORs are valid. The fork performed deep inside the VFS call stack is obviously problematic. As a workaround, you may fix the number of nfsiods. Proper fix might consist of creating a shepherd thread which only task is to act on the requests on creating new nfsiods. Would you try to implement this ? I will provide the assistance, if needed. --X1pyOwMoA9G3DWfK Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (FreeBSD) iEYEARECAAYFAkxpiX8ACgkQC3+MBN1Mb4hvegCgrnKToTzUoPP1aF9vdPsc7NZ8 u9QAoLhcPEITehKNH1Q21vDsI79dxwMA =LVpk -----END PGP SIGNATURE----- --X1pyOwMoA9G3DWfK--