From owner-freebsd-stable@FreeBSD.ORG Fri Jul 25 15:14:54 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 27D6DCC0 for ; Fri, 25 Jul 2014 15:14:54 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BD6F72C37 for ; Fri, 25 Jul 2014 15:14:53 +0000 (UTC) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id s6PFEmQV005076 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 25 Jul 2014 18:14:48 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua s6PFEmQV005076 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id s6PFEmQi005075; Fri, 25 Jul 2014 18:14:48 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 25 Jul 2014 18:14:48 +0300 From: Konstantin Belousov To: Harald Schmalzbauer Subject: Re: panic/lock on 9.3-RELEASE with nullfs/nfs/zfs combination Message-ID: <20140725151448.GY93733@kib.kiev.ua> References: <53D12973.3010805@omnilan.de> <20140724165917.GT93733@kib.kiev.ua> <53D1503B.2030200@omnilan.de> <20140724193048.GU93733@kib.kiev.ua> <53D2006C.7090207@omnilan.de> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="VmIAD9aKo2BNqjfq" Content-Disposition: inline In-Reply-To: <53D2006C.7090207@omnilan.de> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: FreeBSD Stable X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Jul 2014 15:14:54 -0000 --VmIAD9aKo2BNqjfq Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Jul 25, 2014 at 08:59:56AM +0200, Harald Schmalzbauer wrote: > Bez?glich Konstantin Belousov's Nachricht vom 24.07.2014 21:30 > (localtime): > > On Thu, Jul 24, 2014 at 08:28:11PM +0200, Harald Schmalzbauer wrote: > >> Bez??glich Konstantin Belousov's Nachricht vom 24.07.2014 18:59 > >> (localtime): > >>> ??? > >>> The lockmgr flags are LK_SHARE | LK_RETRY, and error 11 =3D=3D EDEADLK > >>> indicates that the lock is already taken by the curthread in the > >>> exclusive mode. I am interested in what line of code did the locking. > >>> > >>> Add ddb, INVARIANTS, WITNESS and DEBUG_VFS_LOCKS options to the kernel > >>> config, reproduce the issue and, after the panic occured and you > >>> get at the ddb prompt, issue command 'show alllocks'. > >>> > >>> Also, do 'show mount', after which do 'show mount ', where > >>> is the address of your nullfs mount point, printed by 'show mount'. > >>> > >>> I need all console output starting from the panic message. > >> FreeBSD/amd64 (mira.inop.mo1.omnilan.net) (ttyu0) > >> > >> login: panic: LK_RETRY set with incompatible flags (0x200400) or an > >> error occured (11) > >> cpuid =3D 1 > >> KDB: stack backtrace: > >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame > >> 0xffffff82e565fc70 > >> kdb_backtrace() at kdb_backtrace+0x37/frame 0xffffff82e565fd30 > >> panic() at panic+0x1cd/frame 0xffffff82e565fe30 > >> _vn_lock() at _vn_lock+0x77/frame 0xffffff82e565fe90 > >> zfs_lookup() at zfs_lookup+0x420/frame 0xffffff82e565ff20 > >> zfs_freebsd_lookup() at zfs_freebsd_lookup+0xa6/frame 0xffffff82e56600= 70 > >> VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0x102/frame > >> 0xffffff82e56600a0 > >> vfs_cache_lookup() at vfs_cache_lookup+0xff/frame 0xffffff82e5660110 > >> VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x102/frame 0xffffff82e5660140 > >> null_lookup() at null_lookup+0x92/frame 0xffffff82e56601c0 > >> VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x102/frame 0xffffff82e56601f0 > >> lookup() at lookup+0x32f/frame 0xffffff82e5660290 > >> namei() at namei+0x3df/frame 0xffffff82e5660340 > >> vn_open_cred() at vn_open_cred+0x1e2/frame 0xffffff82e56604b0 > >> vop_stdvptocnp() at vop_stdvptocnp+0x1af/frame 0xffffff82e56607e0 > >> null_vptocnp() at null_vptocnp+0xf5/frame 0xffffff82e5660850 > >> VOP_VPTOCNP_APV() at VOP_VPTOCNP_APV+0x105/frame 0xffffff82e5660880 > >> vn_vptocnp_locked() at vn_vptocnp_locked+0x15b/frame 0xffffff82e5660910 > >> vn_fullpath1() at vn_fullpath1+0x100/frame 0xffffff82e5660970 > >> kern___getcwd() at kern___getcwd+0xd4/frame 0xffffff82e56609d0 > >> amd64_syscall() at amd64_syscall+0x318/frame 0xffffff82e5660af0 > >> Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff82e5660af0 > >> --- syscall (326, FreeBSD ELF64, sys___getcwd), rip =3D 0x8011a191c, r= sp =3D > >> 0x7fffffffe658, rbp =3D 0x801873400 --- > >> KDB: enter: panic > >> [ thread pid 1918 tid 100919 ] > >> Stopped at kdb_enter+0x3b: movq $0,0x645622(%rip) > >> db> show alllocks > >> db> > You do not have WITNESS in your kernel config, most likely. >=20 > Hmm enabled it is, from the boot msg: > gcc version 4.2.1 20070831 patched [FreeBSD] > WARNING: WITNESS option enabled, expect reduced performance. > CPU: Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz (3292.52-MHz K8-class CPU) >=20 > But there's another WITNESS message: > WITNESS: unable to allocate a new witness object. > Ill ask the relatives what they know about this message (aunt bing, > uncle google aso). Yes, this is the problem. In sys/kern/subr_witness.c, change the #define WITNESS_COUNT from 1024 to e.g. 2048. --VmIAD9aKo2BNqjfq Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBAgAGBQJT0nRoAAoJEJDCuSvBvK1BTK8P/0li4bzqj3SKJ7PfCxLKWh9+ p7Ns/8JoTkYTESlvVVekUypCciHovMgZqxTWICtfQdM5Ibbgq4WdIo1kXq1VBcba VZrSkbGUp9zR2nWs5JKCmU/E6C6i3/6V9FtSxOHGYricpA5loZytOuFBhdT0sDQ3 S/1ZWb8CNmKpkFajnezvx6P9QYzVZ3Ms0cWT3cGieJhzplgvQHNOsi7Aqwr/9lJD ptGsetfm/P6tNDTIVPQXsklHdxwP5ZlxoDCyoAri/oqw/aCuHlkyye/paHd1wKWa pNYw8Xl2arodIAbT9lON6wVRpmSHdJAjkz6u79qmVZEOnP3jOr+DqRstuQScTsQU fTqbEAGu636cpFHxFg+49EbgW8Ah+16l/tnZGXWOSLfm2cYlQ3Qdk+Z2aRD4rS0h +5GSO4oHMFFHyVkD2Lp8adZlwxzHvbu+nK6SicVv9WQJyZCYJihFZL5T3PGoBff4 vlhTLWAd3+/Jn7dOnF189eEnuiKqyQrR8chL/QC6ZEwO0v+5zDWQ8RuElBED3fjR x1Xvjhl6pK1STdVWSOR+WGDk7cMcVWQyw31qso0HvSyAEfZfA1iG3ETTy/VqgEXH qqWPX5rQwjLQyoRJyn2s8cwJ9ld5I75xyn4RWlul847buax1B2YMqNxnPQH4TJbQ 4IESM9X8WpJClNsSBwtG =Z7Pa -----END PGP SIGNATURE----- --VmIAD9aKo2BNqjfq--