From owner-freebsd-stable@freebsd.org Tue Mar 7 23:55:59 2017 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 65299D02002 for ; Tue, 7 Mar 2017 23:55:59 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E79AA107C; Tue, 7 Mar 2017 23:55:58 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id v27NtoKd001740 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 8 Mar 2017 01:55:51 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua v27NtoKd001740 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id v27NtoS1001739; Wed, 8 Mar 2017 01:55:50 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 8 Mar 2017 01:55:50 +0200 From: Konstantin Belousov To: Rick Macklem Cc: Harry Schmalzbauer , FreeBSD Stable , Mark Johnston Subject: Re: unionfs bugs, a partial patch and some comments [Was: Re: 1-BETA3 Panic: __lockmgr_args: downgrade a recursed lockmgr nfs @ /usr/local/share/deploy-tools/RELENG_11/src/sys/fs/unionfs/union_vnops.c:1905] Message-ID: <20170307235550.GP30979@kib.kiev.ua> References: <57A9A6C0.9060609@omnilan.de> <20160812123950.GO83214@kib.kiev.ua> <57B8793E.4070004@omnilan.de> <58BEAAAC.4090303@omnilan.de> <58BEFF83.9010906@omnilan.de> <58BF0DE4.1020300@omnilan.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.8.0 (2017-02-23) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Mar 2017 23:55:59 -0000 On Tue, Mar 07, 2017 at 10:49:01PM +0000, Rick Macklem wrote: > Hmm, this is going to sound dumb, but I don't recall generating any > unionfs patch;-) > I'll go look for it. Maybe it was Kostik's? I did not touched unionfs, and have no plans to. It is equally broken in all relevant versions of FreeBSD. > > rick > ________________________________________ > From: Harry Schmalzbauer > Sent: Tuesday, March 7, 2017 2:45:40 PM > To: Rick Macklem > Cc: Konstantin Belousov; FreeBSD Stable; Mark Johnston; kib@FreeBSD.org > Subject: Re: unionfs bugs, a partial patch and some comments [Was: Re: 1-BETA3 Panic: __lockmgr_args: downgrade a recursed lockmgr nfs @ /usr/local/share/deploy-tools/RELENG_11/src/sys/fs/unionfs/union_vnops.c:1905] > > Bez?glich Harry Schmalzbauer's Nachricht vom 07.03.2017 19:44 (localtime): > > Bez?glich Harry Schmalzbauer's Nachricht vom 07.03.2017 13:42 (localtime): > > ? > >> Something ufs related seems to have tightened the unionfs locking > >> problem in stable/11. Now the machine instantaniously panics during > >> boot after mounting root with Rick's latest patch. > >> > >> Unfortunately I don't have SWAP available on that machine (yet), but > >> maybe shit is a hint for anybody. > >> > >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame > >> 0xfffffe00982220e0 > >> vpanic() at vpanic+0x186/frame 0xfffffe0098222160 > >> kassert_panic() at kassert_panic+0x126/frame 0xfffffe00982221d0 > >> witness_assert() at witness_assert+0x35a/frame 0xfffffe0098222230 > >> __lockmgr_args() at __lockmgr_args+0x517/frame 0xfffffe00982222d0 > >> vop_stdunlock() at vop_stdunlock+0x3b/frame 0xfffffe00982222f0 > >> VOP_UNLOCK_APV() at VOP_UNLOCK_APV+0xe0/frame 0xfffffe0098222320 > >> unionfs_unlock() at unionfs_unlock+0x112/frame 0xfffffe0098222390 > >> VOP_UNLOCK_APV() at VOP_UNLOCK_APV+0xe0/frame 0xfffffe00982223c0 > >> unionfs_nodeget() at unionfs_nodeget+0x3ef/frame 0xfffffe0098222470 > >> unionfs_domount() at unionfs_domount+0x518/frame 0xfffffe00982226b0 > >> vfs_donmount() at vfs_donmount+0xe37/frame 0xfffffe00982228f0 > >> sys_nmount() at sys_nmount+0x72/frame 0xfffffe0098222930 > >> amd64_syscall() at amd64_syscall+0x2f9/frame 0xfffffe0098222ab0 > >> Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0098222ab0 > >> --- syscall (378, FreeBSD ELF64, sys_nmount), rip = 0x80086ecea, rsp = > >> 0x7fffffffe318, rbp = 0x7fffffffeca0 --- > > New discovery: > > Rick's latest patch casues panic only with KDB. If I compile a kernel > > without witenss and KDB, the machine boots fine! > > Also, it's at least not so easy anymore to trigger the deadlock :-) . I > > need to do more testing but until now Rick's approach seems very > > promising :-) . > > My unionfs deadlock problem isn't really solved with Rick's latest > patch, I still can reproduce it: krb5.conf and krb5.keytab are files on > unionfs referenced by /etc. libexec/negotiate_kerberos_auth reads these > and if I have enough helper processes handling requests, the deadlock > occurs. > > _But_: If I move the files outside the unionfs and create a symlink, I > cannot reproduce the deadlock anymore, which was similar easily > reproducable without it or any of the other workarounds. > So it looks like I have an acceptable solution for now, although it's > only usable under certain conditions. > > Unfortunately I can't do tests with a debug kernel since the patch > prevents the system with the debug kernel from starting up. > But if this was ironed out, I'd happily provide more info. > > > Thanks, > > -Harry >