From owner-freebsd-stable@freebsd.org Tue Mar 7 22:49:06 2017 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 05B98D02D03 for ; Tue, 7 Mar 2017 22:49:06 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670052.outbound.protection.outlook.com [40.107.67.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id ACA9A1D9E; Tue, 7 Mar 2017 22:49:04 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM (10.165.218.133) by YTXPR01MB0192.CANPRD01.PROD.OUTLOOK.COM (10.165.218.136) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.947.12; Tue, 7 Mar 2017 22:49:01 +0000 Received: from YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM ([10.165.218.133]) by YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM ([10.165.218.133]) with mapi id 15.01.0947.020; Tue, 7 Mar 2017 22:49:01 +0000 From: Rick Macklem To: Harry Schmalzbauer CC: Konstantin Belousov , FreeBSD Stable , Mark Johnston , "kib@FreeBSD.org" Subject: Re: unionfs bugs, a partial patch and some comments [Was: Re: 1-BETA3 Panic: __lockmgr_args: downgrade a recursed lockmgr nfs @ /usr/local/share/deploy-tools/RELENG_11/src/sys/fs/unionfs/union_vnops.c:1905] Thread-Topic: unionfs bugs, a partial patch and some comments [Was: Re: 1-BETA3 Panic: __lockmgr_args: downgrade a recursed lockmgr nfs @ /usr/local/share/deploy-tools/RELENG_11/src/sys/fs/unionfs/union_vnops.c:1905] Thread-Index: AQHR8Usp3UC6l4vlEEOdyPFAeIRbFKBAJI+AgAA/AgCAA/TscIAA8igAgAiYa2KABCvkAIAZfWvFgR8RkACAAGUjgIAAESQAgAAy89s= Date: Tue, 7 Mar 2017 22:49:01 +0000 Message-ID: References: <57A79E24.8000100@omnilan.de> <57A83C78.1070403@omnilan.de> <20160809060213.GA67664@raichu> <57A9A6C0.9060609@omnilan.de> , <20160812123950.GO83214@kib.kiev.ua> , <57B8793E.4070004@omnilan.de> <58BEAAAC.4090303@omnilan.de> <58BEFF83.9010906@omnilan.de>,<58BF0DE4.1020300@omnilan.de> In-Reply-To: <58BF0DE4.1020300@omnilan.de> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: omnilan.de; dkim=none (message not signed) header.d=none;omnilan.de; dmarc=none action=none header.from=uoguelph.ca; x-ms-office365-filtering-correlation-id: a40e9e66-4b69-480c-440b-08d465ac2668 x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001);SRVR:YTXPR01MB0192; x-microsoft-exchange-diagnostics: 1; YTXPR01MB0192; 7:sdWewN2Iz2r/onH9wQB2OI63CZ5HtdyXKnAIjpnMzqdSQP96jeZMUEC1uWi+HhESKp56nXi/g6CAmzurY93o3LJFkL4pguv8OY5JV13xf6JrpkFsQ/qZ8aT3vU5l3tVbldk7bDBfIltd5t9d5gqSn9o+M/xBs/b/Dh1S0tO5vkXnrpkJMBrA19jt6II5N8hw+5LXn+WFPjBS9R9TAujv9bUKGZ/Bz9zg4z4WBKrwsDJiHq9UHwtai2EPDFGsf897w3mBTV0JSzV8zhugKQbxmw6u8hKz4rnLl3GDUioaRaeWhvRfrnM0C03hPkJU4XSVRjvlOPMvkab2BmmElmSgsQ== x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040375)(2401047)(8121501046)(5005006)(3002001)(10201501046)(6041248)(20161123555025)(20161123562025)(20161123564025)(20161123558025)(20161123560025)(6072148); SRVR:YTXPR01MB0192; BCL:0; PCL:0; RULEID:; SRVR:YTXPR01MB0192; x-forefront-prvs: 0239D46DB6 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(979002)(6009001)(39450400003)(377454003)(9686003)(74316002)(229853002)(189998001)(6506006)(55016002)(54906002)(6436002)(8936002)(7696004)(3660700001)(33656002)(77096006)(86362001)(6246003)(38730400002)(5660300001)(53936002)(110136004)(2900100001)(93886004)(76176999)(74482002)(54356999)(50986999)(4326008)(3280700002)(106116001)(81166006)(6916009)(102836003)(2950100002)(305945005)(122556002)(39060400002)(53546006)(8676002)(2906002)(969003)(989001)(999001)(1009001)(1019001); DIR:OUT; SFP:1101; SCL:1; SRVR:YTXPR01MB0192; H:YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; MLV:ovrnspm; PTR:InfoNoRecords; LANG:en; spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-originalarrivaltime: 07 Mar 2017 22:49:01.7980 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTXPR01MB0192 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Mar 2017 22:49:06 -0000 Hmm, this is going to sound dumb, but I don't recall generating any unionfs patch;-) I'll go look for it. Maybe it was Kostik's? rick ________________________________________ From: Harry Schmalzbauer Sent: Tuesday, March 7, 2017 2:45:40 PM To: Rick Macklem Cc: Konstantin Belousov; FreeBSD Stable; Mark Johnston; kib@FreeBSD.org Subject: Re: unionfs bugs, a partial patch and some comments [Was: Re: 1-BE= TA3 Panic: __lockmgr_args: downgrade a recursed lockmgr nfs @ /usr/local/sh= are/deploy-tools/RELENG_11/src/sys/fs/unionfs/union_vnops.c:1905] Bez=FCglich Harry Schmalzbauer's Nachricht vom 07.03.2017 19:44 (localtime= ): > Bez=FCglich Harry Schmalzbauer's Nachricht vom 07.03.2017 13:42 (localti= me): > =85 >> Something ufs related seems to have tightened the unionfs locking >> problem in stable/11. Now the machine instantaniously panics during >> boot after mounting root with Rick's latest patch. >> >> Unfortunately I don't have SWAP available on that machine (yet), but >> maybe shit is a hint for anybody. >> >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame >> 0xfffffe00982220e0 >> vpanic() at vpanic+0x186/frame 0xfffffe0098222160 >> kassert_panic() at kassert_panic+0x126/frame 0xfffffe00982221d0 >> witness_assert() at witness_assert+0x35a/frame 0xfffffe0098222230 >> __lockmgr_args() at __lockmgr_args+0x517/frame 0xfffffe00982222d0 >> vop_stdunlock() at vop_stdunlock+0x3b/frame 0xfffffe00982222f0 >> VOP_UNLOCK_APV() at VOP_UNLOCK_APV+0xe0/frame 0xfffffe0098222320 >> unionfs_unlock() at unionfs_unlock+0x112/frame 0xfffffe0098222390 >> VOP_UNLOCK_APV() at VOP_UNLOCK_APV+0xe0/frame 0xfffffe00982223c0 >> unionfs_nodeget() at unionfs_nodeget+0x3ef/frame 0xfffffe0098222470 >> unionfs_domount() at unionfs_domount+0x518/frame 0xfffffe00982226b0 >> vfs_donmount() at vfs_donmount+0xe37/frame 0xfffffe00982228f0 >> sys_nmount() at sys_nmount+0x72/frame 0xfffffe0098222930 >> amd64_syscall() at amd64_syscall+0x2f9/frame 0xfffffe0098222ab0 >> Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0098222ab0 >> --- syscall (378, FreeBSD ELF64, sys_nmount), rip =3D 0x80086ecea, rsp = =3D >> 0x7fffffffe318, rbp =3D 0x7fffffffeca0 --- > New discovery: > Rick's latest patch casues panic only with KDB. If I compile a kernel > without witenss and KDB, the machine boots fine! > Also, it's at least not so easy anymore to trigger the deadlock :-) . I > need to do more testing but until now Rick's approach seems very > promising :-) . My unionfs deadlock problem isn't really solved with Rick's latest patch, I still can reproduce it: krb5.conf and krb5.keytab are files on unionfs referenced by /etc. libexec/negotiate_kerberos_auth reads these and if I have enough helper processes handling requests, the deadlock occurs. _But_: If I move the files outside the unionfs and create a symlink, I cannot reproduce the deadlock anymore, which was similar easily reproducable without it or any of the other workarounds. So it looks like I have an acceptable solution for now, although it's only usable under certain conditions. Unfortunately I can't do tests with a debug kernel since the patch prevents the system with the debug kernel from starting up. But if this was ironed out, I'd happily provide more info. Thanks, -Harry