From owner-freebsd-stable@freebsd.org Mon Mar 6 22:31:35 2017 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D059DD00D6F for ; Mon, 6 Mar 2017 22:31:35 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id A482413C5 for ; Mon, 6 Mar 2017 22:31:35 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: by mailman.ysv.freebsd.org (Postfix) id A3459D00D6E; Mon, 6 Mar 2017 22:31:35 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A2DB0D00D6D for ; Mon, 6 Mar 2017 22:31:35 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-eopbgr660084.outbound.protection.outlook.com [40.107.66.84]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3F47413C4 for ; Mon, 6 Mar 2017 22:31:34 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM (10.165.218.133) by YTXPR01MB0191.CANPRD01.PROD.OUTLOOK.COM (10.165.218.135) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.947.12; Mon, 6 Mar 2017 22:31:31 +0000 Received: from YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM ([10.165.218.133]) by YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM ([10.165.218.133]) with mapi id 15.01.0947.018; Mon, 6 Mar 2017 22:31:31 +0000 From: Rick Macklem To: hiren panchasara , Harry Schmalzbauer CC: "stable@freebsd.org" Subject: Re: 'show alllocks' of completely locked machine [Was: Re: Complete IO lockup, state "ufs" from userland, debuging help wanted] Thread-Topic: 'show alllocks' of completely locked machine [Was: Re: Complete IO lockup, state "ufs" from userland, debuging help wanted] Thread-Index: AQHSlrO+zB4j4NMmykeiDtVre1Mil6GIPg4AgAAlPBE= Date: Mon, 6 Mar 2017 22:31:31 +0000 Message-ID: References: <58BC8A44.8020607@omnilan.de> <58BDBEE3.6080601@omnilan.de>,<20170306201042.GB71537@strugglingcoder.info> In-Reply-To: <20170306201042.GB71537@strugglingcoder.info> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: freebsd.org; dkim=none (message not signed) header.d=none;freebsd.org; dmarc=none action=none header.from=uoguelph.ca; x-ms-office365-filtering-correlation-id: 1dd272a7-6cae-4b35-7dc1-08d464e089bc x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001);SRVR:YTXPR01MB0191; x-microsoft-exchange-diagnostics: 1; YTXPR01MB0191; 7:djodsjXGNmQu/glgrGYdxcoOjfk0JStfpoxfEcvQNWAVP1zM5L5I6AvjSfXunfzgn1EMyd9UBmIPJeFb3jMlyIniC+sgH45LxdRcz6nNmkOLvgw7Jni10/1G8aQ025bBvS6gb+6dLfyHD+ck/2EwqUP8M/w9dbSqapmAfEu0rn0eKts4KnjKdVoOWXFFWXqwjn0VU0EaJp5hSBHvpvcW0PeAVVas3eo8ov1osseEzNLjhVtsoEHdRWKL6ssDFvT8FS9fgMZXPQQhbf4nwacuGG9jB2NGhea5PmeTD7TaPpig4hqOd2BCjwJNJ2iM4lOS0XBeeFzOhHyhU75RQVhwQw== x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040375)(2401047)(5005006)(8121501046)(10201501046)(3002001)(6041248)(20161123558025)(20161123562025)(20161123564025)(20161123555025)(20161123560025)(6072148); SRVR:YTXPR01MB0191; BCL:0; PCL:0; RULEID:; SRVR:YTXPR01MB0191; x-forefront-prvs: 0238AEEDB0 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(979002)(6009001)(7916002)(39410400002)(39450400003)(39830400002)(24454002)(2906002)(33656002)(305945005)(4326008)(3280700002)(54356999)(102836003)(2900100001)(92566002)(189998001)(50986999)(76176999)(74316002)(6246003)(9686003)(74482002)(229853002)(55016002)(86362001)(77096006)(106116001)(3660700001)(6506006)(81166006)(5660300001)(38730400002)(2950100002)(7696004)(122556002)(6436002)(8676002)(8936002)(53936002)(969003)(989001)(999001)(1009001)(1019001); DIR:OUT; SFP:1101; SCL:1; SRVR:YTXPR01MB0191; H:YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; MLV:ovrnspm; PTR:InfoNoRecords; LANG:en; spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-originalarrivaltime: 06 Mar 2017 22:31:31.0959 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTXPR01MB0191 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Mar 2017 22:31:35 -0000 hiren panchasara wrote: [sorry, I can't be bothered manually indenting it all, so just see my comme= nt at the end] On 03/06/17 at 08:56P, Harry Schmalzbauer wrote: > Bez?glich Harry Schmalzbauer's Nachricht vom 05.03.2017 22:59 (localtime= ): > > Hello, > > > > I can easily lock up FreeBSD stable/11 from userland. Not that I want t= o... > > I'm running squid, which starts an authentication helper > > "*negotiate_kerberos_auth*", which seems to be the culprit. > > Completely all IO is blocked, there's no way to get anything from any > > filesystem. > > All non IO-requesting processes(threads) run well, including sshd and > > shells. > > There's no load (neither cpu nor io) just any process requesting io > > stucks in state "ufs" > > > > Can anyone help me finding out what's going wrong? > > Serial console is available. > > Dear hackers, > > I managed to get into DDB, but I'm lost from there? > > What information could be usefull to find out the cause of this complete > lockup? > > I'd need someone who could guide me through ? I'd pay for a debuging > lesson! (quiet constrained budget though) > > This happens when the machine got stuck: > > intr_event_handle() at intr_event_handle+0x9c/frame 0xfffffe0093dcb7d0 > intr_execute_handlers() at intr_execute_handlers+0x48/frame > 0xfffffe0093dcb800 > lapic_handle_intr() at lapic_handle_intr+0x68/frame 0xfffffe0093dcb840 > Xapic_isr1() at Xapic_isr1+0xb7/frame 0xfffffe0093dcb840 > --- interrupt, rip =3D 0xffffffff807b9bd6, rsp =3D 0xfffffe0093dcb910, rb= p =3D > 0xfffffe0093dcb910 --- > acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe0093dcb910 > acpi_cpu_idle() at acpi_cpu_idle+0x2ea/frame 0xfffffe0093dcb960 > cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe0093dcb980 > cpu_idle() at cpu_idle+0x8f/frame 0xfffffe0093dcb9a0 > sched_idletd() at sched_idletd+0x436/frame 0xfffffe0093dcba70 > fork_exit() at fork_exit+0x84/frame 0xfffffe0093dcbab0 > fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0093dcbab0 > --- trap 0, rip =3D 0, rsp =3D 0, rbp =3D 0 --- > > > db> show alllocks > Process 1259 (negotiate_kerberos_) thread 0xfffff80005ddea00 (100096) > shared lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1258 (negotiate_kerberos_) thread 0xfffff80005ddc500 (100252) > shared lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1257 (negotiate_kerberos_) thread 0xfffff80005ddda00 (100247) > shared lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1256 (negotiate_kerberos_) thread 0xfffff80065612500 (100261) > shared lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1255 (negotiate_kerberos_) thread 0xfffff80065612a00 (100260) > shared lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1254 (negotiate_kerberos_) thread 0xfffff80065613000 (100257) > shared lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1253 (negotiate_kerberos_) thread 0xfffff80065614000 (100254) > shared lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1252 (negotiate_kerberos_) thread 0xfffff800651e1000 (100246) > shared lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1251 (negotiate_kerberos_) thread 0xfffff80005ddca00 (100251) > shared lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1250 (negotiate_kerberos_) thread 0xfffff800651e2a00 (100241) > shared lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1251 (negotiate_kerberos_) thread 0xfffff80005ddca00 (100251) > shared lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1250 (negotiate_kerberos_) thread 0xfffff800651e2a00 (100241) > shared lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1247 (sqtop) thread 0xfffff80065650a00 (100259) > shared lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1184 (systat) thread 0xfffff80065613a00 (100255) > shared lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1042 (negotiate_kerberos_) thread 0xfffff800651e2500 (100242) > shared lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1041 (negotiate_kerberos_) thread 0xfffff800055e4000 (100078) > shared lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 639 (cron) thread 0xfffff80005afa000 (100095) > shared lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 605 (log_file_daemon) thread 0xfffff800057eda00 (100088) > exclusive sx so_rcv_sx (so_rcv_sx) r =3D 0 (0xfffff80005b547b0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/uipc_sockbuf.c:283 > Process 604 (ext_ldap_group_acl) thread 0xfffff800057dc000 (100087) > exclusive sx so_rcv_sx (so_rcv_sx) r =3D 0 (0xfffff80005b550f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/uipc_sockbuf.c:283 > Process 603 (ext_ldap_group_acl) thread 0xfffff80005478000 (100063) > exclusive sx so_rcv_sx (so_rcv_sx) r =3D 0 (0xfffff80005b410f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/uipc_sockbuf.c:283 > Process 601 (negotiate_kerberos_) thread 0xfffff80005afc000 (100085) > shared lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 600 (negotiate_kerberos_) thread 0xfffff80005afc500 (100084) > shared lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 599 (negotiate_kerberos_) thread 0xfffff80005478a00 (100061) > shared lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 598 (negotiate_kerberos_) thread 0xfffff8000547a000 (100057) > shared lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 597 (negotiate_kerberos_) thread 0xfffff80005afca00 (100083) > exclusive lockmgr ufs (ufs) r =3D 0 (0xfffff800057555f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/fs/unionfs/union_vnops.c:= 1905 > Process 596 (negotiate_kerberos_) thread 0xfffff8000559a500 (100068) > exclusive lockmgr ufs (ufs) r =3D 0 (0xfffff800055125f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/fs/unionfs/union_vnops.c:= 1936 > exclusive lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d068) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/fs/unionfs/union_vnops.c:= 1905 > Process 595 (negotiate_kerberos_) thread 0xfffff800055e3a00 (100079) > shared lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 594 (negotiate_kerberos_) thread 0xfffff80005599000 (100072) > shared lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 unionfs is always broken (see the BUGS section of "man mount_unionfs"). I'd suggest you reconfigure the machine without unionfs. Btw, the "ufs" locks mean the process is waiting for a UFS vnode lock. I'm not sure about the sockbuf ones. If the processes waiting on those hold UFS vnode locks, then that might also explain the deadlock. rick=