From owner-freebsd-stable@freebsd.org Mon Mar 6 20:10:48 2017 Return-Path: <owner-freebsd-stable@freebsd.org> Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C0114CFC54C for <freebsd-stable@mailman.ysv.freebsd.org>; Mon, 6 Mar 2017 20:10:48 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id ACE281A8F for <freebsd-stable@freebsd.org>; Mon, 6 Mar 2017 20:10:48 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: by mailman.ysv.freebsd.org (Postfix) id AC348CFC54B; Mon, 6 Mar 2017 20:10:48 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ABE07CFC54A for <stable@mailman.ysv.freebsd.org>; Mon, 6 Mar 2017 20:10:48 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: from mail.strugglingcoder.info (strugglingcoder.info [104.236.146.68]) by mx1.freebsd.org (Postfix) with ESMTP id 9DF601A8D for <stable@freebsd.org>; Mon, 6 Mar 2017 20:10:48 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: from localhost (unknown [10.1.1.3]) (Authenticated sender: hiren@strugglingcoder.info) by mail.strugglingcoder.info (Postfix) with ESMTPA id C7C8D17964; Mon, 6 Mar 2017 12:10:42 -0800 (PST) Date: Mon, 6 Mar 2017 12:10:42 -0800 From: hiren panchasara <hiren@strugglingcoder.info> To: Harry Schmalzbauer <freebsd@omnilan.de> Cc: stable@freebsd.org Subject: Re: 'show alllocks' of completely locked machine [Was: Re: Complete IO lockup, state "ufs" from userland, debuging help wanted] Message-ID: <20170306201042.GB71537@strugglingcoder.info> References: <58BC8A44.8020607@omnilan.de> <58BDBEE3.6080601@omnilan.de> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="qjNfmADvan18RZcF" Content-Disposition: inline In-Reply-To: <58BDBEE3.6080601@omnilan.de> User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org> List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-stable>, <mailto:freebsd-stable-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-stable/> List-Post: <mailto:freebsd-stable@freebsd.org> List-Help: <mailto:freebsd-stable-request@freebsd.org?subject=help> List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-stable>, <mailto:freebsd-stable-request@freebsd.org?subject=subscribe> X-List-Received-Date: Mon, 06 Mar 2017 20:10:48 -0000 --qjNfmADvan18RZcF Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 03/06/17 at 08:56P, Harry Schmalzbauer wrote: > Bez?glich Harry Schmalzbauer's Nachricht vom 05.03.2017 22:59 (localtime= ): > > Hello, > > > > I can easily lock up FreeBSD stable/11 from userland. Not that I want t= o... > > I'm running squid, which starts an authentication helper > > "*negotiate_kerberos_auth*", which seems to be the culprit. > > Completely all IO is blocked, there's no way to get anything from any > > filesystem. > > All non IO-requesting processes(threads) run well, including sshd and > > shells. > > There's no load (neither cpu nor io) just any process requesting io > > stucks in state "ufs" > > > > Can anyone help me finding out what's going wrong? > > Serial console is available. >=20 > Dear hackers, >=20 > I managed to get into DDB, but I'm lost from there? >=20 > What information could be usefull to find out the cause of this complete > lockup? >=20 > I'd need someone who could guide me through ? I'd pay for a debuging > lesson! (quiet constrained budget though) >=20 > This happens when the machine got stuck: >=20 > intr_event_handle() at intr_event_handle+0x9c/frame 0xfffffe0093dcb7d0 > intr_execute_handlers() at intr_execute_handlers+0x48/frame > 0xfffffe0093dcb800 > lapic_handle_intr() at lapic_handle_intr+0x68/frame 0xfffffe0093dcb840 > Xapic_isr1() at Xapic_isr1+0xb7/frame 0xfffffe0093dcb840 > --- interrupt, rip =3D 0xffffffff807b9bd6, rsp =3D 0xfffffe0093dcb910, rb= p =3D > 0xfffffe0093dcb910 --- > acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe0093dcb910 > acpi_cpu_idle() at acpi_cpu_idle+0x2ea/frame 0xfffffe0093dcb960 > cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe0093dcb980 > cpu_idle() at cpu_idle+0x8f/frame 0xfffffe0093dcb9a0 > sched_idletd() at sched_idletd+0x436/frame 0xfffffe0093dcba70 > fork_exit() at fork_exit+0x84/frame 0xfffffe0093dcbab0 > fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0093dcbab0 > --- trap 0, rip =3D 0, rsp =3D 0, rbp =3D 0 --- >=20 >=20 > db> show alllocks > Process 1259 (negotiate_kerberos_) thread 0xfffff80005ddea00 (100096) > shared lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1258 (negotiate_kerberos_) thread 0xfffff80005ddc500 (100252) > shared lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1257 (negotiate_kerberos_) thread 0xfffff80005ddda00 (100247) > shared lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1256 (negotiate_kerberos_) thread 0xfffff80065612500 (100261) > shared lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1255 (negotiate_kerberos_) thread 0xfffff80065612a00 (100260) > shared lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1254 (negotiate_kerberos_) thread 0xfffff80065613000 (100257) > shared lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1253 (negotiate_kerberos_) thread 0xfffff80065614000 (100254) > shared lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1252 (negotiate_kerberos_) thread 0xfffff800651e1000 (100246) > shared lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1251 (negotiate_kerberos_) thread 0xfffff80005ddca00 (100251) > shared lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1250 (negotiate_kerberos_) thread 0xfffff800651e2a00 (100241) > shared lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1251 (negotiate_kerberos_) thread 0xfffff80005ddca00 (100251) > shared lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1250 (negotiate_kerberos_) thread 0xfffff800651e2a00 (100241) > shared lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1247 (sqtop) thread 0xfffff80065650a00 (100259) > shared lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1184 (systat) thread 0xfffff80065613a00 (100255) > shared lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1042 (negotiate_kerberos_) thread 0xfffff800651e2500 (100242) > shared lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1041 (negotiate_kerberos_) thread 0xfffff800055e4000 (100078) > shared lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 639 (cron) thread 0xfffff80005afa000 (100095) > shared lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 605 (log_file_daemon) thread 0xfffff800057eda00 (100088) > exclusive sx so_rcv_sx (so_rcv_sx) r =3D 0 (0xfffff80005b547b0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/uipc_sockbuf.c:283 > Process 604 (ext_ldap_group_acl) thread 0xfffff800057dc000 (100087) > exclusive sx so_rcv_sx (so_rcv_sx) r =3D 0 (0xfffff80005b550f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/uipc_sockbuf.c:283 > Process 603 (ext_ldap_group_acl) thread 0xfffff80005478000 (100063) > exclusive sx so_rcv_sx (so_rcv_sx) r =3D 0 (0xfffff80005b410f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/uipc_sockbuf.c:283 > Process 601 (negotiate_kerberos_) thread 0xfffff80005afc000 (100085) > shared lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 600 (negotiate_kerberos_) thread 0xfffff80005afc500 (100084) > shared lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 599 (negotiate_kerberos_) thread 0xfffff80005478a00 (100061) > shared lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 598 (negotiate_kerberos_) thread 0xfffff8000547a000 (100057) > shared lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 597 (negotiate_kerberos_) thread 0xfffff80005afca00 (100083) > exclusive lockmgr ufs (ufs) r =3D 0 (0xfffff800057555f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/fs/unionfs/union_vnops.c:= 1905 > Process 596 (negotiate_kerberos_) thread 0xfffff8000559a500 (100068) > exclusive lockmgr ufs (ufs) r =3D 0 (0xfffff800055125f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/fs/unionfs/union_vnops.c:= 1936 > exclusive lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d068) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/fs/unionfs/union_vnops.c:= 1905 > Process 595 (negotiate_kerberos_) thread 0xfffff800055e3a00 (100079) > shared lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 594 (negotiate_kerberos_) thread 0xfffff80005599000 (100072) > shared lockmgr ufs (ufs) r =3D 0 (0xfffff8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 IMO, you'd want to take a crashdump so you can analyze it better. But first while you are at db> prompt, db> show allchains db> show sleepchain Paste that o/p here and then take the dump: db> call doadump Then you can reboot the box: db> reboot Once it comes back up, you can look at the crashdump (usually at /var/crash/) # kgdb /boot/kernel/kernel /var/crash/<vmcore>.N In such situations, https://github.com/bsdjhb/kdbg is really useful.=20 Cheers, Hiren --qjNfmADvan18RZcF Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQF8BAABCgBmBQJYvcI/XxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRBNEUyMEZBMUQ4Nzg4RjNGMTdFNjZGMDI4 QjkyNTBFMTU2M0VERkU1AAoJEIuSUOFWPt/lA/YH/3SFYAKpLHr9sgg4Zp7nsjLj aSACUMkgPmhHCi1cvQEM1qKtBM8tYaDoXpxiU1hka9TA2JrTyu4TRgpvo+p+YNyf S5wevMeIRZ++myQ7IbgPexsTtVIUHxTKC18xXw/Sl0ovK3Ae8dgUAigjXtKhbQQ0 /Pu3SnDrfnRQHHOEqyfwQsZrPF+X1lrP9Zrp6eVNc0HvbMtcELy0WY4G3PtbphqE viqLWqjvMhKpIl78YrAH611ucl4mveBvdjksr8sHJRpRhx3rRaXXQ52SLg8ttZyK SRXCMZa8kRufLe5JPRa0DgyYtUcNCagaJJsdVcjEBtB1YywB7pPhKdEJGZAfC0g= =kRSk -----END PGP SIGNATURE----- --qjNfmADvan18RZcF--