From owner-freebsd-fs@freebsd.org Thu Jan 28 07:23:12 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id CCB5A4FA93B for ; Thu, 28 Jan 2021 07:23:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4DRBlN5G97z4jQ6 for ; Thu, 28 Jan 2021 07:23:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id B49464FA754; Thu, 28 Jan 2021 07:23:12 +0000 (UTC) Delivered-To: fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B45904FAB2F for ; Thu, 28 Jan 2021 07:23:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DRBlN4Z9wz4jKT for ; Thu, 28 Jan 2021 07:23:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 8FC9D37D6 for ; Thu, 28 Jan 2021 07:23:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 10S7NCZW060374 for ; Thu, 28 Jan 2021 07:23:12 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 10S7NCp0060373 for fs@FreeBSD.org; Thu, 28 Jan 2021 07:23:12 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 224292] processes are hanging in state ufs / possible deadlock in file system Date: Thu, 28 Jan 2021 07:23:12 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: ali.abdallah@suse.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Jan 2021 07:23:12 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D224292 Ali Abdallah changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ali.abdallah@suse.com --- Comment #13 from Ali Abdallah --- While upgrading my FreeBSD 13-CURRENT box yesterday, I ran into this issue, when compiling the kernel and then world with -j8, I was ending up always w= ith all the cc processes stuck in biowr and ufs. At the end I've upgraded my box through NFS from my 13-CURRENT server (zfs based). But when upgrading ports from the same server, using pkg upgrade -f, it was also getting stuck with the same symptoms. I've managed to take a kernel du= mp, then booted into single user mode, disabled soft update journaling: (-j), = then booted the system and upgraded my packages, all went ok. Looking at the dump, I see lots of dd processes (launched by pkg I believe), that are in the following state: read 345 (Thread 100752 (PID=3D32594: dd)): #0 sched_switch (td=3D0xfffffe00fdd9be00, flags=3D) at /usr/src/sys/kern/sched_ule.c:2147 #1 0xffffffff803db4be in mi_switch (flags=3Dflags@entry=3D260) at /usr/src/sys/kern/kern_synch.c:542 #2 0xffffffff8042a43d in sleepq_switch (wchan=3Dwchan@entry=3D0x0, pri=3Dpri@entry=3D2122752) at /usr/src/sys/kern/subr_sleepqueue.c:608 #3 0xffffffff8042a327 in sleepq_wait (wchan=3D, pri=3D) at /usr/src/sys/kern/subr_sleepqueue.c:659 #4 0xffffffff803a9f7f in sleeplk (lk=3Dlk@entry=3D0xfffff8000535f810, flags=3Dflags@entry=3D2122752, ilk=3D, ilk@entry=3D0xfffff80= 00535f840, wmesg=3D, wmesg@entry=3D0xffffffff806c8ccb "ufs", pri=3D, pri@entry=3D96, timo=3Dtimo@entry=3D51, queue=3D1) at /usr/src/sys/kern/kern_lock.c:301 #5 0xffffffff803a88e2 in lockmgr_slock_hard (lk=3D0xfffff8000535f810, flags=3Dflags@entry=3D2122752, ilk=3D0xfffff8000535f840, ilk@entry=3D0xfffffe00fdaff658, file=3D, line=3D, lwa=3Dlwa@entry=3D0x0) at /usr/src/sys/kern/kern_lock.c:696 #6 0xffffffff803a84e8 in lockmgr_lock_flags (lk=3D, lk@entry=3D0xfffff8000535f810, flags=3Dflags@entry=3D2122752, ilk=3Dilk@entry=3D0xfffff8000535f840, file=3D, line=3D) at /usr/src/sys/kern/kern_lock.c:1056 #7 0xffffffff805c90e5 in ffs_lock (ap=3D0xfffffe00fdaff658, ap@entry=3D) at /usr/src/sys/ufs/ffs/ffs_vnops.c:494 #8 0xffffffff804be098 in VOP_LOCK1 (vp=3D0xfffff8000535f7a0, flags=3D21063= 68, file=3D0xffffffff806fac3e "/usr/src/sys/kern/vfs_lookup.c", line=3D877) at ./vnode_if.h:1096 #9 _vn_lock (vp=3Dvp@entry=3D0xfffff8000535f7a0, flags=3D2106368, file=3D0xffffffff806fac3e "/usr/src/sys/kern/vfs_lookup.c", line=3Dline@ent= ry=3D877) at /usr/src/sys/kern/vfs_vnops.c:1741 #10 0xffffffff80497a18 in lookup (ndp=3Dndp@entry=3D0xfffffe00fdaff970) at /usr/src/sys/kern/vfs_lookup.c:875 #11 0xffffffff804973dd in namei (ndp=3Dndp@entry=3D0xfffffe00fdaff970) at /usr/src/sys/kern/vfs_lookup.c:631 --Type for more, q to quit, c to continue without paging-- #12 0xffffffff804bd590 in vn_open_cred (ndp=3Dndp@entry=3D0xfffffe00fdaff97= 0, flagp=3D, flagp@entry=3D0xfffffe00fdaffa94, cmode=3D, vn_open_flags=3Dvn_open_flags@entry=3D0, cred=3D, fp=3D0xfffff80003d49140) at /usr/src/sys/kern/vfs_vnops.c:252 #13 0xffffffff804bd44d in vn_open (ndp=3D, ndp@entry=3D0xfffffe00fdaff970, flagp=3D, flagp@entry=3D0xfffffe00fdaffa94, cmode=3D, fp=3D= ) at /usr/src/sys/kern/vfs_vnops.c:193 #14 0xffffffff804b3723 in kern_openat (td=3D0xfffffe00fdd9be00, fd=3D, path=3D, pathseg=3D, flags=3D1539, mode=3D) at /usr/src/sys/kern/vfs_syscalls.c:1143 #15 0xffffffff8068d5cc in syscallenter (td=3D0xfffffe00fdd9be00) at /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:189 #16 amd64_syscall (td=3D0xfffffe00fdd9be00, traced=3D0) at /usr/src/sys/amd64/amd64/trap.c:1156 I will enable back -j, and kernel deadlocks debugging features, to hopefully get more debugging information. --=20 You are receiving this mail because: You are the assignee for the bug.=