From owner-freebsd-current@FreeBSD.ORG Sat Jan 25 04:47:18 2014 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 57ACA2F7 for ; Sat, 25 Jan 2014 04:47:18 +0000 (UTC) Received: from mail-vc0-x22b.google.com (mail-vc0-x22b.google.com [IPv6:2607:f8b0:400c:c03::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 1087A1E1D for ; Sat, 25 Jan 2014 04:47:17 +0000 (UTC) Received: by mail-vc0-f171.google.com with SMTP id le5so2348502vcb.16 for ; Fri, 24 Jan 2014 20:47:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=pqMUcuSsoArXHoGaug0C8TJLWptR1nsZq20TmEGs/y4=; b=tBJ5ud25nAoF3Q5xh1tloe+kCx3WqPvG7cH/obyCR9W2KO9LFgJh2A+qkj3BdxOPq6 qd+Os0r8cLHMNov0tcqZcC1XEsTJ8LpBWNY4WH70EjCVM2ABMKT6kcefguLbttcQCIqX ROKFmh+5vpQpG2DBEXPQJcL08grla8JGnYWqQqWeBMvKhDuWq8a8ADr2o4ASRnObS/c9 Bk9Xvm9vxm35UblrFbUdwkvC5SMM5K0RXFElVQyW4pWCY6PwLN4WXNT+frIPr9jnvVBA kfhLyStGZU2hCuGztmnE9JUb8WVFo57WHggBpvqP7vPd4AXGS/tFCpsDhIImcomRd3NG tiPg== MIME-Version: 1.0 X-Received: by 10.220.139.136 with SMTP id e8mr6564282vcu.34.1390625237035; Fri, 24 Jan 2014 20:47:17 -0800 (PST) Received: by 10.220.168.135 with HTTP; Fri, 24 Jan 2014 20:47:16 -0800 (PST) In-Reply-To: References: Date: Fri, 24 Jan 2014 23:47:16 -0500 Message-ID: Subject: Re: lock order reversals w/ backtrace From: Thomas Hoffmann To: freebsd-current Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.17 Cc: Hans Petter Selasky X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Jan 2014 04:47:18 -0000 On Fri, Jan 24, 2014 at 9:52 AM, Thomas Hoffmann wrote: > On Thu, Jan 23, 2014 at 11:49 AM, Hans Petter Selasky < > hans.petter.selasky@bitfrost.no> wrote: > >> Hi, >> >> Can you see if you can snap some keywords of the backtraces, like usb_xxx usbdx_xxx cam scsi or something like that. >> >> Else I believe there are some sysctl options to prevent the final reboot somehow so that you can write down the messages. >> >> --HPS >> >> >> -----Original message----- >> > From:Thomas Hoffmann >> > Sent: Thursday 23rd January 2014 17:15 >> >> > To: freebsd-current >> > Subject: lock order reversals w/ backtrace >> > >> > A few days ago I started running 11.0-CURRENT at r260971 for the first time. >> >> > >> > The last couple of times I shutdown my system I noticed 2 or 3 short "lock >> > order reversal" messages with accompanying backtraces scroll by. Do these >> > messages represent a problem that I should report or can I ignore them as >> >> > debug in nature? If I should report them, how or where do these messages >> > get logged? I can find no reference to them in syslog or anywhere else upon >> > my subsequent reboot. >> > >> > I also had a couple of these messages pop up the other day while >> >> > mounting/umounting USB thumb drives. I did not think anything of them at >> > the time as the mounts/umounts completed successfully. >> > >> > Please advise. Thanks. >> > >> > -Tom >> >> I managed to snap a photo of my screen during shutdown. > Here is the full text of the first of two lock order reversals I got last > night during system shutdown, both of which are zfs-related to (it > appears?) unmounts. A few lines got chopped as they were out-of-frame when > I took the photo: > > shutting down local daemons: > lock order reversal: > 1st 0xfffff801194f67c8 zfs (zfs) @ /usr/src/sys/kern/vfs_mount.c:1237 > 2nd 0xfffff801194f6420 syncer (syncer) @ > /usr/src/sys/kern/vfs_subr.c:22[..chopped...] > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_delf_wrapper+0x2b/frame > 0xfffffe01ac78[...chopped...] > kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe01ac784650 > witness_checkorder() at witness_checkorder+0xd23/frame 0xfffffe01ac7846e0 > __lockmgr_args() __lockmgr_args+0x878/frame 0xfffffe01ac784810 > vop_stdlock() at vop_stdlock+0x3c/frame 0xfffffe01ac784830 > VOP_LOCK1_APV() at VOP_LOCK1_APV+0xf5/frame 0xfffffe01ac784860 > _vn_lock() at _vn_lock+0xab/frame 0xfffffe01ac7848d0 > vputx() at vputx+0x240/frame 0xfffffe01ac784930 > dounmount at dounmount+0x327/frame 0xfffffe01ac7849b0 > sys_unmount() at sys_unmount+0x356/frame 0xfffffe01ac784ac0 > amd64_syscall() at amd64_syscall+0x265/frame 0xfffffe01ac784bf0 > Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe01ac784bf0 > --- syscall (22, FreeBSD ELF64, sys_unmount, rip = 0x80191c72a, > rsp[...chopped...] > > I have a zpool on an external USB HDD that I export at shutdown via > rc.shutdown.local. Don't know if that is contributing to this or not. When > I get a chance to bring down the system I will manually export it before > shutdown to see if that makes any difference. > Was able to capture the full text of the lock order reversal I've been experiencing at shutdown. Turns out to be associated with the export of one of my zpools that is hosted on an external USB HDD. Normally I export the zpool from rc.shutdown.local, but tonight I exported it manually before I shutdown and got the following. lock order reversal: : 1st 0xfffff8011952b5f0 zfs (zfs) @ /usr/src/sys/kern/vfs_mount.c:1237 : 2nd 0xfffff8011952b068 syncer (syncer) @ /usr/src/sys/kern/vfs_subr.c:2212 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe01add6e5a0 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe01add6e650 witness_checkorder() at witness_checkorder+0xd23/frame 0xfffffe01add6e6e0 __lockmgr_args() at __lockmgr_args+0x878/frame 0xfffffe01add6e810 vop_stdlock() at vop_stdlock+0x3c/frame 0xfffffe01add6e830 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xf5/frame 0xfffffe01add6e860 _vn_lock() at _vn_lock+0xab/frame 0xfffffe01add6e8d0 vputx() at vputx+0x240/frame 0xfffffe01add6e930 dounmount() at dounmount+0x327/frame 0xfffffe01add6e9b0 sys_unmount() at sys_unmount+0x356/frame 0xfffffe01add6eae0 amd64_syscall() at amd64_syscall+0x265/frame 0xfffffe01add6ebf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe01add6ebf0 --- syscall (22, FreeBSD ELF64, sys_unmount), rip = 0x80191c72a, rsp = 0x7fffffffc4c8, rbp = 0x7fffffffc960 --- When imported and mounted, the zpool reports no errors and I have not experienced any anomalies reading or writing to the zpool. Should I be concerned?