From owner-freebsd-current@freebsd.org Tue Dec 29 20:14:47 2015 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 53207A5428F for ; Tue, 29 Dec 2015 20:14:47 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-ob0-x233.google.com (mail-ob0-x233.google.com [IPv6:2607:f8b0:4003:c01::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1E41C1637; Tue, 29 Dec 2015 20:14:47 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-ob0-x233.google.com with SMTP id ba1so165851875obb.3; Tue, 29 Dec 2015 12:14:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type:content-transfer-encoding; bh=UZ306zRnhPMOut+3aFLdWbVqkLcuCpyXau2sxkFMn8c=; b=Hq64xzG+ZxE0vUCx56H9UUc5Nxvc4fUNRicz743VIaDvPFAyXgd/o/w4ZihdaJztSO yMyPHOCAp8KjWTxW6HnAofTqVqd988nI8m0cPUSmTXufSOE9asH3A5ePRxp42JJtVMG2 gr8ANp66IQx1nMTOwiWzLIDYvCzg/xig3pOo9biides6/Qvv4QAV1EzflY/Ap2y1qOFb kcLPuDmEZhHZsuTAeFLTHzXjVEtxCdqRrvDdaPCmomqgq1k0c13RuGnWNbiamsgiOJm3 TO434eSy1XyAie3WKSiwChYhCJQViSd6bjVqTB0tbNcCTw+NV7FanXnGYqDBKu5msYQl 6XAQ== MIME-Version: 1.0 X-Received: by 10.60.159.198 with SMTP id xe6mr36790025oeb.64.1451420086034; Tue, 29 Dec 2015 12:14:46 -0800 (PST) Sender: asomers@gmail.com Received: by 10.202.69.84 with HTTP; Tue, 29 Dec 2015 12:14:45 -0800 (PST) In-Reply-To: <3B435E9A-F93A-407E-AB54-C7B98291269B@gmail.com> References: <5B1F1E9E-7651-4C1D-807F-A875EDEF8705@fsfe.org> <3B435E9A-F93A-407E-AB54-C7B98291269B@gmail.com> Date: Tue, 29 Dec 2015 13:14:45 -0700 X-Google-Sender-Auth: jDISWTjP9s0DEfp-Q1zSHzmiuB8 Message-ID: Subject: Re: Stack backtrace on `zpool export` From: Alan Somers To: NGie Cooper Cc: Florian Ermisch <0xf10e@fsfe.org>, FreeBSD CURRENT , "will@freebsd.org" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Dec 2015 20:14:47 -0000 On Tue, Dec 29, 2015 at 12:54 PM, NGie Cooper wrote= : > >> On Dec 29, 2015, at 11:48, Florian Ermisch <0xf10e@fsfe.org> wrote: >> >> Hi *, >> >> since I've upgraded my laptop from 10.2-RELEASE to 11-CURRENT (r292536, = now r292755) I see this stack backtrace when a zpool is exported: >> >> Dec 27 18:44:02 fuchi-cyber220 kernel: lock order reversal: >> Dec 27 18:44:02 fuchi-cyber220 kernel: 1st 0xfffff800c7472418 zfs (zfs) = @ /usr/src/sys/kern/vfs_mount.c:1224 >> Dec 27 18:44:02 fuchi-cyber220 kernel: 2nd 0xfffff800c73fad50 zfs_gfs (z= fs_gfs) @ /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/gfs.c:494 >> Dec 27 18:44:02 fuchi-cyber220 kernel: stack backtrace: >> Dec 27 18:44:02 fuchi-cyber220 kernel: #0 0xffffffff80a7d6f0 at witness_= debugger+0x70 >> Dec 27 18:44:02 fuchi-cyber220 kernel: #1 0xffffffff80a7d5f1 at witness_= checkorder+0xe71 >> Dec 27 18:44:02 fuchi-cyber220 kernel: #2 0xffffffff809fedcb at __lockmg= r_args+0xd3b >> Dec 27 18:44:02 fuchi-cyber220 kernel: #3 0xffffffff80ac55ec at vop_stdl= ock+0x3c >> Dec 27 18:44:02 fuchi-cyber220 kernel: #4 0xffffffff80fbb220 at VOP_LOCK= 1_APV+0x100 >> Dec 27 18:44:02 fuchi-cyber220 kernel: #5 0xffffffff80ae653a at _vn_lock= +0x9a >> Dec 27 18:44:02 fuchi-cyber220 kernel: #6 0xffffffff8209db13 at gfs_file= _create+0x73 >> Dec 27 18:44:02 fuchi-cyber220 kernel: #7 0xffffffff8209dbbd at gfs_dir_= create+0x1d >> Dec 27 18:44:02 fuchi-cyber220 kernel: #8 0xffffffff821649e7 at zfsctl_m= knode_snapdir+0x47 >> Dec 27 18:44:02 fuchi-cyber220 kernel: #9 0xffffffff8209e135 at gfs_dir_= lookup+0x185 >> Dec 27 18:44:02 fuchi-cyber220 kernel: #10 0xffffffff8209e61d at gfs_vop= _lookup+0x1d >> Dec 27 18:44:02 fuchi-cyber220 kernel: #11 0xffffffff82163a05 at zfsctl_= root_lookup+0xf5 >> Dec 27 18:44:02 fuchi-cyber220 kernel: #12 0xffffffff821648a3 at zfsctl_= umount_snapshots+0x83 >> Dec 27 18:44:02 fuchi-cyber220 kernel: #13 0xffffffff8217d5ab at zfs_umo= unt+0x7b >> Dec 27 18:44:02 fuchi-cyber220 kernel: #14 0xffffffff80acf0b0 at dounmou= nt+0x530 >> Dec 27 18:44:02 fuchi-cyber220 kernel: #15 0xffffffff80aceaed at sys_unm= ount+0x35d >> Dec 27 18:44:02 fuchi-cyber220 kernel: #16 0xffffffff80e6e13b at amd64_s= yscall+0x2db >> Dec 27 18:44:02 fuchi-cyber220 kernel: #17 0xffffffff80e4dd8b at Xfast_s= yscall+0xfb >> >> (From /var/log/messages) >> >> First I've only seen this on the console at shutdown or reboot (after sy= nc I think) but later found I can reproduce it by exporting a zpool. >> While the only pools I can trigger this without a shutdown/reboot are co= nnected via USB(3) I still see it just before poweroff at shutdown when the= system's root pool is synced. >> >> When I try to export a zpool under heavy load (`make -C /use/src -j 4 bu= ildworld` on a 2 core CPU w/ HT) the system locks up completely. I don't th= ink it's related to memory pressure as I haven't seen swap being used durin= g a buildworld (with 8 gigs of RAM). >> >> Has anyone else seen this? > > Please file a bug for this issue and assign it to freebsd-fs. It doesn=E2= =80=99t look familiar (there might be a lock ordering issue with either zfs= or vfs+zfs). > Thanks! > -NGie Adding Will@. I think he was working on this LOR at one point. -Alan