From owner-freebsd-fs@freebsd.org Fri Aug 21 17:34:35 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E8E089BF920 for ; Fri, 21 Aug 2015 17:34:35 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from gw.catspoiler.org (cl-1657.chi-02.us.sixxs.net [IPv6:2001:4978:f:678::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gw.catspoiler.org", Issuer "gw.catspoiler.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 7DD1E34C for ; Fri, 21 Aug 2015 17:34:35 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from FreeBSD.org (mousie.catspoiler.org [192.168.101.2]) by gw.catspoiler.org (8.15.2/8.15.2) with ESMTP id t7LHYQ33096035 for ; Fri, 21 Aug 2015 10:34:30 -0700 (PDT) (envelope-from truckman@FreeBSD.org) Message-Id: <201508211734.t7LHYQ33096035@gw.catspoiler.org> Date: Fri, 21 Aug 2015 10:34:26 -0700 (PDT) From: Don Lewis Subject: Re: solaris assert: avl_is_empty(&dn -> dn_dbufs) panic To: freebsd-fs@FreeBSD.org In-Reply-To: <201508210834.t7L8Yigk093994@gw.catspoiler.org> MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Aug 2015 17:34:36 -0000 On 21 Aug, Don Lewis wrote: > I just started getting this panic: > > solaris assert: avl_is_empty(&dn -> dn_dbufs), file: > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dnode_sync.c, > line 495 > > System info: > FreeBSD zipper.catspoiler.org 11.0-CURRENT FreeBSD 11.0-CURRENT #25 r286923: Wed Aug 19 09:28:53 PDT 2015 dl@zipper.catspoiler.org:/usr/obj/usr/src/sys/GENERIC amd64 > > My zfs pool has one mirrored vdev. Scrub doesn't find any problems. > > %zpool status > pool: zroot > state: ONLINE > scan: scrub repaired 0 in 2h58m with 0 errors on Fri Aug 21 00:44:52 2015 > config: > > NAME STATE READ WRITE CKSUM > zroot ONLINE 0 0 0 > mirror-0 ONLINE 0 0 0 > ada0p3 ONLINE 0 0 0 > ada1p3 ONLINE 0 0 0 > > This panic is reproduceable and happens every time I use poudriere to > build ports using my 9.3-RELEASE amd64 jail and occurs at the end of the > poudriere run when it is unmounting filesystems. > > [00:10:43] ====>> Stopping 4 builders > 93amd64-default-job-01: removed > 93amd64-default-job-01-n: removed > 93amd64-default-job-02: removed > 93amd64-default-job-02-n: removed > 93amd64-default-job-03: removed > 93amd64-default-job-03-n: removed > 93amd64-default-job-04: removed > 93amd64-default-job-04-n: removed > [00:10:46] ====>> Creating pkgng repository > Creating repository in /tmp/packages: 100% > Packing files for repository: 100% > [00:10:55] ====>> Committing packages to repository > [00:10:55] ====>> Removing old packages > [00:10:55] ====>> Built ports: devel/py-pymtbl net/sie-nmsg net/p5-Net-Nmsg net/axa > [93amd64-default] [2015-08-21_00h47m41s] [committing:] Queued: 4 Built: 4 Failed: 0 Skipped: 0 Ignored: 0 Tobuild: 0 Time: 00:10:53 > [00:10:55] ====>> Logs: /var/poudriere/data/logs/bulk/93amd64-default/2015-08-21_00h47m41s > [00:10:55] ====>> Cleaning up > 93amd64-default: removed > 93amd64-default-n: removed > [00:10:55] ====>> Umounting file systems > Write failed: Broken pipe > > Prior to that, I ran poudriere a number of times with a 10.2-STABLE > amd64 jail without incident. > > I've kicked off a bunch of poudriere runs for other jails and > will check on it in the morning. Died the same way after building ports on the first jail, 10.1-RELEASE amd64. Since there have been some zfs commits since r286923, I upgraded to r286998 this morning and tried again with no better luck. I got the same panic again. This machine has mirrored swap, and even though I've done what gmirror(8) says to do in order to capture crash dumps, I've had no luck with that. The dump is getting written, but savecore is unable to find it.