From owner-freebsd-fs@FreeBSD.ORG Fri Apr 11 00:57:27 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id AAA77B51 for ; Fri, 11 Apr 2014 00:57:27 +0000 (UTC) Received: from hergotha.csail.mit.edu (wollman-1-pt.tunnel.tserv4.nyc4.ipv6.he.net [IPv6:2001:470:1f06:ccb::2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4DFCB1993 for ; Fri, 11 Apr 2014 00:57:27 +0000 (UTC) Received: from hergotha.csail.mit.edu (localhost [127.0.0.1]) by hergotha.csail.mit.edu (8.14.7/8.14.7) with ESMTP id s3B0vOOJ091618 for ; Thu, 10 Apr 2014 20:57:24 -0400 (EDT) (envelope-from wollman@hergotha.csail.mit.edu) Received: (from wollman@localhost) by hergotha.csail.mit.edu (8.14.7/8.14.4/Submit) id s3B0vOcS091615; Thu, 10 Apr 2014 20:57:24 -0400 (EDT) (envelope-from wollman) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <21319.15860.827329.334067@hergotha.csail.mit.edu> Date: Thu, 10 Apr 2014 20:57:24 -0400 From: Garrett Wollman To: freebsd-fs@freebsd.org Subject: ZFS panic: solaris assert: BP_EQUAL(bp, &zio->io_bp_orig) X-Mailer: VM 7.17 under 21.4 (patch 22) "Instant Classic" XEmacs Lucid X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (hergotha.csail.mit.edu [127.0.0.1]); Thu, 10 Apr 2014 20:57:25 -0400 (EDT) X-Spam-Status: No, score=-0.8 required=5.0 tests=ALL_TRUSTED, HEADER_FROM_DIFFERENT_DOMAINS autolearn=disabled version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on hergotha.csail.mit.edu X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Apr 2014 00:57:27 -0000 I have a file server that has panic()ed twice today with the following: panic: solaris assert: BP_EQUAL(bp, &zio->io_bp_orig), file: /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c, line: 2955 cpuid = 20 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame 0xffffff98a38d2900 kdb_backtrace() at kdb_backtrace+0x37/frame 0xffffff98a38d29c0 panic() at panic+0x1ce/frame 0xffffff98a38d2ac0 assfail() at assfail+0x1a/frame 0xffffff98a38d2ad0 zio_done() at zio_done+0x120/frame 0xffffff98a38d2b30 zio_execute() at zio_execute+0xc3/frame 0xffffff98a38d2b70 taskqueue_run_locked() at taskqueue_run_locked+0x74/frame 0xffffff98a38d2bc0 taskqueue_thread_loop() at taskqueue_thread_loop+0x46/frame 0xffffff98a38d2be0 fork_exit() at fork_exit+0x11f/frame 0xffffff98a38d2c30 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff98a38d2c30 --- trap 0, rip = 0, rsp = 0xffffff98a38d2cf0, rbp = 0 --- Uptime: 11d15h27m51s Automatic reboot in 15 seconds - press a key on the console to abort --> Press a key on the console to reboot, --> or switch off the system now. (It also failed to reboot automatically, as if a character had been sent to the console.) Anyone have a guess about what's going on here? I obviously can't keep manually restarting a production file server every 14 hours. Here's the code in question: if (bp != NULL) { ASSERT(bp->blk_pad[0] == 0); ASSERT(bp->blk_pad[1] == 0); ASSERT(bcmp(bp, &zio->io_bp_copy, sizeof (blkptr_t)) == 0 || (bp == zio_unique_parent(zio)->io_bp)); if (zio->io_type == ZIO_TYPE_WRITE && !BP_IS_HOLE(bp) && zio->io_bp_override == NULL && !(zio->io_flags & ZIO_FLAG_IO_REPAIR)) { ASSERT(!BP_SHOULD_BYTESWAP(bp)); ASSERT3U(zio->io_prop.zp_copies, <=, BP_GET_NDVAS(bp)); ASSERT(BP_COUNT_GANG(bp) == 0 || (BP_COUNT_GANG(bp) == BP_GET_NDVAS(bp))); } if (zio->io_flags & ZIO_FLAG_NOPWRITE) VERIFY(BP_EQUAL(bp, &zio->io_bp_orig)); } And it just now panicked again. Help! -GAWollman