From owner-freebsd-stable@FreeBSD.ORG Fri Apr 3 09:53:09 2009 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B3E251065673 for ; Fri, 3 Apr 2009 09:53:09 +0000 (UTC) (envelope-from marius@nuenneri.ch) Received: from mail-fx0-f167.google.com (mail-fx0-f167.google.com [209.85.220.167]) by mx1.freebsd.org (Postfix) with ESMTP id 2C8208FC21 for ; Fri, 3 Apr 2009 09:53:08 +0000 (UTC) (envelope-from marius@nuenneri.ch) Received: by fxm11 with SMTP id 11so868320fxm.43 for ; Fri, 03 Apr 2009 02:53:08 -0700 (PDT) MIME-Version: 1.0 Received: by 10.204.120.201 with SMTP id e9mr341362bkr.187.1238751023206; Fri, 03 Apr 2009 02:30:23 -0700 (PDT) In-Reply-To: References: Date: Fri, 3 Apr 2009 11:30:23 +0200 Message-ID: From: =?ISO-8859-1?Q?Marius_N=FCnnerich?= To: Dmitry Morozovsky Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-stable@freebsd.org, Pawel Jakub Dawidek Subject: Re: RELENG_7/i386: ZFS constant panic on file system writes X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Apr 2009 09:53:10 -0000 On Thu, Apr 2, 2009 at 22:32, Dmitry Morozovsky wrote: > Pawel, > > could you please help me a bit with *very* unpleasant situation: one of m= y > servers with very large ZFS reboots on most write requests to one (larges= t, > which effectively prohibits recreating) ZFS file system with > > panic: avl_find() succeeded inside avl_add() > > (kgdb) bt > #0 =A0doadump () at pcpu.h:196 > #1 =A00xc0533227 in boot (howto=3D260) at /usr/src/sys/kern/kern_shutdown= .c:418 > #2 =A00xc0533535 in panic (fmt=3DVariable "fmt" is not available. > ) at /usr/src/sys/kern/kern_shutdown.c:574 > #3 =A00xc0836a20 in avl_add (tree=3DVariable "tree" is not available. > ) at > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/common/avl/avl.c:= 635 > #4 =A00xc088c39f in zap_lockdir (os=3D0xc555a590, obj=3D6108, tx=3D0x0, l= ti=3DRW_READER, > fatreader=3D1, zapp=3D0xfc6907f8) > =A0 =A0at > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs= /zap_micro.c:231 > #5 =A00xc088cc0f in zap_lookup (os=3D0xc555a590, zapobj=3D6108, name=3D0x= fc6908bc > "daily.20080701.gz", integer_size=3D8, num_integers=3D1, buf=3D0xfc69083c= ) > =A0 =A0at > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs= /zap_micro.c:509 > #6 =A00xc089e25d in zfs_dirent_lock (dlpp=3D0xfc690878, dzp=3D0xc709f570, > name=3D0xfc6908bc "daily.20080701.gz", zpp=3D0xfc690874, flag=3DVariable = "flag" is > not available. > ) > =A0 =A0at > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs= /zfs_dir.c:173 > #7 =A00xc089e43e in zfs_dirlook (dzp=3D0xc709f570, name=3D0xfc6908bc > "daily.20080701.gz", vpp=3D0xfc690b5c) > =A0 =A0at > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs= /zfs_dir.c:271 > #8 =A00xc08a8653 in zfs_freebsd_lookup (ap=3D0xfc690a00) > =A0 =A0at > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs= /zfs_vnops.c:1080 > #9 =A00xc06dab42 in VOP_CACHEDLOOKUP_APV (vop=3D0xc08ba7e0, a=3D0xfc690a0= 0) at > vnode_if.c:153 > #10 0xc05a402c in vfs_cache_lookup (ap=3D0xfc690a84) at vnode_if.h:83 > #11 0xc06dc816 in VOP_LOOKUP_APV (vop=3D0xc08ba7e0, a=3D0xfc690a84) at > vnode_if.c:99 > #12 0xc05aa681 in lookup (ndp=3D0xfc690b48) at vnode_if.h:57 > #13 0xc05ab308 in namei (ndp=3D0xfc690b48) at /usr/src/sys/kern/vfs_looku= p.c:215 > #14 0xc05ba07f in kern_lstat (td=3D0xc5186af0, path=3D0xbfbfd088
0xbfbfd088 out of bounds>, pathseg=3DUIO_USERSPACE, sbp=3D0xfc690c18) > =A0 =A0at /usr/src/sys/kern/vfs_syscalls.c:2184 > #15 0xc05ba22f in lstat (td=3D0xc5186af0, uap=3D0xfc690cfc) at > /usr/src/sys/kern/vfs_syscalls.c:2167 > #16 0xc06d0288 in syscall (frame=3D0xfc690d38) at > /usr/src/sys/i386/i386/trap.c:1090 > #17 0xc06b5bc0 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception= .s:255 > #18 0x00000033 in ?? () > Previous frame inner to this frame (corrupt stack?) > > this is fresh RELENG_7/i386 with (I suppose, unrelated) patch to ata from= mav@ > > Thanks in advance. Hi Dmitry, I think the line numbers are misleading, see: http://fxr.watson.org/fxr/source/cddl/contrib/opensolaris/uts/common/fs/zfs= /zap_micro.c?v=3DFREEBSD7;im=3Dbigexcerpts#L262 Could you build a kernel with -O1 or -O0 perhaps that will help. I have no other clue about your situation except maybe try 8-current? - Marius