From owner-freebsd-fs@FreeBSD.ORG Wed May 2 22:24:00 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B8CE316A402 for ; Wed, 2 May 2007 22:24:00 +0000 (UTC) (envelope-from cristi@net.utcluj.ro) Received: from bavaria.utcluj.ro (bavaria.utcluj.ro [193.226.5.35]) by mx1.freebsd.org (Postfix) with ESMTP id 24C0813C45B for ; Wed, 2 May 2007 22:24:00 +0000 (UTC) (envelope-from cristi@net.utcluj.ro) Received: from localhost (localhost [127.0.0.1]) by bavaria.utcluj.ro (Postfix) with ESMTP id 8B37C50834 for ; Thu, 3 May 2007 01:23:58 +0300 (EEST) X-Virus-Scanned: by the daemon playing with your mail on bavaria.utcluj.ro Received: from bavaria.utcluj.ro ([127.0.0.1]) by localhost (bavaria.utcluj.ro [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id n6Pt-r1OHo5S for ; Thu, 3 May 2007 01:23:54 +0300 (EEST) Received: from [172.27.2.200] (c7.campus.utcluj.ro [193.226.6.226]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by bavaria.utcluj.ro (Postfix) with ESMTP id 981DA5082B for ; Thu, 3 May 2007 01:23:54 +0300 (EEST) Message-ID: <46390F78.5080206@net.utcluj.ro> Date: Thu, 03 May 2007 01:23:52 +0300 From: Cristian KLEIN Organization: Data Communication Center - Technical University of Cluj-Napoca User-Agent: Thunderbird 1.5.0.10 (X11/20070306) MIME-Version: 1.0 To: freebsd-fs@freebsd.org References: <59558.86.125.188.48.1177802342.squirrel@intranet.utcluj.ro> <4637A640.6050700@freebsd.org> In-Reply-To: <4637A640.6050700@freebsd.org> X-Enigmail-Version: 0.94.0.0 Content-Type: text/plain; charset=iso-8859-2 Content-Transfer-Encoding: 7bit Subject: Re: panic: softdep_setup_inomapdep: found inode already exists in 6.2 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 May 2007 22:24:00 -0000 On Mar, Mai 1, 2007 11:42 pm, Eric Anderson wrote: > On 04/28/07 18:19, Cristian KLEIN wrote: > >> Hi everybody, >> >> >> I am running a FreeBSD 6.2-p3, on which I am experiencing exactly the >> same simtoms as one item of the TODO list of 6.0: >> http://www.freebsd.org/releases/6.0R/todo.html >> >> >> panic: softdep_setup_inomapdep: found inode Needs testing Tor Egge >> Found by stress tests at >> http://www.holm.cc/stress/log/cons138.html >> >> >> Does anybody know whether this bug should have been solved in 6.2? >> Should >> I file a PR? >> > > > Sorry if I missed it, but were you able to provide a backtrace? If you > can, you should compile your kernel with debugging, so at least you could > make a little more out of the crash. See the handbook if you need help on > that. Hi, I haven't mentioned any technical details yet, as I wasn't sure whether this issue is known or not. First, let me tell you what I did. I wanted to better protect data on /jail/mail/home by doing daily snapshots, saved like /jail/mail/home/.snap/2007-04-03-03-22-02. I mounted one of these snapshots (not the latest) in /mnt/home and rsync'd it to a server. I might have rsync'd while taking a new snapshot. Server started crashing randomly, either while rsync-ing, or in the morning, during heavy load. Now I removed the snapshots and the system is stable. Other random information that might be useful: I am using gmirror and jail. Disabling SMP has not effect, WITNESS didn't say anything. The filesystem has userquotas. The filesystem stores maildir and has about 1.6Minodes. Config is GENERIC + SMP + QUOTA - unused hardware devices. root# uname -a FreeBSD bavaria.xxxxxx 6.2-RELEASE-p3 FreeBSD 6.2-RELEASE-p3 #5: Fri Apr 27 20:01:20 EEST 2007 cristi@bavaria.xxxxxx:/usr/obj/usr/src/sys/BAVARIA-SMP i386 root# kgdb kernel.debug /var/crash/vmcore.3 [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". Unread portion of the kernel message buffer: panic: softdep_setup_inomapdep: found inode already exists cpuid = 0 KDB: enter: panic Uptime: 5h42m12s Dumping 1023 MB (2 chunks) chunk 0: 1MB (159 pages) ... ok chunk 1: 1023MB (261837 pages) 1007 991 975 959 943 927 911 895 879 863 847 831 815 799 783 767 751 735 719 703 687 671 655 639 623 607 591 575 559 543 527 511 495 479 463 447 431 415 399 383 367 351 335 319 303 287 271 255 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15 #0 doadump () at pcpu.h:165 165 __asm __volatile("movl %%fs:0,%0" : "=r" (td)); (kgdb) bt #0 doadump () at pcpu.h:165 #1 0xc0545e18 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 #2 0xc0546149 in panic (fmt=0xc075660c "softdep_setup_inomapdep: found inode already exists") at /usr/src/sys/kern/kern_shutdown.c:565 #3 0xc068c64a in softdep_setup_inomapdep (bp=0xd8c2ba68, ip=0x0, newinum=0) at /usr/src/sys/ufs/ffs/ffs_softdep.c:1527 #4 0xc067d4dd in ffs_nodealloccg (ip=0xc68e818c, cg=0, ipref=Unhandled dwarf expression opcode 0x93 ) at /usr/src/sys/ufs/ffs/ffs_alloc.c:1762 #5 0xc067bb83 in ffs_hashalloc (ip=0xc68e818c, cg=0, pref=Unhandled dwarf expression opcode 0x93 ) at /usr/src/sys/ufs/ffs/ffs_alloc.c:1248 #6 0xc067b232 in ffs_valloc (pvp=0xc6883440, mode=33024, cred=0xc50d3a80, vpp=0xe733d4e8) at /usr/src/sys/ufs/ffs/ffs_alloc.c:932 #7 0xc06a6bd7 in ufs_makeinode (mode=33024, dvp=0xc6883440, vpp=0xe733d8f0, cnp=0xe733d904) at /usr/src/sys/ufs/ufs/ufs_vnops.c:2220 #8 0xc06a348f in ufs_create (ap=0x0) at /usr/src/sys/ufs/ufs/ufs_vnops.c:189 #9 0xc071e879 in VOP_CREATE_APV (vop=0x0, a=0xe733d7fc) at vnode_if.c:204 #10 0xc0684018 in ffs_snapshot (mp=0xc4e28000, snapfile=0xc699c200 "/jail/mail/home/.snap/2007-04-28-03-34-31") at vnode_if.h:111 #11 0xc06964d7 in ffs_mount (mp=0xc4e28000, td=0xc50e1300) at /usr/src/sys/ufs/ffs/ffs_vfsops.c:312 #12 0xc059eb6e in vfs_domount (td=0xc50e1300, fstype=0xc4efac90 "ufs", fspath=0xc4efa670 "/jail/mail/home", fsflags=16842752, fsdata=0xc4efaca0) at /usr/src/sys/kern/vfs_mount.c:928 #13 0xc059e1da in vfs_donmount (td=0x0, fsflags=16842752, fsoptions=0x0) at /usr/src/sys/kern/vfs_mount.c:676 #14 0xc05a0f84 in kernel_mount (ma=0xc4efac50, flags=0) at pcpu.h:162 #15 0xc06966fb in ffs_cmount (ma=0xc4efac50, data=0x0, flags=0, td=0xc50e1300) at /usr/src/sys/ufs/ffs/ffs_vfsops.c:392 #16 0xc059e3f0 in mount (td=0xc50e1300, uap=0xe733dd04) at /usr/src/sys/kern/vfs_mount.c:742 #17 0xc070b5bb in syscall (frame= {tf_fs = 59, tf_es = 59, tf_ds = 59, tf_edi = -1077944004, tf_esi = -1077941276, tf_ebp = -1077943864, tf_isp = -416031388, tf_ebx = -1077943824, tf_edx = -1, tf_ecx = -1077940507, tf_eax = 21, tf_trapno = 12, tf_err = 2, tf_eip = 671885143, tf_cs = 51, tf_eflags = 582, tf_esp = -1077944036, tf_ss = 59}) at /usr/src/sys/i386/i386/trap.c:983 #18 0xc06f60df in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:200 #19 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) Now that I have KDB on serial console I might be able to crash (or even better, test patches) at night on that system. Please don't hesitate to request further information. I will try to elaborate a recipe for crashing the system.