From owner-freebsd-hackers@FreeBSD.ORG Thu Oct 26 08:38:25 2006 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E07A316A412 for ; Thu, 26 Oct 2006 08:38:24 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from fw.zoral.com.ua (fw.zoral.com.ua [213.186.206.134]) by mx1.FreeBSD.org (Postfix) with ESMTP id 354D343D4C for ; Thu, 26 Oct 2006 08:38:23 +0000 (GMT) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by fw.zoral.com.ua (8.13.4/8.13.4) with ESMTP id k9Q8JNmK019596 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 26 Oct 2006 11:19:23 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.13.8/8.13.8) with ESMTP id k9Q8cHef079304; Thu, 26 Oct 2006 11:38:17 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.13.8/8.13.8/Submit) id k9Q8cGrY079303; Thu, 26 Oct 2006 11:38:16 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 26 Oct 2006 11:38:16 +0300 From: Kostik Belousov To: Micah Message-ID: <20061026083816.GS45605@deviant.kiev.zoral.com.ua> References: <453F882B.2040009@ywave.com> <453F98C4.2060402@ywave.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="Po3hG7FLjYU3tiSY" Content-Disposition: inline In-Reply-To: <453F98C4.2060402@ywave.com> User-Agent: Mutt/1.4.2.2i X-Virus-Scanned: ClamAV version 0.88.4, clamav-milter version 0.88.4 on fw.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=1.9 required=5.0 tests=DNS_FROM_RFC_ABUSE, SPF_NEUTRAL,UNPARSEABLE_RELAY autolearn=no version=3.1.4 X-Spam-Level: * X-Spam-Checker-Version: SpamAssassin 3.1.4 (2006-07-25) on fw.zoral.com.ua Cc: FreeBSD Hackers Subject: Re: System panic under load (additional information) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Oct 2006 08:38:25 -0000 --Po3hG7FLjYU3tiSY Content-Type: text/plain; charset=koi8-r Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Oct 25, 2006 at 10:03:00AM -0700, Micah wrote: > Okay, I was reading the developer's handbook and trying stuff. The=20 > following may be helpful: >=20 > (kgdb) up 7 > #7 0xc0593d34 in vfs_hash_get (mp=3D0xc5a53000, hash=3D5347783, flags=3D= 2, > td=3D0xc5ef9d80, vpp=3D0xf02ca99c, fn=3D0, arg=3D0x0) > at /usr/src/sys/kern/vfs_hash.c:73 > 73 if (vp->v_hash !=3D hash) > (kgdb) list *0xc0593d34 > 0xc0593d34 is in vfs_hash_get (/usr/src/sys/kern/vfs_hash.c:73). > 64 int > 65 vfs_hash_get(struct mount *mp, u_int hash, int flags, struct=20 > thread *td, struct vnode **vpp, vfs_hash_cmp_t *fn, void *arg) > 66 { > 67 struct vnode *vp; > 68 int error; > 69 > 70 while (1) { > 71 mtx_lock(&vfs_hash_mtx); > 72 LIST_FOREACH(vp, vfs_hash_index(mp, hash),=20 > v_hashlist) { > 73 if (vp->v_hash !=3D hash) > 74 continue; > 75 if (vp->v_mount !=3D mp) > 76 continue; > 77 if (fn !=3D NULL && fn(vp, arg)) > (kgdb) print vp > $1 =3D (struct vnode *) 0xffffffff >=20 > I'm no expert but that doesn't look like a valid address.... I was able= =20 > to get another crashdump, but kgdb locked up while trying to read it. I saw several similar reports. Please, submit me the output of the "print *mp" in the same frame. Also, I'm interested in kernel config.=20 Is the problem reproducible ? >=20 > Micah wrote: > >Can anyone help me in deciphering what this crashdump is telling me? My= =20 > >i386 6.1p3 system is panicking under heavy IO load and I'm trying to=20 > >isolate the cause. I've never debugged a crashdump before, so if there's= =20 > >more information I can milk from the core file, let me know. > > > >Thanks, > >Micah > > > >trisha# kgdb kernel.debug /home/crash/vmcore.0 > >[GDB will not be able to debug user-mode threads:=20 > >/usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] > >GNU gdb 6.1.1 [FreeBSD] > >Copyright 2004 Free Software Foundation, Inc. > >GDB is free software, covered by the GNU General Public License, and you= =20 > >are > >welcome to change it and/or distribute copies of it under certain=20 > >conditions. > >Type "show copying" to see the conditions. > >There is absolutely no warranty for GDB. Type "show warranty" for detai= ls. > >This GDB was configured as "i386-marcel-freebsd". > > > >Unread portion of the kernel message buffer: > > > > > >Fatal trap 12: page fault while in kernel mode > >fault virtual address =3D 0x27 > >fault code =3D supervisor read, page not present > >instruction pointer =3D 0x20:0xc0593d34 > >stack pointer =3D 0x28:0xf02ca89c > >frame pointer =3D 0x28:0xf02ca8bc > >code segment =3D base 0x0, limit 0xfffff, type 0x1b > > =3D DPL 0, pres 1, def32 1, gran 1 > >processor eflags =3D interrupt enabled, resume, IOPL =3D 0 > >current process =3D 24805 (perl5.8.8) > >trap number =3D 12 > >panic: page fault > >Uptime: 13h16m1s > >Dumping 1534 MB (2 chunks) > > chunk 0: 1MB (159 pages) ... ok > > chunk 1: 1534MB (392672 pages) 1518 1502 1486 1470 1454 1438 1422 1406= =20 > >1390 1374 1358 1342 1326 1310 1294 1278 1262 1246 1230 1214 1198 1182=20 > >1166 1150 1134 1118 1102 1086 1070 1054 1038 1022 1006 990 974 958 942= =20 > >926 910 894 878 862 846 830 814 798 782 766 750 734 718 702 686 670 654= =20 > >638 622 606 590 574 558 542 526 510 494 478 462 446 430 414 398 382 366= =20 > >350 334 318 302 286 270 254 238 222 206 190 174 158 142 126 110 94 78 62= =20 > >46 30 14 > > > >#0 doadump () at pcpu.h:165 > >165 __asm __volatile("movl %%fs:0,%0" : "=3Dr" (td)); > >(kgdb) bt > >#0 doadump () at pcpu.h:165 > >#1 0xc0535a3f in boot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.= c:402 > >#2 0xc0535d66 in panic (fmt=3D0xc0711e06 "%s") > > at /usr/src/sys/kern/kern_shutdown.c:558 > >#3 0xc06ecf2c in trap_fatal (frame=3D0xf02ca85c, eva=3D0) > > at /usr/src/sys/i386/i386/trap.c:836 > >#4 0xc06ecc32 in trap_pfault (frame=3D0xf02ca85c, usermode=3D0, eva=3D3= 9) > > at /usr/src/sys/i386/i386/trap.c:744 > >#5 0xc06ec7fd in trap (frame=3D > > {tf_fs =3D 8, tf_es =3D 40, tf_ds =3D 40, tf_edi =3D 4, tf_esi =3D= 4, tf_ebp=20 > >=3D -265508676, tf_isp =3D -265508728, tf_ebx =3D -1, tf_edx =3D -980369= 408,=20 > >tf_ecx =3D -979030016, tf_eax =3D 5347783, tf_trapno =3D 12, tf_err =3D = 0,=20 > >tf_eip =3D -1067893452, tf_cs =3D 32, tf_eflags =3D 66182, tf_esp =3D=20 > >-979030016, tf_ss =3D 5347783}) > > at /usr/src/sys/i386/i386/trap.c:434 > >#6 0xc06d9e2a in calltrap () at /usr/src/sys/i386/i386/exception.s:139 > >#7 0xc0593d34 in vfs_hash_get (mp=3D0xc5a53000, hash=3D5347783, flags= =3D2, > > td=3D0xc5ef9d80, vpp=3D0xf02ca99c, fn=3D0, arg=3D0x0) > > at /usr/src/sys/kern/vfs_hash.c:73 > >#8 0xc067af59 in ffs_vget (mp=3D0xc5a53000, ino=3D5347783, flags=3D2, > > vpp=3D0xf02ca99c) at pcpu.h:162 > >#9 0xc0683353 in ufs_lookup (ap=3D0xf02caa40) > > at /usr/src/sys/ufs/ufs/ufs_lookup.c:572 > >#10 0xc0700f03 in VOP_CACHEDLOOKUP_APV (vop=3D0x5199c7, a=3D0xc590c000) > > at vnode_if.c:150 > >#11 0xc059006a in vfs_cache_lookup (ap=3D0x5199c7) at vnode_if.h:82 > >#12 0xc0700e78 in VOP_LOOKUP_APV (vop=3D0xc0764f20, a=3D0xf02caaec) > > at vnode_if.c:99 > >#13 0xc059547b in lookup (ndp=3D0xf02cabf4) at vnode_if.h:56 > >#14 0xc0594c08 in namei (ndp=3D0xf02cabf4) at=20 > >/usr/src/sys/kern/vfs_lookup.c:203 > >#15 0xc05a672d in kern_stat (td=3D0xc5ef9d80, > > path=3D0xc590c000 "=9C\033\210? B\215?0???\020?Z?\020?\211?0\003\237= ?PU0?", > > pathseg=3D3314597888, sbp=3D0x5199c7) at=20 > >/usr/src/sys/kern/vfs_syscalls.c:2076 > >#16 0xc05a66bf in stat (td=3D0x5199c7, uap=3D0xf02cad04) > > at /usr/src/sys/kern/vfs_syscalls.c:2061 > >#17 0xc06ed302 in syscall (frame=3D > > {tf_fs =3D 59, tf_es =3D 59, tf_ds =3D 59, tf_edi =3D 1209537912, = tf_esi =3D=20 > >140130588, tf_ebp =3D -1077941096, tf_isp =3D -265507484, tf_ebx =3D=20 > >1209530852, tf_edx =3D 68, tf_ecx =3D 0, tf_eax =3D 188, tf_trapno =3D 1= 2,=20 > >tf_err =3D 2, tf_eip =3D 1210540343, tf_cs =3D 51, tf_eflags =3D 582, tf= _esp =3D=20 > >-1077941140, tf_ss =3D 59}) > > at /usr/src/sys/i386/i386/trap.c:981 > >#18 0xc06d9e7f in Xint0x80_syscall () at=20 > >/usr/src/sys/i386/i386/exception.s:200 > >---Type to continue, or q to quit--- > >#19 0x00000033 in ?? () > >Previous frame inner to this frame (corrupt stack?) > >_______________________________________________ > >freebsd-hackers@freebsd.org mailing list > >http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > >To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.or= g" >=20 > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" --Po3hG7FLjYU3tiSY Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (FreeBSD) iD8DBQFFQHP3C3+MBN1Mb4gRAo/cAJ4hKUa0spNekUDK2bfrg2eAA9mzOwCfYeWW 01UsJX7jBRFj/nFIm4+Y+KM= =J6LK -----END PGP SIGNATURE----- --Po3hG7FLjYU3tiSY--