From owner-freebsd-current@FreeBSD.ORG Sun Feb 25 05:54:36 2007 Return-Path: X-Original-To: current@freebsd.org Delivered-To: freebsd-current@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4CD3616A402 for ; Sun, 25 Feb 2007 05:54:36 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from relay02.kiev.sovam.com (relay02.kiev.sovam.com [62.64.120.197]) by mx1.freebsd.org (Postfix) with ESMTP id D480C13C478 for ; Sun, 25 Feb 2007 05:54:35 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from [212.82.216.227] (helo=fw.zoral.com.ua) by relay02.kiev.sovam.com with esmtps (TLSv1:AES256-SHA:256) (Exim 4.60) (envelope-from ) id 1HLCL3-0008qd-VZ for current@freebsd.org; Sun, 25 Feb 2007 07:54:34 +0200 Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by fw.zoral.com.ua (8.13.4/8.13.4) with ESMTP id l1P5qt9I018126 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sun, 25 Feb 2007 07:52:55 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.13.8/8.13.8) with ESMTP id l1P5qt37054356; Sun, 25 Feb 2007 07:52:55 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.13.8/8.13.8/Submit) id l1P5qtcu054355; Sun, 25 Feb 2007 07:52:55 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Sun, 25 Feb 2007 07:52:54 +0200 From: Kostik Belousov To: Alexandre Sunny Kovalenko Message-ID: <20070225055254.GD77131@deviant.kiev.zoral.com.ua> References: <1171414959.906.16.camel@RabbitsDen.RabbitsLawn.verizon.net> <20070214011755.GA73381@xor.obsecurity.org> <1171500531.780.6.camel@RabbitsDen.RabbitsLawn.verizon.net> <20070215011400.GA10455@xor.obsecurity.org> <1171982928.751.1.camel@RabbitsDen.RabbitsLawn.verizon.net> <1172156865.848.14.camel@RabbitsDen.RabbitsLawn.verizon.net> <20070224195540.GB77131@deviant.kiev.zoral.com.ua> <1172382104.17603.2.camel@twinhead.rabbitslawn.verizon.net> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="cQXOx3fnlpmgJsTP" Content-Disposition: inline In-Reply-To: <1172382104.17603.2.camel@twinhead.rabbitslawn.verizon.net> User-Agent: Mutt/1.4.2.2i X-Virus-Scanned: ClamAV version 0.88.7, clamav-milter version 0.88.7 on fw.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-0.1 required=5.0 tests=ALL_TRUSTED,SPF_NEUTRAL autolearn=failed version=3.1.8 X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on fw.zoral.com.ua X-Scanner-Signature: feb28a81ec0850514d099b7346a2573b X-DrWeb-checked: yes X-SpamTest-Envelope-From: kostikbel@gmail.com X-SpamTest-Group-ID: 00000000 X-SpamTest-Info: Profiles 803 [Feb 24 2007] X-SpamTest-Info: helo_type=3 X-SpamTest-Info: {received from trusted relay: not dialup} X-SpamTest-Method: none X-SpamTest-Method: Local Lists X-SpamTest-Rate: 0 X-SpamTest-Status: Not detected X-SpamTest-Status-Extended: not_detected X-SpamTest-Version: SMTP-Filter Version 3.0.0 [0255], KAS30/Release Cc: current@freebsd.org, Kris Kennaway Subject: Re: -CURRENT panics on intensive fs operations. X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 Feb 2007 05:54:36 -0000 --cQXOx3fnlpmgJsTP Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Feb 25, 2007 at 12:41:44AM -0500, Alexandre Sunny Kovalenko wrote: > On Sat, 2007-02-24 at 21:55 +0200, Kostik Belousov wrote: > > On Thu, Feb 22, 2007 at 10:07:45AM -0500, Alexandre Sunny Kovalenko wro= te: > > > On Tue, 2007-02-20 at 09:48 -0500, Alexandre "Sunny" Kovalenko wrote: > > > > On Wed, 2007-02-14 at 20:14 -0500, Kris Kennaway wrote: > > > > > On Wed, Feb 14, 2007 at 07:48:51PM -0500, Alexandre Sunny Kovalen= ko wrote: > > > > > > On Tue, 2007-02-13 at 20:17 -0500, Kris Kennaway wrote: > > > > > > > On Tue, Feb 13, 2007 at 08:02:39PM -0500, Alexandre Sunny Kov= alenko wrote: > > > > > > > > I can reliably panic -CURRENT (Feb 11, noon EST) with the s= omething that > > > > > > > > excersises the file system. I have currently settled on (cd= /usr/ports; > > > > > > > > make clean), but it all started out as doing some "emerges"= to test the > > > > > > > > latest linuxolator. In the case of the "make clean" I have = seen it > > > > > > > > crashing as early as /usr/ports/audio and as late > > > > > > > > as /usr/ports/textproc.=20 > > > > > > > >=20 > > > I am still not capable to get good backtrace from the kernel dump, bu= t I > > > have managed to hook up remote console to this machine, so here are > > > results: > > >=20 > > > db> bt > > > Tracing pid 33 tid 100032 td 0xc4cee510 > > > kdb_enter(c067c69d) at kdb_enter+0x2b > > > panic(c0667ba3,c306d5c0,c306d5c0,e38a2cfc,c0619fd9,...) at panic+0x11c > > > vm_pageq_remove_nowakeup(c306d5c0,c061a0b8,e38a2d04,c061a0ee,e38a2d24= ,...) at vm_pageq_remove_nowakeup+0x35 > > > vm_page_zero_idle(e38a2d24,c04c7fe4,0,e38a2d38,c4ef8900,...) at > > > vm_page_zero_idle+0x49 > > > vm_pagezero(0,e38a2d38) at vm_pagezero+0x36 > > > fork_exit(c061a0b8,0,e38a2d38) at fork_exit+0xac > > > fork_trampoline() at fork_trampoline+0x8 > > > --- trap 0, eip =3D 0, esp =3D 0xe38a2d70, ebp =3D 0 --- > > > db> ps > > > pid ppid pgrp uid state wmesg wchan cmd > > > > > > 33 0 0 0 RL CPU 0 [pagezero] > > > > > >=20 > > > ... and (hopefully) relevant bits from the source > > > (kgdb) list *vm_pageq_remove_nowakeup+0x35 > > > 0xc06192f9 is in vm_pageq_remove_nowakeup > > > (/usr/src/sys/vm/vm_pageq.c:223). > > > 218 struct vpgqueues *pq; > > > 219 > > > 220 if (queue !=3D PQ_NONE) { > > > 221 pq =3D &vm_page_queues[queue]; > > > 222 VM_PAGE_SETQUEUE2(m, PQ_NONE); > > > 223 TAILQ_REMOVE(&pq->pl, m, pageq); > > There, please, show the output of "p/x *m" and "p/x *pq". > >=20 > Unfortunately, with the latest -CURRENT the end result is different: >=20 > Kernel page fault with the following non-sleepable locks held: > exclusive sleep mutex sellck r =3D 0 (0xc0741284) locked > @ /usr/src/sys/kern/sys_generic.c:776 > KDB: stack backtrace: > db_trace_self_wrapper(c067f599) at db_trace_self_wrapper+0x25 > kdb_backtrace(1,c4fb46c0,c,e5447ab4,e5447aa8,...) at kdb_backtrace+0x29 > witness_warn(5,0,c069ed6f) at witness_warn+0x192 > trap(e5447ab4) at trap+0x10f > calltrap() at calltrap+0x6 > --- trap 0xc, eip =3D 0x80bfe48d, esp =3D 0xe5447af4, ebp =3D 0xe5447c54 = --- > kernload(c4e02d80,5,bfbfedb0,0,bfbfedb0,...) at -0x7f401b73 > select(c4e02d80,e5447d00) at select+0x44 > syscall(e5447d38) at syscall+0x256 > Xint0x80_syscall() at Xint0x80_syscall+0x20 > --- syscall (93, FreeBSD ELF32, select), eip =3D 0x2815f273, esp =3D > 0xbfbfe82c, ebp =3D 0xbfbfee48 --- >=20 >=20 > Fatal trap 12: page fault while in kernel mode > cpuid =3D 0; apic id =3D 00 > fault virtual address =3D 0x80bfe48d > fault code =3D supervisor read, page not present > instruction pointer =3D 0x20:0x80bfe48d > stack pointer =3D 0x28:0xe5447af4 > frame pointer =3D 0x28:0xe5447c54 > code segment =3D base 0x0, limit 0xfffff, type 0x1b > =3D DPL 0, pres 1, def32 1, gran 1 > processor eflags =3D interrupt enabled, resume, IOPL =3D 0 > current process =3D 520 (powerd) > [thread pid 520 tid 100052 ] > Stopped at -0x7f401b73: *** error reading from address 80bfe48d > *** > db>=20 >=20 > db> where > Tracing pid 520 tid 100052 td 0xc4e02d80 > kern_select(c4e02d80,5,bfbfedb0,0,bfbfedb0,...) at kern_select+0x4e5 > select(c4e02d80,e5447d00) at select+0x44 > syscall(e5447d38) at syscall+0x256 > Xint0x80_syscall() at Xint0x80_syscall+0x20 > --- syscall (93, FreeBSD ELF32, select), eip =3D 0x2815f273, esp =3D > 0xbfbfe82c, ebp =3D 0xbfbfee48 --- > db>=20 >=20 > what should I look for here? Did you used some memory tester ? This looks like (random) memory corruptio= n. Reported fault address 0x80bfe48d belongs to user part of VA. Could you, please, show the source line that corresponds the kern_select+0x4e5 in your compiled kernel ? --cQXOx3fnlpmgJsTP Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (FreeBSD) iD8DBQFF4SQ2C3+MBN1Mb4gRAsmxAKCx8ZRmF7tb6ddTMRD/3ItWCFMc/QCg3ffn 47z8hMoGjL7TFUj36CZ0r8g= =n5/Q -----END PGP SIGNATURE----- --cQXOx3fnlpmgJsTP--