Date: Sun, 25 Feb 2007 07:52:54 +0200 From: Kostik Belousov <kostikbel@gmail.com> To: Alexandre Sunny Kovalenko <alex.kovalenko@verizon.net> Cc: current@freebsd.org, Kris Kennaway <kris@obsecurity.org> Subject: Re: -CURRENT panics on intensive fs operations. Message-ID: <20070225055254.GD77131@deviant.kiev.zoral.com.ua> In-Reply-To: <1172382104.17603.2.camel@twinhead.rabbitslawn.verizon.net> References: <1171414959.906.16.camel@RabbitsDen.RabbitsLawn.verizon.net> <20070214011755.GA73381@xor.obsecurity.org> <1171500531.780.6.camel@RabbitsDen.RabbitsLawn.verizon.net> <20070215011400.GA10455@xor.obsecurity.org> <1171982928.751.1.camel@RabbitsDen.RabbitsLawn.verizon.net> <1172156865.848.14.camel@RabbitsDen.RabbitsLawn.verizon.net> <20070224195540.GB77131@deviant.kiev.zoral.com.ua> <1172382104.17603.2.camel@twinhead.rabbitslawn.verizon.net>
next in thread | previous in thread | raw e-mail | index | archive | help
--cQXOx3fnlpmgJsTP
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On Sun, Feb 25, 2007 at 12:41:44AM -0500, Alexandre Sunny Kovalenko wrote:
> On Sat, 2007-02-24 at 21:55 +0200, Kostik Belousov wrote:
> > On Thu, Feb 22, 2007 at 10:07:45AM -0500, Alexandre Sunny Kovalenko wro=
te:
> > > On Tue, 2007-02-20 at 09:48 -0500, Alexandre "Sunny" Kovalenko wrote:
> > > > On Wed, 2007-02-14 at 20:14 -0500, Kris Kennaway wrote:
> > > > > On Wed, Feb 14, 2007 at 07:48:51PM -0500, Alexandre Sunny Kovalen=
ko wrote:
> > > > > > On Tue, 2007-02-13 at 20:17 -0500, Kris Kennaway wrote:
> > > > > > > On Tue, Feb 13, 2007 at 08:02:39PM -0500, Alexandre Sunny Kov=
alenko wrote:
> > > > > > > > I can reliably panic -CURRENT (Feb 11, noon EST) with the s=
omething that
> > > > > > > > excersises the file system. I have currently settled on (cd=
/usr/ports;
> > > > > > > > make clean), but it all started out as doing some "emerges"=
to test the
> > > > > > > > latest linuxolator. In the case of the "make clean" I have =
seen it
> > > > > > > > crashing as early as /usr/ports/audio and as late
> > > > > > > > as /usr/ports/textproc.=20
> > > > > > > >=20
> > > I am still not capable to get good backtrace from the kernel dump, bu=
t I
> > > have managed to hook up remote console to this machine, so here are
> > > results:
> > >=20
> > > db> bt
> > > Tracing pid 33 tid 100032 td 0xc4cee510
> > > kdb_enter(c067c69d) at kdb_enter+0x2b
> > > panic(c0667ba3,c306d5c0,c306d5c0,e38a2cfc,c0619fd9,...) at panic+0x11c
> > > vm_pageq_remove_nowakeup(c306d5c0,c061a0b8,e38a2d04,c061a0ee,e38a2d24=
,...) at vm_pageq_remove_nowakeup+0x35
> > > vm_page_zero_idle(e38a2d24,c04c7fe4,0,e38a2d38,c4ef8900,...) at
> > > vm_page_zero_idle+0x49
> > > vm_pagezero(0,e38a2d38) at vm_pagezero+0x36
> > > fork_exit(c061a0b8,0,e38a2d38) at fork_exit+0xac
> > > fork_trampoline() at fork_trampoline+0x8
> > > --- trap 0, eip =3D 0, esp =3D 0xe38a2d70, ebp =3D 0 ---
> > > db> ps
> > > pid ppid pgrp uid state wmesg wchan cmd
> > > <snip>
> > > 33 0 0 0 RL CPU 0 [pagezero]
> > > <snip>
> > >=20
> > > ... and (hopefully) relevant bits from the source
> > > (kgdb) list *vm_pageq_remove_nowakeup+0x35
> > > 0xc06192f9 is in vm_pageq_remove_nowakeup
> > > (/usr/src/sys/vm/vm_pageq.c:223).
> > > 218 struct vpgqueues *pq;
> > > 219
> > > 220 if (queue !=3D PQ_NONE) {
> > > 221 pq =3D &vm_page_queues[queue];
> > > 222 VM_PAGE_SETQUEUE2(m, PQ_NONE);
> > > 223 TAILQ_REMOVE(&pq->pl, m, pageq);
> > There, please, show the output of "p/x *m" and "p/x *pq".
> >=20
> Unfortunately, with the latest -CURRENT the end result is different:
>=20
> Kernel page fault with the following non-sleepable locks held:
> exclusive sleep mutex sellck r =3D 0 (0xc0741284) locked
> @ /usr/src/sys/kern/sys_generic.c:776
> KDB: stack backtrace:
> db_trace_self_wrapper(c067f599) at db_trace_self_wrapper+0x25
> kdb_backtrace(1,c4fb46c0,c,e5447ab4,e5447aa8,...) at kdb_backtrace+0x29
> witness_warn(5,0,c069ed6f) at witness_warn+0x192
> trap(e5447ab4) at trap+0x10f
> calltrap() at calltrap+0x6
> --- trap 0xc, eip =3D 0x80bfe48d, esp =3D 0xe5447af4, ebp =3D 0xe5447c54 =
---
> kernload(c4e02d80,5,bfbfedb0,0,bfbfedb0,...) at -0x7f401b73
> select(c4e02d80,e5447d00) at select+0x44
> syscall(e5447d38) at syscall+0x256
> Xint0x80_syscall() at Xint0x80_syscall+0x20
> --- syscall (93, FreeBSD ELF32, select), eip =3D 0x2815f273, esp =3D
> 0xbfbfe82c, ebp =3D 0xbfbfee48 ---
>=20
>=20
> Fatal trap 12: page fault while in kernel mode
> cpuid =3D 0; apic id =3D 00
> fault virtual address =3D 0x80bfe48d
> fault code =3D supervisor read, page not present
> instruction pointer =3D 0x20:0x80bfe48d
> stack pointer =3D 0x28:0xe5447af4
> frame pointer =3D 0x28:0xe5447c54
> code segment =3D base 0x0, limit 0xfffff, type 0x1b
> =3D DPL 0, pres 1, def32 1, gran 1
> processor eflags =3D interrupt enabled, resume, IOPL =3D 0
> current process =3D 520 (powerd)
> [thread pid 520 tid 100052 ]
> Stopped at -0x7f401b73: *** error reading from address 80bfe48d
> ***
> db>=20
>=20
> db> where
> Tracing pid 520 tid 100052 td 0xc4e02d80
> kern_select(c4e02d80,5,bfbfedb0,0,bfbfedb0,...) at kern_select+0x4e5
> select(c4e02d80,e5447d00) at select+0x44
> syscall(e5447d38) at syscall+0x256
> Xint0x80_syscall() at Xint0x80_syscall+0x20
> --- syscall (93, FreeBSD ELF32, select), eip =3D 0x2815f273, esp =3D
> 0xbfbfe82c, ebp =3D 0xbfbfee48 ---
> db>=20
>=20
> what should I look for here?
Did you used some memory tester ? This looks like (random) memory corruptio=
n.
Reported fault address 0x80bfe48d belongs to user part of VA. Could you,
please, show the source line that corresponds the kern_select+0x4e5 in
your compiled kernel ?
--cQXOx3fnlpmgJsTP
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (FreeBSD)
iD8DBQFF4SQ2C3+MBN1Mb4gRAsmxAKCx8ZRmF7tb6ddTMRD/3ItWCFMc/QCg3ffn
47z8hMoGjL7TFUj36CZ0r8g=
=n5/Q
-----END PGP SIGNATURE-----
--cQXOx3fnlpmgJsTP--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20070225055254.GD77131>