Date: Fri, 11 Jan 2013 15:09:58 +0000 From: Po-Li Soong <polis@spectralogic.com> To: Konstantin Belousov <kostikbel@gmail.com> Cc: "stable@FreeBSD.org" <stable@FreeBSD.org> Subject: RE: zio_done panic on unadulterated FreeBSD Release 9.1 Message-ID: <0C4D65F6A0FC9E4B95EA114508C7E0FE5F66E4F3@reactor.sldomain.com> In-Reply-To: <20130109234924.GM2561@kib.kiev.ua> References: <0C4D65F6A0FC9E4B95EA114508C7E0FE5F66DDB6@reactor.sldomain.com> <20130109234924.GM2561@kib.kiev.ua>
next in thread | previous in thread | raw e-mail | index | archive | help
(kgdb) p/x *(struct vm_object *)0xffffffff81281580
$1 =3D {mtx =3D {lock_object =3D {lo_name =3D 0xffffffff80e54bbd,
lo_flags =3D 0x1430000, lo_data =3D 0x0, lo_witness =3D 0x0},
mtx_lock =3D 0xfffffe0006f44000}, object_list =3D {
tqe_next =3D 0xffffffff81281240, tqe_prev =3D 0xffffffff812814a0},
shadow_head =3D {lh_first =3D 0x0}, shadow_list =3D {le_next =3D 0x0,
le_prev =3D 0x0}, memq =3D {tqh_first =3D 0xfffffe00cfd3f880,
tqh_last =3D 0xfffffe00c9cac398}, root =3D 0xfffffe00cd733ab0,
size =3D 0x7ffffff, generation =3D 0x1, ref_count =3D 0x3f8, shadow_count=
=3D 0x0,
memattr =3D 0x6, type =3D 0x4, flags =3D 0x1000, pg_color =3D 0x0, pad1 =
=3D 0x0,
resident_page_count =3D 0x9b729, backing_object =3D 0x0,
backing_object_offset =3D 0x0, pager_object_list =3D {tqe_next =3D 0x0,
tqe_prev =3D 0x0}, rvq =3D {lh_first =3D 0xfffffe00c7dd2140}, cache =3D=
0x0,
handle =3D 0x0, un_pager =3D {vnp =3D {vnp_size =3D 0x0, writemappings =
=3D 0x0},
devp =3D {devp_pglist =3D {tqh_first =3D 0x0, tqh_last =3D 0x0}, ops =
=3D 0x0},
sgp =3D {sgp_pglist =3D {tqh_first =3D 0x0, tqh_last =3D 0x0}}, swp =3D=
{
swp_bcount =3D 0x0}}, cred =3D 0x0, charge =3D 0x0, paging_in_progres=
s =3D 0x1}
(kgdb) p/x *(struct vm_page *)0xfffffe00cd733ab0
$2 =3D {pageq =3D {tqe_next =3D 0x0, tqe_prev =3D 0xfffffe00c7e7d678}, list=
q =3D {
tqe_next =3D 0xfffffe00cd733b28, tqe_prev =3D 0xfffffe00cd7331d8},
left =3D 0xfffffe00c9b31c38, right =3D 0xfffffe00cd735c70,
object =3D 0xfffffffb81281580, pindex =3D 0x7495a, phys_addr =3D 0xbe95a0=
00, md =3D {
pv_list =3D {tqh_first =3D 0x0, tqh_last =3D 0xfffffe00cd733af8},
pat_mode =3D 0x6}, queue =3D 0xff, segind =3D 0x2, hold_count =3D 0x0,
order =3D 0xd, pool =3D 0x0, cow =3D 0x0, wire_count =3D 0x0, aflags =3D =
0x0,
flags =3D 0x0, oflags =3D 0x4, act_count =3D 0x0, busy =3D 0x0, valid =3D=
0xff,
dirty =3D 0x0}
(kgdb) list *vm_page_free_toq+0x45
0xffffffff80b506f5 is in vm_page_free_toq (/usr/src/sys/vm/vm_page.c:1878).
warning: Source file is more recent than executable.
1873
1874 /*
1875 * If fictitious remove object association and
1876 * return, otherwise delay object association removal.
1877 */
1878 if ((m->flags & PG_FICTITIOUS) !=3D 0) {
1879 return;
1880 }
1881
1882 m->valid =3D 0;
(kgdb)
-----Original Message-----
From: Konstantin Belousov [mailto:kostikbel@gmail.com]=20
Sent: Wednesday, January 09, 2013 4:49 PM
To: Po-Li Soong
Cc: stable@FreeBSD.org
Subject: Re: zio_done panic on unadulterated FreeBSD Release 9.1
On Wed, Jan 09, 2013 at 08:03:38PM +0000, Po-Li Soong wrote:
> Hi,
>=20
> My name is Po-Li Soong. I ran into a crash not long after installing the =
9.1 release on my home machine. I was performing a test run of file transfe=
r with samba server running on the FreeBSD installation. The transfer rate =
was about 70-80 MB/sec. The core.txt is attached. If there are other crash =
dumps needed, please let me know.
>=20
> I first discussed this panic with Justin Gibbs, a coworker of mine at Spe=
ctra Logic. He referred me to this email address, suggesting that the infor=
mation should be relevant to you. Thanks for the help.
>=20
> Regards,
>=20
> Po-Li Soong
>=20
> maestoso dumped core - see /var/crash/vmcore.0
>=20
> Sat Jan 5 19:53:24 MST 2013
>=20
> FreeBSD maestoso 9.1-RELEASE FreeBSD 9.1-RELEASE #0 r243825: Tue Dec 4 0=
9:23:10 UTC 2012 root@farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENE=
RIC amd64
>=20
> panic: page fault
>=20
> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and=20
> you are welcome to change it and/or distribute copies of it under certain=
conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB. Type "show warranty" for detail=
s.
> This GDB was configured as "amd64-marcel-freebsd"...
>=20
> Unread portion of the kernel message buffer:
>=20
>=20
> Fatal trap 12: page fault while in kernel mode cpuid =3D 1; apic id =3D 0=
1
> fault virtual address =3D 0xfffffffb812815d8
> fault code =3D supervisor read data, page not present
> instruction pointer =3D 0x20:0xffffffff80b50597
> stack pointer =3D 0x28:0xffffff80fa3bc8d0
> frame pointer =3D 0x28:0xffffff80fa3bc900
> code segment =3D base 0x0, limit 0xfffff, type 0x1b
> =3D DPL 0, pres 1, long 1, def32 0, gran 1
> processor eflags =3D interrupt enabled, resume, IOPL =3D 0
> current process =3D 0 (zio_write_intr_5)
> trap number =3D 12
> panic: page fault
> cpuid =3D 3
> KDB: stack backtrace:
> #0 0xffffffff809208a6 at kdb_backtrace+0x66
> #1 0xffffffff808ea8be at panic+0x1ce
> #2 0xffffffff80bd8240 at trap_fatal+0x290
> #3 0xffffffff80bd857d at trap_pfault+0x1ed
> #4 0xffffffff80bd8b9e at trap+0x3ce
> #5 0xffffffff80bc315f at calltrap+0x8
> #6 0xffffffff80b506f5 at vm_page_free_toq+0x45
> #7 0xffffffff80b4f276 at vm_object_page_remove+0x196
> #8 0xffffffff80b46b06 at vm_map_delete+0x316
> #9 0xffffffff80b46c11 at vm_map_remove+0x51
> #10 0xffffffff80b3a70a at uma_large_free+0x3a
> #11 0xffffffff808d589a at free+0x5a
> #12 0xffffffff8169b4ce at zio_done+0x2ee
> #13 0xffffffff81699063 at zio_execute+0xc3
> #14 0xffffffff8092cf55 at taskqueue_run_locked+0x85
> #15 0xffffffff8092ded6 at taskqueue_thread_loop+0x46
> #16 0xffffffff808bb9ef at fork_exit+0x11f
> #17 0xffffffff80bc368e at fork_trampoline+0xe
> Uptime: 3h19m34s
> Dumping 571 out of 3561=20
> MB:..3%..12%..23%..31%..42%..51%..62%..73%..82%..93%
>=20
> Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /boot/ker=
nel/zfs.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/zfs.ko Reading symbols from=20
> /boot/kernel/opensolaris.ko...Reading symbols from /boot/kernel/opensolar=
is.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/opensolaris.ko
> #0 doadump (textdump=3DVariable "textdump" is not available.
> ) at pcpu.h:224
> 224 pcpu.h: No such file or directory.
> in pcpu.h
> (kgdb) #0 doadump (textdump=3DVariable "textdump" is not available.
> ) at pcpu.h:224
> #1 0xffffffff808ea3a1 in kern_reboot (howto=3D260)
> at /usr/src/sys/kern/kern_shutdown.c:448
> #2 0xffffffff808ea897 in panic (fmt=3D0x1 <Address 0x1 out of bounds>)
> at /usr/src/sys/kern/kern_shutdown.c:636
> #3 0xffffffff80bd8240 in trap_fatal (frame=3D0xc, eva=3DVariable "eva" i=
s not available.
> )
> at /usr/src/sys/amd64/amd64/trap.c:857
> #4 0xffffffff80bd857d in trap_pfault (frame=3D0xffffff80fa3bc820, usermo=
de=3D0)
> at /usr/src/sys/amd64/amd64/trap.c:773
> #5 0xffffffff80bd8b9e in trap (frame=3D0xffffff80fa3bc820)
> at /usr/src/sys/amd64/amd64/trap.c:456
P
> #6 0xffffffff80bc315f in calltrap ()
> at /usr/src/sys/amd64/amd64/exception.S:228
> #7 0xffffffff80b50597 in vm_page_remove (m=3D0xfffffe00cd733ab0)
> at /usr/src/sys/vm/vm_page.c:975
> #8 0xffffffff80b506f5 in vm_page_free_toq (m=3D0xfffffe00cd733ab0)
> at /usr/src/sys/vm/vm_page.c:1872
> #9 0xffffffff80b4f276 in vm_object_page_remove (object=3D0xffffffff81281=
580,=20
> start=3D477512, end=3D477539, options=3DVariable "options" is not ava=
ilable.
> ) at /usr/src/sys/vm/vm_object.c:1899
> #10 0xffffffff80b46b06 in vm_map_delete (map=3D0xfffffe00020000e8, start=
=3DVariable "start" is not available.
> )
> at /usr/src/sys/vm/vm_map.c:2739
> #11 0xffffffff80b46c11 in vm_map_remove (map=3D0xfffffe00020000e8,=20
> start=3D18446743525909626880, end=3D18446743525909737472)
> at /usr/src/sys/vm/vm_map.c:2871
> #12 0xffffffff80b3a70a in uma_large_free (slab=3D0xfffffe00aceff8e0)
> at /usr/src/sys/vm/uma_core.c:3085
> #13 0xffffffff808d589a in free (addr=3D0xffffff8074948000,=20
> mtp=3D0xffffffff81747c20) at /usr/src/sys/kern/kern_malloc.c:572
> #14 0xffffffff8169b4ce in zio_done (zio=3D0xfffffe007a9906e0)
> at=20
> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/
> zfs/zio.c:2960
> #15 0xffffffff81699063 in zio_execute (zio=3D0xfffffe007a9906e0)
> at=20
> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/
> zfs/zio.c:1196
> #16 0xffffffff8092cf55 in taskqueue_run_locked (queue=3D0xfffffe0006ed9a0=
0)
> at /usr/src/sys/kern/subr_taskqueue.c:308
> #17 0xffffffff8092ded6 in taskqueue_thread_loop (arg=3DVariable "arg" is =
not available.
> )
> at /usr/src/sys/kern/subr_taskqueue.c:497
> #18 0xffffffff808bb9ef in fork_exit (
> callout=3D0xffffffff8092de90 <taskqueue_thread_loop>,=20
> arg=3D0xfffffe0006c072e0, frame=3D0xffffff80fa3bcc40)
> at /usr/src/sys/kern/kern_fork.c:992
> #19 0xffffffff80bc368e in fork_trampoline ()
> at /usr/src/sys/amd64/amd64/exception.S:602
> #20 0x0000000000000000 in ?? ()
> #21 0x0000000000000000 in ?? ()
> #22 0x0000000000000000 in ?? ()
> #23 0x0000000000000000 in ?? ()
> #24 0x0000000000000000 in ?? ()
> #25 0x0000000000000000 in ?? ()
> #26 0x0000000000000000 in ?? ()
> #27 0x0000000000000000 in ?? ()
> #28 0x0000000000000000 in ?? ()
> #29 0x0000000000000000 in ?? ()
> #30 0x0000000000000000 in ?? ()
> #31 0x0000000000000000 in ?? ()
> #32 0x0000000000000000 in ?? ()
> #33 0x0000000000000000 in ?? ()
> #34 0x0000000000000000 in ?? ()
> #35 0x0000000000000000 in ?? ()
> #36 0x0000000000000000 in ?? ()
> #37 0x0000000000000000 in ?? ()
> #38 0x0000000000000000 in ?? ()
> #39 0x0000000000000000 in ?? ()
> #40 0x0000000000000000 in ?? ()
> #41 0x0000000000000000 in ?? ()
> #42 0x0000000000000000 in ?? ()
> #43 0x0000000000000000 in ?? ()
> #44 0xffffffff81242880 in tdq_cpu ()
> #45 0xffffffff81242880 in tdq_cpu ()
> #46 0xfffffe0006f44000 in ?? ()
> #47 0x0000000000000000 in ?? ()
> #48 0xffffff80fa3bc290 in ?? ()
> #49 0xffffff80fa3bc238 in ?? ()
> #50 0xfffffe00049a88e0 in ?? ()
> #51 0xffffffff8091352e in sched_switch (td=3D0xffffffff812228a0,=20
> newtd=3D0xfffffe0006c072e0, flags=3DVariable "flags" is not available=
.
> ) at /usr/src/sys/kern/sched_ule.c:1921
> Previous frame inner to this frame (corrupt stack?)
> (kgdb)
Please, at the kgdb prompt, do
p/x *(struct vm_object *)0xffffffff81281580 p/x *(struct vm_page *)0xfffffe=
00cd733ab0 list *vm_page_free_toq+0x45
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?0C4D65F6A0FC9E4B95EA114508C7E0FE5F66E4F3>