From owner-freebsd-stable@FreeBSD.ORG Sun Jan 13 17:55:41 2013 Return-Path: Delivered-To: stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 9903CC86 for ; Sun, 13 Jan 2013 17:55:41 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id D4CB6171 for ; Sun, 13 Jan 2013 17:55:40 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.6/8.14.6) with ESMTP id r0DHtRke073161; Sun, 13 Jan 2013 19:55:27 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.7.4 kib.kiev.ua r0DHtRke073161 Received: (from kostik@localhost) by tom.home (8.14.6/8.14.6/Submit) id r0DHtRNZ073160; Sun, 13 Jan 2013 19:55:27 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sun, 13 Jan 2013 19:55:27 +0200 From: Konstantin Belousov To: Po-Li Soong Subject: Re: zio_done panic on unadulterated FreeBSD Release 9.1 Message-ID: <20130113175527.GB2561@kib.kiev.ua> References: <0C4D65F6A0FC9E4B95EA114508C7E0FE5F66DDB6@reactor.sldomain.com> <20130109234924.GM2561@kib.kiev.ua> <0C4D65F6A0FC9E4B95EA114508C7E0FE5F66E4F3@reactor.sldomain.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="hTyD2MSx3HZlTIuF" Content-Disposition: inline In-Reply-To: <0C4D65F6A0FC9E4B95EA114508C7E0FE5F66E4F3@reactor.sldomain.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: "stable@FreeBSD.org" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Jan 2013 17:55:41 -0000 --hTyD2MSx3HZlTIuF Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Jan 11, 2013 at 03:09:58PM +0000, Po-Li Soong wrote: > (kgdb) p/x *(struct vm_object *)0xffffffff81281580 > $1 =3D {mtx =3D {lock_object =3D {lo_name =3D 0xffffffff80e54bbd, > lo_flags =3D 0x1430000, lo_data =3D 0x0, lo_witness =3D 0x0}, > mtx_lock =3D 0xfffffe0006f44000}, object_list =3D { > tqe_next =3D 0xffffffff81281240, tqe_prev =3D 0xffffffff812814a0}, > shadow_head =3D {lh_first =3D 0x0}, shadow_list =3D {le_next =3D 0x0, > le_prev =3D 0x0}, memq =3D {tqh_first =3D 0xfffffe00cfd3f880, > tqh_last =3D 0xfffffe00c9cac398}, root =3D 0xfffffe00cd733ab0, > size =3D 0x7ffffff, generation =3D 0x1, ref_count =3D 0x3f8, shadow_cou= nt =3D 0x0, > memattr =3D 0x6, type =3D 0x4, flags =3D 0x1000, pg_color =3D 0x0, pad1= =3D 0x0, > resident_page_count =3D 0x9b729, backing_object =3D 0x0, > backing_object_offset =3D 0x0, pager_object_list =3D {tqe_next =3D 0x0, > tqe_prev =3D 0x0}, rvq =3D {lh_first =3D 0xfffffe00c7dd2140}, cache = =3D 0x0, > handle =3D 0x0, un_pager =3D {vnp =3D {vnp_size =3D 0x0, writemappings = =3D 0x0}, > devp =3D {devp_pglist =3D {tqh_first =3D 0x0, tqh_last =3D 0x0}, ops = =3D 0x0}, > sgp =3D {sgp_pglist =3D {tqh_first =3D 0x0, tqh_last =3D 0x0}}, swp = =3D { > swp_bcount =3D 0x0}}, cred =3D 0x0, charge =3D 0x0, paging_in_progr= ess =3D 0x1} >=20 > (kgdb) p/x *(struct vm_page *)0xfffffe00cd733ab0 > $2 =3D {pageq =3D {tqe_next =3D 0x0, tqe_prev =3D 0xfffffe00c7e7d678}, li= stq =3D { > tqe_next =3D 0xfffffe00cd733b28, tqe_prev =3D 0xfffffe00cd7331d8}, > left =3D 0xfffffe00c9b31c38, right =3D 0xfffffe00cd735c70, > object =3D 0xfffffffb81281580, pindex =3D 0x7495a, phys_addr =3D 0xbe95= a000, md =3D { > pv_list =3D {tqh_first =3D 0x0, tqh_last =3D 0xfffffe00cd733af8}, > pat_mode =3D 0x6}, queue =3D 0xff, segind =3D 0x2, hold_count =3D 0x0, > order =3D 0xd, pool =3D 0x0, cow =3D 0x0, wire_count =3D 0x0, aflags = =3D 0x0, > flags =3D 0x0, oflags =3D 0x4, act_count =3D 0x0, busy =3D 0x0, valid = =3D 0xff, > dirty =3D 0x0} >=20 > (kgdb) list *vm_page_free_toq+0x45 > 0xffffffff80b506f5 is in vm_page_free_toq (/usr/src/sys/vm/vm_page.c:1878= ). > warning: Source file is more recent than executable. >=20 > 1873 > 1874 /* > 1875 * If fictitious remove object association and > 1876 * return, otherwise delay object association removal. > 1877 */ > 1878 if ((m->flags & PG_FICTITIOUS) !=3D 0) { > 1879 return; > 1880 } > 1881 > 1882 m->valid =3D 0; > (kgdb) This is strange. Can you disassemble your instance of the vm_page_free_toq() and show me the assembler listing ? The line you show has nothing to cause page fault if the m pointer itself is valid. >=20 >=20 > -----Original Message----- > From: Konstantin Belousov [mailto:kostikbel@gmail.com]=20 > Sent: Wednesday, January 09, 2013 4:49 PM > To: Po-Li Soong > Cc: stable@FreeBSD.org > Subject: Re: zio_done panic on unadulterated FreeBSD Release 9.1 >=20 > On Wed, Jan 09, 2013 at 08:03:38PM +0000, Po-Li Soong wrote: > > Hi, > >=20 > > My name is Po-Li Soong. I ran into a crash not long after installing th= e 9.1 release on my home machine. I was performing a test run of file trans= fer with samba server running on the FreeBSD installation. The transfer rat= e was about 70-80 MB/sec. The core.txt is attached. If there are other cras= h dumps needed, please let me know. > >=20 > > I first discussed this panic with Justin Gibbs, a coworker of mine at S= pectra Logic. He referred me to this email address, suggesting that the inf= ormation should be relevant to you. Thanks for the help. > >=20 > > Regards, > >=20 > > Po-Li Soong > >=20 >=20 > > maestoso dumped core - see /var/crash/vmcore.0 > >=20 > > Sat Jan 5 19:53:24 MST 2013 > >=20 > > FreeBSD maestoso 9.1-RELEASE FreeBSD 9.1-RELEASE #0 r243825: Tue Dec 4= 09:23:10 UTC 2012 root@farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GE= NERIC amd64 > >=20 > > panic: page fault > >=20 > > GNU gdb 6.1.1 [FreeBSD] > > Copyright 2004 Free Software Foundation, Inc. > > GDB is free software, covered by the GNU General Public License, and=20 > > you are welcome to change it and/or distribute copies of it under certa= in conditions. > > Type "show copying" to see the conditions. > > There is absolutely no warranty for GDB. Type "show warranty" for deta= ils. > > This GDB was configured as "amd64-marcel-freebsd"... > >=20 > > Unread portion of the kernel message buffer: > >=20 > >=20 > > Fatal trap 12: page fault while in kernel mode cpuid =3D 1; apic id =3D= 01 > > fault virtual address =3D 0xfffffffb812815d8 > > fault code =3D supervisor read data, page not present > > instruction pointer =3D 0x20:0xffffffff80b50597 > > stack pointer =3D 0x28:0xffffff80fa3bc8d0 > > frame pointer =3D 0x28:0xffffff80fa3bc900 > > code segment =3D base 0x0, limit 0xfffff, type 0x1b > > =3D DPL 0, pres 1, long 1, def32 0, gran 1 > > processor eflags =3D interrupt enabled, resume, IOPL =3D 0 > > current process =3D 0 (zio_write_intr_5) > > trap number =3D 12 > > panic: page fault > > cpuid =3D 3 > > KDB: stack backtrace: > > #0 0xffffffff809208a6 at kdb_backtrace+0x66 > > #1 0xffffffff808ea8be at panic+0x1ce > > #2 0xffffffff80bd8240 at trap_fatal+0x290 > > #3 0xffffffff80bd857d at trap_pfault+0x1ed > > #4 0xffffffff80bd8b9e at trap+0x3ce > > #5 0xffffffff80bc315f at calltrap+0x8 > > #6 0xffffffff80b506f5 at vm_page_free_toq+0x45 > > #7 0xffffffff80b4f276 at vm_object_page_remove+0x196 > > #8 0xffffffff80b46b06 at vm_map_delete+0x316 > > #9 0xffffffff80b46c11 at vm_map_remove+0x51 > > #10 0xffffffff80b3a70a at uma_large_free+0x3a > > #11 0xffffffff808d589a at free+0x5a > > #12 0xffffffff8169b4ce at zio_done+0x2ee > > #13 0xffffffff81699063 at zio_execute+0xc3 > > #14 0xffffffff8092cf55 at taskqueue_run_locked+0x85 > > #15 0xffffffff8092ded6 at taskqueue_thread_loop+0x46 > > #16 0xffffffff808bb9ef at fork_exit+0x11f > > #17 0xffffffff80bc368e at fork_trampoline+0xe > > Uptime: 3h19m34s > > Dumping 571 out of 3561=20 > > MB:..3%..12%..23%..31%..42%..51%..62%..73%..82%..93% > >=20 > > Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /boot/k= ernel/zfs.ko.symbols...done. > > done. > > Loaded symbols for /boot/kernel/zfs.ko Reading symbols from=20 > > /boot/kernel/opensolaris.ko...Reading symbols from /boot/kernel/opensol= aris.ko.symbols...done. > > done. > > Loaded symbols for /boot/kernel/opensolaris.ko > > #0 doadump (textdump=3DVariable "textdump" is not available. > > ) at pcpu.h:224 > > 224 pcpu.h: No such file or directory. > > in pcpu.h > > (kgdb) #0 doadump (textdump=3DVariable "textdump" is not available. > > ) at pcpu.h:224 > > #1 0xffffffff808ea3a1 in kern_reboot (howto=3D260) > > at /usr/src/sys/kern/kern_shutdown.c:448 > > #2 0xffffffff808ea897 in panic (fmt=3D0x1
) > > at /usr/src/sys/kern/kern_shutdown.c:636 > > #3 0xffffffff80bd8240 in trap_fatal (frame=3D0xc, eva=3DVariable "eva"= is not available. > > ) > > at /usr/src/sys/amd64/amd64/trap.c:857 > > #4 0xffffffff80bd857d in trap_pfault (frame=3D0xffffff80fa3bc820, user= mode=3D0) > > at /usr/src/sys/amd64/amd64/trap.c:773 > > #5 0xffffffff80bd8b9e in trap (frame=3D0xffffff80fa3bc820) > > at /usr/src/sys/amd64/amd64/trap.c:456 > P > > #6 0xffffffff80bc315f in calltrap () > > at /usr/src/sys/amd64/amd64/exception.S:228 > > #7 0xffffffff80b50597 in vm_page_remove (m=3D0xfffffe00cd733ab0) > > at /usr/src/sys/vm/vm_page.c:975 > > #8 0xffffffff80b506f5 in vm_page_free_toq (m=3D0xfffffe00cd733ab0) > > at /usr/src/sys/vm/vm_page.c:1872 > > #9 0xffffffff80b4f276 in vm_object_page_remove (object=3D0xffffffff812= 81580,=20 > > start=3D477512, end=3D477539, options=3DVariable "options" is not a= vailable. > > ) at /usr/src/sys/vm/vm_object.c:1899 > > #10 0xffffffff80b46b06 in vm_map_delete (map=3D0xfffffe00020000e8, star= t=3DVariable "start" is not available. > > ) > > at /usr/src/sys/vm/vm_map.c:2739 > > #11 0xffffffff80b46c11 in vm_map_remove (map=3D0xfffffe00020000e8,=20 > > start=3D18446743525909626880, end=3D18446743525909737472) > > at /usr/src/sys/vm/vm_map.c:2871 > > #12 0xffffffff80b3a70a in uma_large_free (slab=3D0xfffffe00aceff8e0) > > at /usr/src/sys/vm/uma_core.c:3085 > > #13 0xffffffff808d589a in free (addr=3D0xffffff8074948000,=20 > > mtp=3D0xffffffff81747c20) at /usr/src/sys/kern/kern_malloc.c:572 > > #14 0xffffffff8169b4ce in zio_done (zio=3D0xfffffe007a9906e0) > > at=20 > > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/ > > zfs/zio.c:2960 > > #15 0xffffffff81699063 in zio_execute (zio=3D0xfffffe007a9906e0) > > at=20 > > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/ > > zfs/zio.c:1196 > > #16 0xffffffff8092cf55 in taskqueue_run_locked (queue=3D0xfffffe0006ed9= a00) > > at /usr/src/sys/kern/subr_taskqueue.c:308 > > #17 0xffffffff8092ded6 in taskqueue_thread_loop (arg=3DVariable "arg" i= s not available. > > ) > > at /usr/src/sys/kern/subr_taskqueue.c:497 > > #18 0xffffffff808bb9ef in fork_exit ( > > callout=3D0xffffffff8092de90 ,=20 > > arg=3D0xfffffe0006c072e0, frame=3D0xffffff80fa3bcc40) > > at /usr/src/sys/kern/kern_fork.c:992 > > #19 0xffffffff80bc368e in fork_trampoline () > > at /usr/src/sys/amd64/amd64/exception.S:602 > > #20 0x0000000000000000 in ?? () > > #21 0x0000000000000000 in ?? () > > #22 0x0000000000000000 in ?? () > > #23 0x0000000000000000 in ?? () > > #24 0x0000000000000000 in ?? () > > #25 0x0000000000000000 in ?? () > > #26 0x0000000000000000 in ?? () > > #27 0x0000000000000000 in ?? () > > #28 0x0000000000000000 in ?? () > > #29 0x0000000000000000 in ?? () > > #30 0x0000000000000000 in ?? () > > #31 0x0000000000000000 in ?? () > > #32 0x0000000000000000 in ?? () > > #33 0x0000000000000000 in ?? () > > #34 0x0000000000000000 in ?? () > > #35 0x0000000000000000 in ?? () > > #36 0x0000000000000000 in ?? () > > #37 0x0000000000000000 in ?? () > > #38 0x0000000000000000 in ?? () > > #39 0x0000000000000000 in ?? () > > #40 0x0000000000000000 in ?? () > > #41 0x0000000000000000 in ?? () > > #42 0x0000000000000000 in ?? () > > #43 0x0000000000000000 in ?? () > > #44 0xffffffff81242880 in tdq_cpu () > > #45 0xffffffff81242880 in tdq_cpu () > > #46 0xfffffe0006f44000 in ?? () > > #47 0x0000000000000000 in ?? () > > #48 0xffffff80fa3bc290 in ?? () > > #49 0xffffff80fa3bc238 in ?? () > > #50 0xfffffe00049a88e0 in ?? () > > #51 0xffffffff8091352e in sched_switch (td=3D0xffffffff812228a0,=20 > > newtd=3D0xfffffe0006c072e0, flags=3DVariable "flags" is not availab= le. > > ) at /usr/src/sys/kern/sched_ule.c:1921 > > Previous frame inner to this frame (corrupt stack?) > > (kgdb) >=20 > Please, at the kgdb prompt, do > p/x *(struct vm_object *)0xffffffff81281580 p/x *(struct vm_page *)0xffff= fe00cd733ab0 list *vm_page_free_toq+0x45 --hTyD2MSx3HZlTIuF Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iQIcBAEBAgAGBQJQ8vUNAAoJEJDCuSvBvK1BQsAP/A51XZ9Yvs0gWK828YVaSuha we8+xDHCzH3u7OTi7zAj95P+lm3d+V6dw+7ihQAE+3OliXpdytW7bP/R8aJrByz5 DrvLsrdZFOGEEJR3LS60RR5wQzUFD5S/8wboiudoIzaI9RCqgssSbi7uoPYpf7vJ Y8Bo3dHRPhVJy4RXwJwhVi6DfCt+Hge037mnkYctS02DQHfbsOZRiPhKvvpirk4n 6xkO7mcOEA5zqy9M2QwGOjyWzr1ImEMzF5lDFr+v1O7Q98fyLBgQHIw7EjaK/kco KFIWFQacT1wwMy+b4cYocOb18KvNrO9xsbfZ4W3WXvSyiRgicGlijTO1uUifnsoi 0YBRl0CRKOxyIkfh+DsYfr7aAWpuINni37DKYhnE1qGlni8HphQAzfJBgtdgc1OI 78/h+mI8zrJVcGNApyGmqdvGSbnL13S+tYVQsiV+I5uSq7Wzo0LIYLSqVngsp4jF 2vR3VwDxwlXX0AyEouUhmDi9uIG+YCO052PwqeZiQsP1f3/Cygx1KZ/Du27Ake3T /p4nAQ/UMD/OE6QEpnG12pqNXuCNygSRernyMSK2gIhMk1dUf70XZvv0bXO+8wHI 29tauVgwTZSNRJ2thv5gAy3igchsyliq0hZLYOD7w6Qhx2lAaLUKXZWmKH9bDqWV Q7im8ewwqAinRfkMOCVS =hWUL -----END PGP SIGNATURE----- --hTyD2MSx3HZlTIuF--