From owner-freebsd-stable@FreeBSD.ORG Fri Nov 7 17:52:47 2003 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1F58B16A4CE; Fri, 7 Nov 2003 17:52:47 -0800 (PST) Received: from carver.gumbysoft.com (carver.gumbysoft.com [66.220.23.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6E88943F85; Fri, 7 Nov 2003 17:52:46 -0800 (PST) (envelope-from dwhite@gumbysoft.com) Received: by carver.gumbysoft.com (Postfix, from userid 1000) id 5E8AA72DB5; Fri, 7 Nov 2003 17:52:46 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by carver.gumbysoft.com (Postfix) with ESMTP id 590AC72DAD; Fri, 7 Nov 2003 17:52:46 -0800 (PST) Date: Fri, 7 Nov 2003 17:52:46 -0800 (PST) From: Doug White To: Prafulla Deuskar In-Reply-To: <20031107172535.A35573@hub.freebsd.org> Message-ID: <20031107174754.R27719@carver.gumbysoft.com> References: <20031107172535.A35573@hub.freebsd.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-stable@freebsd.org Subject: Re: Panic on -stable with 10 Gigabit driver X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 08 Nov 2003 01:52:47 -0000 On Fri, 7 Nov 2003, Prafulla Deuskar wrote: > We are having a reproducible panic with our 10 Gigabit driver on -stable > under stress. (4.8-RELEASE, 4.9-RELEASE) We have not been able to > reproduce the panic on 5.1-RELEASE with same load. What revision of uipc_mbuf.c generated this crashdump? I'm going to guess 4.8 RELEASE since it points to an actual line of code. I'm going to take a guess and shoot for cluster corruption of some sort. The offending line is MCLGET(mp, how); Which would explode if the free list is wrecked perhaps. If you can, load up the crashdump and poke around the freelist and see if something got spammed perhaps. > > The test is nttcp with 1500 byte MTU. > > If anyone on the list has any ideas please let me know. > > Thanks, > Prafulla > > Here is the crash dump output: > ------------------------------ > > #0 dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487 > 487 if (dumping++) { > (kgdb) where > #0 dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487 > #1 0xc021a823 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:316 > #2 0xc021ac61 in panic (fmt=0xc03b1944 "from debugger") > at /usr/src/sys/kern/kern_shutdown.c:595 > #3 0xc014be59 in db_panic (addr=-1071421816, have_addr=0, count=-1, > modif=0xc0417874 "") at /usr/src/sys/ddb/db_command.c:435 > #4 0xc014bdf9 in db_command (last_cmdp=0xc041daa4, cmd_table=0xc041d8e4, > aux_cmd_tablep=0xc0460d98) at /usr/src/sys/ddb/db_command.c:333 > #5 0xc014bebe in db_command_loop () at /usr/src/sys/ddb/db_command.c:457 > #6 0xc014dffb in db_trap (type=12, code=0) at /usr/src/sys/ddb/db_trap.c:71 > #7 0xc0379c8e in kdb_trap (type=12, code=0, regs=0xc04179c8) > at /usr/src/sys/i386/i386/db_interface.c:158 > #8 0xc038a4a4 in trap_fatal (frame=0xc04179c8, eva=117471232) > at /usr/src/sys/i386/i386/trap.c:969 > #9 0xc038a17d in trap_pfault (frame=0xc04179c8, usermode=0, eva=117471232) > at /usr/src/sys/i386/i386/trap.c:867 > #10 0xc0389cf7 in trap (frame={tf_fs = -1069481968, tf_es = -1069481968, tf_ds = 16, > tf_edi = 6756506, tf_esi = 1, tf_ebp = -1069450724, tf_isp = -1069450764, > tf_ebx = -1039395840, tf_edx = -1039732736, tf_ecx = 117471232, tf_eax = 565021, > tf_trapno = 12, tf_err = 0, tf_eip = -1071421816, tf_cs = 8, tf_eflags = 66050, > tf_esp = 0, tf_ss = -955602944}) at /usr/src/sys/i386/i386/trap.c:466 > #11 0xc0236688 in m_getcl (how=1, type=1, flags=2) at /usr/src/sys/kern/uipc_mbuf.c:589 > #12 0xc03a4282 in ixgb_get_buf (i=254, adapter=0xc70aa800, nmp=0x0) > at /usr/src/sys/dev/ixgb/if_ixgb.c:1768 > #13 0xc03a495d in ixgb_process_receive_interrupts (adapter=0xc70aa800, count=-35) > at /usr/src/sys/dev/ixgb/if_ixgb.c:2095 > #14 0xc03a310e in ixgb_intr (arg=0xc70aa800) at /usr/src/sys/dev/ixgb/if_ixgb.c:798 > #15 0xc0394cf9 in intr_mux (arg=0xc70acfe0) at /usr/src/sys/i386/isa/intr_machdep.c:582 > #16 0xc0380e02 in cpu_idle () at /usr/src/sys/i386/i386/machdep.c:1022 > (kgdb) up 10 > #10 0xc0389cf7 in trap (frame={tf_fs = -1069481968, tf_es = -1069481968, tf_ds = 16, > tf_edi = 6756506, tf_esi = 1, tf_ebp = -1069450724, tf_isp = -1069450764, > tf_ebx = -1039395840, tf_edx = -1039732736, tf_ecx = 117471232, tf_eax = 565021, > tf_trapno = 12, tf_err = 0, tf_eip = -1071421816, tf_cs = 8, tf_eflags = 66050, > tf_esp = 0, tf_ss = -955602944}) at /usr/src/sys/i386/i386/trap.c:466 > 466 (void) trap_pfault(&frame, FALSE, eva); > (kgdb) frame frame->tf_ebp frame->tf_eip > #0 0xc0236688 in m_getcl (how=1, type=1, flags=2) at /usr/src/sys/kern/uipc_mbuf.c:589 > 589 MCLGET(mp, how); > (kgdb) list > 584 } else > 585 MGETHDR(mp, how, type); > 586 } else > 587 MGET(mp, how, type); > 588 if (mp) { > 589 MCLGET(mp, how); > 590 if ( (mp->m_flags & M_EXT) == 0) { > 591 m_free(mp); > 592 mp = NULL; > 593 } > (kgdb) print mp > $1 = (struct mbuf *) 0xc20c1400 > (kgdb) print how > $2 = 1 > (kgdb) up > #1 0xc03a4282 in ixgb_get_buf (i=254, adapter=0xc70aa800, nmp=0x0) > at /usr/src/sys/dev/ixgb/if_ixgb.c:1768 > 1768 mp->m_next = NULL; > (kgdb) list > 1763 } > 1764 mp->m_len = mp->m_pkthdr.len = MCLBYTES; > 1765 } else { > 1766 mp->m_len = mp->m_pkthdr.len = MCLBYTES; > 1767 mp->m_data = mp->m_ext.ext_buf; > 1768 mp->m_next = NULL; > 1769 } > 1770 > 1771 if (ifp->if_mtu <= ETHERMTU) { > 1772 m_adj(mp, ETHER_ALIGN); > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > -- Doug White | FreeBSD: The Power to Serve dwhite@gumbysoft.com | www.FreeBSD.org