From owner-freebsd-bugs@FreeBSD.ORG Sun Nov 27 17:00:11 2005 Return-Path: X-Original-To: freebsd-bugs@hub.freebsd.org Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 30E5316A420 for ; Sun, 27 Nov 2005 17:00:11 +0000 (GMT) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id AA85843D55 for ; Sun, 27 Nov 2005 17:00:09 +0000 (GMT) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.13.3/8.13.3) with ESMTP id jARH09bM034553 for ; Sun, 27 Nov 2005 17:00:09 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.13.3/8.13.1/Submit) id jARH09nJ034551; Sun, 27 Nov 2005 17:00:09 GMT (envelope-from gnats) Resent-Date: Sun, 27 Nov 2005 17:00:09 GMT Resent-Message-Id: <200511271700.jARH09nJ034551@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Travis Mikalson Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0F89716A41F for ; Sun, 27 Nov 2005 16:49:56 +0000 (GMT) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (www.freebsd.org [216.136.204.117]) by mx1.FreeBSD.org (Postfix) with ESMTP id A110043D5A for ; Sun, 27 Nov 2005 16:49:55 +0000 (GMT) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (localhost [127.0.0.1]) by www.freebsd.org (8.13.1/8.13.1) with ESMTP id jARGntg5046120 for ; Sun, 27 Nov 2005 16:49:55 GMT (envelope-from nobody@www.freebsd.org) Received: (from nobody@localhost) by www.freebsd.org (8.13.1/8.13.1/Submit) id jARGnt48046119; Sun, 27 Nov 2005 16:49:55 GMT (envelope-from nobody) Message-Id: <200511271649.jARGnt48046119@www.freebsd.org> Date: Sun, 27 Nov 2005 16:49:55 GMT From: Travis Mikalson To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-2.3 Cc: Subject: kern/89633: [panic] if_sis panic under extended load in 6.0-RELEASE X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Nov 2005 17:00:11 -0000 >Number: 89633 >Category: kern >Synopsis: [panic] if_sis panic under extended load in 6.0-RELEASE >Confidential: no >Severity: critical >Priority: high >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Sun Nov 27 17:00:09 GMT 2005 >Closed-Date: >Last-Modified: >Originator: Travis Mikalson >Release: 6.0-RELEASE >Organization: TerraNovaNet, Inc. >Environment: FreeBSD tnn1.wlb.terranova.net 6.0-RELEASE FreeBSD 6.0-RELEASE #2: Mon Nov 26 12:50:26 EST 2005 root@freebsd6.tog.net:/usr/cfobj/usr/src/sys/cfbsd-wrap-debug i386 >Description: If I am understanding this correctly, if_sis seems to be panicking after a time. The load does not have to be that much, in fact an hour of 20+ mbit does not seem to reproduce the problem. The panic occurs after some time (every 2 - 16 hours) with just a couple mbit going in and out of the sis0 interface. The ethernet controller is embedded in a SBC called a "WRAP" board (http://www.pcengines.ch/wrap.htm) sis0: port 0x1000-0x10ff mem 0x80040000-0x80040fff irq 10 at device 14.0 on pci0 sis0: Silicon Revision: DP83816A The WRAP board has an ath minipci card plugged into it: ath0: mem 0x80000000-0x8000ffff irq 12 at device 13.0 on pci0 ath0: Ethernet address: 00:0b:6b:34:35:ee ath0: mac 5.9 phy 4.3 radio 3.6 This particular system's job in life is to use if_bridge to shuffle packets from ath0 to sis0 and vice-versa (basically an 802.11 access point) so that's what it's doing when the panics occur. For troubleshooting purposes I have disabled everything possible (pf and ipfw are disabled, net.link.bridge.ipfw and net.link.ether.ipfw are 0) Let me just mention I'm new to kernel debugging. Here's a backtrace from my dump: # kgdb kernel.debug /home/tog/crashes/vmcore.0 .. Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode fault virtual address = 0xbffffffc fault code = supervisor read, page not present instruction pointer = 0x20:0xc0625d25 stack pointer = 0x28:0xc571ec10 frame pointer = 0x28:0xc571ec60 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 20 (irq10: sis0) trap number = 12 panic: page fault Uptime: 1h54m5s Dumping 63 MB (2 chunks) chunk 0: 1MB (160 pages) ... ok chunk 1: 63MB (16128 pages) 48 32 16 <3>stray irq7 <3>stray irq7 <3>stray irq7 #0 doadump () at pcpu.h:165 165 __asm __volatile("movl %%fs:0,%0" : "=r" (td)); (kgdb) bt full #0 doadump () at pcpu.h:165 No locals. #1 0xc04cf2e2 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:399 first_buf_printf = 1 #2 0xc04cf578 in panic (fmt=0xc065cbe8 "%s") at /usr/src/sys/kern/kern_shutdown.c:555 td = (struct thread *) 0xc09b3000 bootopt = 260 newpanic = 0 ap = 0xc09b3000 "H,\233À \n\233À" buf = "page fault", '\0' #3 0xc06364c4 in trap_fatal (frame=0xc571ebd0, eva=3221225468) at /usr/src/sys/i386/i386/trap.c:831 code = 40 type = 12 ss = 40 esp = 0 softseg = {ssd_base = 0, ssd_limit = 1048575, ssd_type = 27, ssd_dpl = 0, ssd_p = 1, ssd_xx = 12, ssd_xx1 = 2, ssd_def32 = 1, ssd_gran = 1} #4 0xc063622f in trap_pfault (frame=0xc571ebd0, usermode=0, eva=3221225468) at /usr/src/sys/i386/i386/trap.c:742 va = 3221221376 vm = (struct vmspace *) 0x0 map = 0xc06add00 rv = 1 ftype = 1 '\001' td = (struct thread *) 0xc09b3000 p = (struct proc *) 0xc09b2c48 #5 0xc0635e6d in trap (frame= {tf_fs = -1064894456, tf_es = -982450136, tf_ds = -1067581400, tf_edi = -1063159808, tf_esi = -1062973440, tf_ebp = -982389664, tf_isp = -982389764, tf_ebx = -1061820672, tf_edx = 0, tf_ecx = -1, tf_eax = 1048575, tf_trapno = 12, tf_err = 0, tf_eip = -1067295451, tf_cs = -982450144, tf_eflags = 66055, tf_esp = -1063639808, tf_ss = -982389724}) at /usr/src/sys/i386/i386/trap.c:432 td = (struct thread *) 0xc09b3000 p = (struct proc *) 0xc09b2c48 sticks = 0 i = 0 ucode = 0 type = 12 code = 0 eva = 3221225468 #6 0xc0628aca in calltrap () at /usr/src/sys/i386/i386/exception.s:139 No locals. #7 0xc0625d25 in bus_dmamap_load (dmat=0xc09e5480, map=0xfffff, buf=0xffffffff, buflen=2048, callback=0xc05b9614 , callback_arg=0xc0a45000, flags=0) at pmap.h:200 lastaddr = 0 error = 0 nsegs = 0 #8 0xc05bbaa9 in sis_newbuf (sc=0xc0a17800, c=0xc0a45000, m=0xc0b5e700) at /usr/src/sys/pci/if_sis.c:1391 No locals. #9 0xc05bbbb7 in sis_rxeof (sc=0xc0a17800) at /usr/src/sys/pci/if_sis.c:1460 m = (struct mbuf *) 0xc0c13500 ifp = (struct ifnet *) 0xc0a1f400 cur_rx = (struct sis_desc *) 0xc0a45000 total_len = 60 rxstat = 2575302720 #10 0xc05bc0bb in sis_intr (arg=0xc0a17800) at /usr/src/sys/pci/if_sis.c:1668 sc = (struct sis_softc *) 0xc0a17800 ifp = (struct ifnet *) 0xc0a1f400 status = 9 #11 0xc04bbb11 in ithread_loop (arg=0xc09a2800) at /usr/src/sys/kern/kern_intr.c:547 ithd = (struct ithd *) 0xc09a2800 ih = (struct intrhand *) 0xc0a47800 td = (struct thread *) 0xc09b3000 p = (struct proc *) 0xc09b2c48 count = 0 warned = 0 #12 0xc04badc0 in fork_exit (callout=0xc04bb9b8 , arg=0xc09a2800, frame=0xc571ed38) at /usr/src/sys/kern/kern_fork.c:789 p = (struct proc *) 0xc09b2c48 td = (struct thread *) 0x0 #13 0xc0628b2c in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:208 No locals. (kgdb) up .. (kgdb) up #13 0xc0628b2c in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:208 208 call fork_exit Current language: auto; currently asm (kgdb) list 203 204 ENTRY(fork_trampoline) 205 pushl %esp /* trapframe pointer */ 206 pushl %ebx /* arg1 */ 207 pushl %esi /* function */ 208 call fork_exit 209 addl $12,%esp 210 /* cut from syscall */ 211 212 /* (kgdb) down #12 0xc04badc0 in fork_exit (callout=0xc04bb9b8 , arg=0xc09a2800, frame=0xc571ed38) at /usr/src/sys/kern/kern_fork.c:789 789 callout(arg, frame); Current language: auto; currently c (kgdb) list 784 * cpu_set_fork_handler intercepts this function call to 785 * have this call a non-return function to stay in kernel mode. 786 * initproc has its own fork handler, but it does return. 787 */ 788 KASSERT(callout != NULL, ("NULL callout in fork_exit")); 789 callout(arg, frame); 790 791 /* 792 * Check if a kernel thread misbehaved and returned from its main 793 * function. (kgdb) down #11 0xc04bbb11 in ithread_loop (arg=0xc09a2800) at /usr/src/sys/kern/kern_intr.c:547 547 ih->ih_handler(ih->ih_argument); (kgdb) list 542 mtx_unlock(&ithd->it_lock); 543 goto restart; 544 } 545 if ((ih->ih_flags & IH_MPSAFE) == 0) 546 mtx_lock(&Giant); 547 ih->ih_handler(ih->ih_argument); 548 if ((ih->ih_flags & IH_MPSAFE) == 0) 549 mtx_unlock(&Giant); 550 } 551 if (!(ithd->it_flags & IT_SOFT)) (kgdb) down #10 0xc05bc0bb in sis_intr (arg=0xc0a17800) at /usr/src/sys/pci/if_sis.c:1668 1668 sis_rxeof(sc); (kgdb) list 1663 (SIS_ISR_TX_DESC_OK | SIS_ISR_TX_ERR | 1664 SIS_ISR_TX_OK | SIS_ISR_TX_IDLE) ) 1665 sis_txeof(sc); 1666 1667 if (status & (SIS_ISR_RX_DESC_OK|SIS_ISR_RX_OK|SIS_ISR_RX_IDLE)) 1668 sis_rxeof(sc); 1669 1670 if (status & (SIS_ISR_RX_ERR | SIS_ISR_RX_OFLOW)) 1671 sis_rxeoc(sc); 1672 (kgdb) down #9 0xc05bbbb7 in sis_rxeof (sc=0xc0a17800) at /usr/src/sys/pci/if_sis.c:1460 1460 if (sis_newbuf(sc, cur_rx, NULL) == 0) (kgdb) list 1455 * copy done in m_devget(). 1456 * If we are on an architecture with alignment problems, or 1457 * if the allocation fails, then use m_devget and leave the 1458 * existing buffer in the receive ring. 1459 */ 1460 if (sis_newbuf(sc, cur_rx, NULL) == 0) 1461 m->m_pkthdr.len = m->m_len = total_len; 1462 else 1463 #endif 1464 { (kgdb) down #8 0xc05bbaa9 in sis_newbuf (sc=0xc0a17800, c=0xc0a45000, m=0xc0b5e700) at /usr/src/sys/pci/if_sis.c:1391 1391 bus_dmamap_load(sc->sis_tag, c->sis_map, (kgdb) list 1386 1387 c->sis_mbuf = m; 1388 c->sis_ctl = SIS_RXLEN; 1389 1390 bus_dmamap_create(sc->sis_tag, 0, &c->sis_map); 1391 bus_dmamap_load(sc->sis_tag, c->sis_map, 1392 mtod(m, void *), MCLBYTES, 1393 sis_dma_map_desc_ptr, c, 0); 1394 bus_dmamap_sync(sc->sis_tag, c->sis_map, BUS_DMASYNC_PREREAD); 1395 (kgdb) down #7 0xc0625d25 in bus_dmamap_load (dmat=0xc09e5480, map=0xfffff, buf=0xffffffff, buflen=2048, callback=0xc05b9614 , callback_arg=0xc0a45000, flags=0) at pmap.h:200 200 pa = *vtopte(va); (kgdb) list 195 vm_paddr_t pa; 196 197 if ((pa = PTD[va >> PDRSHIFT]) & PG_PS) { 198 pa = (pa & ~(NBPDR - 1)) | (va & (NBPDR - 1)); 199 } else { 200 pa = *vtopte(va); 201 pa = (pa & PG_FRAME) | (va & PAGE_MASK); 202 } 203 return pa; 204 } >How-To-Repeat: Run a NatSemi DP83816A with if_sis for 2 - 16 hours under some constant light load. >Fix: >Release-Note: >Audit-Trail: >Unformatted: