From owner-freebsd-net@FreeBSD.ORG Tue Sep 4 16:06:22 2012 Return-Path: Delivered-To: freebsd-net@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 19D69106567B; Tue, 4 Sep 2012 16:06:22 +0000 (UTC) (envelope-from krassi@bulinfo.net) Received: from mx.bulinfo.net (mx.bulinfo.net [193.194.156.1]) by mx1.freebsd.org (Postfix) with ESMTP id 72BDD8FC1C; Tue, 4 Sep 2012 16:06:21 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mx.bulinfo.net (Postfix) with ESMTP id 5AEE75CA7C; Tue, 4 Sep 2012 19:06:14 +0300 (EEST) X-Virus-Scanned: amavisd-new at bulinfo.net Received: from mx.bulinfo.net ([127.0.0.1]) by localhost (mx.bulinfo.net [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id cCnReuFO-QrI; Tue, 4 Sep 2012 19:06:14 +0300 (EEST) Received: from [192.168.2.187] (pythia.bulinfo.net [212.72.195.5]) by mx.bulinfo.net (Postfix) with ESMTP id F131D5CA57; Tue, 4 Sep 2012 19:06:13 +0300 (EEST) Message-ID: <504626F6.3010009@bulinfo.net> Date: Tue, 04 Sep 2012 19:06:14 +0300 From: Krassimir Slavchev User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:10.0.6esrpre) Gecko/20120802 Thunderbird/10.0.6 MIME-Version: 1.0 To: freebsd-stable@FreeBSD.ORG, freebsd-net@FreeBSD.org References: <5044BA4D.2010403@bulinfo.net> In-Reply-To: <5044BA4D.2010403@bulinfo.net> X-Enigmail-Version: 1.4.3 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Subject: Re: bce related panic on 8.3-STABLE [updated] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Sep 2012 16:06:22 -0000 Hi, Crash dump files are available here: http://193.194.156.21/_debug/ #0 doadump () at /usr/src/sys/kern/kern_shutdown.c:266 #1 0xffffffff80223b9c in db_fncall (dummy1=Variable "dummy1" is not available. ) at /usr/src/sys/ddb/db_command.c:548 #2 0xffffffff80223ed1 in db_command (last_cmdp=0xffffffff80d06280, cmd_table=Variable "cmd_table" is not available. ) at /usr/src/sys/ddb/db_command.c:445 #3 0xffffffff80224120 in db_command_loop () at /usr/src/sys/ddb/db_command.c:498 #4 0xffffffff802261e9 in db_trap (type=Variable "type" is not available. ) at /usr/src/sys/ddb/db_main.c:231 #5 0xffffffff80696361 in kdb_trap (type=12, code=0, tf=0xffffff86c3de2680) at /usr/src/sys/kern/subr_kdb.c:654 #6 0xffffffff8093140d in trap_fatal (frame=0xffffff86c3de2680, eva=Variable "eva" is not available. ) at /usr/src/sys/amd64/amd64/trap.c:843 #7 0xffffffff8093178e in trap_pfault (frame=0xffffff86c3de2680, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:764 #8 0xffffffff80931c5e in trap (frame=0xffffff86c3de2680) at /usr/src/sys/amd64/amd64/trap.c:457 #9 0xffffffff809180b4 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:228 #10 0xffffffff80653198 in _thread_lock_flags (td=0xffffff000479f8d0, opts=Variable "opts" is not available. ) at /usr/src/sys/kern/kern_mutex.c:582 #11 0xffffffff806a4401 in propagate_priority (td=0xffffff000479f8d0) at /usr/src/sys/kern/subr_turnstile.c:210 #12 0xffffffff806a51ef in turnstile_wait (ts=Variable "ts" is not available. ) at /usr/src/sys/kern/subr_turnstile.c:743 #13 0xffffffff806610d1 in _rw_rlock (rw=0xffffffff80d66088, file=Variable "file" is not available. ) at /usr/src/sys/kern/kern_rwlock.c:470 #14 0xffffffff807e5ce1 in tcp_input (m=0xffffff003a2cda00, off0=20) at /usr/src/sys/netinet/tcp_input.c:745 #15 0xffffffff8077bf5c in ip_input (m=0xffffff003a2cda00) at /usr/src/sys/netinet/ip_input.c:787 #16 0xffffffff8072439e in netisr_dispatch_src (proto=1, source=Variable "source" is not available. ) at /usr/src/sys/net/netisr.c:859 #17 0xffffffff8071a36c in ether_demux (ifp=0xffffff00049b0000, m=0xffffff003a2cda00) at /usr/src/sys/net/if_ethersubr.c:899 #18 0xffffffff8071a757 in ether_input (ifp=0xffffff00049b0000, m=0xffffff003a2cda00) at /usr/src/sys/net/if_ethersubr.c:758 #19 0xffffffff80330e77 in bce_intr (xsc=Variable "xsc" is not available. ) at /usr/src/sys/dev/bce/if_bce.c:6903 #20 0xffffffff8063a824 in intr_event_execute_handlers (p=Variable "p" is not available. ) at /usr/src/sys/kern/kern_intr.c:1219 #21 0xffffffff8063beb5 in ithread_loop (arg=0xffffff00049c6ba0) at /usr/src/sys/kern/kern_intr.c:1232 #22 0xffffffff806374bf in fork_exit (callout=0xffffffff8063be20 , arg=0xffffff00049c6ba0, frame=0xffffff86c3de2c40) at /usr/src/sys/kern/kern_fork.c:876 #23 0xffffffff809185fe in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:602 #24 0x0000000000000000 in ?? () #25 0x0000000000000000 in ?? () #26 0x0000000000000001 in ?? () On 09/03/12 17:10, Krassimir Slavchev wrote: > Hi All, > > Today, after upgrading an HP Proliant DL380 G6 to 8.3-STABLE we had the > following panic few minutes after going to multiuser mode: > > http://193.194.156.21/bce_crash.jpg > > dmesg from 8.3-STABLE kernel (Note the link up/down events): > ... > Sep 3 14:19:54 m kernel: bce0: Server Adapter (C0)> mem 0xf4000000-0xf5ffffff irq 16 at device 0.0 on pci2 > Sep 3 14:19:54 m kernel: miibus0: on bce0 > Sep 3 14:19:54 m kernel: brgphy0: PHY > 1 on miibus0 > Sep 3 14:19:54 m kernel: brgphy0: 10baseT, 10baseT-FDX, 100baseTX, > 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, > 1000baseT-FDX-master, auto, auto-flow > Sep 3 14:19:54 m kernel: bce0: Ethernet address: 00:26:55:52:27:06 > Sep 3 14:19:54 m kernel: bce0: [ITHREAD] > Sep 3 14:19:54 mkernel: bce0: ASIC (0x57092003); Rev (C0); Bus (PCIe > x2, 2.5Gbps); B/C (4.6.4); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); > MFW (NCSI 1.0.3) > Sep 3 14:19:54 m kernel: Coal (RX:6,6,18,18; TX:20,20,80,80) > ... > Sep 3 14:19:54 m kernel: Trying to mount root from ufs:/dev/da0s1a > Sep 3 14:19:57 m kernel: bce0: > Sep 3 14:19:57 m kernel: bce0: link state changed to UP > Sep 3 14:19:57 m kernel: Gigabit link up! > Sep 3 14:19:57 m kernel: bce0: Gigabit link up! > Sep 3 14:19:58 m kernel: bce1: > Sep 3 14:19:58 m kernel: bce1: link state changed to UP > Sep 3 14:19:58 m kernel: Gigabit link up! > Sep 3 14:19:58 m kernel: bce1: Gigabit link up! > Sep 3 14:20:01 m kernel: bce1: Gigabit link up! > Sep 3 14:20:27 m kernel: bce0: Gigabit link up! > Sep 3 14:24:25 m syslogd: kernel boot file is /boot/kernel/kernel > ... > > > The previous kernel from the middle of May last year (8.2-STABLE): > > $sysctl dev.bce.0 > dev.bce.0.%desc: HP NC382i DP Multifunction Gigabit Server Adapter (C0) > dev.bce.0.%driver: bce > dev.bce.0.%location: slot=0 function=0 > dev.bce.0.%pnpinfo: vendor=0x14e4 device=0x1639 subvendor=0x103c > subdevice=0x7055 class=0x020000 > dev.bce.0.%parent: pci2 > dev.bce.0.l2fhdr_error_count: 0 > dev.bce.0.mbuf_alloc_failed_count: 0 > dev.bce.0.mbuf_frag_count: 0 > dev.bce.0.dma_map_addr_rx_failed_count: 0 > dev.bce.0.dma_map_addr_tx_failed_count: 51 > dev.bce.0.unexpected_attention_count: 0 > dev.bce.0.stat_IfHcInOctets: 8862469148 > dev.bce.0.stat_IfHCInBadOctets: 329986 > dev.bce.0.stat_IfHCOutOctets: 89884604332 > dev.bce.0.stat_IfHCOutBadOctets: 0 > dev.bce.0.stat_IfHCInUcastPkts: 47972963 > dev.bce.0.stat_IfHCInMulticastPkts: 0 > dev.bce.0.stat_IfHCInBroadcastPkts: 301 > dev.bce.0.stat_IfHCOutUcastPkts: 72217877 > dev.bce.0.stat_IfHCOutMulticastPkts: 0 > dev.bce.0.stat_IfHCOutBroadcastPkts: 45 > dev.bce.0.stat_emac_tx_stat_dot3statsinternalmactransmiterrors: 0 > dev.bce.0.stat_Dot3StatsCarrierSenseErrors: 0 > dev.bce.0.stat_Dot3StatsFCSErrors: 0 > dev.bce.0.stat_Dot3StatsAlignmentErrors: 0 > dev.bce.0.stat_Dot3StatsSingleCollisionFrames: 0 > dev.bce.0.stat_Dot3StatsMultipleCollisionFrames: 0 > dev.bce.0.stat_Dot3StatsDeferredTransmissions: 0 > dev.bce.0.stat_Dot3StatsExcessiveCollisions: 0 > dev.bce.0.stat_Dot3StatsLateCollisions: 0 > dev.bce.0.stat_EtherStatsCollisions: 0 > dev.bce.0.stat_EtherStatsFragments: 0 > dev.bce.0.stat_EtherStatsJabbers: 0 > dev.bce.0.stat_EtherStatsUndersizePkts: 0 > dev.bce.0.stat_EtherStatsOversizePkts: 0 > dev.bce.0.stat_EtherStatsPktsRx64Octets: 28900335 > dev.bce.0.stat_EtherStatsPktsRx65Octetsto127Octets: 11130062 > dev.bce.0.stat_EtherStatsPktsRx128Octetsto255Octets: 94457 > dev.bce.0.stat_EtherStatsPktsRx256Octetsto511Octets: 268122 > dev.bce.0.stat_EtherStatsPktsRx512Octetsto1023Octets: 6647988 > dev.bce.0.stat_EtherStatsPktsRx1024Octetsto1522Octets: 932300 > dev.bce.0.stat_EtherStatsPktsRx1523Octetsto9022Octets: 0 > dev.bce.0.stat_EtherStatsPktsTx64Octets: 2695217 > dev.bce.0.stat_EtherStatsPktsTx65Octetsto127Octets: 2635924 > dev.bce.0.stat_EtherStatsPktsTx128Octetsto255Octets: 2697153 > dev.bce.0.stat_EtherStatsPktsTx256Octetsto511Octets: 4127448 > dev.bce.0.stat_EtherStatsPktsTx512Octetsto1023Octets: 2505593 > dev.bce.0.stat_EtherStatsPktsTx1024Octetsto1522Octets: 57556587 > dev.bce.0.stat_EtherStatsPktsTx1523Octetsto9022Octets: 0 > dev.bce.0.stat_XonPauseFramesReceived: 0 > dev.bce.0.stat_XoffPauseFramesReceived: 0 > dev.bce.0.stat_OutXonSent: 0 > dev.bce.0.stat_OutXoffSent: 0 > dev.bce.0.stat_FlowControlDone: 0 > dev.bce.0.stat_MacControlFramesReceived: 0 > dev.bce.0.stat_XoffStateEntered: 0 > dev.bce.0.stat_IfInFramesL2FilterDiscards: 4331 > dev.bce.0.stat_IfInRuleCheckerDiscards: 0 > dev.bce.0.stat_IfInFTQDiscards: 0 > dev.bce.0.stat_IfInMBUFDiscards: 0 > dev.bce.0.stat_IfInRuleCheckerP4Hit: 301 > dev.bce.0.stat_CatchupInRuleCheckerDiscards: 0 > dev.bce.0.stat_CatchupInFTQDiscards: 0 > dev.bce.0.stat_CatchupInMBUFDiscards: 0 > dev.bce.0.stat_CatchupInRuleCheckerP4Hit: 0 > dev.bce.0.com_no_buffers: 0 > > $vmstat -i > > interrupt total rate > irq1: atkbd0 2035 0 > irq17: atapci0 35 0 > irq22: uhci2 uhci4 10250 1 > cpu0: timer 4649590 799 > irq256: ciss0 2682286 461 > irq257: bce0 47790473 8221 > irq258: bce1 2746014 472 > cpu1: timer 4646382 799 > cpu9: timer 4646256 799 > cpu6: timer 4646397 799 > cpu15: timer 4646263 799 > cpu7: timer 4646397 799 > cpu10: timer 4646250 799 > cpu5: timer 4646397 799 > cpu11: timer 4646263 799 > cpu4: timer 4646397 799 > cpu13: timer 4646227 799 > cpu3: timer 4646396 799 > cpu12: timer 4646244 799 > cpu2: timer 4646372 799 > cpu14: timer 4646263 799 > cpu8: timer 4646163 799 > Total 127575350 21946 > > $pciconv -lv > ... > bce0@pci0:2:0:0: class=0x020000 card=0x7055103c chip=0x163914e4 > rev=0x20 hdr=0x00 > vendor = 'Broadcom Corporation' > device = 'NetXtreme II Gigabit Ethernet (BCM5709)' > class = network > subclass = ethernet > bce1@pci0:2:0:1: class=0x020000 card=0x7055103c chip=0x163914e4 > rev=0x20 hdr=0x00 > vendor = 'Broadcom Corporation' > device = 'NetXtreme II Gigabit Ethernet (BCM5709)' > class = network > subclass = ethernet > bce2@pci0:3:0:0: class=0x020000 card=0x7055103c chip=0x163914e4 > rev=0x20 hdr=0x00 > vendor = 'Broadcom Corporation' > device = 'NetXtreme II Gigabit Ethernet (BCM5709)' > class = network > subclass = ethernet > bce3@pci0:3:0:1: class=0x020000 card=0x7055103c chip=0x163914e4 > rev=0x20 hdr=0x00 > vendor = 'Broadcom Corporation' > device = 'NetXtreme II Gigabit Ethernet (BCM5709)' > class = network > subclass = ethernet > ... > > > We had to increase RX_PAGES/TX_PAGES 2 -> 64 to suppress Ierrs shown by > nenstat -ni. > > > Best Regards > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >