Date: Tue, 23 Feb 2010 08:18:10 -0700 From: "Kirk Davis" <kirk.davis@epsb.ca> To: "Jack Vogel" <jfvogel@gmail.com> Cc: freebsd-net@freebsd.org, Mike Tancsa <mike@sentex.net> Subject: RE: Intel em0: watchdog timeout Message-ID: <529374128DC1B04D9D037911B8E8F05301C17A5C@Exchange26.EDU.epsb.ca> In-Reply-To: <2a41acea1002221629vbe7548am7b5f1ba94d7efa9f@mail.gmail.com> References: <529374128DC1B04D9D037911B8E8F05301C17A51@Exchange26.EDU.epsb.ca> <43416_1266864062_4B82CFBE_43416_81_1_2a41acea1002221043k1b8742c9m8fb484a8e8a4fdda@mail.gmail.com> <529374128DC1B04D9D037911B8E8F05301C17A54@Exchange26.EDU.epsb.ca> <43669_1266865888_4B82D6E0_43669_263_1_2a41acea1002221113v26804200q4f3971c3359dffab@mail.gmail.com> <529374128DC1B04D9D037911B8E8F05301C17A55@Exchange26.EDU.epsb.ca> <201002222107.o1ML7v3Z059734@lava.sentex.ca> <529374128DC1B04D9D037911B8E8F05301C17A56@Exchange26.EDU.epsb.ca> <2a41acea1002221444o6e449602m1830761b21837c41@mail.gmail.com> <529374128DC1B04D9D037911B8E8F05301C17A57@Exchange26.EDU.epsb.ca> <2a41acea1002221629vbe7548am7b5f1ba94d7efa9f@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
The driver is static right now. I have another Dell 2950 for backup so I'm going to split the mirror on the production server and put one of the drives into the development server. Then I can do the upgrade to 7.2 and test. I don't think I will be able to get a window to switch out the production box until this weekend at the earliest. =20 How far behind is the driver in 7.2-RELEASE with any patches applied? Would this get me close enough to the latest driver? I'm not sure if I'm ready to go to 7.3-RC1 on my production server. =20 ----- Kirk ________________________________ From: Jack Vogel [mailto:jfvogel@gmail.com]=20 Sent: Monday, February 22, 2010 5:30 PM To: Kirk Davis Cc: Mike Tancsa; freebsd-net@freebsd.org Subject: Re: Intel em0: watchdog timeout =09 =09 Is your driver static, ie builtin, to the kernel, or do you load/unload it as a module? I ask because perhaps we could try a later driver, and being a module makes that easier.=20 =09 Jack =09 =09 =09 On Mon, Feb 22, 2010 at 3:37 PM, Kirk Davis <kirk.davis@epsb.ca> wrote: =09 OK. I have the following in /boot/loader.conf (and rebooted) hw.em.rxd=3D1024 hw.em.txd=3D1024 =20 Should this be hw.em2.rxd? Is it set per interface or across all interfaces? =20 nmbcluster=3D262144 =20 # sysctl dev.em.2.stats=3D1 Feb 22 16:29:57 inet-gw kernel: em2: Defer count =3D 20 Feb 22 16:29:57 inet-gw kernel: em2: Missed Packets =3D 119947 =20 Feb 22 16:29:57 inet-gw kernel: em2: Receive No Buffers =3D 276762 Feb 22 16:29:57 inet-gw kernel: em2: Receive Length Errors =3D 0=20 Feb 22 16:29:57 inet-gw kernel: em2: Receive errors =3D 0 Feb 22 16:29:57 inet-gw kernel: em2: Crc errors =3D 0 Feb 22 16:29:57 inet-gw kernel: em2: Alignment errors =3D 0 Feb 22 16:29:57 inet-gw kernel: em2: Collision/Carrier extension errors =3D 0 Feb 22 16:29:57 inet-gw kernel: em2: RX overruns =3D 21 Feb 22 16:29:57 inet-gw kernel: em2: watchdog timeouts =3D 47 Feb 22 16:29:57 inet-gw kernel: em2: RX MSIX IRQ =3D 0 TX MSIX IRQ =3D 0 LINK MSIX IRQ =3D 0 Feb 22 16:29:57 inet-gw kernel: em2: XON Rcvd =3D 22 Feb 22 16:29:57 inet-gw kernel: em2: XON Xmtd =3D 8349 Feb 22 16:29:57 inet-gw kernel: em2: XOFF Rcvd =3D 31 Feb 22 16:29:57 inet-gw kernel: em2: XOFF Xmtd =3D 15779 Feb 22 16:29:57 inet-gw kernel: em2: Good Packets Rcvd =3D 966101852 Feb 22 16:29:57 inet-gw kernel: em2: Good Packets Xmtd =3D 755993237 Feb 22 16:29:57 inet-gw kernel: em2: TSO Contexts Xmtd =3D 0 Feb 22 16:29:57 inet-gw kernel: em2: TSO Contexts Failed =3D 0 =20 still seeing the watchdog timer and link up/down messages. =20 Should I try going higher than 1024 on the hw.em.rxd? I'm not sure the next time I can schedule another reboot on this production server. =20 ---- Kirk =20 Kirk Davis=20 Senior Network Analyst, ITS=20 Edmonton Public Schools=20 One Kingsway Ave.=20 Edmonton, Alberta, Canada=20 T5H 4G9=20 phone: 1-780-429-8308=20 =20 ________________________________ =09 From: Jack Vogel [mailto:jfvogel@gmail.com]=20 =09 Sent: Monday, February 22, 2010 3:45 PM To: Kirk Davis Cc: Mike Tancsa; freebsd-net@freebsd.org=20 Subject: Re: Intel em0: watchdog timeout =09 OK, so you are still failing to get mbufs in the RX side, increase the nmbcluster value, and then what size is your RX ring (number of rx descriptors)? =09 If you havent already done so, change that to 1024.=20 =09 I am developing a change in the RX code right now that will help this situation, but am doing so in the 10G driver, once its solid there I will be backporting it into the 1G drivers, it will make discards almost unnecessary. =09 Jack =09 =09 On Mon, Feb 22, 2010 at 1:43 PM, Kirk Davis <kirk.davis@epsb.ca> wrote: =09 > -----Original Message----- > From: Mike Tancsa [mailto:mike@sentex.net] > Subject: Re: Intel em0: watchdog timeout > > At 03:46 PM 2/22/2010, Kirk Davis wrote: > >Does this need to be done in loader.conf? It doesn't seem > to take from > >the command line. > ># sysctl dev.em.2.stats=3D1 > >dev.em.2.stats: -1 -> -1 > > > ># sysctl dev.em.2.stats > >dev.em.2.stats: -1 > > Hi, > After you issue those commands, the driver will spit out a > lot of useful stats to syslog. It will report something like the > following in /var/log/messages > > Feb 22 16:06:31 offsite kernel: em0: Excessive collisions =3D 0 > Feb 22 16:06:31 offsite kernel: em0: Sequence errors =3D 0 > Feb 22 16:06:31 offsite kernel: em0: Defer count =3D 0 > Feb 22 16:06:31 offsite kernel: em0: Missed Packets =3D 0 > Feb 22 16:06:31 offsite kernel: em0: Receive No Buffers =3D 0 > Feb 22 16:06:31 offsite kernel: em0: Receive Length Errors =3D 0 > Feb 22 16:06:31 offsite kernel: em0: Receive errors =3D 0 > Feb 22 16:06:31 offsite kernel: em0: Crc errors =3D 0 > Feb 22 16:06:31 offsite kernel: em0: Alignment errors =3D 0 > Feb 22 16:06:31 offsite kernel: em0: Collision/Carrier > extension errors =3D 0 > Feb 22 16:06:31 offsite kernel: em0: RX overruns =3D 0 > Feb 22 16:06:31 offsite kernel: em0: watchdog timeouts =3D 0 > Feb 22 16:06:31 offsite kernel: em0: RX MSIX IRQ =3D 0 TX MSIX IRQ =3D 0 > LINK MSIX IRQ =3D 0 > Feb 22 16:06:31 offsite kernel: em0: XON Rcvd =3D 0 > Feb 22 16:06:31 offsite kernel: em0: XON Xmtd =3D 0 > Feb 22 16:06:31 offsite kernel: em0: XOFF Rcvd =3D 0 > Feb 22 16:06:31 offsite kernel: em0: XOFF Xmtd =3D 0 > Feb 22 16:06:31 offsite kernel: em0: Good Packets Rcvd =3D 2559032551 > Feb 22 16:06:31 offsite kernel: em0: Good Packets Xmtd =3D 1568751141 > Feb 22 16:06:31 offsite kernel: em0: TSO Contexts Xmtd =3D 0 > Feb 22 16:06:31 offsite kernel: em0: TSO Contexts Failed =3D 0 =09 =09 Thanks Mike and Jack. I don't know why I didn'ty notice the output in /var/log/messages =09 Here is the output for the two interfaces that are causing this issue. =09 Feb 22 13:33:52 inet-gw kernel: em0: Excessive collisions =3D 0 Feb 22 13:33:52 inet-gw kernel: em0: Sequence errors =3D 0 Feb 22 13:33:52 inet-gw kernel: em0: Defer count =3D 0 Feb 22 13:33:52 inet-gw kernel: em0: Missed Packets =3D 24296 Feb 22 13:33:52 inet-gw kernel: em0: Receive No Buffers =3D 0 Feb 22 13:33:52 inet-gw kernel: em0: Receive Length Errors =3D 0 Feb 22 13:33:52 inet-gw kernel: em0: Receive errors =3D 0 Feb 22 13:33:52 inet-gw kernel: em0: Crc errors =3D 0 Feb 22 13:33:52 inet-gw kernel: em0: Alignment errors =3D 0 Feb 22 13:33:52 inet-gw kernel: em0: Collision/Carrier extension errors =3D 0 Feb 22 13:33:52 inet-gw kernel: em0: RX overruns =3D 0 Feb 22 13:33:52 inet-gw kernel: em0: watchdog timeouts =3D 6 Feb 22 13:33:52 inet-gw kernel: em0: RX MSIX IRQ =3D 0 TX MSIX IRQ =3D 0 LINK MSIX IRQ =3D 0 Feb 22 13:33:52 inet-gw kernel: em0: XON Rcvd =3D 0 Feb 22 13:33:52 inet-gw kernel: em0: XON Xmtd =3D 0 Feb 22 13:33:52 inet-gw kernel: em0: XOFF Rcvd =3D 0 Feb 22 13:33:52 inet-gw kernel: em0: XOFF Xmtd =3D 0 Feb 22 13:33:52 inet-gw kernel: em0: Good Packets Rcvd =3D 424303810 Feb 22 13:33:52 inet-gw kernel: em0: Good Packets Xmtd =3D 576529136 Feb 22 13:33:52 inet-gw kernel: em0: TSO Contexts Xmtd =3D 0 Feb 22 13:33:52 inet-gw kernel: em0: TSO Contexts Failed =3D 0 Feb 22 13:34:12 inet-gw kernel: em2: Excessive collisions =3D 0 Feb 22 13:34:12 inet-gw kernel: em2: Sequence errors =3D 0 Feb 22 13:34:12 inet-gw kernel: em2: Defer count =3D 20 Feb 22 13:34:12 inet-gw kernel: em2: Missed Packets =3D 68059 Feb 22 13:34:12 inet-gw kernel: em2: Receive No Buffers =3D 275612 Feb 22 13:34:12 inet-gw kernel: em2: Receive Length Errors =3D 0 Feb 22 13:34:12 inet-gw kernel: em2: Receive errors =3D 0 Feb 22 13:34:12 inet-gw kernel: em2: Crc errors =3D 0 Feb 22 13:34:12 inet-gw kernel: em2: Alignment errors =3D 0 Feb 22 13:34:12 inet-gw kernel: em2: Collision/Carrier extension errors =3D 0 Feb 22 13:34:12 inet-gw kernel: em2: RX overruns =3D 17 Feb 22 13:34:12 inet-gw kernel: em2: watchdog timeouts =3D 38 Feb 22 13:34:12 inet-gw kernel: em2: RX MSIX IRQ =3D 0 TX MSIX IRQ =3D 0 LINK MSIX IRQ =3D 0 Feb 22 13:34:12 inet-gw kernel: em2: XON Rcvd =3D 21 Feb 22 13:34:12 inet-gw kernel: em2: XON Xmtd =3D 8344 Feb 22 13:34:12 inet-gw kernel: em2: XOFF Rcvd =3D 30 Feb 22 13:34:12 inet-gw kernel: em2: XOFF Xmtd =3D 9159 Feb 22 13:34:12 inet-gw kernel: em2: Good Packets Rcvd =3D 713607509 Feb 22 13:34:12 inet-gw kernel: em2: Good Packets Xmtd =3D 569694020 Feb 22 13:34:12 inet-gw kernel: em2: TSO Contexts Xmtd =3D 0 Feb 22 13:34:12 inet-gw kernel: em2: TSO Contexts Failed =3D 0 Feb 22 13:35:10 inet-gw kernel: em2: Excessive collisions =3D 0 Feb 22 13:35:10 inet-gw kernel: em2: Sequence errors =3D 0 Feb 22 13:35:10 inet-gw kernel: em2: Defer count =3D 20 Feb 22 13:35:10 inet-gw kernel: em2: Missed Packets =3D 68059 Feb 22 13:35:10 inet-gw kernel: em2: Receive No Buffers =3D 275612 Feb 22 13:35:10 inet-gw kernel: em2: Receive Length Errors =3D 0 Feb 22 13:35:10 inet-gw kernel: em2: Receive errors =3D 0 Feb 22 13:35:10 inet-gw kernel: em2: Crc errors =3D 0 Feb 22 13:35:10 inet-gw kernel: em2: Alignment errors =3D 0 Feb 22 13:35:10 inet-gw kernel: em2: Collision/Carrier extension errors =3D 0 Feb 22 13:35:10 inet-gw kernel: em2: RX overruns =3D 17 Feb 22 13:35:10 inet-gw kernel: em2: watchdog timeouts =3D 38 Feb 22 13:35:10 inet-gw kernel: em2: RX MSIX IRQ =3D 0 TX MSIX IRQ =3D 0 LINK MSIX IRQ =3D 0 Feb 22 13:35:10 inet-gw kernel: em2: XON Rcvd =3D 21 Feb 22 13:35:10 inet-gw kernel: em2: XON Xmtd =3D 8344 Feb 22 13:35:10 inet-gw kernel: em2: XOFF Rcvd =3D 30 Feb 22 13:35:10 inet-gw kernel: em2: XOFF Xmtd =3D 9159 Feb 22 13:35:10 inet-gw kernel: em2: Good Packets Rcvd =3D 715555016 Feb 22 13:35:10 inet-gw kernel: em2: Good Packets Xmtd =3D 571157561 Feb 22 13:35:10 inet-gw kernel: em2: TSO Contexts Xmtd =3D 0 Feb 22 13:35:10 inet-gw kernel: em2: TSO Contexts Failed =3D 0 Feb 22 13:39:12 inet-gw kernel: em2: Excessive collisions =3D 0 Feb 22 13:39:12 inet-gw kernel: em2: Sequence errors =3D 0 Feb 22 13:39:12 inet-gw kernel: em2: Defer count =3D 20 Feb 22 13:39:12 inet-gw kernel: em2: Missed Packets =3D 68059 Feb 22 13:39:12 inet-gw kernel: em2: Receive No Buffers =3D 275612 Feb 22 13:39:12 inet-gw kernel: em2: Receive Length Errors =3D 0 Feb 22 13:39:12 inet-gw kernel: em2: Receive errors =3D 0 Feb 22 13:39:12 inet-gw kernel: em2: Crc errors =3D 0 Feb 22 13:39:12 inet-gw kernel: em2: Alignment errors =3D 0 Feb 22 13:39:12 inet-gw kernel: em2: Collision/Carrier extension errors =3D 0 Feb 22 13:39:12 inet-gw kernel: em2: RX overruns =3D 17 Feb 22 13:39:12 inet-gw kernel: em2: watchdog timeouts =3D 38 Feb 22 13:39:12 inet-gw kernel: em2: RX MSIX IRQ =3D 0 TX MSIX IRQ =3D 0 LINK MSIX IRQ =3D 0 Feb 22 13:39:12 inet-gw kernel: em2: XON Rcvd =3D 21 Feb 22 13:39:12 inet-gw kernel: em2: XON Xmtd =3D 8344 Feb 22 13:39:12 inet-gw kernel: em2: XOFF Rcvd =3D 30 Feb 22 13:39:12 inet-gw kernel: em2: XOFF Xmtd =3D 9159 Feb 22 13:39:12 inet-gw kernel: em2: Good Packets Rcvd =3D 723521981 Feb 22 13:39:12 inet-gw kernel: em2: Good Packets Xmtd =3D 577211431 Feb 22 13:39:12 inet-gw kernel: em2: TSO Contexts Xmtd =3D 0 Feb 22 13:39:12 inet-gw kernel: em2: TSO Contexts Failed =3D 0 =09 =09 Can this be the problem? "Receive No Buffers =3D 275612" =09 ---- Kirk Kirk Davis Senior Network Analyst, ITS Edmonton Public Schools One Kingsway Ave. Edmonton, Alberta, Canada T5H 4G9 =09 phone: 1-780-429-8308 =09 =09 =09
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?529374128DC1B04D9D037911B8E8F05301C17A5C>