From owner-freebsd-net@FreeBSD.ORG Tue Feb 23 00:29:47 2010 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 996F01065672 for ; Tue, 23 Feb 2010 00:29:47 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-wy0-f182.google.com (mail-wy0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id 08C238FC12 for ; Tue, 23 Feb 2010 00:29:46 +0000 (UTC) Received: by wyb40 with SMTP id 40so442517wyb.13 for ; Mon, 22 Feb 2010 16:29:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=oW+uOWH2bxM2Uj4GdefiUzgF/KP7LGwq96VlT7I+xDg=; b=c8NKyhjTDw0qpHy3ubnLnpOvczV+HV4a7GO0EhKbGUeNTwyGNfkkIkvUUBNig0vfG8 jGSgoRJ8hlT5SFC8yMxgFDJqdvthprvBqEMCcqtf1CvDj/B0m4Ex0vtD5OGudD6HpZOh hPyxYBzqpVSbS/L2ikxe6YUXA9l3r6XRPqgNo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=LA/W/9qkKJFG7Hb3G4LRs6EHVGRz5uTUNI14isOPXloFcdto8a7rfr0V+9aFQSy297 r4smUBvWN1htao5MYISWRcel5EDa0zYinL9kQOydIV7QefL4vzNdSOq7f2hFrZ982ZSH 6QqnJvhy5Fn0kvQQs5Xe2+VkgajCj3ICjkvZo= MIME-Version: 1.0 Received: by 10.216.177.82 with SMTP id c60mr208141wem.25.1266884980973; Mon, 22 Feb 2010 16:29:40 -0800 (PST) In-Reply-To: <529374128DC1B04D9D037911B8E8F05301C17A57@Exchange26.EDU.epsb.ca> References: <529374128DC1B04D9D037911B8E8F05301C17A51@Exchange26.EDU.epsb.ca> <43416_1266864062_4B82CFBE_43416_81_1_2a41acea1002221043k1b8742c9m8fb484a8e8a4fdda@mail.gmail.com> <529374128DC1B04D9D037911B8E8F05301C17A54@Exchange26.EDU.epsb.ca> <43669_1266865888_4B82D6E0_43669_263_1_2a41acea1002221113v26804200q4f3971c3359dffab@mail.gmail.com> <529374128DC1B04D9D037911B8E8F05301C17A55@Exchange26.EDU.epsb.ca> <201002222107.o1ML7v3Z059734@lava.sentex.ca> <529374128DC1B04D9D037911B8E8F05301C17A56@Exchange26.EDU.epsb.ca> <2a41acea1002221444o6e449602m1830761b21837c41@mail.gmail.com> <529374128DC1B04D9D037911B8E8F05301C17A57@Exchange26.EDU.epsb.ca> Date: Mon, 22 Feb 2010 16:29:40 -0800 Message-ID: <2a41acea1002221629vbe7548am7b5f1ba94d7efa9f@mail.gmail.com> From: Jack Vogel To: Kirk Davis Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-net@freebsd.org, Mike Tancsa Subject: Re: Intel em0: watchdog timeout X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Feb 2010 00:29:47 -0000 Is your driver static, ie builtin, to the kernel, or do you load/unload it as a module? I ask because perhaps we could try a later driver, and being a module makes that easier. Jack On Mon, Feb 22, 2010 at 3:37 PM, Kirk Davis wrote: > OK. I have the following in /boot/loader.conf (and rebooted) > hw.em.rxd=1024 > hw.em.txd=1024 > > Should this be hw.em2.rxd? Is it set per interface or across all > interfaces? > > nmbcluster=262144 > > # sysctl dev.em.2.stats=1 > Feb 22 16:29:57 inet-gw kernel: em2: Defer count = 20 > Feb 22 16:29:57 inet-gw kernel: em2: Missed Packets = 119947 > Feb 22 16:29:57 inet-gw kernel: em2: Receive No Buffers = 276762 > Feb 22 16:29:57 inet-gw kernel: em2: Receive Length Errors = 0 > Feb 22 16:29:57 inet-gw kernel: em2: Receive errors = 0 > Feb 22 16:29:57 inet-gw kernel: em2: Crc errors = 0 > Feb 22 16:29:57 inet-gw kernel: em2: Alignment errors = 0 > Feb 22 16:29:57 inet-gw kernel: em2: Collision/Carrier extension errors = 0 > Feb 22 16:29:57 inet-gw kernel: em2: RX overruns = 21 > Feb 22 16:29:57 inet-gw kernel: em2: watchdog timeouts = 47 > Feb 22 16:29:57 inet-gw kernel: em2: RX MSIX IRQ = 0 TX MSIX IRQ = 0 LINK > MSIX IRQ = 0 > Feb 22 16:29:57 inet-gw kernel: em2: XON Rcvd = 22 > Feb 22 16:29:57 inet-gw kernel: em2: XON Xmtd = 8349 > Feb 22 16:29:57 inet-gw kernel: em2: XOFF Rcvd = 31 > Feb 22 16:29:57 inet-gw kernel: em2: XOFF Xmtd = 15779 > Feb 22 16:29:57 inet-gw kernel: em2: Good Packets Rcvd = 966101852 > Feb 22 16:29:57 inet-gw kernel: em2: Good Packets Xmtd = 755993237 > Feb 22 16:29:57 inet-gw kernel: em2: TSO Contexts Xmtd = 0 > Feb 22 16:29:57 inet-gw kernel: em2: TSO Contexts Failed = 0 > > still seeing the watchdog timer and link up/down messages. > > Should I try going higher than 1024 on the hw.em.rxd? I'm not sure the > next time I can schedule another reboot on this production server. > > ---- Kirk > > > *Kirk Davis*** > *Senior Network Analyst, ITS* > *Edmonton Public Schools* > *One Kingsway Ave. * > *Edmonton, Alberta, Canada* > *T5H 4G9* > *phone: 1-780-429-8308* > > > ------------------------------ > *From:* Jack Vogel [mailto:jfvogel@gmail.com] > *Sent:* Monday, February 22, 2010 3:45 PM > *To:* Kirk Davis > *Cc:* Mike Tancsa; freebsd-net@freebsd.org > > *Subject:* Re: Intel em0: watchdog timeout > > OK, so you are still failing to get mbufs in the RX side, increase the > nmbcluster > value, and then what size is your RX ring (number of rx descriptors)? > > If you havent already done so, change that to 1024. > > I am developing a change in the RX code right now that will help > this situation, but am doing so in the 10G driver, once its solid there > I will be backporting it into the 1G drivers, it will make discards > almost unnecessary. > > Jack > > On Mon, Feb 22, 2010 at 1:43 PM, Kirk Davis wrote: > >> >> >> > -----Original Message----- >> > From: Mike Tancsa [mailto:mike@sentex.net] >> > Subject: Re: Intel em0: watchdog timeout >> > >> > At 03:46 PM 2/22/2010, Kirk Davis wrote: >> > >Does this need to be done in loader.conf? It doesn't seem >> > to take from >> > >the command line. >> > ># sysctl dev.em.2.stats=1 >> > >dev.em.2.stats: -1 -> -1 >> > > >> > ># sysctl dev.em.2.stats >> > >dev.em.2.stats: -1 >> > >> > Hi, >> > After you issue those commands, the driver will spit out a >> > lot of useful stats to syslog. It will report something like the >> > following in /var/log/messages >> > >> > Feb 22 16:06:31 offsite kernel: em0: Excessive collisions = 0 >> > Feb 22 16:06:31 offsite kernel: em0: Sequence errors = 0 >> > Feb 22 16:06:31 offsite kernel: em0: Defer count = 0 >> > Feb 22 16:06:31 offsite kernel: em0: Missed Packets = 0 >> > Feb 22 16:06:31 offsite kernel: em0: Receive No Buffers = 0 >> > Feb 22 16:06:31 offsite kernel: em0: Receive Length Errors = 0 >> > Feb 22 16:06:31 offsite kernel: em0: Receive errors = 0 >> > Feb 22 16:06:31 offsite kernel: em0: Crc errors = 0 >> > Feb 22 16:06:31 offsite kernel: em0: Alignment errors = 0 >> > Feb 22 16:06:31 offsite kernel: em0: Collision/Carrier >> > extension errors = 0 >> > Feb 22 16:06:31 offsite kernel: em0: RX overruns = 0 >> > Feb 22 16:06:31 offsite kernel: em0: watchdog timeouts = 0 >> > Feb 22 16:06:31 offsite kernel: em0: RX MSIX IRQ = 0 TX MSIX IRQ = 0 >> > LINK MSIX IRQ = 0 >> > Feb 22 16:06:31 offsite kernel: em0: XON Rcvd = 0 >> > Feb 22 16:06:31 offsite kernel: em0: XON Xmtd = 0 >> > Feb 22 16:06:31 offsite kernel: em0: XOFF Rcvd = 0 >> > Feb 22 16:06:31 offsite kernel: em0: XOFF Xmtd = 0 >> > Feb 22 16:06:31 offsite kernel: em0: Good Packets Rcvd = 2559032551 >> > Feb 22 16:06:31 offsite kernel: em0: Good Packets Xmtd = 1568751141 >> > Feb 22 16:06:31 offsite kernel: em0: TSO Contexts Xmtd = 0 >> > Feb 22 16:06:31 offsite kernel: em0: TSO Contexts Failed = 0 >> >> Thanks Mike and Jack. I don't know why I didn'ty notice the output in >> /var/log/messages >> >> Here is the output for the two interfaces that are causing this issue. >> >> Feb 22 13:33:52 inet-gw kernel: em0: Excessive collisions = 0 >> Feb 22 13:33:52 inet-gw kernel: em0: Sequence errors = 0 >> Feb 22 13:33:52 inet-gw kernel: em0: Defer count = 0 >> Feb 22 13:33:52 inet-gw kernel: em0: Missed Packets = 24296 >> Feb 22 13:33:52 inet-gw kernel: em0: Receive No Buffers = 0 >> Feb 22 13:33:52 inet-gw kernel: em0: Receive Length Errors = 0 >> Feb 22 13:33:52 inet-gw kernel: em0: Receive errors = 0 >> Feb 22 13:33:52 inet-gw kernel: em0: Crc errors = 0 >> Feb 22 13:33:52 inet-gw kernel: em0: Alignment errors = 0 >> Feb 22 13:33:52 inet-gw kernel: em0: Collision/Carrier extension errors >> = 0 >> Feb 22 13:33:52 inet-gw kernel: em0: RX overruns = 0 >> Feb 22 13:33:52 inet-gw kernel: em0: watchdog timeouts = 6 >> Feb 22 13:33:52 inet-gw kernel: em0: RX MSIX IRQ = 0 TX MSIX IRQ = 0 >> LINK MSIX IRQ = 0 >> Feb 22 13:33:52 inet-gw kernel: em0: XON Rcvd = 0 >> Feb 22 13:33:52 inet-gw kernel: em0: XON Xmtd = 0 >> Feb 22 13:33:52 inet-gw kernel: em0: XOFF Rcvd = 0 >> Feb 22 13:33:52 inet-gw kernel: em0: XOFF Xmtd = 0 >> Feb 22 13:33:52 inet-gw kernel: em0: Good Packets Rcvd = 424303810 >> Feb 22 13:33:52 inet-gw kernel: em0: Good Packets Xmtd = 576529136 >> Feb 22 13:33:52 inet-gw kernel: em0: TSO Contexts Xmtd = 0 >> Feb 22 13:33:52 inet-gw kernel: em0: TSO Contexts Failed = 0 >> Feb 22 13:34:12 inet-gw kernel: em2: Excessive collisions = 0 >> Feb 22 13:34:12 inet-gw kernel: em2: Sequence errors = 0 >> Feb 22 13:34:12 inet-gw kernel: em2: Defer count = 20 >> Feb 22 13:34:12 inet-gw kernel: em2: Missed Packets = 68059 >> Feb 22 13:34:12 inet-gw kernel: em2: Receive No Buffers = 275612 >> Feb 22 13:34:12 inet-gw kernel: em2: Receive Length Errors = 0 >> Feb 22 13:34:12 inet-gw kernel: em2: Receive errors = 0 >> Feb 22 13:34:12 inet-gw kernel: em2: Crc errors = 0 >> Feb 22 13:34:12 inet-gw kernel: em2: Alignment errors = 0 >> Feb 22 13:34:12 inet-gw kernel: em2: Collision/Carrier extension errors >> = 0 >> Feb 22 13:34:12 inet-gw kernel: em2: RX overruns = 17 >> Feb 22 13:34:12 inet-gw kernel: em2: watchdog timeouts = 38 >> Feb 22 13:34:12 inet-gw kernel: em2: RX MSIX IRQ = 0 TX MSIX IRQ = 0 >> LINK MSIX IRQ = 0 >> Feb 22 13:34:12 inet-gw kernel: em2: XON Rcvd = 21 >> Feb 22 13:34:12 inet-gw kernel: em2: XON Xmtd = 8344 >> Feb 22 13:34:12 inet-gw kernel: em2: XOFF Rcvd = 30 >> Feb 22 13:34:12 inet-gw kernel: em2: XOFF Xmtd = 9159 >> Feb 22 13:34:12 inet-gw kernel: em2: Good Packets Rcvd = 713607509 >> Feb 22 13:34:12 inet-gw kernel: em2: Good Packets Xmtd = 569694020 >> Feb 22 13:34:12 inet-gw kernel: em2: TSO Contexts Xmtd = 0 >> Feb 22 13:34:12 inet-gw kernel: em2: TSO Contexts Failed = 0 >> Feb 22 13:35:10 inet-gw kernel: em2: Excessive collisions = 0 >> Feb 22 13:35:10 inet-gw kernel: em2: Sequence errors = 0 >> Feb 22 13:35:10 inet-gw kernel: em2: Defer count = 20 >> Feb 22 13:35:10 inet-gw kernel: em2: Missed Packets = 68059 >> Feb 22 13:35:10 inet-gw kernel: em2: Receive No Buffers = 275612 >> Feb 22 13:35:10 inet-gw kernel: em2: Receive Length Errors = 0 >> Feb 22 13:35:10 inet-gw kernel: em2: Receive errors = 0 >> Feb 22 13:35:10 inet-gw kernel: em2: Crc errors = 0 >> Feb 22 13:35:10 inet-gw kernel: em2: Alignment errors = 0 >> Feb 22 13:35:10 inet-gw kernel: em2: Collision/Carrier extension errors >> = 0 >> Feb 22 13:35:10 inet-gw kernel: em2: RX overruns = 17 >> Feb 22 13:35:10 inet-gw kernel: em2: watchdog timeouts = 38 >> Feb 22 13:35:10 inet-gw kernel: em2: RX MSIX IRQ = 0 TX MSIX IRQ = 0 >> LINK MSIX IRQ = 0 >> Feb 22 13:35:10 inet-gw kernel: em2: XON Rcvd = 21 >> Feb 22 13:35:10 inet-gw kernel: em2: XON Xmtd = 8344 >> Feb 22 13:35:10 inet-gw kernel: em2: XOFF Rcvd = 30 >> Feb 22 13:35:10 inet-gw kernel: em2: XOFF Xmtd = 9159 >> Feb 22 13:35:10 inet-gw kernel: em2: Good Packets Rcvd = 715555016 >> Feb 22 13:35:10 inet-gw kernel: em2: Good Packets Xmtd = 571157561 >> Feb 22 13:35:10 inet-gw kernel: em2: TSO Contexts Xmtd = 0 >> Feb 22 13:35:10 inet-gw kernel: em2: TSO Contexts Failed = 0 >> Feb 22 13:39:12 inet-gw kernel: em2: Excessive collisions = 0 >> Feb 22 13:39:12 inet-gw kernel: em2: Sequence errors = 0 >> Feb 22 13:39:12 inet-gw kernel: em2: Defer count = 20 >> Feb 22 13:39:12 inet-gw kernel: em2: Missed Packets = 68059 >> Feb 22 13:39:12 inet-gw kernel: em2: Receive No Buffers = 275612 >> Feb 22 13:39:12 inet-gw kernel: em2: Receive Length Errors = 0 >> Feb 22 13:39:12 inet-gw kernel: em2: Receive errors = 0 >> Feb 22 13:39:12 inet-gw kernel: em2: Crc errors = 0 >> Feb 22 13:39:12 inet-gw kernel: em2: Alignment errors = 0 >> Feb 22 13:39:12 inet-gw kernel: em2: Collision/Carrier extension errors >> = 0 >> Feb 22 13:39:12 inet-gw kernel: em2: RX overruns = 17 >> Feb 22 13:39:12 inet-gw kernel: em2: watchdog timeouts = 38 >> Feb 22 13:39:12 inet-gw kernel: em2: RX MSIX IRQ = 0 TX MSIX IRQ = 0 >> LINK MSIX IRQ = 0 >> Feb 22 13:39:12 inet-gw kernel: em2: XON Rcvd = 21 >> Feb 22 13:39:12 inet-gw kernel: em2: XON Xmtd = 8344 >> Feb 22 13:39:12 inet-gw kernel: em2: XOFF Rcvd = 30 >> Feb 22 13:39:12 inet-gw kernel: em2: XOFF Xmtd = 9159 >> Feb 22 13:39:12 inet-gw kernel: em2: Good Packets Rcvd = 723521981 >> Feb 22 13:39:12 inet-gw kernel: em2: Good Packets Xmtd = 577211431 >> Feb 22 13:39:12 inet-gw kernel: em2: TSO Contexts Xmtd = 0 >> Feb 22 13:39:12 inet-gw kernel: em2: TSO Contexts Failed = 0 >> >> >> Can this be the problem? "Receive No Buffers = 275612" >> >> ---- Kirk >> Kirk Davis >> Senior Network Analyst, ITS >> Edmonton Public Schools >> One Kingsway Ave. >> Edmonton, Alberta, Canada >> T5H 4G9 >> phone: 1-780-429-8308 >> >> >> >