From owner-freebsd-net@FreeBSD.ORG Sun Jun 22 22:16:08 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1ACAB10656AE for ; Sun, 22 Jun 2008 22:16:08 +0000 (UTC) (envelope-from crapsh@monkeybrains.net) Received: from ape.monkeybrains.net (ape.monkeybrains.net [208.69.40.11]) by mx1.freebsd.org (Postfix) with ESMTP id F408A8FC1A for ; Sun, 22 Jun 2008 22:16:07 +0000 (UTC) (envelope-from crapsh@monkeybrains.net) Received: from monchichi.monkeybrains.net (adsl-76-203-175-234.dsl.pltn13.sbcglobal.net [76.203.175.234]) (authenticated bits=0) by ape.monkeybrains.net (8.14.1/8.14.1) with ESMTP id m5MMG5HQ060571 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sun, 22 Jun 2008 15:16:07 -0700 (PDT) (envelope-from crapsh@monkeybrains.net) Message-ID: <485ECF20.6000607@monkeybrains.net> Date: Sun, 22 Jun 2008 15:16:00 -0700 From: Rudy User-Agent: Thunderbird 2.0.0.12 (X11/20080310) MIME-Version: 1.0 To: Jack Vogel References: <20080516185813.H866@logos.sky.od.ua> <2a41acea0805160904g7dcf9f58rf69ca5d0612945cc@mail.gmail.com> <4853055C.2030303@MonkeyBrains.NET> <48535A11.4020003@monkeybrains.net> <48582C29.8030307@monkeybrains.net> <2a41acea0806191055w5e112b8bsa57a8db2b177adbe@mail.gmail.com> <485C0F07.7000408@monkeybrains.net> <2a41acea0806201355y3b123462wc37280f28a9f4216@mail.gmail.com> <485C41C5.4090706@monkeybrains.net> <2a41acea0806201713m7a083b9dr10894c8a0845df8e@mail.gmail.com> <485D5B9E.8020908@monkeybrains.net> In-Reply-To: <485D5B9E.8020908@monkeybrains.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV version 0.93, clamav-milter version 0.93 on pita.monkeybrains.net X-Virus-Status: Clean Cc: freebsd-net@freebsd.org Subject: Re: Seeking help understanding my "emX: watchdog timeout" messages X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Jun 2008 22:16:08 -0000 Reading if_em.h 77 * EM_TIDV - Transmit Interrupt Delay Value 78 * Valid Range: 0-65535 (0=off) 79 * Default Value: 64 80 * This value delays the generation of transmit interrupts in units of 81 * 1.024 microseconds. Transmit interrupt reduction can improve CPU 82 * efficiency if properly tuned for specific network traffic. If the 83 * system is reporting dropped transmits, this value may be set too high 84 * causing the driver to run out of available transmit descriptors. seems to indicate that 'dropped transmits' may be due to the 'tx_int_delay' being too high. So, I lowered the Transmit Interrupt Delay Values 33 useconds. I will report if this helps. Does anyone have experience using tuning these values? dev.em.2.tx_int_delay: 33 dev.em.2.tx_abs_int_delay: 33 The Intel docs explain the tx_int_delay vs the tx_abs_int_delay: "In [high traffic] situations, the absolute timer is the source of most device interrupts. Under light loads or brief bursts of traffic, the packet timers are the primary source of interrupts. In these situations, the packet timers determine the latency suffered by most packets. The packet timers also determine the minimum traffic rate required to trigger the absolute timer interrupts. For example, if the traffic rate is high enough to prevent the packet timer from ever expiring, then the GbE controller does not interrupt until the absolute timer has expired." That is why I picked a tx_abs_int_delay == tx_int_delay. The source code references running out of available transmit descriptors... Is there a way to bump up the number of available transmit descriptors or to change the packet buffer size? here is the dev.em.2.debug info: kernel: em2: Adapter hardware address = 0xc4d91224 kernel: em2: CTRL = 0x400c0241 RCTL = 0x8002 kernel: em2: Packet buffer = Tx=16k Rx=32k kernel: em2: Flow control watermarks high = 30720 low = 29220 kernel: em2: tx_int_delay = 32, tx_abs_int_delay = 32 kernel: em2: rx_int_delay = 0, rx_abs_int_delay = 66 kernel: em2: fifo workaround = 0, fifo_reset_count = 0 kernel: em2: hw tdh = 41, hw tdt = 41 kernel: em2: hw rdh = 130, hw rdt = 129 kernel: em2: Num Tx descriptors avail = 254 (Can I increase?) kernel: em2: Tx Descriptors not avail1 = 15 (Why not available?) kernel: em2: Tx Descriptors not avail2 = 0 kernel: em2: Std mbuf failed = 0 kernel: em2: Std mbuf cluster failed = 0 kernel: em2: Driver dropped packets = 0 kernel: em2: Driver tx dma failure in encap = 0 (not sure why tx_int_delay is 32 when I set it to 33... starts counting at 0?) Rudy Reference for if_em.h: http://fxr.watson.org/fxr/source/dev/em/if_em.h?v=FREEBSD7 Referenve for Absolute vs Packet timer: http://www.intel.com/design/network/applnots/ap450.pdf (page 12)