From owner-freebsd-net@FreeBSD.ORG Fri Oct 7 21:14:54 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 84F341065678 for ; Fri, 7 Oct 2011 21:14:54 +0000 (UTC) (envelope-from nitroboost@gmail.com) Received: from mail-ey0-f182.google.com (mail-ey0-f182.google.com [209.85.215.182]) by mx1.freebsd.org (Postfix) with ESMTP id 07B918FC0A for ; Fri, 7 Oct 2011 21:14:53 +0000 (UTC) Received: by eyz10 with SMTP id 10so2588449eyz.13 for ; Fri, 07 Oct 2011 14:14:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=PWh9CYKNNP3Ej9k+j/P1/zLD/GnUyMYJcQxoEN8BBSQ=; b=kLMT+UMTecpcI1+M/AyMcaR43kLHQDEt3JgVO6TTChVVpzc2JaceCrk5DjExDQdPTt hKQh1vReMM0E75ZqwTkggHp7rYlOtmjSIGRGzY2KEB64JlNAgQE/Nx4BkZTncgIlYo4V FvH8tAKCWhagvjFxF497Av+Ro8bk2MyxgNN9I= MIME-Version: 1.0 Received: by 10.223.76.11 with SMTP id a11mr8142487fak.1.1318022092759; Fri, 07 Oct 2011 14:14:52 -0700 (PDT) Received: by 10.152.36.102 with HTTP; Fri, 7 Oct 2011 14:14:52 -0700 (PDT) In-Reply-To: References: <4E8F157A.40702@sentex.net> <4E8F51D4.1060509@sentex.net> Date: Fri, 7 Oct 2011 14:14:52 -0700 Message-ID: From: Jason Wolfe To: Mike Tancsa , freebsd-net@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Subject: Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Oct 2011 21:14:54 -0000 On Fri, Oct 7, 2011 at 12:55 PM, Arnaud Lacombe wrote: > Hi, > > On Fri, Oct 7, 2011 at 3:24 PM, Mike Tancsa wrote: > > When you disable MSI-X, you mean via hw.pci.enable_msix=0 across the > > board, or you disable multi-queue for the NIC, so it uses just one > > interrupt, rather than separate ones for xmit and recv ? > > > As noticed it was the prior, all were disabled via hw.em.enable_msix=0 > em(4)'s multiqueue is misleading. By default, with MSI-X enabled, > before AFAIK, April 2010 it used 2 (RX+TX) queue + 1, ie. 5 MSI-X > vectors[0]. After April 2010, it uses 1 * (RX+TX) queue + 1, ie. 3 > MSI-X vectors. There is no logic for the driver to use 1 vector with > MSI-X enabled. > > As a side note, the only gain of EM_MULTIQUEUE, now, is to allow the > driver to use the buf_ring(9) lockless queue API, compared to the > locked ifq. Today, em(4) should waste about 16k of memory for when > !EM_MULTIQUEUE. This is the memory, 4096 * sizeof(void *), allocated > for the buf_ring(9) structure which is not used in the !EM_MULTIQUEUE > case. > > > Also, what is the purpose of > > hw.pci.do_power_nodriver=3 vs 0 (3 means put absolutely everything > > in D3 state.) > 0 is the default state and means no 'power management' is done. My stripped down kernels don't have usb support so '3' just reduces the consumption by not powering those devices. If you load up the modules power will be supplied and the devices function. > > > > net.link.ifqmaxlen 1024 vs 50 (does anything else need to be adjusted of > > this value is increased?) > I haven't seen any evidence that there are other supporting adjustments for this, this was actually something I stumbled across while researching the issue. > > > He might as well try to enable EM_MULTIQUEUE. > > > hw.em.rxd="2048" > > hw.em.txd="2048" > > > As it starts to be well known here, I am not a fan of bumping a limit > to hide a bug. So I'd rather lower this to 512 or 256, and hope it > triggers the issue more often, so that it could be diagnosticed and > fixed for good. > > - Arnaud > > [0]: actually it depends on a field in the chip NVM, which can be up > to 4 (0 based accounting, this would translate in 5 vectors), but > happened to be 2 (3 vector) in 82574 I've got access to. Last time I > checked, this setting could not be seen with the standard NVM dump > sysctl, which limit the output's size. On those chip, the > pre-April-2010 code would falls back on MSI even if 3 were available. > > > Have you tried leaving these two at the default on 7.2.3 ? > > if_em.h implies 1024 for each. > > > > ---Mike > Bumping rx/tx descriptors to 2048 was actually for performance reasons and not to try to get around the issue. I did some fairly in depth testing and found under heavy load it performed the best with those settings. As mentioned on the other thread I'll re enable MSI-X on a few servers here and collect uptime and the kernel msgbuf in addition. I'll bump the descriptors down to 512 to try and increase our chances and compile the driver with EM_MULTIQUEUE also. Jason