From owner-freebsd-net@FreeBSD.ORG Wed Nov 9 08:55:08 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7A658106566C for ; Wed, 9 Nov 2011 08:55:08 +0000 (UTC) (envelope-from hoomanfazaeli@gmail.com) Received: from mail-bw0-f54.google.com (mail-bw0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id E53808FC23 for ; Wed, 9 Nov 2011 08:55:07 +0000 (UTC) Received: by bkbzs8 with SMTP id zs8so1586079bkb.13 for ; Wed, 09 Nov 2011 00:55:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type; bh=HdifiK+eyWmIviM/hLOKVuk/Ep2V2O2wYOS1RitFKp4=; b=seqEDZjexXjCgx+LHAt1NIZ8q8BQm34OohiQtLR/LwzEuqlAWx4jagvup/LEvZMQEO bTYaM9ULghyuGHNAYFbhpKUSyZigRUq8nfWlyuUa2x6Z70FFkh5v0zcYSHHfclWc9iJy QYlCZ6+WlBiytxFjxWTEp1pIfnn6wbfRlskvw= Received: by 10.204.156.133 with SMTP id x5mr963597bkw.87.1320828906814; Wed, 09 Nov 2011 00:55:06 -0800 (PST) Received: from [127.0.0.1] ([84.241.57.181]) by mx.google.com with ESMTPS id q6sm4072437bka.6.2011.11.09.00.55.03 (version=SSLv3 cipher=OTHER); Wed, 09 Nov 2011 00:55:06 -0800 (PST) Message-ID: <4EBA3FE4.3050106@gmail.com> Date: Wed, 09 Nov 2011 12:25:00 +0330 From: Hooman Fazaeli User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.23) Gecko/20110920 Thunderbird/3.1.15 MIME-Version: 1.0 To: Jason Wolfe References: <4E8F51D4.1060509@sentex.net> <4EA7E203.3020306@sepehrs.com> <4EA80818.3030504@sentex.net> <4EA80F88.4000400@hotplug.ru> <4EA82715.2000404@gmail.com> <4EA8FA40.7010504@hotplug.ru> <4EA91836.2040508@gmail.com> <4EA959EE.2070806@hotplug.ru> <4EAD116A.8090006@gmail.com> <4EAE58A2.9040803@gmail.com> <4EB96511.50701@gmail.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: pyunyh@gmail.com, freebsd-net@freebsd.org, Jack Vogel , Emil Muratov Subject: Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Nov 2011 08:55:08 -0000 On 11/8/2011 10:23 PM, Jason Wolfe wrote: > On Tue, Nov 8, 2011 at 10:21 AM, Hooman Fazaeli > wrote: > > I have allocated more time to the problem and guess I can explain what > your problem is. > > With MSIX disabled, the driver uses fast interrupt handler (em_irq_fast) > which calls rx/tx task and then checks for link status change. This > implies that rx/tx task is executed with every link state change. This is > not efficient, as it is a waste of time to start transmission when link is down. > However, it has the effect that after a temporary link loss (active->inactive->active), > _start is executed and transmission continues normally. The value of link_toggles (3) > clearly indicates that you had such a transition when the problem occured. > > With MSIX enabled, the link task (em_handle_link) does _not_ triggers > _start when the link changes state from inactive to active (which it should). > If if_snd quickly fills up during a temporary link loss, transmission is > stopped forever and the driver never recovers from that state. > > The last patch should have reduced the frequency of the problem > but it assumes every IFQ_ENQUEUE is followed by a if_start which > is not a true assumption. > > If you are willing to test, I can prepare another patch for you to fix > the issue in a different and more reliable way. > > > Hooman, > > Thanks again for the assist, it sounds like this may also be why we see a bit higher latency with MSI-X disabled on this chipset. > > I'm happy to test any patches as I have a handful of boxes set aside to 'research' this issue. Hopefully the testing here helps along any patches to the tree for others benefit also. > > Jason Latency may or may not be related. I am doing more tests and will post my findings soon.