From owner-freebsd-stable@FreeBSD.ORG Fri Apr 9 01:48:13 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4B9FF106567A for ; Fri, 9 Apr 2010 01:48:13 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-wy0-f182.google.com (mail-wy0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id CEFDF8FC1B for ; Fri, 9 Apr 2010 01:48:12 +0000 (UTC) Received: by wyi11 with SMTP id 11so55691wyi.13 for ; Thu, 08 Apr 2010 18:48:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:received:message-id:subject:from:to:cc:content-type; bh=EnmCxNsMEajtc5nh1xTvI3LWwKIwvBb7OE4kzw3iqWo=; b=oWxLiJ4vuVL9Kl+HAmJVhfEC0IAxGbyvDkbwZgN8pd70JG+w5B098z7z5Q4WqA+nwg qTzRbbwccSrxTy99yURluvR3gRtQi3aNvHbeZ/4zh2zda6XA+MrQvs+YpDCYJt5kKbLO JQLW1fGtYoCyex20FXuLTrjxzjrfa2g+Yk60U= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=YJaFrtzi361zFQwe31W1FbqiC8YNlzfxU+gIFQjC3MCOl4JWt1B0YoHwtiWOBJeO0A oJOym5MPAgT6VIdx/7tHqaAzmKEbwxUcWg2hDhBAXPo/GDEHPDX5Yi3h8cao5fKAiycU t7XZiQ7cZ3x06Mcy42QJ0hQqGWEu91F0AAdc8= MIME-Version: 1.0 Received: by 10.216.11.8 with HTTP; Thu, 8 Apr 2010 18:48:11 -0700 (PDT) In-Reply-To: <20100408230750.GR5734@michelle.cdnetworks.com> References: <201004081313.o38DD4JM041821@lava.sentex.ca> <7.1.0.9.0.20100408091756.10652be0@sentex.net> <201004081446.o38EkU7h042296@lava.sentex.ca> <20100408181741.GI5734@michelle.cdnetworks.com> <201004081831.o38IVR3s043434@lava.sentex.ca> <20100408205626.GN5734@michelle.cdnetworks.com> <201004082105.o38L5DCH044187@lava.sentex.ca> <20100408230750.GR5734@michelle.cdnetworks.com> Date: Thu, 8 Apr 2010 18:48:11 -0700 Received: by 10.216.90.7 with SMTP id d7mr502585wef.81.1270777691752; Thu, 08 Apr 2010 18:48:11 -0700 (PDT) Message-ID: From: Jack Vogel To: pyunyh@gmail.com Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-stable@freebsd.org Subject: Re: em driver regression X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Apr 2010 01:48:13 -0000 Ah, ok, let me play around with it a bit, perhaps I'll make it definable, of course if there is no positive benefit from using it it would seem silly to leave it around :) Will look at your patch changes and that issue tomorrow. Thanks for your efforts! Jack On Thu, Apr 8, 2010 at 4:07 PM, Pyun YongHyeon wrote: > On Thu, Apr 08, 2010 at 02:06:09PM -0700, Jack Vogel wrote: > > Only one device support by em does multiqueue right now, and that is > > Hartwell, 82574. > > > > Thanks for the info. > > Mike, here is updated patch. Now UDP bulk TX transfer performance > recovered a lot(about 890Mbps) but it still shows bad numbers > compared to other controllers. For example, bce(4) shows about > 958Mbps for the same load. > During the testing I found a strong indication of packet reordering > issue of drbr interface. If I forcibly change to use single TX > queue, em(4) got 950Mbps as it used to be. > > Jack, as we talked about possible drbr issue with igb(4), UDP > transfer seems to suffer from packet reordering issue here. Can we > make em(4)/igb(4) use single TX queue until we solve drbr interface > issue? Given that only one em(4) controller supports multiqueue, > dropping multiqueue support for em(4) does not look bad to me. > > > Jack > > > > > > On Thu, Apr 8, 2010 at 2:05 PM, Mike Tancsa wrote: > > > > > At 04:56 PM 4/8/2010, Pyun YongHyeon wrote: > > > > > >> On Thu, Apr 08, 2010 at 02:31:18PM -0400, Mike Tancsa wrote: > > >> > At 02:17 PM 4/8/2010, Pyun YongHyeon wrote: > > >> > > > >> > >Try this patch. It should fix the issue. It seems Jack forgot to > > >> > >strip CRC bytes as old em(4) didn't strip it, probably to > > >> > >workaround silicon bug of old em(4) controllers. > > >> > > > >> > Thanks! The attached patch does indeed fix the dhclient issue. > > >> > > > >> > > > >> > >It seems there are also TX issues here. The system load is too high > > >> > >and sometimes system is not responsive while TX is in progress. > > >> > >Because I initiated TCP bulk transfers, TSO should reduce CPU load > > >> > >a lot but it didn't so I guess it could also be related watchdog > > >> > >timeouts you've seen. I'll see what can be done. > > >> > > > >> > Thanks for looking into that as well!! > > >> > > > >> > ---Mike > > >> > > > >> > > >> Mike, > > >> > > >> Here is patch I'm working on. This patch fixes high system load and > > >> system is very responsive as before. But it seems there is still > > >> some TX issue here. Bulk UDP performance is very poor(< 700Mbps) > > >> and I have no idea what caused this at this moment. > > >> > > >> BTW, I have trouble to reproduce watchdog timeouts. I'm not sure > > >> whether latest fix from Jack cured it. By chance does your > > >> controller support multi TX/RX queues? You can check whether em(4) > > >> uses multi-queues with "vmstat -i". If em(4) use multi-queue you > > >> may have multiple irq output for em0. > > >> > > > > > > Hi, > > > I will give it a try later tonight! This one does not seem to. > > > > > > 0(ich10)# vmstat -i > > > interrupt total rate > > > irq16: uhci0+ 30 0 > > > irq18: ehci0 uhci5 158419 17 > > > irq19: fwohci0++ 86 0 > > > irq21: uhci1 17 0 > > > irq23: uhci3 ehci1 2 0 > > > cpu0: timer 18570305 1994 > > > irq256: igb0 80 0 > > > irq257: igb0 255 0 > > > irq258: igb0 66 0 > > > irq259: igb0 32 0 > > > irq260: igb0 2 0 > > > irq261: igb1 2679 0 > > > irq262: igb1 998 0 > > > irq263: igb1 2468 0 > > > irq264: igb1 6361 0 > > > irq265: igb1 2 0 > > > irq266: em0 33910 3 > > > irq267: ahci1 15317 1 > > > cpu1: timer 18557074 1993 > > > cpu3: timer 18557168 1993 > > > cpu2: timer 18557108 1993 > > > Total 74462379 7998 > > > 0(ich10)# > > > >