From owner-freebsd-current@FreeBSD.ORG Wed Nov 3 23:27:22 2010 Return-Path: <owner-freebsd-current@FreeBSD.ORG> Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A06C4106566B for <freebsd-current@freebsd.org>; Wed, 3 Nov 2010 23:27:22 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 33A308FC13 for <freebsd-current@freebsd.org>; Wed, 3 Nov 2010 23:27:21 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ApwEAN+M0UyDaFvO/2dsb2JhbACDI583rxiRFYRTcwSKVQ X-IronPort-AV: E=Sophos;i="4.58,291,1286164800"; d="scan'208";a="99541892" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-jnhn-pri.mail.uoguelph.ca with ESMTP; 03 Nov 2010 19:27:20 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id BB754B3F35; Wed, 3 Nov 2010 19:27:20 -0400 (EDT) Date: Wed, 3 Nov 2010 19:27:20 -0400 (EDT) From: Rick Macklem <rmacklem@uoguelph.ca> To: pyunyh@gmail.com Message-ID: <1320759482.60041.1288826840754.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <20101101234001.GD1433@michelle.cdnetworks.com> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_60040_1528688248.1288826840752" X-Originating-IP: [99.225.56.115] X-Mailer: Zimbra 6.0.7_GA_2476.RHEL4 (ZimbraWebClient - IE8 (Win)/6.0.7_GA_2473.RHEL4_64) Cc: freebsd-current@freebsd.org Subject: Re: re(4) driver dropping packets when reading NFS files X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current <freebsd-current.freebsd.org> List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-current>, <mailto:freebsd-current-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-current> List-Post: <mailto:freebsd-current@freebsd.org> List-Help: <mailto:freebsd-current-request@freebsd.org?subject=help> List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-current>, <mailto:freebsd-current-request@freebsd.org?subject=subscribe> X-List-Received-Date: Wed, 03 Nov 2010 23:27:22 -0000 ------=_Part_60040_1528688248.1288826840752 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit > > I'm more interested in number of dropped frames. See below how to > extract that information. > I've attached the stats. I'm guessing that the Rx missed frames : 14792 is the culprit. This was for a read of a fairly large file via NFS over TCP, getting a read rate of about 450Kbytes/sec. (No DEVICE_POLLING option.) (with your patch applied) > > > (It almost looks like it only handles the first received packet, > > although > > it appears to be using a receive ring of 64 buffers.) > > > > No, re(4) uses 256 TX/RX buffers for RTL810xE controllers. > Oops, my mistake. At a quick glance, I had thought rl_type was set to 8139, but I now see it's 8169. Btw, I printed out the hwrev and its a RL_HWREV_8102EL_SPIN1, if that is of any use to you. > > Ok, here is patch. > http://people.freebsd.org/~yongari/re/re.intr.patch > > The patch has the following changes. > o 64bit DMA support for PCIe controllers. > o Hardware MAC statistics counter support. You can extract these > counters with "sysctl dev.re.0.stats=1". You can check the > output on console or dmesg. It seems extracting these counters > take a lot of time so I didn't try to accumulate the counters. > You can see how many frames are dropped from the output. I saw a > lot FAE(frame alignment errors) under high RX load and I can't > explain how this can happen. This may indicate PHY hardware is > poor or it may need DSP fixups. Realtek seems to maintain large > set of DSP fixups for each PHY revisions and re(4) does not > have the magic code at this moment. > o Overhaul MSI interrupt handler such that make it give fairness > to TX as well as serving RX. Because re(4) controllers do not > have interrupt moderation mechanism, naive interrupt handler can > generate more than 125k intrs/sec under high load. Fortunately, > Bill implemented TX interrupt moderation with a timer register > and it seems to work well on TX path. One drawback of the > approach is it will require extra timer register accesses in > fast path. There is no second timer register to use in RX path > so no RX interrupt moderation is done in driver such that it can > generate about 25k intrs/sec under high RX load. However, I > think most systems can handle that interrupt load. Note, this > feature is activated only when MSI is in use and DEVICE_POLLING > is not defined. > > >From my limited testing, it seems it works as expected. Would you > give it try and let me know how well it behaves with NFS? > Without DEVICE_POLLING it behaves just like the unpatched one. I'm going to look at the driver tomorrow and try some hacks on it, rick ------=_Part_60040_1528688248.1288826840752 Content-Type: text/plain; name=re.stats Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename=re.stats cmUwIHN0YXRpc3RpY3M6DQpUcmFuc21pdCBnb29kIGZyYW1lcyA6IDEwMDk2Ng0KUmVjZWl2ZSBn b29kIGZyYW1lcyA6IDEzMzQ3MA0KVHggZXJyb3JzIDogMA0KUnggZXJyb3JzIDogMA0KUnggbWlz c2VkIGZyYW1lcyA6IDE0NzkyDQpSeCBmcmFtZSBhbGlnbm1lbnQgZXJycyA6IDANClR4IHNpbmds ZSBjb2xsaXNpb25zIDogMA0KVHggbXVsdGlwbGUgY29sbGlzaW9ucyA6IDANClJ4IHVuaWNhc3Qg ZnJhbWVzIDogMTMzNDYzDQpSeCBicm9hZGNhc3QgZnJhbWVzIDogMA0KUnggbXVsdGljYXN0IGZy YW1lcyA6IDcNClR4IGFib3J0cyA6IDANClR4IHVuZGVycnVucyA6IDANCg== ------=_Part_60040_1528688248.1288826840752--