From owner-freebsd-current@FreeBSD.ORG Thu Nov 4 00:39:17 2010 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 73EA1106566C for ; Thu, 4 Nov 2010 00:39:17 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-px0-f182.google.com (mail-px0-f182.google.com [209.85.212.182]) by mx1.freebsd.org (Postfix) with ESMTP id 426098FC0A for ; Thu, 4 Nov 2010 00:39:17 +0000 (UTC) Received: by pxi1 with SMTP id 1so76001pxi.13 for ; Wed, 03 Nov 2010 17:39:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:received:from:date:to:cc :subject:message-id:reply-to:references:mime-version:content-type :content-disposition:in-reply-to:user-agent; bh=/ZI1owUWBO1y+Jv3AYgOKfH190iWiWvMifuM3+Ouijg=; b=JvVVGhJAkhyuFFQ6HP06zfVqRSyVot23qBbL4Id14+4tkTehFicrUKbq9AtTxeoyvP /PyzWIiEM8w9acHjWvlNmqPLZfObBBeR81+/eRw1YvqIqR88n9ZYfXj/P2n0IdHl4BjP hakMoh9nOR4Rf/PEgkjFEQ/Q/2w/iSY3hEHMM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=RmZgrk85miLh0WoMef9UZnsXPf9W5ttRl1l5h8nhoAQNGtGVjxDHSo6Zex/yh5t7Xd rAMmZp3X7fr540XuknXxB4D2yDwnzw7xbTyCLqo2YM18JWg9o4M5nw7aUoW/7rj1hJEP wjXMHQCOU47k+dWFIgTF+8FujheM5ERvdZda4= Received: by 10.142.153.4 with SMTP id a4mr1031465wfe.240.1288831153574; Wed, 03 Nov 2010 17:39:13 -0700 (PDT) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id w22sm14482596wfd.7.2010.11.03.17.39.10 (version=TLSv1/SSLv3 cipher=RC4-MD5); Wed, 03 Nov 2010 17:39:11 -0700 (PDT) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Wed, 3 Nov 2010 17:39:06 -0700 From: Pyun YongHyeon Date: Wed, 3 Nov 2010 17:39:06 -0700 To: Rick Macklem Message-ID: <20101104003906.GC14740@michelle.cdnetworks.com> References: <20101101234001.GD1433@michelle.cdnetworks.com> <1320759482.60041.1288826840754.JavaMail.root@erie.cs.uoguelph.ca> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1320759482.60041.1288826840754.JavaMail.root@erie.cs.uoguelph.ca> User-Agent: Mutt/1.4.2.3i Cc: freebsd-current@freebsd.org Subject: Re: re(4) driver dropping packets when reading NFS files X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Nov 2010 00:39:17 -0000 On Wed, Nov 03, 2010 at 07:27:20PM -0400, Rick Macklem wrote: > > > > I'm more interested in number of dropped frames. See below how to > > extract that information. > > > > I've attached the stats. I'm guessing that the > Rx missed frames : 14792 > is the culprit. > Because that counter is 16bit it's also possible it wrapped multiple times. Could you verify that? > This was for a read of a fairly large file via NFS over TCP, > getting a read rate of about 450Kbytes/sec. (No DEVICE_POLLING option.) > (with your patch applied) > > > > > > (It almost looks like it only handles the first received packet, > > > although > > > it appears to be using a receive ring of 64 buffers.) > > > > > > > No, re(4) uses 256 TX/RX buffers for RTL810xE controllers. > > > > Oops, my mistake. At a quick glance, I had thought rl_type was > set to 8139, but I now see it's 8169. > > Btw, I printed out the hwrev and its a RL_HWREV_8102EL_SPIN1, > if that is of any use to you. > > > > > Ok, here is patch. > > http://people.freebsd.org/~yongari/re/re.intr.patch > > > > The patch has the following changes. > > o 64bit DMA support for PCIe controllers. > > o Hardware MAC statistics counter support. You can extract these > > counters with "sysctl dev.re.0.stats=1". You can check the > > output on console or dmesg. It seems extracting these counters > > take a lot of time so I didn't try to accumulate the counters. > > You can see how many frames are dropped from the output. I saw a > > lot FAE(frame alignment errors) under high RX load and I can't > > explain how this can happen. This may indicate PHY hardware is > > poor or it may need DSP fixups. Realtek seems to maintain large > > set of DSP fixups for each PHY revisions and re(4) does not > > have the magic code at this moment. > > o Overhaul MSI interrupt handler such that make it give fairness > > to TX as well as serving RX. Because re(4) controllers do not > > have interrupt moderation mechanism, naive interrupt handler can > > generate more than 125k intrs/sec under high load. Fortunately, > > Bill implemented TX interrupt moderation with a timer register > > and it seems to work well on TX path. One drawback of the > > approach is it will require extra timer register accesses in > > fast path. There is no second timer register to use in RX path > > so no RX interrupt moderation is done in driver such that it can > > generate about 25k intrs/sec under high RX load. However, I > > think most systems can handle that interrupt load. Note, this > > feature is activated only when MSI is in use and DEVICE_POLLING > > is not defined. > > > > >From my limited testing, it seems it works as expected. Would you > > give it try and let me know how well it behaves with NFS? > > > Without DEVICE_POLLING it behaves just like the unpatched one. > Hmm, that's strange. Are you sure you rebuilt kernel without polling option? Just disabling polling with ifconfig(8) has no effect to make patch work. > I'm going to look at the driver tomorrow and try some hacks on it, rick > re0 statistics: > Transmit good frames : 100966 > Receive good frames : 133470 > Tx errors : 0 > Rx errors : 0 > Rx missed frames : 14792 If the counter was not wrapped, it seem you lost more than 10% out of total RX frames. This is a lot loss and there should be a way to mitigate it. > Rx frame alignment errs : 0 > Tx single collisions : 0 > Tx multiple collisions : 0 > Rx unicast frames : 133463 > Rx broadcast frames : 0 > Rx multicast frames : 7 > Tx aborts : 0 > Tx underruns : 0