From owner-freebsd-current@FreeBSD.ORG Fri Nov 5 02:32:01 2010 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4BCE21065670 for ; Fri, 5 Nov 2010 02:32:01 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-iw0-f182.google.com (mail-iw0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 073A68FC12 for ; Fri, 5 Nov 2010 02:32:00 +0000 (UTC) Received: by iwn39 with SMTP id 39so2233628iwn.13 for ; Thu, 04 Nov 2010 19:32:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:received:from:date:to:cc :subject:message-id:reply-to:references:mime-version:content-type :content-disposition:in-reply-to:user-agent; bh=n9xecjTwCH0yLw3qMMVieYQ0TWPpEG6xjjeNC+TXt3U=; b=VtDxWFJKtsutlj9aEkwcJCiBhEXsgXd0HnspnTyVzjUvN/tlj7oLcAEeSBQr6++juY nNJD5O6AeTLxpHZ8XKuGbj16ydZb7L6jMLcPEmLYlkHpBWCzA5OA1nQnw5oozGz4Br/T FExqOLFqvioozBKUQFY6DJuEHCNq0eqY6Tu6A= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=DZKYUU3vMEd4aRT6pdpgDCEwLP21Ti2Aii8rb5wTEl01cDUaKFWvIbgPYsYwb5/bM8 RW62K7UG5pP3iRuBqDCgqwXRBrg3RmA92HiHo/Qsy2mv+pFxTc4w9Mk77WHy7RT0T4gU qqVsPgChRSoko7q/M3CWk+NoR7QnOsZR34sFE= Received: by 10.231.85.138 with SMTP id o10mr1237461ibl.165.1288924320394; Thu, 04 Nov 2010 19:32:00 -0700 (PDT) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id fw4sm623288ibb.19.2010.11.04.19.31.57 (version=TLSv1/SSLv3 cipher=RC4-MD5); Thu, 04 Nov 2010 19:31:58 -0700 (PDT) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Thu, 4 Nov 2010 19:31:53 -0700 From: Pyun YongHyeon Date: Thu, 4 Nov 2010 19:31:53 -0700 To: Rick Macklem Message-ID: <20101105023153.GA20301@michelle.cdnetworks.com> References: <20101104003906.GC14740@michelle.cdnetworks.com> <430617549.130565.1288920690389.JavaMail.root@erie.cs.uoguelph.ca> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="mP3DRpeJDSE+ciuQ" Content-Disposition: inline In-Reply-To: <430617549.130565.1288920690389.JavaMail.root@erie.cs.uoguelph.ca> User-Agent: Mutt/1.4.2.3i Cc: freebsd-current@freebsd.org Subject: Re: re(4) driver dropping packets when reading NFS files X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Nov 2010 02:32:01 -0000 --mP3DRpeJDSE+ciuQ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Thu, Nov 04, 2010 at 09:31:30PM -0400, Rick Macklem wrote: > > > > If the counter was not wrapped, it seem you lost more than 10% out of > > total RX frames. This is a lot loss and there should be a way to > > mitigate it. > > > I've attached a patch (to the if_re.c in head, not your patched variant) > that works a lot better (about 5Mbytes/sec read rate). To get that, I > had to disable msi and not clear the RL_IMR register in re_intr(). I > suspect that a packet would be received between when the bits in RL_IMR > were cleared and when they were set at the end of re_int_task() and those > were getting lost. > > This patch doesn't completely fix the problem. (I added your stats collecting > stuff to the if_re.c in head and attached the result, which still shows some lost packets. One > thought is clearing the bits in RL_ISR in re_intr() instead of re_int_task(), > but then I can't see a good way to pass the old value of the status reg. > through to re_int_task()? > Hmm, I still don't understand how the patch mitigates the issue. :-( The patch does not disable interrupts in interrupt handler so taskqueue runs with interrupt enabled. This may ensure not loosing interrupts but it may also generate many interrupts. By chance, did you check number of interrupts generated with/without your patch? The only guess I have at the moment is the writing RL_IMR in interrupt handler at the end of taskqueue might be not immediately reflected so controller can loose interrupts for the time window. Would you try attach patch and let me know it makes any difference? > The patch doesn't help when msi is enabled and when I played with your > patched variant, I got it to hang when RL_IMR wasn't cleared. > > I've attached the patch and stats. > > I might play around with it some more tomorrow, rick Thanks for your work. > ps: If you have hardware to test re with, you want to do an NFS mount > and then read a large file when nothing else is happening on the > machine, to see if you can reproduce the problem. It seems I'm not able to reproduce it on my box(8168B GigE). > pss: All tests done with a kernel that does not have option DEVICE_POLLING. Ok, I have to commit statistics counter patch since it seems to help narrowing down driver issue. > re0 statistics: > Transmit good frames : 83320 > Receive good frames : 136158 > Tx errors : 0 > Rx errors : 0 > Rx missed frames : 2666 > Rx frame alignment errs : 0 > Tx single collisions : 0 > Tx multiple collisions : 0 > Rx unicast frames : 136157 > Rx broadcast frames : 0 > Rx multicast frames : 1 > Tx aborts : 0 > Tx underruns : 0 --mP3DRpeJDSE+ciuQ Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="re.intr.patch2" Index: sys/dev/re/if_re.c =================================================================== --- sys/dev/re/if_re.c (revision 214692) +++ sys/dev/re/if_re.c (working copy) @@ -2254,6 +2254,7 @@ } CSR_WRITE_2(sc, RL_IMR, RL_INTRS_CPLUS); + CSR_READ_2(sc, RL_IMR); } static int --mP3DRpeJDSE+ciuQ--