From owner-freebsd-net@FreeBSD.ORG  Sun Mar 18 22:40:57 2012
Return-Path: <owner-freebsd-net@FreeBSD.ORG>
Delivered-To: freebsd-net@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id 18E93106566B
	for <freebsd-net@freebsd.org>; Sun, 18 Mar 2012 22:40:57 +0000 (UTC)
	(envelope-from longwitz@incore.de)
Received: from dss.incore.de (dss.incore.de [195.145.1.138])
	by mx1.freebsd.org (Postfix) with ESMTP id 9D1078FC08
	for <freebsd-net@freebsd.org>; Sun, 18 Mar 2012 22:40:56 +0000 (UTC)
Received: from inetmail.dmz (inetmail.dmz [10.3.0.3])
	by dss.incore.de (Postfix) with ESMTP id D73DA5D29F;
	Sun, 18 Mar 2012 23:40:48 +0100 (CET)
X-Virus-Scanned: amavisd-new at incore.de
Received: from dss.incore.de ([10.3.0.3])
	by inetmail.dmz (inetmail.dmz [10.3.0.3]) (amavisd-new, port 10024)
	with LMTP id FnDmzidkQTzB; Sun, 18 Mar 2012 23:40:47 +0100 (CET)
Received: from mail.incore (fwintern.dmz [10.0.0.253])
	by dss.incore.de (Postfix) with ESMTP id DC2AE5D29D;
	Sun, 18 Mar 2012 23:40:47 +0100 (CET)
Received: from bsdmhs.longwitz (unknown [192.168.99.6])
	by mail.incore (Postfix) with ESMTP id 38CDA450A8;
	Sun, 18 Mar 2012 23:40:47 +0100 (CET)
Message-ID: <4F66646E.4080603@incore.de>
Date: Sun, 18 Mar 2012 23:40:46 +0100
From: Andreas Longwitz <longwitz@incore.de>
User-Agent: Thunderbird 2.0.0.19 (X11/20090113)
MIME-Version: 1.0
To: pyunyh@gmail.com
References: <4F594856.3030303@incore.de>
	<20120312211907.GC3671@michelle.cdnetworks.com>
	<4F5E0AF7.30302@incore.de>
	<20120313202559.GA3360@michelle.cdnetworks.com>
In-Reply-To: <20120313202559.GA3360@michelle.cdnetworks.com>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Cc: freebsd-net@freebsd.org
Subject: Re: Intel 82550 Pro/100 Ethernet and Microcode
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
	<mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
	<mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Sun, 18 Mar 2012 22:40:57 -0000

YongHyeon PYUN wrote:
> 
> The microcode is normally used to reduce high number of interrupts
> under heavy network load by bundling multiple RX frames.

ok, I understand. But I have DEVICE_POLLING and HZ=1000 in the kernel
source, so interrupts do not occur.

> However
> your reason to use microcode for i82550C looks weird since the
> microcode used for i82550C does not have a fix for TCO bug.
> The microcode for i82550(fxp2 in your system) indeed has fix for
> TCO bug and includes additional feature for bundling.

According to the comments in rcvbundl.h it is just directly opposed.

> If you're
> suffering from TCO bug of i82550, NFS over UDP issue should happen
> only on i82550(fxp2). Can you check whether the NFS issue happens
> on i82550C (fxp0 and fxp1) without loading the microcode?

The NFS issue happens on fxp0 (i82550C) without loading of microcode:
Mar 16 16:12:52 <kern.info> dssresv1 kernel: nfs server
                dsscam:/prod/mobotix: not responding
Mar 16 16:13:24 <kern.info> dssresv1 kernel: nfs server
                dsscam:/prod/mobotix: not responding
Mar 16 16:13:56 <kern.info> dssresv1 kernel: nfs server
                dsscam:/prod/mobotix: not responding
Mar 16 16:14:28 <kern.info> dssresv1 kernel: nfs server
                dsscam:/prod/mobotix: not responding
Mar 16 16:14:59 <kern.info> dssresv1 kernel: nfs server
                dsscam:/prod/mobotix: is alive again

The last message appears immediately after loading the microcode with
            ifconfig fxp0 link0.

> I still can't explain why your i82550C with the loaded microcode
> does not generates SCB timeouts because mine always shows the error
> right after loading the microcode.

In FreeBSD 6 and 8 I need my bzero-patch to get fxp working, In FreeBSD
4 - on the same hardware - this was not necessary.

> Are you actively using fxp0 or fxp1 after loading the microcode?

Yes I have 6 server with fxp0/fxp1 i82550C and 2 server with fxp0/fxp1
i82550 on motherboard in production, additionally some fxp2 i82550
external cards. All run with microcode loaded and polling.

> If yes, could you check whether
> the CPU Saver feature of the microcode really works on i82550C?
> You may be able to use netperf UDP stream test to verify that.

I did some tests with netperf 2.5.0, At first I had to patch some format
strings in netperf from %d to %lld for uint64_t variables to get
readable output from netperf on my i386 systems.

The test command
      netperf -H host_with_fxp -t UDP_STREAM
gives nearly always the same output

Socket  Message  Elapsed      Messages
Size    Size     Time         Okay Errors   Throughput
bytes   bytes    secs            #      #   10^6bits/sec

  9216    9216   10.00       13069 1515880     96.34
 41600           10.00       13069             96.34

And this output did not considerably change for the test cases fxp-type
i82550C or i82550, microcode loaded or not and polling yes or no.

But I can answer your question concerning the CPU Saver funktion on
i82550C and its a little suprise (for me). In all my tests without
polling I checked the irq's on host_with_fxp using "vmstat -i". The
result is that the CPU Saver feature of the microcode does not work on
i82250C and it works on i82250. On i82250C the value of
dev.fxp.0.bundle_max is irrelevant, the i82250C always needs ca. 91000
irq's for the netperf test. The same number of irq's needs the i82250
with bundle_max=1, but when I set bundle_max=6 (the default), then the
number of irq's for the same test goes down to 91000/7 = 13000.


Regards

Andreas Longwitz