From owner-freebsd-net@FreeBSD.ORG  Wed Aug  9 11:36:07 2006
Return-Path: <owner-freebsd-net@FreeBSD.ORG>
X-Original-To: freebsd-net@freebsd.org
Delivered-To: freebsd-net@freebsd.org
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 5376A16A4DA
	for <freebsd-net@freebsd.org>; Wed,  9 Aug 2006 11:36:07 +0000 (UTC)
	(envelope-from roshind@mail.ru)
Received: from f49.mail.ru (f49.mail.ru [194.67.57.85])
	by mx1.FreeBSD.org (Postfix) with ESMTP id E20D743D4C
	for <freebsd-net@freebsd.org>; Wed,  9 Aug 2006 11:36:06 +0000 (GMT)
	(envelope-from roshind@mail.ru)
Received: from mail by f49.mail.ru with local id 1GAmM1-000I0z-00
	for freebsd-net@freebsd.org; Wed, 09 Aug 2006 15:36:05 +0400
Received: from [81.94.128.242] by win.mail.ru with HTTP;
	Wed, 09 Aug 2006 15:36:05 +0400
From: Dima Roshin <roshind@mail.ru>
To: freebsd-net@freebsd.org
Mime-Version: 1.0
X-Mailer: mPOP Web-Mail 2.19
X-Originating-IP: [81.94.128.242]
Date: Wed, 09 Aug 2006 15:36:05 +0400
Content-Type: text/plain; charset=koi8-r
Content-Transfer-Encoding: 8bit
Message-Id: <E1GAmM1-000I0z-00.roshind-mail-ru@f49.mail.ru>
Subject: Gigabit ethernet questions?
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
Reply-To: Dima Roshin <roshind@mail.ru>
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
	<mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
	<mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 09 Aug 2006 11:36:07 -0000

 Greeting colleagues. I've got two DL-360(pciX bus) servers, with BCM5704 NetXtreme Dual Gigabit Adapters(bge). The Uname is 6.1-RELEASE-p3. The bge interfaces of the both servers are connected with each other with a cat6 patchcord.
   Here are my settings:
kernel config:
options DEVICE_POLLING 
options HZ=1000 #

sysctl.conf:
kern.polling.enable=1
net.inet.ip.intr_queue_maxlen=5000
kern.ipc.maxsockbuf=8388608
net.inet.tcp.sendspace=3217968
net.inet.tcp.recvspace=3217968
net.inet.tcp.rfc1323=1

bge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 9000
        options=5b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,POLLING>
        inet 192.168.0.1 netmask 0xffffff00 broadcast 192.168.0.255
        ether 00:17:a4:3a:e1:81
        media: Ethernet autoselect (1000baseTX <full-duplex>)
        status: active
(note mtu 9000)

and here are tests results:

netperf:

TCP STREAM TEST to 192.168.0.1
Recv   Send    Send
Socket Socket  Message  Elapsed
Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    10^6bits/sec

6217968 6217968 6217968    10.22     320.04

UDP UNIDIRECTIONAL SEND TEST to 192.168.0.1
Socket  Message  Elapsed      Messages
Size    Size     Time         Okay Errors   Throughput
bytes   bytes    secs            #      #   10^6bits/sec

  9216    9216   10.00      118851 1724281     876.20
41600           10.00           0              0.00]



iperf:
gate2# iperf -s -N
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 3.07 MByte (default)
------------------------------------------------------------
[  4] local 192.168.0.2 port 5001 connected with 192.168.0.1 port 52597
[  4]  0.0-10.1 sec    384 MBytes    319 Mbits/sec

Also I can say, that I've managed to achieve about 500mbit.s by tuning tcp window with -w key in iperf.

How can we explain such a low tcp performance? What else is to tune? Is there somebody who achieved gigabit speed with tcp on freebsd?

Thanks.