From owner-freebsd-net@FreeBSD.ORG Mon Jun 20 20:11:30 2005 Return-Path: X-Original-To: freebsd-net@freebsd.org Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9BD1916A41C for ; Mon, 20 Jun 2005 20:11:30 +0000 (GMT) (envelope-from pieter@thedarkside.nl) Received: from mail.thelostparadise.com (129pc197.sshunet.nl [145.97.197.129]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3F29043D49 for ; Mon, 20 Jun 2005 20:11:29 +0000 (GMT) (envelope-from pieter@thedarkside.nl) Received: from [195.16.84.90] (serkoon@jura.thelostparadise.com [195.16.84.90] (may be forged)) by mail.thelostparadise.com (8.12.11/8.12.8) with ESMTP id j5KKBRpL080546; Mon, 20 Jun 2005 22:11:27 +0200 (CEST) (envelope-from pieter@thedarkside.nl) Message-ID: <42B722EF.2090203@thedarkside.nl> Date: Mon, 20 Jun 2005 22:11:27 +0200 From: Pieter de Boer User-Agent: Debian Thunderbird 1.0.2 (X11/20050602) X-Accept-Language: en-us, en MIME-Version: 1.0 To: freebsd-net@freebsd.org X-Enigmail-Version: 0.91.0.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Issues with a Large Fat pipe Network simulation X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Jun 2005 20:11:30 -0000 Hello there, For a project about TCP (performance) enhancements, we have been trying to simulate a network with a high bandwidth*delay product. Although we haven't started our real tests just yet, we already stumbled upon some issues :). For one (advertising an invalid window scale in some situations), we'll file a PR soon. We have three systems: 'client', 'network' and 'server'. All three systems have two intel gigabit NICs (em) in them. They run 5.4-RELEASE using the SMP-kernel. 'network' has HZ bumped to 2000 and nmbclusters to 128*1024. The setup is as follows: 'client' <-----> 'network' <-----> 'server' 100.2 100.1 200.1 200.2 'network' routes traffic between 192.168.100.0/24 and 192.168.200.0/24 and is equipped with ipfw/dummynet for simulation purposes. We had the following ipfw pipes on 'network': pipe 1 ip from client to server via em0 pipe 2 ip from server to client via em1 We're testing using iperf ('client' actually runs the iperf server) client# iperf -s -l64K -N server# iperf -c client -i 5 -N -t 900 -l 64k When testing without any extra delay on 'network' and send/recvspaces of 65535 bytes, we can sustain around 800mbit/s. The interrupts on 'network' may be the limiting factor here. However, when we set the send/recv space to 65535*2, we can only sustain around 200-300mbit/s. It seems the speed isn't as 'stable' either (peaks of more than 300mbit/s, sometimes up to 500mbit/s). We also used read/write sizes of 128KB using the -l option on iperf, but this didn't seem to have any noticeable effect. When adding extra latency on 'network' and adjusting the send/recv-spaces to correct for the greater bandwidth*delay product, we weren't able to sustain rates much higher than 200mbit/s either. Can anyone shed some light on what we're seeing here? -- With regards, Pieter de Boer