From owner-freebsd-net Fri Nov 1 8: 4:41 2002 Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5883B37B401 for ; Fri, 1 Nov 2002 08:04:39 -0800 (PST) Received: from seraph3.grc.nasa.gov (seraph3.lerc.nasa.gov [128.156.10.12]) by mx1.FreeBSD.org (Postfix) with ESMTP id AA71B43E77 for ; Fri, 1 Nov 2002 08:04:38 -0800 (PST) (envelope-from Fran.Lawas-Grodek@lerc.nasa.gov) Received: from lombok-fi.lerc.nasa.gov (lombok-fi.lerc.nasa.gov [139.88.112.33]) by seraph3.grc.nasa.gov (Postfix) with ESMTP id 102D0640BE for ; Fri, 1 Nov 2002 11:04:33 -0500 (EST) Received: from jamaica.lerc.nasa.gov (IDENT:KJbAQ0xQ3Dg6HQA4dZKqx+5ZxoODgMxg@jamaica.lerc.nasa.gov [139.88.38.84]) by lombok-fi.lerc.nasa.gov (NASA GRC 8.12.3/8.12.3) with ESMTP id gA1G4Wwg019614; Fri, 1 Nov 2002 11:04:32 -0500 (EST) Received: (fmfran@localhost) by jamaica.lerc.nasa.gov (8.11.6/2.01-local) id gA1G4Ws03393; Fri, 1 Nov 2002 11:04:32 -0500 Date: Fri, 1 Nov 2002 11:04:32 -0500 From: Fran Lawas-Grodek To: Luigi Rizzo Cc: Cindy.Tran@grc.nasa.gov, freebsd-net@FreeBSD.ORG, Mark.Allman@grc.nasa.gov Subject: Re: Problem in High Speed and Long Delay with FreeBSD Message-ID: <20021101110432.D3259@grc.nasa.gov> Reply-To: Fran.Lawas-Grodek@grc.nasa.gov References: <20021031135601.B23802@grc.nasa.gov> <20021031150217.A35590@carp.icir.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i In-Reply-To: <20021031150217.A35590@carp.icir.org>; from rizzo@icir.org on Thu, Oct 31, 2002 at 03:02:17PM -0800 Sender: owner-freebsd-net@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org These are our sysctl settings: kern.ipc.maxsockbuf=4194304 net.inet.tcp.sendspace=3125000 net.inet.tcp.recvspace=3125000 net.inet.ip.intr_queue_maxlen=500 nmbclusters=32768 After reading your suggestion, we were able to achieve a slightly better throughput from 32Mbps on the 250ms delayed network, to 46Mbps overall throughput, by increasing the -l buffer length from 1024 to 8192 bytes. Increasing the above intr_queue_maxlen from the default 50 to 500 also helped a bit. Our time sequence plot now shows a sender buffer window of 1.5MB being used, from the 1MB of the earlier tests. (BTW, a typo in the my posted ttcp example -- "-b" should be "-b 3125000", not "-b 312500".) We still do not understand why we cannot not get a better window usage of our requested socket buffer of 3MB, a better throughput of 60+Mbps. Any other thoughts? Fran Lawas-Grodek Cindy Tran NASA Glenn Research Center ________________________________________________________________ Frances J. Lawas-Grodek | NASA Glenn Research Center | phone: (216) 433-5052 21000 Brookpark Rd, MS 142-4 | fax : (216) 433-8000 Cleveland, Ohio 44135 | email: fran@grc.nasa.gov ________________________________________________________________ On Thu, Oct 31, 2002 at 03:02:17PM -0800, Luigi Rizzo wrote: > you might want to have a look at the sysctl variable > kern.ipc.sockbuf_waste_factor too. > > Remember that memory is charged to socket buffers depending on how > many clusters are allocated, even if they are not fully used. > E.g. in your example you are probably doing 1KB writes each of > which consumes a 2KB cluster plus a 256byte mbuf, so no > matter what you will never manage to reach more than (roughly) > a window larger than kern.ipc.maxsockbuf/2. > > The max raw amounf of memory allocated in a socket buffer is > typically > > min( tcp_{send|recv}buf * kern.ipc.sockbuf_waste_factor, > kern.ipc.maxsockbuf) > > and probably you are hitting the roof on the second one. > > cheers > luigi To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-net" in the body of the message