From owner-freebsd-current@FreeBSD.ORG Sat Sep 23 01:47:39 2006 Return-Path: X-Original-To: freebsd-current@freebsd.org Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7AA0716A403; Sat, 23 Sep 2006 01:47:39 +0000 (UTC) (envelope-from gallatin@cs.duke.edu) Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) by mx1.FreeBSD.org (Postfix) with ESMTP id 04C7C43D45; Sat, 23 Sep 2006 01:47:38 +0000 (GMT) (envelope-from gallatin@cs.duke.edu) Received: from grasshopper.cs.duke.edu (grasshopper.cs.duke.edu [152.3.145.30]) by duke.cs.duke.edu (8.13.6/8.13.6) with ESMTP id k8N1lbVl012325 (version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=NO); Fri, 22 Sep 2006 21:47:37 -0400 (EDT) Received: (from gallatin@localhost) by grasshopper.cs.duke.edu (8.12.9p2/8.12.9/Submit) id k8N1lWQl043880; Fri, 22 Sep 2006 21:47:32 -0400 (EDT) (envelope-from gallatin) From: Andrew Gallatin MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <17684.37427.990749.849229@grasshopper.cs.duke.edu> Date: Fri, 22 Sep 2006 21:47:31 -0400 (EDT) To: Andre Oppermann In-Reply-To: <45145F1D.8020005@freebsd.org> References: <4511B9B1.2000903@freebsd.org> <17683.63162.919620.114649@grasshopper.cs.duke.edu> <45145F1D.8020005@freebsd.org> X-Mailer: VM 6.75 under 21.1 (patch 12) "Channel Islands" XEmacs Lucid Cc: alc@freebsd.org, freebsd-net@freebsd.org, freebsd-current@freebsd.org, tegge@freebsd.org Subject: Re: Much improved sendfile(2) kernel implementation X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Sep 2006 01:47:39 -0000 Andre Oppermann writes: > Andrew Gallatin wrote: > > > > Between TSO and your sendfile changes, things are looking up! > > > > Here are some Myri10GbE 1500 byte results from a 1.8GHz UP > > FreeBSD/amd64 machine (AMD Athlon(tm) 64 Processor 3000+) sending to a > > 2.0GHz SMP Linux/x86_64 machine (AMD Athlon(tm) 64 X2 Dual Core Processor > > 3800+) running 26.17.7smp and our 1.1.0 Myri10GE driver (with LRO). > > I used a linux receiver because LRO is the only way to receive > > standard frames at line rate (without a TOE). > > > > These tests are all for sendfile of a 10MB file in /var/tmp: > > > > % netperf242 -Hrome-my -tTCP_SENDFILE -F /var//tmp/zot -T,1 -c -C -- -s393216 > > You should use -m5M as well. netperf is kinda dumb and does only I will try this. > > The -T,1 is required to force the netserver to use a different core > > than the interrupt handler is bound to on the linux machine. BTW, > > it would be really nice if FreeBSD supported CPU affinity for processes > > and interrupt handlers.. > > I have a gross version of that in my tree. The kernel itself supports > it but it's not yet exposed to userland for manual intervention. Sweet. > Be a bit careful with the CPU usage figures. The numbers netperf reports > differ quite a bit from those reported by time(1) on the high side. And > there are some differences in the approach how FreeBSD and Linux do their > statistical measurements of user and system time. This doesn't change the Netperf essentially subtracts idle time from all other time. The numbers I see reported from netperf have historically matched what vmstat says. BTW, I use netperf from svn. I don't use the FreeBSD port. > throughput number though. But see the -m5M option. New sendfile is really > optimized to chew on a large file (larger than the socket buffer size) as > it normally happens in reality. > > > For comparision, if I reboot the sender into RHEL (Linux 2.6.9-11.EL x86_64): > > 87380 65536 65536 10.01 9333.00 28.98 75.23 0.254 1.321 > > > > > > The above results are the median result for 5 runs at each setting. > > How large is the variance between the runs? Much bigger for FreeBSD than for Linux. I think I had one 9.1 and 4 9.3s for linux. I had a 6.2 a 6.5 and 3 around 6.9 for FreeBSD, Drew