From owner-freebsd-current@FreeBSD.ORG Fri Sep 22 14:44:19 2006 Return-Path: X-Original-To: freebsd-current@freebsd.org Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 394A016A412; Fri, 22 Sep 2006 14:44:19 +0000 (UTC) (envelope-from gallatin@cs.duke.edu) Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) by mx1.FreeBSD.org (Postfix) with ESMTP id BF0C243D58; Fri, 22 Sep 2006 14:44:18 +0000 (GMT) (envelope-from gallatin@cs.duke.edu) Received: from grasshopper.cs.duke.edu (grasshopper.cs.duke.edu [152.3.145.30]) by duke.cs.duke.edu (8.13.6/8.13.6) with ESMTP id k8MEiGOZ027656 (version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=NO); Fri, 22 Sep 2006 10:44:16 -0400 (EDT) Received: (from gallatin@localhost) by grasshopper.cs.duke.edu (8.12.9p2/8.12.9/Submit) id k8MEiA4Z042996; Fri, 22 Sep 2006 10:44:11 -0400 (EDT) (envelope-from gallatin) From: Andrew Gallatin MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <17683.63162.919620.114649@grasshopper.cs.duke.edu> Date: Fri, 22 Sep 2006 10:44:10 -0400 (EDT) To: Andre Oppermann In-Reply-To: <4511B9B1.2000903@freebsd.org> References: <4511B9B1.2000903@freebsd.org> X-Mailer: VM 6.75 under 21.1 (patch 12) "Channel Islands" XEmacs Lucid Cc: alc@freebsd.org, freebsd-net@freebsd.org, freebsd-current@freebsd.org, tegge@freebsd.org Subject: Re: Much improved sendfile(2) kernel implementation X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Sep 2006 14:44:19 -0000 Between TSO and your sendfile changes, things are looking up! Here are some Myri10GbE 1500 byte results from a 1.8GHz UP FreeBSD/amd64 machine (AMD Athlon(tm) 64 Processor 3000+) sending to a 2.0GHz SMP Linux/x86_64 machine (AMD Athlon(tm) 64 X2 Dual Core Processor 3800+) running 26.17.7smp and our 1.1.0 Myri10GE driver (with LRO). I used a linux receiver because LRO is the only way to receive standard frames at line rate (without a TOE). These tests are all for sendfile of a 10MB file in /var/tmp: % netperf242 -Hrome-my -tTCP_SENDFILE -F /var//tmp/zot -T,1 -c -C -- -s393216 The -T,1 is required to force the netserver to use a different core than the interrupt handler is bound to on the linux machine. BTW, it would be really nice if FreeBSD supported CPU affinity for processes and interrupt handlers.. I did a number of runs with TSO and the patch applied and found that setting the send-side socket buffer size to 393216 gave the best performance in that case. I used this size for all tests, but it is possible there is a different sweet spot for other configurations. Note that linux auto-tunes socket buffer sizes, so I omitted the -- -s393216 for linux. Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB Without patch: 87380 393216 393216 10.00 2163.08 100.00 19.35 3.787 1.466 Without patch + TSO: 87380 393216 393216 10.00 4367.18 71.54 42.07 1.342 1.578 With patch: 87380 393216 393216 10.01 1882.73 86.15 18.43 3.749 1.604 With patch + TSO: 87380 393216 393216 10.00 6961.08 47.69 60.11 0.561 1.415 For comparision, if I reboot the sender into RHEL (Linux 2.6.9-11.EL x86_64): 87380 65536 65536 10.01 9333.00 28.98 75.23 0.254 1.321 The above results are the median result for 5 runs at each setting. Drew