From owner-freebsd-net@FreeBSD.ORG Mon Mar 5 18:48:58 2007 Return-Path: X-Original-To: freebsd-net@FreeBSD.org Delivered-To: freebsd-net@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 00F2716A404; Mon, 5 Mar 2007 18:48:57 +0000 (UTC) (envelope-from gallatin@cs.duke.edu) Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) by mx1.freebsd.org (Postfix) with ESMTP id B401213C4B3; Mon, 5 Mar 2007 18:48:57 +0000 (UTC) (envelope-from gallatin@cs.duke.edu) Received: from grasshopper.cs.duke.edu (grasshopper.cs.duke.edu [152.3.145.30]) by duke.cs.duke.edu (8.14.0/8.14.0) with ESMTP id l25Imvt6016866 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 5 Mar 2007 13:48:57 -0500 (EST) Received: (from gallatin@localhost) by grasshopper.cs.duke.edu (8.12.9p2/8.12.9/Submit) id l25ImpG3056078; Mon, 5 Mar 2007 13:48:51 -0500 (EST) (envelope-from gallatin) From: Andrew Gallatin MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <17900.26131.918618.853907@grasshopper.cs.duke.edu> Date: Mon, 5 Mar 2007 13:48:51 -0500 (EST) To: Robert Watson In-Reply-To: <20070305182755.S31701@fledge.watson.org> References: <45E8276D.60105@freebsd.org> <17900.24574.751134.397740@grasshopper.cs.duke.edu> <20070305182755.S31701@fledge.watson.org> X-Mailer: VM 6.75 under 21.1 (patch 12) "Channel Islands" XEmacs Lucid Cc: freebsd-net@FreeBSD.org, freebsd-current@FreeBSD.org, Andre Oppermann , kmacy@FreeBSD.org, jhb@FreeBSD.org Subject: Re: New optimized soreceive_stream() for TCP sockets, proof of concept X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Mar 2007 18:48:58 -0000 Robert Watson writes: > On Mon, 5 Mar 2007, Andrew Gallatin wrote: > > > With the patch, we finally seem to be performance competative on the receive > > side with Linux x86_64 and Solaris/amd64 on this same hardware. Both of > > those OSes do much better (saturate the link with jumbos) when CPU affinity > > is used to bind the interrupt handler and netserver process to different > > cores on the same socket. I imagine FreeBSD may be able to do even better > > if it ever grows CPU affinity support for both interrupt handlers and > > processes. With the patch, it performs at least as well, if not better > > than, Solaris and Linux do without CPU affinity. > > I don't have numbers in front of me, and am currently packing for a trip to > Tokyo so won't find them before traveling, but my experience has been that > binding the ithread to a specific CPU is very helpful in improving receive > performance. You can slap a sched_bind(0) into the interrupt handler the > first time it runs and it should stick appropriately, and add a sysctl to > sched_bind() for a user process as a hack to test it out. You lost me at adding the sysctl for the user process.. Does FreeBSD have, or plan to have, an interface to bind threads to CPUs? > > John has a patch that pins interrupt threads, etc, not sure what the status of > that is. CC'd. He wanted me to test it and I dropped the ball. By the time I got a chance (some months later), it was so stale it did not come close to applying. I've asked him to regen. Drew