From owner-freebsd-net@FreeBSD.ORG Wed Oct 12 21:17:20 2005 Return-Path: X-Original-To: net@FreeBSD.org Delivered-To: freebsd-net@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9BEA116A41F; Wed, 12 Oct 2005 21:17:20 +0000 (GMT) (envelope-from gallatin@cs.duke.edu) Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) by mx1.FreeBSD.org (Postfix) with ESMTP id E3FAB43D45; Wed, 12 Oct 2005 21:17:19 +0000 (GMT) (envelope-from gallatin@cs.duke.edu) Received: from grasshopper.cs.duke.edu (grasshopper.cs.duke.edu [152.3.145.30]) by duke.cs.duke.edu (8.13.4/8.13.4) with ESMTP id j9CLHHn1020537 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 12 Oct 2005 17:17:18 -0400 (EDT) Received: (from gallatin@localhost) by grasshopper.cs.duke.edu (8.12.9p2/8.12.9/Submit) id j9CLHCqn007199; Wed, 12 Oct 2005 17:17:12 -0400 (EDT) (envelope-from gallatin) From: Andrew Gallatin MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <17229.32088.696346.868182@grasshopper.cs.duke.edu> Date: Wed, 12 Oct 2005 17:17:12 -0400 (EDT) To: Robert Watson In-Reply-To: <20051012212915.E66014@fledge.watson.org> References: <20051008143854.B84936@fledge.watson.org> <17229.29164.891534.200216@grasshopper.cs.duke.edu> <20051012212915.E66014@fledge.watson.org> X-Mailer: VM 6.75 under 21.1 (patch 12) "Channel Islands" XEmacs Lucid Cc: net@FreeBSD.org Subject: Re: Call for performance evaluation: net.isr.direct (fwd) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Oct 2005 21:17:20 -0000 Robert Watson writes: > > On Wed, 12 Oct 2005, Andrew Gallatin wrote: > > > Speaking of net.isr, is there any reason why if_simloop() calls > > netisr_queue() rather than netisr_dispatch()? > > Yes -- it's basically to prevent recursion for loopback traffic, which can > result in both lock orders and general concerns regarding reentrance. To > be specific: if you send a packet on a loopback TCP socket, it gets > processes asynchronously in the netisr rather than immediately walking > back into the TCP code again. Right now WITNESS would warn about this, > but there were also quite bad things that could happen before we did the > locking work -- for example, when connections are torn down. It also > avoids Really Deep Stacks. Right now, at least, it seems to work OK. I haven't tried witness, but a non-debug kernel shows a big speedup from enabling it. Do you think there is a chance that it could be made to work in FreeBSD? For what its worth, our TCP RR loopback latency on a dual-core AMD64 3800+ is roughly 2x as much as the solaris and linux latency. Heck, we're even slower than Darwin. And I didn't think it was *possible* to be slower than Darwin: --------------------------------------------------------------------- Host OS 2p/0K Pipe AF UDP RPC/ TCP RPC/ TCP ctxsw UNIX UDP TCP conn --------- ------------- ----- ----- ---- ----- ----- ----- ----- ---- rome FreeBSD 7.0-C 7.220 15.7 18.1 36.1 rome.sw.m Darwin 8.2.0 1.250 6.531 19.9 33.7 48.4 33.0 48.0 63. rome SunOS 5.11 5.140 12.8 15.4 22.0 17.0 43.1 141. rome Linux 2.6.9-1 3.760 9.780 11.7 17.0 20.5 18.9 23.2 32. Using net.isr.direct over the loopback brings the latency down to 25us. vmstat 1 with direct disabled: procs memory page disk faults cpu r b w avm fre flt re pi po fr sr ad0 in sy cs us sy id 2 0 0 33216 457016 0 0 0 0 0 0 4 53593 107075 214030 5 68 27 2 0 0 33216 457016 0 0 0 0 0 0 0 53694 107279 214356 5 72 24 <...> And with it enabled: 2 0 0 33796 456620 0 0 0 0 0 0 0 106 153987 150570 2 59 39 2 0 0 33796 456620 0 0 0 0 0 0 0 107 154100 150945 3 61 36 Drew