From owner-freebsd-net@FreeBSD.ORG Thu Apr 27 13:54:23 2006 Return-Path: X-Original-To: freebsd-net@freebsd.org Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A599E16A401 for ; Thu, 27 Apr 2006 13:54:23 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id 244A743D48 for ; Thu, 27 Apr 2006 13:54:23 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 012C246C3B; Thu, 27 Apr 2006 09:54:22 -0400 (EDT) Date: Thu, 27 Apr 2006 14:54:21 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Jeremie Le Hen In-Reply-To: <20060427093916.GC84148@obiwan.tataz.chchile.org> Message-ID: <20060427145252.I75848@fledge.watson.org> References: <7bb8f24157080b6aaacb897a99259df9@madhaus.cns.utoronto.ca> <20060427093916.GC84148@obiwan.tataz.chchile.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Marcos Bedinelli , freebsd-net@freebsd.org Subject: Re: [fbsd] Network performance in a dual CPU system X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Apr 2006 13:54:23 -0000 On Thu, 27 Apr 2006, Jeremie Le Hen wrote: >> PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND >> 60 root 1 -44 -163 0K 8K WAIT 355.6H 72.17% swi1: >> net >> 39 root 1 -68 -187 0K 8K WAIT 52.3H 5.22% irq28: >> bge0 >> 40 root 1 -68 -187 0K 8K WAIT 28.3H 2.25% irq29: >> bge1 >> 11 root 1 171 52 0K 8K RUN 166.6H 0.00% idle >> 63 root 1 -16 0 0K 8K - 121:55 0.00% yarrow >> 61 root 1 -32 -151 0K 8K WAIT 46:21 0.00% swi4: >> clock sio >> [...] >> >> Does anyone know whether a dual CPU system can help us improve the >> situation? I was wondering if the software interrupt threads would be >> divided between the two processors. > > I am a few weeks late, I just saw this very interesting thread. What > solution did you finally employ to circumvent your high interrupt load ? I missed the original thread, but in answer to the question: if you set net.isr.direct=1, then FreeBSD 6.x will run the netisr code in the ithread of the network device driver. This will allow the IP forwarding and related paths in two threads instead of one, potentially allowing greater parallelism. Of course, you also potentially contend more locks, you may increase the time it takes for the ithread to respond to new interrupts, etc, so it's not quite cut and dry, but with a workload like the one shown above, it might make quite a difference. Robert N M Watson