From owner-freebsd-net@FreeBSD.ORG Thu Apr 27 14:38:20 2006 Return-Path: X-Original-To: freebsd-net@freebsd.org Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0C0A116A401; Thu, 27 Apr 2006 14:38:20 +0000 (UTC) (envelope-from tataz@tataz.chchile.org) Received: from smtp2-g19.free.fr (smtp2-g19.free.fr [212.27.42.28]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6108C43D49; Thu, 27 Apr 2006 14:38:19 +0000 (GMT) (envelope-from tataz@tataz.chchile.org) Received: from tatooine.tataz.chchile.org (tataz.chchile.org [82.233.239.98]) by smtp2-g19.free.fr (Postfix) with ESMTP id 9D823731A5; Thu, 27 Apr 2006 16:38:18 +0200 (CEST) Received: from obiwan.tataz.chchile.org (unknown [192.168.1.25]) by tatooine.tataz.chchile.org (Postfix) with ESMTP id 2967A9CA08; Thu, 27 Apr 2006 14:38:14 +0000 (UTC) Received: by obiwan.tataz.chchile.org (Postfix, from userid 1000) id 218C1405B; Thu, 27 Apr 2006 16:38:14 +0200 (CEST) Date: Thu, 27 Apr 2006 16:38:14 +0200 From: Jeremie Le Hen To: Robert Watson Message-ID: <20060427143814.GD84148@obiwan.tataz.chchile.org> References: <7bb8f24157080b6aaacb897a99259df9@madhaus.cns.utoronto.ca> <20060427093916.GC84148@obiwan.tataz.chchile.org> <20060427145252.I75848@fledge.watson.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20060427145252.I75848@fledge.watson.org> User-Agent: Mutt/1.5.11 Cc: Marcos Bedinelli , freebsd-net@freebsd.org Subject: Re: [fbsd] Re: [fbsd] Network performance in a dual CPU system X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Apr 2006 14:38:20 -0000 Hi, Robert, On Thu, Apr 27, 2006 at 02:54:21PM +0100, Robert Watson wrote: > > On Thu, 27 Apr 2006, Jeremie Le Hen wrote: > > >> PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND > >> 60 root 1 -44 -163 0K 8K WAIT 355.6H 72.17% swi1: > >>net > >> 39 root 1 -68 -187 0K 8K WAIT 52.3H 5.22% irq28: > >>bge0 > >> 40 root 1 -68 -187 0K 8K WAIT 28.3H 2.25% irq29: > >>bge1 > >> 11 root 1 171 52 0K 8K RUN 166.6H 0.00% idle > >> 63 root 1 -16 0 0K 8K - 121:55 0.00% yarrow > >> 61 root 1 -32 -151 0K 8K WAIT 46:21 0.00% swi4: > >>clock sio > >>[...] > >> > >>Does anyone know whether a dual CPU system can help us improve the > >>situation? I was wondering if the software interrupt threads would be > >>divided between the two processors. > > > >I am a few weeks late, I just saw this very interesting thread. What > >solution did you finally employ to circumvent your high interrupt load ? > > I missed the original thread, but in answer to the question: if you set > net.isr.direct=1, then FreeBSD 6.x will run the netisr code in the ithread > of the network device driver. This will allow the IP forwarding and > related paths in two threads instead of one, potentially allowing greater > parallelism. Of course, you also potentially contend more locks, you may > increase the time it takes for the ithread to respond to new interrupts, > etc, so it's not quite cut and dry, but with a workload like the one shown > above, it might make quite a difference. Actually you already replied in the original thread, explaining mostly the same thing. The whole thread [1] brought up multiple valuable network performance tuning knobs, such as polling, fastforwarding, net.isr.direct but there is no happy end to the thread. Given this is a real world situation, I wanted to know how Marcos revolved his problem. BTW, what I understand is that net.isr.direct=1 prevents from multiplexing all packets on the netisr thread and instead makes the ithread do the job. In this case, what happens to the netisr thread ? Does it still have some work to do or is it removed ? Thank you. Regards, [1] http://lists.freebsd.org/pipermail/freebsd-net/2006-February/thread.html#9725 -- Jeremie Le Hen < jeremie at le-hen dot org >< ttz at chchile dot org >