From owner-freebsd-net@FreeBSD.ORG Fri Apr 28 13:02:48 2006 Return-Path: X-Original-To: freebsd-net@freebsd.org Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7871116A403 for ; Fri, 28 Apr 2006 13:02:48 +0000 (UTC) (envelope-from bedinelli@madhaus.cns.utoronto.ca) Received: from madhaus.cns.utoronto.ca (madhaus.cns.utoronto.ca [128.100.103.10]) by mx1.FreeBSD.org (Postfix) with SMTP id C6B7943D48 for ; Fri, 28 Apr 2006 13:02:47 +0000 (GMT) (envelope-from bedinelli@madhaus.cns.utoronto.ca) Received: (qmail 7685 invoked by uid 31014); 28 Apr 2006 13:02:46 -0000 Received: from [128.100.103.148] (HELO [128.100.103.148]) (128.100.103.148) by madhaus.cns.utoronto.ca (qpsmtpd/0.30) with ESMTP; Fri, 28 Apr 2006 09:02:46 -0400 In-Reply-To: <20060427143814.GD84148@obiwan.tataz.chchile.org> References: <7bb8f24157080b6aaacb897a99259df9@madhaus.cns.utoronto.ca> <20060427093916.GC84148@obiwan.tataz.chchile.org> <20060427145252.I75848@fledge.watson.org> <20060427143814.GD84148@obiwan.tataz.chchile.org> Mime-Version: 1.0 (Apple Message framework v623) Content-Type: text/plain; charset=US-ASCII; format=flowed Message-Id: <31c0d864b5b8cd45fdc01e30e6685488@madhaus.cns.utoronto.ca> Content-Transfer-Encoding: 7bit From: Marcos Bedinelli Date: Fri, 28 Apr 2006 09:02:45 -0400 To: Jeremie Le Hen X-Mailer: Apple Mail (2.623) Cc: freebsd-net@freebsd.org Subject: Re: [fbsd] Re: [fbsd] Network performance in a dual CPU system X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Apr 2006 13:02:48 -0000 Hi, On 27-Apr-06, at 10:38, Jeremie Le Hen wrote: > Hi, Robert, > > On Thu, Apr 27, 2006 at 02:54:21PM +0100, Robert Watson wrote: >> >> On Thu, 27 Apr 2006, Jeremie Le Hen wrote: >> >>>> PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU >>>> COMMAND >>>> 60 root 1 -44 -163 0K 8K WAIT 355.6H 72.17% >>>> swi1: >>>> net >>>> 39 root 1 -68 -187 0K 8K WAIT 52.3H 5.22% >>>> irq28: >>>> bge0 >>>> 40 root 1 -68 -187 0K 8K WAIT 28.3H 2.25% >>>> irq29: >>>> bge1 >>>> 11 root 1 171 52 0K 8K RUN 166.6H 0.00% idle >>>> 63 root 1 -16 0 0K 8K - 121:55 0.00% >>>> yarrow >>>> 61 root 1 -32 -151 0K 8K WAIT 46:21 0.00% >>>> swi4: >>>> clock sio >>>> [...] >>>> >>>> Does anyone know whether a dual CPU system can help us improve the >>>> situation? I was wondering if the software interrupt threads would >>>> be >>>> divided between the two processors. >> >> I missed the original thread, but in answer to the question: if you >> set >> net.isr.direct=1, then FreeBSD 6.x will run the netisr code in the >> ithread >> of the network device driver. This will allow the IP forwarding and >> related paths in two threads instead of one, potentially allowing >> greater >> parallelism. Of course, you also potentially contend more locks, you >> may >> increase the time it takes for the ithread to respond to new >> interrupts, >> etc, so it's not quite cut and dry, but with a workload like the one >> shown >> above, it might make quite a difference. > > Actually you already replied in the original thread, explaining mostly > the same thing. The whole thread [1] brought up multiple valuable > network performance tuning knobs, such as polling, fastforwarding, > net.isr.direct but there is no happy end to the thread. Given this is > a real world situation, I wanted to know how Marcos revolved his > problem. Shortly after I opened the thread, most replies I received (from the list or privately) were saying that a second CPU either wouldn't help to improve the situation, or that the performance gain was unclear. We had an immediate problem to solve and the management wasn't comfortable in throwing money at new and expensive hardware that, in the end, might or might not solve the issue. The performance gain with fastforwarding enabled was almost 20% and that's what we've been running since. Note, however, that I am also using virtual interface disc0 and that fastforwarding doesn't work with that interface (at least prior to and including FreeBSD 6.0-release-p4). Gleb Smirnoff sent me a one-line patch that fixed the problem. Hope this helps. Regards, -- Marcos