From owner-freebsd-net@FreeBSD.ORG Fri Mar 3 01:31:08 2006 Return-Path: X-Original-To: freebsd-net@freebsd.org Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 08D2816A420 for ; Fri, 3 Mar 2006 01:31:08 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id ACC6643D45 for ; Fri, 3 Mar 2006 01:31:07 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 9640F46B29; Thu, 2 Mar 2006 20:30:47 -0500 (EST) Date: Fri, 3 Mar 2006 01:35:41 +0000 (GMT) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Iasen Kostov In-Reply-To: <1141306007.70735.16.camel@DraGoN.OTEL.net> Message-ID: <20060303012652.A54458@fledge.watson.org> References: <1141306007.70735.16.camel@DraGoN.OTEL.net> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: FreeBSD Net Subject: Re: SMP NAT X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Mar 2006 01:31:08 -0000 On Thu, 2 Mar 2006, Iasen Kostov wrote: > I'm now using a MP system (dual opteron) to do NAT for about 1500 clients at > once at speed above 200Mbit/sec full-duplex (e.g about 400Mbit/sec) and I'm > using PF to do the NAT. Bad thing is that the second CPU is idle. As I can > see from top - about 50% of the cpu is used by irq handler for the ethernet > adapter (irq27: bge0 bge1 - I'm using only bge0 to route via VLANs) and > about 30% by the network interrupt handler. I guess that the swi1:net is > handling the NAT (via PF) and if swi1 and irq27 are in different handlers > why they don't get executed on different CPUs (second CPU is 98% idle and > top show that both handlers run on same CPU). Aren't both handlers in > different kernel threads ? If they are not - is it possible to be in > different threads on different CPUs ? In general, yes -- I frequently look at top -S and see the ithreads running on different CPUs from each other. As you surmise, the hardware ithread is handling the hardware interrupts up through link layer processing, and then the netisr is doing the IP layer processing including NAT. On recent FreeBSD, generally if a second CPU is idle, we will generally wake up the netisr on another idle CPU. However, that's a property of the scheduler, and the details of how that happens vary a bit by FreeBSD version. You don't include information on which FreeBSD version you're using. It's also worth keeping in mind that if you have idle CPU time on your first CPU even with both threads going as fast as the hardware is driving, it's not necessarily "better" to be running the two tasks on different CPUs, for reasons of memory caching -- i.e., the second CPU won't have to cache miss and read the packets in from memory a second time when it begins processing the mbufs previously brought into memory on the first CPU by the interrupt handler. So a few questions: (1) What version of FreeBSD are you running? (2) Is your network stack running MPSAFE? "sysctl debug.mpsafenet" will return either 0 or 1. If you're running with certain network features, such as the KAME IPSEC stack, you may be running with single processor network stack. (3) Are you using SCHED_4BSD (or rather, have you changed to SCHED_ULE)? (4) Are you running with PREEMPTION compiled into the kernel? When a thread, such as the netisr, is preempted by a hardware ithread, it won't necessarily be bounced to the other CPU immediately. Robert N M Watson