From owner-freebsd-net@FreeBSD.ORG Sun Jun 29 09:07:32 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 28B511065675 for ; Sun, 29 Jun 2008 09:07:32 +0000 (UTC) (envelope-from andrew@modulus.org) Received: from email.octopus.com.au (host-122-100-2-232.octopus.com.au [122.100.2.232]) by mx1.freebsd.org (Postfix) with ESMTP id DEB238FC13 for ; Sun, 29 Jun 2008 09:07:31 +0000 (UTC) (envelope-from andrew@modulus.org) Received: by email.octopus.com.au (Postfix, from userid 1002) id 251A717214; Sun, 29 Jun 2008 19:07:29 +1000 (EST) X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on email.octopus.com.au X-Spam-Level: X-Spam-Status: No, score=-1.4 required=10.0 tests=ALL_TRUSTED autolearn=failed version=3.2.3 Received: from [10.1.50.60] (138.21.96.58.exetel.com.au [58.96.21.138]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: admin@email.octopus.com.au) by email.octopus.com.au (Postfix) with ESMTP id E4FD2171BF; Sun, 29 Jun 2008 19:07:24 +1000 (EST) Message-ID: <48675095.7020008@modulus.org> Date: Sun, 29 Jun 2008 19:06:29 +1000 From: Andrew Snow User-Agent: Thunderbird 2.0.0.14 (X11/20080523) MIME-Version: 1.0 To: Paul References: <4867420D.7090406@gtcomm.net> In-Reply-To: <4867420D.7090406@gtcomm.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Net Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Jun 2008 09:07:32 -0000 The "em" driver currently only has a single worker/queue so will only use one CPU to process packets. However I remember reading that multi-threaded version of the driver is being worked on and is "coming soon", but there is no known ETA yet. I see you mentioned that you played with the receive descriptors and set that to 4096, which is good if your chipset supports it. But I can't see that you mentioned playing around with hw.em.rx_int_delay or tx_int_delay - have you tried this? By default rx_int_delay is disabled but it has the capability to lower CPU consumption if you enable it and it works on your chipset. By my reckoning, if you are aiming for a million pps, thats microsecond per packet, so you can afford to increase the delay quite alot, try 50 or 100 or even more? - Andrew