From owner-freebsd-net@FreeBSD.ORG Mon Jul 7 11:13:22 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 884101065672 for ; Mon, 7 Jul 2008 11:13:22 +0000 (UTC) (envelope-from andre@freebsd.org) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) by mx1.freebsd.org (Postfix) with ESMTP id B170C8FC1F for ; Mon, 7 Jul 2008 11:13:20 +0000 (UTC) (envelope-from andre@freebsd.org) Received: (qmail 31286 invoked from network); 7 Jul 2008 10:03:48 -0000 Received: from localhost (HELO [127.0.0.1]) ([127.0.0.1]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 7 Jul 2008 10:03:48 -0000 Message-ID: <4871FA4F.40206@freebsd.org> Date: Mon, 07 Jul 2008 13:13:19 +0200 From: Andre Oppermann User-Agent: Thunderbird 1.5.0.14 (Windows/20071210) MIME-Version: 1.0 To: Robert Watson References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> <486A9A0E.6060308@elischer.org> <486B41D5.3060609@gtcomm.net> <4871E85C.8090907@freebsd.org> <20080707114538.K63144@fledge.watson.org> In-Reply-To: <20080707114538.K63144@fledge.watson.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Net , Paul Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Jul 2008 11:13:22 -0000 Robert Watson wrote: > > On Mon, 7 Jul 2008, Andre Oppermann wrote: > >> Distributing the interrupts and taskqueues among the available CPUs >> gives concurrent forwarding with bi- or multi-directional traffic. All >> incoming traffic from any particular interface is still serialized >> though. > > ... although not on multiple input queue-enabled hardware and drivers. > While I've really only focused on local traffic performance with my > 10gbps Chelsio setup, it should be possible to do packet forwarding from > multiple input queues using that hardware and driver today. > > I'll update the netisr2 patches, which allow work to be pushed to > multiple CPUs from a single input queue. However, these necessarily > take a cache miss or two on packet header data in order to break out the > packets from the input queue into flows that can be processed > independently without ordering constraints, so if those cache misses on > header data are a big part of the performance of a configuration, load > balancing in this manner may not help. What would be neat is if the > cards without multiple input queues could still tag receive descriptors > with a flow identifier generated from the IP/TCP/etc layers that could > be used for work placement. The cache miss is really the elephant in the room. If the network card supports multiple RX rings with separate interrupts and a stable hash based (that includes IP+Port src+dst) distribution they can be bound to different CPUs. It is very important to maintain the packet order for flows that go through the router. Otherwise TCP and VoIP will suffer. -- Andre