From owner-freebsd-current@FreeBSD.ORG Fri Nov 16 00:43:34 2007 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 41B7316A41B for ; Fri, 16 Nov 2007 00:43:34 +0000 (UTC) (envelope-from andre@freebsd.org) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) by mx1.freebsd.org (Postfix) with ESMTP id 8AE7213C457 for ; Fri, 16 Nov 2007 00:43:25 +0000 (UTC) (envelope-from andre@freebsd.org) Received: (qmail 67423 invoked from network); 16 Nov 2007 00:17:18 -0000 Received: from c00l3r.networx.ch (HELO [127.0.0.1]) ([62.48.2.2]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 16 Nov 2007 00:17:18 -0000 Message-ID: <473CE7A2.6080804@freebsd.org> Date: Fri, 16 Nov 2007 01:43:14 +0100 From: Andre Oppermann User-Agent: Thunderbird 1.5.0.13 (Windows/20070809) MIME-Version: 1.0 To: Jack Vogel References: <200711151729.lAFHTbiq024351@ambrisko.com> <473CE2B9.8010201@freebsd.org> <2a41acea0711151625q5a994cf3ja0d5f14b0670ca3@mail.gmail.com> In-Reply-To: <2a41acea0711151625q5a994cf3ja0d5f14b0670ca3@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org, Scott Ullrich , freebsd-current@freebsd.org, Julian Elischer , "Wilkinson, Alex" Subject: Re: I/OAT ... Coming Soon ? X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Nov 2007 00:43:34 -0000 Jack Vogel wrote: > On Nov 15, 2007 4:22 PM, Andre Oppermann wrote: >> Scott Ullrich wrote: >>> On 11/15/07, Doug Ambrisko wrote: >>>> Hmm, I forgot about the 2970 which are AMD based. Can you check the >>>> BIOS to see if there is an option to turn it on? I think this is an >>>> Intel feature. AMD might have something close? We have one 2970 >>>> that we've played with a little but not much. I can't say for sure >>>> if it has it. >>> Right you are. As of BIOS 1.2.2 I do not see a I/OAT option. Guess >>> I will need to pick up a different server as we are interested in what >>> kind of packet forwarding rate increase that this feature might bring >>> on a heavily loaded firewall. >> Not much. Unless your firewall is in usermode. Otherwise the data >> stays in the kernel and I/OAT is of not help as no copying happens. >> Your CPU is probably spending half of its clock cycles waiting on >> cache misses from newly arrived packets. Some Intel chipset integrated >> gige ports have a cache prefetch feature (duno whether our driver >> supports it) that would help quite a bit for your case. > > What might help this is multiqueue support on the receive AND send, > and stack support for the same. Not sure what the stack changes > would look like, but I know there's interest in this sort of thing, so > naturally I'd be into it :) Dunno if multiqueue is a big win here. You have to make sure that packet order is maintained which kinda implies a single queue. Of course one could spread some load with fixed hashes to keep flows together. The reason a small 1GHz embedded MIPS CPU with integrated GigE ports can do more than 1Mpps is the cache prefetching feature. The thingies generally move the first 128bytes of every packet received into the L2 cache. This is enough for the headers and to perform a lookup on the routing table or the TCP/UDP control block table without much delay. The normal PC architecture is quite broken in that regard as everything that comes in through DMA is in cold main memory. Once the CPU wants to look at it, it has to wait an insane amount of time. That times the number of packets. In pure forwarding applications (routing) it wastes half of all CPU cycles with waiting on main memory. -- Andre