From owner-freebsd-current@FreeBSD.ORG Mon Jul 13 09:20:33 2009 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A176B106564A for ; Mon, 13 Jul 2009 09:20:33 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from web63903.mail.re1.yahoo.com (web63903.mail.re1.yahoo.com [69.147.97.118]) by mx1.freebsd.org (Postfix) with SMTP id 5D6118FC19 for ; Mon, 13 Jul 2009 09:20:33 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: (qmail 83988 invoked by uid 60001); 13 Jul 2009 09:20:32 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1247476832; bh=3uHymm/pHSIUH6f1mpVEyzXj70ESUi1f9UjmW3CX6lM=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=EL6aNcQh6IsUrewVzIxt35pFKWde2ML5J9+8ONJw54f1kKeFIhmJTeEB1X+UXxkR2skKeb4ZQMpr9ZhVZgcqZ1OdHTVZ4Puo3Wywn4ka91basqFlyQ4mswOsdoKYBN0b4+UfNqpotK5ZdpgiRx/Eq+4VM3lOnD71fEpo78YVmEc= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=j8wLsYD8m/iaCZ+xZZVTXLtwVrr2fi8i+27xK2wFAn5waOtLw2fciV1tpocl88/NjoZbNEGW2Zzhfz0y4IRjoNYy+DjyWItjryGdU93xAIJDkp5B7+sm1CqBexUhZw8qsSMUJe4Ctgvwu9Y2tuS81HT4f/zOX4ghhLmZiS1FnhM=; Message-ID: <837660.83798.qm@web63903.mail.re1.yahoo.com> X-YMail-OSG: 1OupKBcVM1l0IX.EvSesIrJA2EovoT.hv24BuSK16zxnxZ3zD9CTbrlG6grXuN5TdG0epBzkSu5oPjti7mesntB2JtgjL1IghhdnS0cL30woI8lq15Fq2AYC9d8aULw8VPsKZOqWhUGHdEfDh8lJZsPVmANLyjBG2n_2TyfFEdbNz4x41pxEEt4dyfpQptI9urED838.dU9YvJC03hkRoq_K3ZH4qctxGH6gZ2BC5AbY9ihv4cuWaDbpfdRkn9vovps2f0ZQytP6ZBs1kMUAEQMaySRjIx23bNzM2P392f7gxHpNngbyXLE- Received: from [66.176.162.245] by web63903.mail.re1.yahoo.com via HTTP; Mon, 13 Jul 2009 02:20:32 PDT X-Mailer: YahooMailClassic/5.4.17 YahooMailWebService/0.7.289.15 Date: Mon, 13 Jul 2009 02:20:32 -0700 (PDT) From: Barney Cordoba To: freebsd-current@freebsd.org, Chris Buechler MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Cc: Subject: Re: Flowtables -- any tuning hints? X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Jul 2009 09:20:34 -0000 =0A=0A--- On Sun, 7/12/09, Chris Buechler wrote:=0A=0A> F= rom: Chris Buechler =0A> Subject: Re: Flowtables -- any tu= ning hints?=0A> To: freebsd-current@freebsd.org=0A> Date: Sunday, July 12, = 2009, 6:41 PM=0A> Kip Macy wrote:=0A> > On Sat, Jul 11, 2009 at 10:24 AM, S= cott Ullrich=0A> wrote:=0A> >=A0=A0=A0=0A> >> Hello Fre= ebsd-current@ folks,=0A> >> =0A> >> I see with the commit "svn commit: r191= 259 -=0A> head/sys/netinet"=0A> >> flowtables have been added..=A0 Cool!=0A= > >> =0A> >> Does anyone have any tuning hints for this=0A> addition -- spe= cifically=0A> >> how much memory does the hash table=0A> consume?=A0=A0=A0O= r better yet does any=0A> >> documentation exist for this newly added featu= re?=0A> >> =0A> >> Looking for an easy way to calculate max flows for=0A> t= he amount of=0A> >> memory installed in a FreeBSD machine.=0A> >>=A0 =A0=A0= =A0=0A> > =0A> > =0A> > You want to avoid hash collisions. So, generally=0A= > speaking you want the=0A> > hash table to be sized 2x larger than the num= ber of=0A> unique connection=0A> > destinations.=A0 You want the maximum nu= mber of=0A> flows to be as large as=0A> > the maximum number of unique dest= inations x number of=0A> cores. When you=0A> > get to the case of hundreds = of thousands of unique=0A> destinations as in=0A> > the case of a small ISP= doing IP forwarding, you're=0A> probably better=0A> > off disabling the fl= owtable. For most other workloads=0A> its likely to be=0A> > a clear win. R= unning a process on an 8-core system=0A> with 8 threads each=0A> > calling = sendto(...) with 10 bytes I can push 3.5 -=0A> 4Mpps (with cxgb -=0A> > you= won't get this with most cards) with the flowtable=0A> enabled. With=0A> >= the flowtable disabled lock contention causes=0A> performance to degrade= =0A> > to 330kpps with the aforementioned workload.=0A> >=A0=A0=A0=0A> This= is interesting functionality, but I think we need to=0A> look at it a bit = closer for our use case. Is there any=0A> benefit in running this in a fire= wall scenario? That's=0A> primarily what Scott and I (pfsense) are interest= ed in. In=0A> our world, if you're pushing 50Kpps+, you're almost=0A> certa= inly falling into the "small ISP doing IP forwarding"=0A> scenario with hun= dreds of thousands of unique destinations.=0A> Where we usually see these k= inds of loads are small ISPs,=0A> web hosting companies, or universities (w= hich are=0A> functionally not much diff from a small ISP), all of which=0A>= I'm familiar with falling into the "better off disabling"=0A> category. I = also suspect pf's locking negates some or all of=0A> the benefits here.=0A>= =0A> I suspect it's not applicable to the specific workload our=0A> users = normally have, where you're almost entirely doing IP=0A> forwarding, and in= itiating very little if any traffic. bz@=0A> said it's not something you wa= nt on a router. Is that a fair=0A> assessment? =0A> best,=0A> Chris=0A> =0A= =0AI don't see the need for flowtables or any sort of hashing with=0Amulti = queue boards in a pass-through environment (such as a firewall). =0AThe flo= ws are already hashed by the hardware so doing all of the =0Agobbledygook i= n flowtables seems like a waste of cycles. Just matching=0Ayour receive and= transmit queues in a properly designed driver is all=0Athats necessary for= lock avoidance.=0A=0ABarney=0A=0A=0A