From owner-freebsd-current@FreeBSD.ORG Sun Jul 12 22:58:06 2009 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0F101106564A for ; Sun, 12 Jul 2009 22:58:06 +0000 (UTC) (envelope-from cmb@pfsense.org) Received: from mail.pfsense.org (mail.pfsense.org [69.64.6.29]) by mx1.freebsd.org (Postfix) with ESMTP id DA7C08FC08 for ; Sun, 12 Jul 2009 22:58:05 +0000 (UTC) (envelope-from cmb@pfsense.org) Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.pfsense.org (Postfix) with ESMTP id 7BD7B28231 for ; Sun, 12 Jul 2009 17:42:01 -0500 (EST) X-Virus-Scanned: amavisd-new at mail.pfsense.org Received: from mail.pfsense.org ([127.0.0.1]) by localhost (localhost [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id da1vQN8QeZGg for ; Sun, 12 Jul 2009 17:42:00 -0500 (EST) Received: from [10.0.64.15] (96-28-38-25.dhcp.insightbb.com [96.28.38.25]) by mail.pfsense.org (Postfix) with ESMTP id E04C725F50 for ; Sun, 12 Jul 2009 17:41:59 -0500 (EST) Message-ID: <4A5A66B7.6060206@pfsense.org> Date: Sun, 12 Jul 2009 18:41:59 -0400 From: Chris Buechler User-Agent: Thunderbird 2.0.0.22 (Windows/20090605) MIME-Version: 1.0 To: freebsd-current@freebsd.org References: <3c1674c90907120009o330da19ds68c45d0dab6ef81f@mail.gmail.com> In-Reply-To: <3c1674c90907120009o330da19ds68c45d0dab6ef81f@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Mon, 13 Jul 2009 00:05:47 +0000 Subject: Re: Flowtables -- any tuning hints? X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Jul 2009 22:58:06 -0000 Kip Macy wrote: > On Sat, Jul 11, 2009 at 10:24 AM, Scott Ullrich wrote: > >> Hello Freebsd-current@ folks, >> >> I see with the commit "svn commit: r191259 - head/sys/netinet" >> flowtables have been added.. Cool! >> >> Does anyone have any tuning hints for this addition -- specifically >> how much memory does the hash table consume? Or better yet does any >> documentation exist for this newly added feature? >> >> Looking for an easy way to calculate max flows for the amount of >> memory installed in a FreeBSD machine. >> > > > You want to avoid hash collisions. So, generally speaking you want the > hash table to be sized 2x larger than the number of unique connection > destinations. You want the maximum number of flows to be as large as > the maximum number of unique destinations x number of cores. When you > get to the case of hundreds of thousands of unique destinations as in > the case of a small ISP doing IP forwarding, you're probably better > off disabling the flowtable. For most other workloads its likely to be > a clear win. Running a process on an 8-core system with 8 threads each > calling sendto(...) with 10 bytes I can push 3.5 - 4Mpps (with cxgb - > you won't get this with most cards) with the flowtable enabled. With > the flowtable disabled lock contention causes performance to degrade > to 330kpps with the aforementioned workload. > This is interesting functionality, but I think we need to look at it a bit closer for our use case. Is there any benefit in running this in a firewall scenario? That's primarily what Scott and I (pfsense) are interested in. In our world, if you're pushing 50Kpps+, you're almost certainly falling into the "small ISP doing IP forwarding" scenario with hundreds of thousands of unique destinations. Where we usually see these kinds of loads are small ISPs, web hosting companies, or universities (which are functionally not much diff from a small ISP), all of which I'm familiar with falling into the "better off disabling" category. I also suspect pf's locking negates some or all of the benefits here. I suspect it's not applicable to the specific workload our users normally have, where you're almost entirely doing IP forwarding, and initiating very little if any traffic. bz@ said it's not something you want on a router. Is that a fair assessment? best, Chris