From owner-freebsd-net@FreeBSD.ORG Fri Jul 4 11:16:16 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 901521065676 for ; Fri, 4 Jul 2008 11:16:16 +0000 (UTC) (envelope-from if@xip.at) Received: from chile.gbit.at (ns1.xip.at [193.239.188.99]) by mx1.freebsd.org (Postfix) with ESMTP id DE5C28FC18 for ; Fri, 4 Jul 2008 11:16:15 +0000 (UTC) (envelope-from if@xip.at) Received: (qmail 1868 invoked from network); 4 Jul 2008 13:16:13 +0200 Received: from unknown (HELO filebunker.xip.at) (86.59.10.180) by chile.gbit.at with (DHE-RSA-AES256-SHA encrypted) SMTP; 4 Jul 2008 13:16:13 +0200 Date: Fri, 4 Jul 2008 13:16:13 +0200 (CEST) From: Ingo Flaschberger To: Paul In-Reply-To: <486DF1A3.9000409@gtcomm.net> Message-ID: References: <4867420D.7090406@gtcomm.net> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> <486A9A0E.6060308@elischer.org> <486B41D5.3060609@gtcomm.net> <486B4F11.6040906@gtcomm.net> <486BC7F5.5070604@gtcomm.net> <20080703160540.W6369@delplex.bde.org> <486C7F93.7010308@gtcomm.net> <20080703195521.O6973@delplex.bde.org> <486D35A0.4000302@gtcomm.net> <486DF1A3.9000409@gtcomm.net> User-Agent: Alpine 1.10 (LFD 962 2008-03-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: FreeBSD Net Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Jul 2008 11:16:16 -0000 Dear Paul, >>> what could cause this? >> >> *) kern.polling.idle_poll enabled? >> *) kern.polling.user_frac ? >> *) kern.polling.reg_frac ? >> *) kern.polling.burst_max ? >> *) kern.polling.each_burst ? > > I tried tons of different values for these and nothing made any significant > difference. > Idle polling makes a difference, allows more pps, but still errors. > Without idle polling it seems PPS is limited to HZ * descriptors, or 1000 * > 256 or 512 > but 1000 * 1024 is the same as 512.. 4000 * 256 or 2000 * 512 works but > starts erroring 600kpps (SMP right now but it happens in UP too) I have patched src/sys/kern/kern_poll.c to support higher burst_max values: #define MAX_POLL_BURST_MAX 10000 When setting kern.polling.burst_max to higher values, the server reach a point, where cpu-usage goes up without load, so try to keep below this values. I also have set the network card to 4096 rx-"ram", to have more room for late polls. Kind regards, Ingo Flaschberger