From owner-svn-src-stable@FreeBSD.ORG Wed Nov 17 18:54:19 2010 Return-Path: Delivered-To: svn-src-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 74D10106564A; Wed, 17 Nov 2010 18:54:19 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail05.syd.optusnet.com.au (mail05.syd.optusnet.com.au [211.29.132.186]) by mx1.freebsd.org (Postfix) with ESMTP id 0C4A38FC17; Wed, 17 Nov 2010 18:54:18 +0000 (UTC) Received: from c122-106-146-145.carlnfd1.nsw.optusnet.com.au (c122-106-146-145.carlnfd1.nsw.optusnet.com.au [122.106.146.145]) by mail05.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id oAHIsEEG004106 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 18 Nov 2010 05:54:15 +1100 Date: Thu, 18 Nov 2010 05:54:14 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Maxim Sobolev In-Reply-To: <4CE3DFB3.4060809@sippysoft.com> Message-ID: <20101118053830.K1202@besplex.bde.org> References: <201011160440.oAG4e3YU039413@svn.freebsd.org> <20101117030118.X1203@besplex.bde.org> <4CE3DFB3.4060809@sippysoft.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: svn-src-stable-7@FreeBSD.org, svn-src-stable@FreeBSD.org, svn-src-all@FreeBSD.org, src-committers@FreeBSD.org, Bruce Evans Subject: Re: svn commit: r215368 - in stable/7/sys: arm/at91 arm/xscale/ixp425 contrib/dev/oltr dev/ae dev/an dev/ar dev/arl dev/ath dev/awi dev/ce dev/cm dev/cnw dev/cp dev/cs dev/ctau dev/cx dev/cxgb dev/ed d... X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Nov 2010 18:54:19 -0000 On Wed, 17 Nov 2010, Maxim Sobolev wrote: > On 11/16/2010 8:12 AM, Bruce Evans wrote: >> This was quite low for yestdeay's uses (starting in about 1995), but today >> it is little missed since only yesterday's low-end hardware uses it. Most >> of today's interfaces are 1Gbps, and for this it is almost essential for >> the hardware to have a ring buffer with > 50 entries, so most of today's >> drivers ignore ifqmaxlen and set the queue length to the almost equally >> bogus value of the ring buffer size (-1). I set it to about 10000 instead >> in bge and em (10000 is too large, but fixes streaming under certain loads >> when hz is small). > > One of those interfaces is if_rl, which is still quite popular these days and > supports speeds up to 1gbps (which I believe triggered this change). But in It is the only one on the list that I used. Maybe it should be handled specially. Just bump up its queue lengths to maybe 128 for 100 Mbps and 512 for 1 Gbps in all cases, or tune this depending on the amount of memory? > general I agree, unfortunately FreeBSD network subsystem is tuned for > yesteday's speeds. We are seeing lot of lookups and other issues under high > PPS. I wish somebody could stand and pick up the task of cleaning it up and > re-tuning eventually for 2010. We could probably even sponsor in part such a > work (anyone). I haven't seen any lockups, but just the maximum pps on fixed hardware decreasing with every increase in the FreeBSD version number (about 30% since FreeBSD-5). My hardware CPU and bus are saturated by low-end em 1 Gbps and medium-end bge 1 Gbps, so bloat in the stack translates into lower pps. I tuned bge a lot to make it fast under the version of FreeBSD-5 that I usually run, but barely touched upper layers. > Apart from interface tuning for Gbps speeds, another area that needs more > work is splitting up memory pool for the IPC from the memory pool for the > other networking. Today's software is highly distributed and rock-solid IPC > is a must for the FreeBSD being a solid server application platform. That's > OK when under the load we drop some packets, but it's not OK when extreme > network activity can bring down communications between application and > database system within the host itself. And that's exactly what can happen in > FreeBSD. Does flow control help here? I think it should prevent most dropped packets, but be actively harmful if it stops the flow when IPC packets are queued behind non-IPC ones. Large queue lengths are also bad for latency. Bruce