From owner-freebsd-net@FreeBSD.ORG Sun Oct 4 15:04:39 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DF3421065670 for ; Sun, 4 Oct 2009 15:04:39 +0000 (UTC) (envelope-from luigi@onelab2.iet.unipi.it) Received: from onelab2.iet.unipi.it (onelab2.iet.unipi.it [131.114.59.238]) by mx1.freebsd.org (Postfix) with ESMTP id A497F8FC08 for ; Sun, 4 Oct 2009 15:04:39 +0000 (UTC) Received: by onelab2.iet.unipi.it (Postfix, from userid 275) id 955AF730DA; Sun, 4 Oct 2009 17:11:17 +0200 (CEST) Date: Sun, 4 Oct 2009 17:11:17 +0200 From: Luigi Rizzo To: rihad Message-ID: <20091004151117.GA42877@onelab2.iet.unipi.it> References: <4AC8A76B.3050502@mail.ru> <20091004144909.GA42503@onelab2.iet.unipi.it> <4AC8B77D.6070703@mail.ru> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4AC8B77D.6070703@mail.ru> User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Oct 2009 15:04:40 -0000 On Sun, Oct 04, 2009 at 07:55:57PM +0500, rihad wrote: > Luigi Rizzo wrote: > >On Sun, Oct 04, 2009 at 06:47:23PM +0500, rihad wrote: > >>Hi, we have around 500-600 mbit/s traffic flowing through a 7.1R Dell > >>PowerEdge w/ 2 GigE bce cards. There are currently around 4 thousand ISP > >>users online limited by dummynet pipes of various speeds. According to > >>netstat -s output around 500-1000 packets are being dropped every second > >>(this accounts for wasting around 7-12 mbit/s worth of traffic according > >>to systat -ifstat): > > > >At those speeds you might be hitting various limits with your > >config (e.g. 50k nmbclusters is probably way too small for > >4k users -- means you have an average of 10-15 buffers per user; > >the queue size of 350kbytes = 2.6Mbits means 2.6 seconds of buffering, > >which is quite high besides the fact that in order to scale to 4k users > >you would need over 1GB of kernel memory just for the buffers). > > > top output: > Mem: 2037M Active, 1248M Inact, 450M Wired, 184M Cache, 214M Buf, 17M Free > > I guess we're quite far from reaching 1GB of kernel memory. of course, you'd have to configure also 500k nmbclusters (and then probably this would not fit)