From owner-freebsd-net@FreeBSD.ORG Sun Oct 4 14:56:00 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0B6921065670 for ; Sun, 4 Oct 2009 14:56:00 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx76.mail.ru (mx76.mail.ru [94.100.176.91]) by mx1.freebsd.org (Postfix) with ESMTP id BCFEF8FC17 for ; Sun, 4 Oct 2009 14:55:59 +0000 (UTC) Received: from [217.25.27.27] (port=41510 helo=[217.25.27.27]) by mx76.mail.ru with asmtp id 1MuSVC-0002ht-00; Sun, 04 Oct 2009 18:55:58 +0400 Message-ID: <4AC8B77D.6070703@mail.ru> Date: Sun, 04 Oct 2009 19:55:57 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Luigi Rizzo References: <4AC8A76B.3050502@mail.ru> <20091004144909.GA42503@onelab2.iet.unipi.it> In-Reply-To: <20091004144909.GA42503@onelab2.iet.unipi.it> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Oct 2009 14:56:00 -0000 Luigi Rizzo wrote: > On Sun, Oct 04, 2009 at 06:47:23PM +0500, rihad wrote: >> Hi, we have around 500-600 mbit/s traffic flowing through a 7.1R Dell >> PowerEdge w/ 2 GigE bce cards. There are currently around 4 thousand ISP >> users online limited by dummynet pipes of various speeds. According to >> netstat -s output around 500-1000 packets are being dropped every second >> (this accounts for wasting around 7-12 mbit/s worth of traffic according >> to systat -ifstat): > > At those speeds you might be hitting various limits with your > config (e.g. 50k nmbclusters is probably way too small for > 4k users -- means you have an average of 10-15 buffers per user; > the queue size of 350kbytes = 2.6Mbits means 2.6 seconds of buffering, > which is quite high besides the fact that in order to scale to 4k users > you would need over 1GB of kernel memory just for the buffers). top output: Mem: 2037M Active, 1248M Inact, 450M Wired, 184M Cache, 214M Buf, 17M Free I guess we're quite far from reaching 1GB of kernel memory.