From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 08:53:23 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2518B1065676 for ; Mon, 5 Oct 2009 08:53:23 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx30.mail.ru (mx30.mail.ru [94.100.176.44]) by mx1.freebsd.org (Postfix) with ESMTP id B27DC8FC19 for ; Mon, 5 Oct 2009 08:53:22 +0000 (UTC) Received: from [217.25.27.27] (port=47720 helo=[217.25.27.27]) by mx30.mail.ru with asmtp id 1MujJo-0008p2-00; Mon, 05 Oct 2009 12:53:21 +0400 Message-ID: <4AC9B400.9020400@mail.ru> Date: Mon, 05 Oct 2009 13:53:20 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Luigi Rizzo References: <4AC8A76B.3050502@mail.ru> <20091005025521.GA52702@svzserv.kemerovo.su> <20091005061025.GB55845@onelab2.iet.unipi.it> In-Reply-To: <20091005061025.GB55845@onelab2.iet.unipi.it> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 08:53:23 -0000 Luigi Rizzo wrote: > On Mon, Oct 05, 2009 at 10:55:21AM +0800, Eugene Grosbein wrote: >> On Sun, Oct 04, 2009 at 06:47:23PM +0500, rihad wrote: >> >>> sysctls: >>> kern.ipc.nmbclusters=50000 >>> net.inet.ip.dummynet.io_fast=1 >> I guess you should also try to increase pipes length: >> >> net.inet.ip.dummynet.hash_size=65536 >> net.inet.ip.dummynet.pipe_slot_limit=1000 > > in fact, i forgot to ask, we'd need to know the output of > "ipfw pipe show" just to get the idea if there is any > known reason for the drop (e.g. queues too short, etc.) > The pipes are fine, each normally having 100-120 concurrent consumers (i.e. active users). (The max number of consumers allowed per pipe is hash_size*max_chain_len, normally 64*16==1024.) kern.ipc.nmbclusters=111111 And these are by default: #net.inet.ip.dummynet.hash_size=64 #net.inet.ip.dummynet.max_chain_len=16 Yesterday I set up a cronjob logging the number of drops in the past 15 minutes: */15 * * * * (echo -n "$(date) "; netstat -z -s 2>/dev/null | fgrep -w "output packets dropped") >> /tmp/bufs.log And here's bufs.log: Sun Oct 4 21:45:00 AZST 2009 418869 output packets dropped due to no bufs, etc. Sun Oct 4 22:00:00 AZST 2009 851693 output packets dropped due to no bufs, etc. Sun Oct 4 22:15:01 AZST 2009 932885 output packets dropped due to no bufs, etc. Sun Oct 4 22:30:00 AZST 2009 890522 output packets dropped due to no bufs, etc. Sun Oct 4 22:45:00 AZST 2009 1065931 output packets dropped due to no bufs, etc. Sun Oct 4 23:00:00 AZST 2009 937863 output packets dropped due to no bufs, etc. Sun Oct 4 23:15:01 AZST 2009 1018822 output packets dropped due to no bufs, etc. Sun Oct 4 23:30:00 AZST 2009 981922 output packets dropped due to no bufs, etc. Sun Oct 4 23:45:00 AZST 2009 1015124 output packets dropped due to no bufs, etc. Mon Oct 5 00:00:01 AZST 2009 1123926 output packets dropped due to no bufs, etc. Mon Oct 5 00:15:01 AZST 2009 948161 output packets dropped due to no bufs, etc. Mon Oct 5 00:30:00 AZST 2009 937277 output packets dropped due to no bufs, etc. Mon Oct 5 00:45:00 AZST 2009 875218 output packets dropped due to no bufs, etc. Mon Oct 5 01:00:00 AZST 2009 803527 output packets dropped due to no bufs, etc. Mon Oct 5 01:15:00 AZST 2009 728639 output packets dropped due to no bufs, etc. Mon Oct 5 01:30:00 AZST 2009 626154 output packets dropped due to no bufs, etc. Mon Oct 5 01:45:00 AZST 2009 519441 output packets dropped due to no bufs, etc. Mon Oct 5 02:00:00 AZST 2009 371098 output packets dropped due to no bufs, etc. Mon Oct 5 02:15:00 AZST 2009 681243 output packets dropped due to no bufs, etc. Mon Oct 5 02:30:00 AZST 2009 562909 output packets dropped due to no bufs, etc. Mon Oct 5 02:45:00 AZST 2009 426734 output packets dropped due to no bufs, etc. Mon Oct 5 03:00:00 AZST 2009 344619 output packets dropped due to no bufs, etc. Mon Oct 5 03:15:00 AZST 2009 90006 output packets dropped due to no bufs, etc. Mon Oct 5 03:30:00 AZST 2009 17064 output packets dropped due to no bufs, etc. Mon Oct 5 03:45:00 AZST 2009 3851 output packets dropped due to no bufs, etc. Mon Oct 5 04:00:00 AZST 2009 1323 output packets dropped due to no bufs, etc. Mon Oct 5 04:15:00 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 04:30:00 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 04:45:00 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 05:00:00 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 05:15:00 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 05:30:01 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 05:45:01 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 06:00:00 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 06:15:01 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 06:30:00 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 06:45:00 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 07:00:00 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 07:15:00 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 07:30:00 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 07:45:00 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 08:00:00 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 08:15:00 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 08:30:00 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 08:45:00 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 09:00:00 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 09:15:00 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 09:30:00 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 09:45:00 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 10:00:00 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 10:15:00 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 10:30:00 AZST 2009 177 output packets dropped due to no bufs, etc. Mon Oct 5 10:45:00 AZST 2009 1701 output packets dropped due to no bufs, etc. Mon Oct 5 11:00:01 AZST 2009 19933 output packets dropped due to no bufs, etc. Mon Oct 5 11:15:00 AZST 2009 30003 output packets dropped due to no bufs, etc. Mon Oct 5 11:30:00 AZST 2009 56712 output packets dropped due to no bufs, etc. Mon Oct 5 11:45:00 AZST 2009 78721 output packets dropped due to no bufs, etc. Mon Oct 5 12:00:01 AZST 2009 112518 output packets dropped due to no bufs, etc. Mon Oct 5 12:15:00 AZST 2009 7229 output packets dropped due to no bufs, etc. Mon Oct 5 12:30:01 AZST 2009 24965 output packets dropped due to no bufs, etc. Mon Oct 5 12:45:00 AZST 2009 75900 output packets dropped due to no bufs, etc. Mon Oct 5 13:00:00 AZST 2009 45002 output packets dropped due to no bufs, etc. Mon Oct 5 13:15:00 AZST 2009 67161 output packets dropped due to no bufs, etc. Mon Oct 5 13:30:00 AZST 2009 112591 output packets dropped due to no bufs, etc. As you can see the drops gradually went away completely at about 4:00 a.m., and started coming up at about 10:30 a.m., although at a lower rate, probably thanks to me bumping "ipfw ... queue NNN" up to 5000 at 10a.m. this morning. The traffic flow between 4a.m. and 10:30a.m., the "quiet" times, is about 200-330 mbit/s 5 minute average, without a single drop. But after that, in come the drops, no matter how high I set the queue. Should I try 10000 slots? 20000? I'm really sure there are plenty of heavy downloaders between 4a.m. and 10a.m., but still without a single drop before approx. 330 mbit/s! Strange, isn't it? This makes me believe I'm hitting some other global memory limit, rather than per-user limit, at around 340-350 mbit/s, causing drops. top and netstat -m are OK right now, though: Mem: 1870M Active, 1220M Inact, 481M Wired, 201M Cache, 214M Buf, 164M Free 15595/5385/20980/111112 mbuf clusters in use (current/cache/total/max)