From owner-freebsd-net@FreeBSD.ORG Thu Oct 8 16:45:17 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6A88A1065672 for ; Thu, 8 Oct 2009 16:45:17 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outA.internet-mail-service.net (outa.internet-mail-service.net [216.240.47.224]) by mx1.freebsd.org (Postfix) with ESMTP id 0C8E88FC14 for ; Thu, 8 Oct 2009 16:45:16 +0000 (UTC) Received: from idiom.com (mx0.idiom.com [216.240.32.160]) by out.internet-mail-service.net (Postfix) with ESMTP id A0BB0ACB80; Thu, 8 Oct 2009 09:45:16 -0700 (PDT) X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e Received: from julian-mac.elischer.org (home.elischer.org [216.240.48.38]) by idiom.com (Postfix) with ESMTP id 9A5182D60BE; Thu, 8 Oct 2009 09:45:14 -0700 (PDT) Message-ID: <4ACE171D.9080707@elischer.org> Date: Thu, 08 Oct 2009 09:45:17 -0700 From: Julian Elischer User-Agent: Thunderbird 2.0.0.23 (Macintosh/20090812) MIME-Version: 1.0 To: rihad References: <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <4ACA2CC6.70201@elischer.org> <4ACAFF2A.1000206@mail.ru> <4ACB0C22.4000008@mail.ru> <20091006100726.GA26426@svzserv.kemerovo.su> <4ACB42D2.2070909@mail.ru> <20091006142152.GA42350@svzserv.kemerovo.su> <4ACB6223.1000709@mail.ru> <20091006161240.GA49940@svzserv.kemerovo.su> <4ACC5563.602@mail.ru> <4ACC56A6.1030808@mail.ru> <4ACC5DEC.1010006@mail.ru> <4ACC65A0.7030900@mail.ru> <4ACC8CC8.8050403@mail.ru> <4ACDE1FB.6020602@mail.ru> In-Reply-To: <4ACDE1FB.6020602@mail.ru> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Eugene Grosbein , Robert Watson , Luigi Rizzo , freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Oct 2009 16:45:17 -0000 rihad wrote: > Robert Watson wrote: >> I would suggest making just the HZ -> 4000 change for now and see how >> it goes. >> > 2018 users online, 73 drops have just occurred. > p.s.: already 123 drops. > It will only get worse after some time. > > Traffic load: 440-450 mbps. > > top -HS: > last pid: 68314; load averages: 1.35, 1.22, 1.25 > up 0+05:13:28 17:53:49 > 145 processes: 11 running, 118 sleeping, 16 waiting > CPU: 1.4% user, 0.0% nice, 2.8% system, 10.3% interrupt, 85.5% idle > Mem: 1337M Active, 1683M Inact, 355M Wired, 40K Cache, 214M Buf, 560M Free > Swap: 2048M Total, 2048M Free > > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 14 root 171 ki31 0K 16K CPU4 4 257:35 99.41% idle: cpu4 > 12 root 171 ki31 0K 16K RUN 6 286:39 98.14% idle: cpu6 > 18 root 171 ki31 0K 16K RUN 0 225:16 92.72% idle: cpu0 > 15 root 171 ki31 0K 16K RUN 3 255:35 90.04% idle: cpu3 > 16 root 171 ki31 0K 16K CPU2 2 272:04 87.40% idle: cpu2 > 13 root 171 ki31 0K 16K CPU5 5 260:52 81.69% idle: cpu5 > 17 root 171 ki31 0K 16K CPU1 1 239:06 75.29% idle: cpu1 > 21 root -44 - 0K 16K CPU7 7 108:49 57.37% swi1: net > 11 root 171 ki31 0K 16K CPU7 7 267:48 49.02% idle: cpu7 > 29 root -68 - 0K 16K WAIT 1 41:45 20.90% irq256: bce0 > 470 root -68 - 0K 16K - 5 27:01 9.18% dummynet > 19 root -32 - 0K 16K WAIT 1 16:13 6.59% swi4: > clock sio > 31 root -68 - 0K 16K WAIT 2 9:35 4.35% irq257: bce1 > >> Robert Watson wrote: >> Suggestions like increasing timer resolution are intended to spread >> out the injection of packets by dummynet to attempt to reduce the >> peaks of burstiness that occur when multiple queues inject packets in >> a burst that exceeds the queue depth supported by combined hardware >> descriptor rings and software transmit queue. > > How to tweak the software transmit queue? > > > P.S.: still only 123 drops (while io_pkt_drop: 0, intr_queue_drops: 0), > but it was a warning. you can not do anything about it if one of the custommers sends a burst of 3000 udp packets at their maximum speed(or maybe some combination of custommers to something which results in an aggreagate burst rate like that. In other words you may always continue to get moments when the pipe releases a bunch of stuff that has a potential to over-run something down stream. Think of it as a dam in a stream... If you have no dam, teh water level goes up and down gradually and by small amounts, but if you have a dam, you can release water in such a way that the stream is flooded higher than it would normally ever get. work out how large a burst of data the pipe will release in 1/4000th of a second and using a small packet size, work out how many packets that is. Then make sure that the driver software input queue is capable of holding that many packets.