From owner-freebsd-net@FreeBSD.ORG Fri Feb 19 18:48:28 2010 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7135E106566C for ; Fri, 19 Feb 2010 18:48:28 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx39.mail.ru (mx39.mail.ru [94.100.176.53]) by mx1.freebsd.org (Postfix) with ESMTP id 2BC6B8FC15 for ; Fri, 19 Feb 2010 18:48:28 +0000 (UTC) Received: from [217.25.27.27] (port=60340 helo=[217.25.27.27]) by mx39.mail.ru with asmtp id 1NiXtq-000KeX-00 for freebsd-net@freebsd.org; Fri, 19 Feb 2010 21:48:26 +0300 Message-ID: <4B7EDD00.5080308@mail.ru> Date: Fri, 19 Feb 2010 22:48:32 +0400 From: rihad User-Agent: Thunderbird 2.0.0.23 (X11/20090817) MIME-Version: 1.0 To: freebsd-net@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Subject: Slow speeds experienced with Dummynet X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: rihad List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Feb 2010 18:48:28 -0000 Hi, all, Recalling my old posting "dummynet dropping too many packets" dated October 4, 2009, the problem isn't over just yet. This time, there are no interface i/o drops (just a reminder: we have 2 bce(4) GigE cards connected to a Cisco router, one for input, and one for output. The box itself does some traffic accounting and enforces speed limits w/ ipfw/dummynet. There are normally around 5-6k users online). Traffic shaping is accomplished by this ipfw rule: pipe tablearg ip from any to table(0) out where table(0) contains those 5-6k IP addresses. The pipes themselves are GRED (or taildrop, it doesn't matter): ipfw pipe 512 config bw 512kbit/s mask dst-ip 0xffffffff gred 0.002/900/1000/0.1 queue 1000 Taking this template the speeds range from 512 to tens of mbps. With the setup as above very many users complain about very slow downloads, slow browsing. systat -ifstat, refreshed every 5 seconds, does reveal large differences between subsequent displays: if around 800-850 mbps is what's to be expected, it doesn't stay within those limits, also jumping to as low as 620+, and to somewhere in the 700's, Now imagine this: once I turn dummynet off (by short-circuiting a "allow ip from any to any" before the "pipe tablearg" rule) all user complaints stop, with traffic load staying stably at around 930-950 mbps. Does this have anything to do with "dummynet bursts"? How can I beat that? If I keep the pipe queue size at 2000 slots, the net.inet.ip.dummynet.io_pkt_drops sysctl stops increasing, once I start tweaking the value to as low as 100 slots, the value starts raising constantly at about 300-500 pps. I had hoped that smaller queue sizes would battle the negative effects of dummynet burstiness, it did so, I guess, but not in a very decisive manner. FreeBSD 7.1-RELEASE-p10 kern.hz=4000 kern.ipc.nmbclusters=111111 net.inet.ip.fastforwarding=1 net.inet.ip.dummynet.io_fast=1 net.isr.direct=0 net.inet.ip.intr_queue_maxlen=5000 net.inet.ip.dummynet.hash_size=512 net.inet.ip.dummynet.max_chain_len=16 net.inet.ip.intr_queue_drops: 0 systat -ip shows zero output drops at times of trouble. netstat -s's "output packets dropped due to no bufs, etc." is also fine. netstat -m shows nothing suspicious. p.s: Two "bloody expensive" Intel 10 GigE's are on their way to us to replace the Broadcom cards, meanwhile what should I try doing? Thanks for reading.