From owner-freebsd-ports@FreeBSD.ORG Fri Feb 21 19:13:03 2014 Return-Path: Delivered-To: freebsd-ports@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7B3582AB for ; Fri, 21 Feb 2014 19:13:03 +0000 (UTC) Received: from shrewd.pub.knigma.org (shrewd.ipv6.pub.knigma.org [IPv6:2001:8b0:b0:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id F2A67125D for ; Fri, 21 Feb 2014 19:13:02 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) (authenticated bits=0) by shrewd.pub.knigma.org (8.14.7/8.14.7) with ESMTP id s1LJD02i039242 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO) for ; Fri, 21 Feb 2014 19:13:00 GMT (envelope-from lists@knigma.org) Message-ID: <5307A53B.1040406@knigma.org> Date: Fri, 21 Feb 2014 19:12:59 +0000 From: Mark Knight User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.3.0 MIME-Version: 1.0 To: freebsd-ports@freebsd.org Subject: dummynet queues hanging Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.4.3 (shrewd.pub.knigma.org [127.0.0.1]); Fri, 21 Feb 2014 19:13:00 +0000 (GMT) X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list Reply-To: markk@knigma.org List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Feb 2014 19:13:03 -0000 I'm trying to use Dummynet to throttle bandwidth at "peak" times. However my configuration seems to be behaving very oddly. Before I go too much further debugging this, can anyone see anything obviously wrong with my configuration? I have tried two similar configurations. The first works very reliably but doesn't make any attempt to distribute the available bandwidth between different flows: queue 40 config pipe 40 queue 5 mask src-ip 0xffffffff src-port 0xffff (this line is essentially redundant) pipe 40 config bw 600Kbit/s type QFQ queue 5 mask dst-ip 0xffffffff add 525 pipe 40 ip from any to any via em0 out In this second configuration I pass traffic through a queue first, rather than directly through the pipe. My end goal is to limit bandwidth to each host but to also try and distribute the available bandwidth fairly between the applications on each host. queue 40 config pipe 40 queue 5 mask src-ip 0xffffffff src-port 0xffff pipe 40 config bw 600Kbit/s type QFQ queue 5 mask dst-ip 0xffffffff add 525 queue 40 ip from any to any via em0 out When I switch to using the queue I start to see very odd behaviour when traffic levels increase. Typically after just a few minutes the scheduler or queues seem to get "stuck": mkn@shrewd$ sudo ipfw sched list 00040: 600.000 Kbit/s 0 ms burst 0 sched 40 type QFQ flags 0x1 256 buckets 1 active mask: 0x00 0x00000000/0x0000 -> 0xffffffff/0x0000 Children flowsets: 40 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 181 ip 0.0.0.0/0 217.169.23.231/0 11701 3844619 1123 72289 2112 If I look at "ipfw queue list", I see that many the flows have full buffers and are dropping packets like crazy - essentially the queues seem to have stopped draining. I also get a few kernel message when I list the queues: Feb 21 18:38:00 shrewd kernel: [29168] copy_obj_q ERROR type 5 queue -1 have 32 need 96 I saw the same behaviour with FreeBSD 9.2 and now FreeBSD 10.0. I have another queue running with a slightly different configuration on another interface and it's absolutely rock solid. The only obvious difference is that I do have a inbound "ipfw fwd" rule running on on em0 but that could be a red herring: 00476 fwd 81.2.102.154,8090 tcp from any to any dst-port 80 via em0 in not tagged 2 // lan Thanks in advance for any insight. PS: The problem offers using either QFQ or WF2Q+. -- Mark Knight Mobile: +44 7753 250584. http://www.knigma.org/ Email: markk@knigma.org. Skype: knigma