From owner-freebsd-questions@FreeBSD.ORG Sat Oct 13 14:00:24 2007 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DCDA416A421 for ; Sat, 13 Oct 2007 14:00:24 +0000 (UTC) (envelope-from fbsd.questions@rachie.is-a-geek.net) Received: from snoogles.rachie.is-a-geek.net (rachie.is-a-geek.net [66.230.99.27]) by mx1.freebsd.org (Postfix) with ESMTP id 62EE913C465 for ; Sat, 13 Oct 2007 14:00:24 +0000 (UTC) (envelope-from fbsd.questions@rachie.is-a-geek.net) Received: from localhost (localhost [127.0.0.1]) by snoogles.rachie.is-a-geek.net (Postfix) with ESMTP id 78BC01CDFC for ; Sat, 13 Oct 2007 06:00:23 -0800 (AKDT) From: Mel To: freebsd-questions@freebsd.org Date: Sat, 13 Oct 2007 16:00:21 +0200 User-Agent: KMail/1.9.7 References: <470FF27C.7050806@unixservers.us> <200710130049.28349.fbsd.questions@rachie.is-a-geek.net> <4710D05A.707@unixservers.us> In-Reply-To: <4710D05A.707@unixservers.us> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200710131600.22040.fbsd.questions@rachie.is-a-geek.net> Subject: Re: PF ALTQ CBQ rules X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Oct 2007 14:00:25 -0000 On Saturday 13 October 2007 16:04:10 Ovi wrote: > Mel wrote: > >On Saturday 13 October 2007 00:17:32 Ovi wrote: > >>Hello guys > >> > >>I have this example from OpenBSD: > >> > >>altq on $br1_if cbq bandwidth 20Mb qlimit 100 tbrsize 1000 queue { std1, > >>customer_1 } queue customer_1 bandwidth 1Mb cbq(red,ecn) { > >> customer_1_bulk, customer_1_ack } queue customer_1_ack priority 7 > >>queue customer_1_bulk priority 0 > >> > >>I want to use CBQ on FreeBSD, with similar rules still I have the > >>following problem: > >>On a 20Mb internet line I have 100 users. I want to limit (cap) > >>bandwidth per user at 1 Mb and to add queues for all 100 users. > >>The problem is that on FreeBSD this rules are not working, instead I > >>must use this: > >> > >>altq on $br1_if cbq bandwidth 20Mb qlimit 100 tbrsize 1000 queue { std1, > >>customer_1 } queue customer_1 bandwidth 1Mb cbq(red,ecn) { > >> customer_1_bulk, customer_1_ack } queue customer_1_ack bandwidth 800Kb > >> priority 7 queue customer_1_bulk bandwidth 128Kb priority 0 > >> > >> > >>This "bandwidth" option does not help me because I must not exceed 1 Mb. > > > >This can't be done with cbq, because in cbq the sum of child queues must > > match the interface bandwidth or less. Use hfsc for this. > >Also, you reserve 80% for ack and 20% for bulk, you might wanna reverse > > that. > > > >>So my question is: how I do bandwidthupper limit with CBQ per user, like > >>no more than 1 Mb, and add rules for 100 users? > > > >See above. > > > >A good resource on HFSC: > >http://www.probsd.net/pf/index.php/HFSC > > > >It would look something like this: > ># Use interface bandwidth, so your interface doesn't get limited > >altq on $br1_if bandwidth 100Mb hfsc(upperlimit 100Mb) queue { \ > > NO_CUSTOMER, \ > > CUSTOMERS > >} > ># Any traffic not assigned to a customer comes on the NO_CUSTOMER queue. > ># Backlogged traffic consumes a maximum of 80Mbit > ># There's always 55Mbit available. > ># Realtime values may not exceed 75% of root queue > >queue NO_CUSTOMER bandwidth 80Mb hfsc(realtime 55Mb default) > > > ># Create a customers root queue, setting hard limits for any customers > ># It ensures the entire internet connection is available at all times > ># and also limits it to that ammount > >queue CUSTOMERS bandwidth 20Mb hfsc(realtime 20Mb upperlimit 20Mb) { \ > > customer_1, \ > > customer_2, \ > > ..., \ > >} > ># Assign 1% per customer (100 customers) for backlogged traffic. > ># No realtime guarantees, let hfsc figure out how to spend the 20Mbit > ># from the CUSTOMERS parent queue. > ># No customer gets more then 1Mbit even if he's alone surfing the net. > >queue customer_1 bandwidth 1% hfsc(linkshare 1% upperlimit 1Mb) \ > > { customer_1_bulk, customer_1_ack } > ># default priority is 1 > >queue customer_1_bulk bandwidth 80% hfsc > >queue customer_1_ack bandwidth 20% priority 2 hfsc > > Thank you for your answer! > > Well, I've simplified the example. My real situation is that I have 2500 > users, sharing 100 Mb fiber optic line.. > For 2500 users 1% will be too much. Can I use 0.25% ? > > Also regarding HFSC, I know is an linear algorytm, which means many does > not scale well for lots of users. I know is hardcoded at 64, I've > modified and used with success for up to 500 queues. For more queues it > is working very slow, even with Xeon CPUs. So for 2500 users I would > probably need 4-5 separate machines. > > I've used rules similar to your HFSC example, and I had to switch to > ipfw + dummynet because of poor performance (on one machine) of HFSC > with so many queues. (using ipfw+dummynet, multiple pass, to a limiting > pipe then to queues to share load is working ok for browsing but I have > 2 issues: first the traffic is not stable, it variates to much, second > the traffic does not reach 100 Mbps which is my bandwidth (if i disable > the firewall,when trafic goes to 100 Mbps). I have to mention that CPU > is 95% idle. > > So my question is, what would be a choice to shape bandwidth with > FreeBSD for like 2000 users, with bandwidth limiting, and sharing using > clases. On linux it works for 1000-2000 users using HTB, (tc, traffic > control) and scales well, but I want to use FreeBSD, there must be a way > to get similar performance on FreeBSD too. Hmm, have you tried freebsd-isp@ list? That many users/queues, I can't comment on with real-life experience and don't think I can simulate that either ;). Also, there's more then one factor impacting performance here (nic driver, kernel threading concurrency, locking, pf/ipfw implementation) - I'm sure folks over @current are very interested in hearing your results with 7-RELEASE coming up. -- Mel