From owner-freebsd-net@FreeBSD.ORG Fri Jul 30 23:07:07 2010 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5A962106564A for ; Fri, 30 Jul 2010 23:07:07 +0000 (UTC) (envelope-from mahan@mahan.org) Received: from ns.mahan.org (ns.mahan.org [67.116.10.138]) by mx1.freebsd.org (Postfix) with ESMTP id 36DA88FC13 for ; Fri, 30 Jul 2010 23:07:05 +0000 (UTC) Received: from Gypsy.mahan.org (crowTrobot [67.116.10.140]) by ns.mahan.org (8.13.6/8.13.6) with ESMTP id o6UNALHo003177 for ; Fri, 30 Jul 2010 16:10:22 -0700 (PDT) (envelope-from mahan@mahan.org) Message-ID: <4C535B18.8020205@mahan.org> Date: Fri, 30 Jul 2010 16:07:04 -0700 From: Patrick Mahan User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.4) Gecko/20100608 Thunderbird/3.1 MIME-Version: 1.0 To: freebsd-net@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: AltQ throughput issues (long message) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Jul 2010 23:07:07 -0000 All, I am looking for (again) some understanding of AltQ and how it works w.r.t. packet through put. I posted earlier this month regarding how to initially configure AltQ (thanks to everyone's help) and now have it working over the em(4) drive on a FreeBSD 8.0 platform (HP DL350 G5). I had to bring the em(4) driver from the 8-Stable branch, but it is working just fine so far (needed to add the drbr_needs_enqueue() to if_var.h). I have now gone back to trying to setup up one queue with a bandwith of 1900 Kbs (1.9 Mbs). I ran a test with 'iperf' using udp and setting the bandwidth to 25 Mbs. I then ran a test setting the queue bandwith to 20 Mbs and running 'iperf' again using udp and 25 Mbs bandwith. In both cases, the throughput only seems to be 89% of the requested throughput. Test 1 AltQ queue bandwidth 1.9 Mbs, iperf -b 25M pfctl -vv -s queue reported: queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 28298 bytes: 42771988 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 140.8 packets/s, 1.70Mb/s ] queue test7788 on em0 bandwidth 1.90Mb cbq( default ) [ pkts: 28298 bytes: 42771988 dropped pkts: 397077 bytes: 600380424 ] [ qlength: 50/ 50 borrows: 0 suspends: 3278 ] [ measured: 140.8 packets/s, 1.70Mb/s ] iperf reported [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 3] 0.0-200.4 sec 39.7 MBytes 1.66 Mbits/sec 6.998 ms 397190/425533 (93%) Test 2 AltQ queue bandwidth 20 Mbs, iperf -b 25M pfctl -vv -s queue reported: queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 356702 bytes: 539329126 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 1500.2 packets/s, 18.15Mb/s ] queue test7788 on em0 bandwidth 20Mb cbq( default ) [ pkts: 356702 bytes: 539329126 dropped pkts: 149198 bytes: 225587376 ] [ qlength: 46/ 50 borrows: 0 suspends: 39629 ] [ measured: 1500.2 packets/s, 18.15Mb/s ] iperf reported [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 3] 0.0-240.0 sec 505 MBytes 17.6 Mbits/sec 0.918 ms 150584/510637 (29%) Why can AltQ not drive it at full bandwidth? This is just some preliminary testing, but I want to scale this up to use all available AltQ CBQ queues for various operations. As always, my knowledge is always increased with I ask questions on this list. Thanks, Patrick ======================= Test Results =========================== Network topology: +--------------+ +--------------+ | | | | | NPX8 | | NPX3 | | (em1)+ ===================== +(em3) | | | | | | | | (em0) | +--------------+ +------+-------+ I I I I I I I I +------+-------+ | (em0) | | | | NPX6 | | | | | +--------------+ NPX8: em1: 172.16.38.80/24 em1: flags=8843 metric 0 mtu 1500 options=19b ether 00:1c:c4:48:93:10 inet 172.16.38.80 netmask 0xffffff00 broadcast 172.16.38.255 media: Ethernet autoselect (1000baseT ) status: active NPX3: em3: 172.16.38.30/24 em3: flags=8843 metric 0 mtu 1500 options=19b ether 00:1f:29:5f:c6:aa inet 172.16.38.30 netmask 0xffffff00 broadcast 172.16.38.255 media: Ethernet autoselect (1000baseT ) status: active em0: 172.16.13.30/24 em0: flags=8843 metric 0 mtu 1500 options=19b ether 00:1f:29:5f:c6:a9 inet 172.16.13.30 netmask 0xffffff00 broadcast 172.16.13.255 media: Ethernet autoselect (1000baseT ) status: active NPX6: em0: 172.16.13.60/24 em0: flags=8843 metric 0 mtu 1500 options=19b ether 00:1c:c4:48:95:d1 inet 172.16.13.60 netmask 0xffffff00 broadcast 172.16.13.255 media: Ethernet autoselect (1000baseT ) status: active NPX8 IPv4 Routing table npx8# netstat -nr Routing tables Internet: Destination Gateway Flags Refs Use Netif Expire default 10.10.1.1 UGS 0 0 bce0 10.10.0.0/16 link#5 U 7 44 bce0 10.10.20.48 link#5 UHS 0 0 lo0 127.0.0.1 link#7 UH 0 0 lo0 172.16.13.0/24 172.16.38.30 UGS 0 0 em1 172.16.38.0/24 link#2 U 0 0 em1 172.16.38.80 link#2 UHS 0 0 lo0 NPX3 IPv4 Routing Table npx3# netstat -nr Routing tables Internet: Destination Gateway Flags Refs Use Netif Expire default 10.10.1.1 UGS 0 0 bce0 10.10.0.0/16 link#5 U 8 499 bce0 10.10.20.43 link#5 UHS 0 0 lo0 127.0.0.1 link#7 UH 0 0 lo0 172.16.13.0/24 link#1 U 0 0 em0 172.16.13.30 link#1 UHS 0 0 lo0 172.16.38.0/24 link#4 U 0 0 em3 172.16.38.30 link#4 UHS 0 0 lo0 NPX6 IPv4 Routing Table npx6# netstat -nr Routing tables Internet: Destination Gateway Flags Refs Use Netif Expire default 10.10.1.1 UGS 0 0 bce0 10.10.0.0/16 link#5 U 7 243 bce0 10.10.20.46 link#5 UHS 0 0 lo0 127.0.0.1 link#7 UH 0 0 lo0 172.16.13.0/24 link#1 U 0 0 em0 172.16.13.60 link#1 UHS 0 0 lo0 172.16.38.0/24 172.16.13.30 UGS 0 0 em0 NPX3 has forwarding enabled npx3# sysctl net.inet.ip.forwarding net.inet.ip.forwarding: 1 NPX8 has a route to NPX6 via em1 172.16.13.0/24 172.16.38.30 UGS 0 1754682 em1 NPX6 has a route to NPX8 via em0 172.16.38.0/24 172.16.13.30 UGS 0 2 em0 On NPX3 the following /etc/pf.conf is enabled npx3# cat /etc/pf.conf ####################################################################### # ALT-Q Configuration ####################################################################### #====================================================================== # Cluster configuration #====================================================================== altq on em0 cbq bandwidth 1000Mb queue { test7788 } #====================================================================== # Queueu configuration #====================================================================== queue test7788 bandwidth 1900Kb priority 1 cbq (default) #====================================================================== # Filter rules #====================================================================== pass out quick on em0 inet proto udp from any to any port 7788 no state queue test7788 On NPX6 I start iperf in server mode npx6# iperf -s -p 7788 -u -i 30 ------------------------------------------------------------ Server listening on UDP port 7788 Receiving 1470 byte datagrams UDP buffer size: 41.1 KByte (default) ------------------------------------------------------------ [ 3] local 172.16.38.80 port 41064 connected with 172.16.13.60 port 7788 [ ID] Interval Transfer Bandwidth [ 3] 0.0-30.0 sec 89.5 MBytes 25.0 Mbits/sec [ ID] Interval Transfer Bandwidth [ 3] 30.0-60.0 sec 89.5 MBytes 25.0 Mbits/sec [ ID] Interval Transfer Bandwidth [ 3] 60.0-90.0 sec 89.5 MBytes 25.0 Mbits/sec [ ID] Interval Transfer Bandwidth [ 3] 90.0-120.0 sec 89.5 MBytes 25.0 Mbits/sec [ ID] Interval Transfer Bandwidth [ 3] 120.0-150.0 sec 89.5 MBytes 25.0 Mbits/sec [ ID] Interval Transfer Bandwidth [ 3] 150.0-180.0 sec 89.5 MBytes 25.0 Mbits/sec [ ID] Interval Transfer Bandwidth [ 3] 0.0-200.0 sec 597 MBytes 25.0 Mbits/sec [ 3] Sent 425533 datagrams [ 3] Server Report: [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 3] 0.0-200.4 sec 39.7 MBytes 1.66 Mbits/sec 6.998 ms 397190/425533 (93%) On NPX8 I start the iperf client npx8# iperf -c 172.16.13.10 -p 7788 -u -b 25M -u -i 30 -t 200 ------------------------------------------------------------ Client connecting to 172.16.13.60, UDP port 7788 Sending 1470 byte datagrams UDP buffer size: 9.00 KByte (default) ------------------------------------------------------------ [ 3] local 172.16.13.60 port 7788 connected with 172.16.38.80 port 41064 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 3] 0.0-30.0 sec 5.96 MBytes 1.67 Mbits/sec 0.710 ms 59453/63706 (93%) [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 3] 30.0-60.0 sec 5.95 MBytes 1.66 Mbits/sec 0.736 ms 59616/63859 (93%) [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 3] 60.0-90.0 sec 5.94 MBytes 1.66 Mbits/sec 0.721 ms 59529/63768 (93%) [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 3] 90.0-120.0 sec 5.95 MBytes 1.66 Mbits/sec 0.760 ms 59616/63859 (93%) [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 3] 120.0-150.0 sec 5.95 MBytes 1.66 Mbits/sec 0.732 ms 59618/63862 (93%) [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 3] 150.0-180.0 sec 5.95 MBytes 1.66 Mbits/sec 0.726 ms 59608/63852 (93%) [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 3] 0.0-200.4 sec 39.7 MBytes 1.66 Mbits/sec 6.998 ms 397190/425533 (93%) On NPX3 I start monitoring of the Altq npx3: pfctl -vv -s queue queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 3 bytes: 238 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 0.0 packets/s, 0 b/s ] queue test7788 on em0 bandwidth 1.90Mb cbq( default ) [ pkts: 3 bytes: 238 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 0.0 packets/s, 0 b/s ] queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 10 bytes: 532 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 0.2 packets/s, 58.80 b/s ] queue test7788 on em0 bandwidth 1.90Mb cbq( default ) [ pkts: 10 bytes: 532 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 0.2 packets/s, 58.80 b/s ] queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 10 bytes: 532 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 0.2 packets/s, 51.45 b/s ] queue test7788 on em0 bandwidth 1.90Mb cbq( default ) [ pkts: 10 bytes: 532 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 0.2 packets/s, 51.45 b/s ] queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 710 bytes: 1058932 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 17.6 packets/s, 211.73Kb/s ] queue test7788 on em0 bandwidth 1.90Mb cbq( default ) [ pkts: 710 bytes: 1058932 dropped pkts: 9600 bytes: 14515200 ] [ qlength: 50/ 50 borrows: 0 suspends: 80 ] [ measured: 17.6 packets/s, 211.73Kb/s ] queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 1418 bytes: 2129428 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 33.1 packets/s, 399.36Kb/s ] queue test7788 on em0 bandwidth 1.90Mb cbq( default ) [ pkts: 1418 bytes: 2129428 dropped pkts: 19534 bytes: 29535408 ] [ qlength: 50/ 50 borrows: 0 suspends: 162 ] [ measured: 33.1 packets/s, 399.36Kb/s ] queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 2125 bytes: 3198412 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 46.7 packets/s, 563.24Kb/s ] queue test7788 on em0 bandwidth 1.90Mb cbq( default ) [ pkts: 2125 bytes: 3198412 dropped pkts: 29470 bytes: 44558640 ] [ qlength: 50/ 50 borrows: 0 suspends: 244 ] [ measured: 46.7 packets/s, 563.24Kb/s ] queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 2832 bytes: 4267396 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 58.5 packets/s, 706.63Kb/s ] queue test7788 on em0 bandwidth 1.90Mb cbq( default ) [ pkts: 2832 bytes: 4267396 dropped pkts: 39406 bytes: 59581872 ] [ qlength: 50/ 50 borrows: 0 suspends: 326 ] [ measured: 58.5 packets/s, 706.63Kb/s ] queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 3539 bytes: 5336380 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 68.9 packets/s, 832.10Kb/s ] queue test7788 on em0 bandwidth 1.90Mb cbq( default ) [ pkts: 3539 bytes: 5336380 dropped pkts: 49341 bytes: 74603592 ] [ qlength: 50/ 50 borrows: 0 suspends: 408 ] [ measured: 68.9 packets/s, 832.10Kb/s ] queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 4246 bytes: 6405364 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 77.9 packets/s, 941.88Kb/s ] queue test7788 on em0 bandwidth 1.90Mb cbq( default ) [ pkts: 4246 bytes: 6405364 dropped pkts: 59277 bytes: 89626824 ] [ qlength: 50/ 50 borrows: 0 suspends: 490 ] [ measured: 77.9 packets/s, 941.88Kb/s ] queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 4954 bytes: 7475860 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 85.9 packets/s, 1.04Mb/s ] queue test7788 on em0 bandwidth 1.90Mb cbq( default ) [ pkts: 4954 bytes: 7475860 dropped pkts: 69211 bytes: 104647032 ] [ qlength: 50/ 50 borrows: 0 suspends: 572 ] [ measured: 85.9 packets/s, 1.04Mb/s ] queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 5661 bytes: 8544844 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 92.8 packets/s, 1.12Mb/s ] queue test7788 on em0 bandwidth 1.90Mb cbq( default ) [ pkts: 5661 bytes: 8544844 dropped pkts: 79147 bytes: 119670264 ] [ qlength: 50/ 50 borrows: 0 suspends: 654 ] [ measured: 92.8 packets/s, 1.12Mb/s ] queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 6368 bytes: 9613828 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 98.9 packets/s, 1.20Mb/s ] queue test7788 on em0 bandwidth 1.90Mb cbq( default ) [ pkts: 6368 bytes: 9613828 dropped pkts: 89082 bytes: 134691984 ] [ qlength: 50/ 50 borrows: 0 suspends: 736 ] [ measured: 98.9 packets/s, 1.20Mb/s ] queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 7075 bytes: 10682812 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 104.2 packets/s, 1.26Mb/s ] queue test7788 on em0 bandwidth 1.90Mb cbq( default ) [ pkts: 7075 bytes: 10682812 dropped pkts: 99018 bytes: 149715216 ] [ qlength: 50/ 50 borrows: 0 suspends: 818 ] [ measured: 104.2 packets/s, 1.26Mb/s ] queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 7783 bytes: 11753308 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 108.9 packets/s, 1.32Mb/s ] queue test7788 on em0 bandwidth 1.90Mb cbq( default ) [ pkts: 7783 bytes: 11753308 dropped pkts: 108953 bytes: 164736936 ] [ qlength: 50/ 50 borrows: 0 suspends: 900 ] [ measured: 108.9 packets/s, 1.32Mb/s ] queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 8490 bytes: 12822292 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 113.0 packets/s, 1.37Mb/s ] queue test7788 on em0 bandwidth 1.90Mb cbq( default ) [ pkts: 8490 bytes: 12822292 dropped pkts: 118888 bytes: 179758656 ] [ qlength: 50/ 50 borrows: 0 suspends: 982 ] [ measured: 113.0 packets/s, 1.37Mb/s ] queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 9199 bytes: 13894300 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 116.6 packets/s, 1.41Mb/s ] queue test7788 on em0 bandwidth 1.90Mb cbq( default ) [ pkts: 9199 bytes: 13894300 dropped pkts: 128823 bytes: 194780376 ] [ qlength: 50/ 50 borrows: 0 suspends: 1064 ] [ measured: 116.6 packets/s, 1.41Mb/s ] queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 9906 bytes: 14963284 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 119.7 packets/s, 1.45Mb/s ] queue test7788 on em0 bandwidth 1.90Mb cbq( default ) [ pkts: 9906 bytes: 14963284 dropped pkts: 138759 bytes: 209803608 ] [ qlength: 50/ 50 borrows: 0 suspends: 1146 ] [ measured: 119.7 packets/s, 1.45Mb/s ] queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 10613 bytes: 16032268 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 122.4 packets/s, 1.48Mb/s ] queue test7788 on em0 bandwidth 1.90Mb cbq( default ) [ pkts: 10613 bytes: 16032268 dropped pkts: 148694 bytes: 224825328 ] [ qlength: 50/ 50 borrows: 0 suspends: 1228 ] [ measured: 122.4 packets/s, 1.48Mb/s ] queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 11321 bytes: 17102764 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 124.8 packets/s, 1.51Mb/s ] queue test7788 on em0 bandwidth 1.90Mb cbq( default ) [ pkts: 11321 bytes: 17102764 dropped pkts: 158629 bytes: 239847048 ] [ qlength: 50/ 50 borrows: 0 suspends: 1310 ] [ measured: 124.8 packets/s, 1.51Mb/s ] queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 12028 bytes: 18171748 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 126.9 packets/s, 1.53Mb/s ] queue test7788 on em0 bandwidth 1.90Mb cbq( default ) [ pkts: 12028 bytes: 18171748 dropped pkts: 168565 bytes: 254870280 ] [ qlength: 50/ 50 borrows: 0 suspends: 1392 ] [ measured: 126.9 packets/s, 1.53Mb/s ] queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 12737 bytes: 19243756 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 128.7 packets/s, 1.56Mb/s ] queue test7788 on em0 bandwidth 1.90Mb cbq( default ) [ pkts: 12737 bytes: 19243756 dropped pkts: 178498 bytes: 269888976 ] [ qlength: 50/ 50 borrows: 0 suspends: 1474 ] [ measured: 128.7 packets/s, 1.56Mb/s ] queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 13445 bytes: 20314252 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 130.3 packets/s, 1.58Mb/s ] queue test7788 on em0 bandwidth 1.90Mb cbq( default ) [ pkts: 13445 bytes: 20314252 dropped pkts: 188433 bytes: 284910696 ] [ qlength: 50/ 50 borrows: 0 suspends: 1556 ] [ measured: 130.3 packets/s, 1.58Mb/s ] queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 14152 bytes: 21383236 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 131.7 packets/s, 1.59Mb/s ] queue test7788 on em0 bandwidth 1.90Mb cbq( default ) [ pkts: 14152 bytes: 21383236 dropped pkts: 198368 bytes: 299932416 ] [ qlength: 50/ 50 borrows: 0 suspends: 1638 ] [ measured: 131.7 packets/s, 1.59Mb/s ] queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 14859 bytes: 22452220 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 132.9 packets/s, 1.61Mb/s ] queue test7788 on em0 bandwidth 1.90Mb cbq( default ) [ pkts: 14859 bytes: 22452220 dropped pkts: 208304 bytes: 314955648 ] [ qlength: 50/ 50 borrows: 0 suspends: 1720 ] [ measured: 132.9 packets/s, 1.61Mb/s ] queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 15566 bytes: 23521204 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 134.0 packets/s, 1.62Mb/s ] queue test7788 on em0 bandwidth 1.90Mb cbq( default ) [ pkts: 15566 bytes: 23521204 dropped pkts: 218240 bytes: 329978880 ] [ qlength: 50/ 50 borrows: 0 suspends: 1802 ] [ measured: 134.0 packets/s, 1.62Mb/s ] queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 16274 bytes: 24591700 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 134.9 packets/s, 1.63Mb/s ] queue test7788 on em0 bandwidth 1.90Mb cbq( default ) [ pkts: 16274 bytes: 24591700 dropped pkts: 228174 bytes: 344999088 ] [ qlength: 50/ 50 borrows: 0 suspends: 1884 ] [ measured: 134.9 packets/s, 1.63Mb/s ] queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 16981 bytes: 25660684 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 135.7 packets/s, 1.64Mb/s ] queue test7788 on em0 bandwidth 1.90Mb cbq( default ) [ pkts: 16981 bytes: 25660684 dropped pkts: 238110 bytes: 360022320 ] [ qlength: 50/ 50 borrows: 0 suspends: 1966 ] [ measured: 135.7 packets/s, 1.64Mb/s ] queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 17688 bytes: 26729668 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 136.5 packets/s, 1.65Mb/s ] queue test7788 on em0 bandwidth 1.90Mb cbq( default ) [ pkts: 17688 bytes: 26729668 dropped pkts: 248047 bytes: 375047064 ] [ qlength: 50/ 50 borrows: 0 suspends: 2048 ] [ measured: 136.5 packets/s, 1.65Mb/s ] queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 18396 bytes: 27800164 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 137.1 packets/s, 1.66Mb/s ] queue test7788 on em0 bandwidth 1.90Mb cbq( default ) [ pkts: 18396 bytes: 27800164 dropped pkts: 257981 bytes: 390067272 ] [ qlength: 50/ 50 borrows: 0 suspends: 2130 ] [ measured: 137.1 packets/s, 1.66Mb/s ] queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 19103 bytes: 28869148 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 137.6 packets/s, 1.66Mb/s ] queue test7788 on em0 bandwidth 1.90Mb cbq( default ) [ pkts: 19103 bytes: 28869148 dropped pkts: 267917 bytes: 405090504 ] [ qlength: 50/ 50 borrows: 0 suspends: 2212 ] [ measured: 137.6 packets/s, 1.66Mb/s ] queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 19810 bytes: 29938132 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 138.1 packets/s, 1.67Mb/s ] queue test7788 on em0 bandwidth 1.90Mb cbq( default ) [ pkts: 19810 bytes: 29938132 dropped pkts: 277853 bytes: 420113736 ] [ qlength: 50/ 50 borrows: 0 suspends: 2294 ] [ measured: 138.1 packets/s, 1.67Mb/s ] queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 20517 bytes: 31007116 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 138.5 packets/s, 1.68Mb/s ] queue test7788 on em0 bandwidth 1.90Mb cbq( default ) [ pkts: 20517 bytes: 31007116 dropped pkts: 287788 bytes: 435135456 ] [ qlength: 50/ 50 borrows: 0 suspends: 2376 ] [ measured: 138.5 packets/s, 1.68Mb/s ] queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 21225 bytes: 32077612 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 138.9 packets/s, 1.68Mb/s ] queue test7788 on em0 bandwidth 1.90Mb cbq( default ) [ pkts: 21225 bytes: 32077612 dropped pkts: 297723 bytes: 450157176 ] [ qlength: 50/ 50 borrows: 0 suspends: 2458 ] [ measured: 138.9 packets/s, 1.68Mb/s ] queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 21932 bytes: 33146596 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 139.2 packets/s, 1.68Mb/s ] queue test7788 on em0 bandwidth 1.90Mb cbq( default ) [ pkts: 21932 bytes: 33146596 dropped pkts: 307659 bytes: 465180408 ] [ qlength: 50/ 50 borrows: 0 suspends: 2540 ] [ measured: 139.2 packets/s, 1.68Mb/s ] queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 22639 bytes: 34215580 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 139.5 packets/s, 1.69Mb/s ] queue test7788 on em0 bandwidth 1.90Mb cbq( default ) [ pkts: 22639 bytes: 34215580 dropped pkts: 317594 bytes: 480202128 ] [ qlength: 50/ 50 borrows: 0 suspends: 2622 ] [ measured: 139.5 packets/s, 1.69Mb/s ] queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 23346 bytes: 35284564 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 139.7 packets/s, 1.69Mb/s ] queue test7788 on em0 bandwidth 1.90Mb cbq( default ) [ pkts: 23346 bytes: 35284564 dropped pkts: 327530 bytes: 495225360 ] [ qlength: 50/ 50 borrows: 0 suspends: 2704 ] [ measured: 139.7 packets/s, 1.69Mb/s ] queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 24054 bytes: 36355060 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 140.0 packets/s, 1.69Mb/s ] queue test7788 on em0 bandwidth 1.90Mb cbq( default ) [ pkts: 24054 bytes: 36355060 dropped pkts: 337465 bytes: 510247080 ] [ qlength: 50/ 50 borrows: 0 suspends: 2786 ] [ measured: 140.0 packets/s, 1.69Mb/s ] queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 24761 bytes: 37424044 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 140.1 packets/s, 1.70Mb/s ] queue test7788 on em0 bandwidth 1.90Mb cbq( default ) [ pkts: 24761 bytes: 37424044 dropped pkts: 347400 bytes: 525268800 ] [ qlength: 50/ 50 borrows: 0 suspends: 2868 ] [ measured: 140.1 packets/s, 1.70Mb/s ] queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 25468 bytes: 38493028 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 140.3 packets/s, 1.70Mb/s ] queue test7788 on em0 bandwidth 1.90Mb cbq( default ) [ pkts: 25468 bytes: 38493028 dropped pkts: 357336 bytes: 540292032 ] [ qlength: 50/ 50 borrows: 0 suspends: 2950 ] [ measured: 140.3 packets/s, 1.70Mb/s ] queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 26176 bytes: 39563524 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 140.5 packets/s, 1.70Mb/s ] queue test7788 on em0 bandwidth 1.90Mb cbq( default ) [ pkts: 26176 bytes: 39563524 dropped pkts: 367272 bytes: 555315264 ] [ qlength: 50/ 50 borrows: 0 suspends: 3032 ] [ measured: 140.5 packets/s, 1.70Mb/s ] queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 26883 bytes: 40632508 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 140.6 packets/s, 1.70Mb/s ] queue test7788 on em0 bandwidth 1.90Mb cbq( default ) [ pkts: 26883 bytes: 40632508 dropped pkts: 377207 bytes: 570336984 ] [ qlength: 50/ 50 borrows: 0 suspends: 3114 ] [ measured: 140.6 packets/s, 1.70Mb/s ] queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 27590 bytes: 41701492 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 140.7 packets/s, 1.70Mb/s ] queue test7788 on em0 bandwidth 1.90Mb cbq( default ) [ pkts: 27590 bytes: 41701492 dropped pkts: 387143 bytes: 585360216 ] [ qlength: 50/ 50 borrows: 0 suspends: 3196 ] [ measured: 140.7 packets/s, 1.70Mb/s ] queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 28298 bytes: 42771988 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 140.8 packets/s, 1.70Mb/s ] queue test7788 on em0 bandwidth 1.90Mb cbq( default ) [ pkts: 28298 bytes: 42771988 dropped pkts: 397077 bytes: 600380424 ] [ qlength: 50/ 50 borrows: 0 suspends: 3278 ] [ measured: 140.8 packets/s, 1.70Mb/s ] queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 28355 bytes: 42858172 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 124.6 packets/s, 1.51Mb/s ] queue test7788 on em0 bandwidth 1.90Mb cbq( default ) [ pkts: 28355 bytes: 42858172 dropped pkts: 397190 bytes: 600551280 ] [ qlength: 0/ 50 borrows: 0 suspends: 3284 ] [ measured: 124.6 packets/s, 1.51Mb/s ] ============================================================================ With the queue set to 20Mb throughput. NPX3# pfctl -vv -s queue queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 334193 bytes: 505295518 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 1500.0 packets/s, 18.14Mb/s ] queue test7788 on em0 bandwidth 20Mb cbq( default ) [ pkts: 334193 bytes: 505295518 dropped pkts: 139779 bytes: 211345848 ] [ qlength: 46/ 50 borrows: 0 suspends: 37128 ] [ measured: 1500.0 packets/s, 18.14Mb/s ] queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 341694 bytes: 516637030 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 1500.1 packets/s, 18.14Mb/s ] queue test7788 on em0 bandwidth 20Mb cbq( default ) [ pkts: 341697 bytes: 516641566 dropped pkts: 142920 bytes: 216095040 ] [ qlength: 43/ 50 borrows: 0 suspends: 37961 ] [ measured: 1500.1 packets/s, 18.15Mb/s ] queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 349196 bytes: 527980054 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 1500.1 packets/s, 18.15Mb/s ] queue test7788 on em0 bandwidth 20Mb cbq( default ) [ pkts: 349196 bytes: 527980054 dropped pkts: 146057 bytes: 220838184 ] [ qlength: 50/ 50 borrows: 0 suspends: 38795 ] [ measured: 1500.1 packets/s, 18.14Mb/s ] queue root_em0 on em0 bandwidth 1Gb priority 0 cbq( wrr root ) {test7788} [ pkts: 356702 bytes: 539329126 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 1500.2 packets/s, 18.15Mb/s ] queue test7788 on em0 bandwidth 20Mb cbq( default ) [ pkts: 356702 bytes: 539329126 dropped pkts: 149198 bytes: 225587376 ] [ qlength: 46/ 50 borrows: 0 suspends: 39629 ] [ measured: 1500.2 packets/s, 18.15Mb/s ] NPX8: iperf -c 172.16.13.60 -p 7788 -u -b 25M -u -i 30 -t 240 ------------------------------------------------------------ Client connecting to 172.16.13.60, UDP port 7788 Sending 1470 byte datagrams UDP buffer size: 9.00 KByte (default) ------------------------------------------------------------ [ 3] local 172.16.38.80 port 17106 connected with 172.16.13.60 port 7788 [ ID] Interval Transfer Bandwidth [ 3] 0.0-30.0 sec 89.5 MBytes 25.0 Mbits/sec [ ID] Interval Transfer Bandwidth [ 3] 30.0-60.0 sec 89.5 MBytes 25.0 Mbits/sec [ ID] Interval Transfer Bandwidth [ 3] 60.0-90.0 sec 89.5 MBytes 25.0 Mbits/sec [ ID] Interval Transfer Bandwidth [ 3] 90.0-120.0 sec 89.5 MBytes 25.0 Mbits/sec [ ID] Interval Transfer Bandwidth [ 3] 120.0-150.0 sec 89.5 MBytes 25.0 Mbits/sec [ ID] Interval Transfer Bandwidth [ 3] 150.0-180.0 sec 89.5 MBytes 25.0 Mbits/sec [ ID] Interval Transfer Bandwidth [ 3] 180.0-210.0 sec 89.5 MBytes 25.0 Mbits/sec [ ID] Interval Transfer Bandwidth [ 3] 210.0-240.0 sec 89.5 MBytes 25.0 Mbits/sec [ ID] Interval Transfer Bandwidth [ 3] 0.0-240.0 sec 716 MBytes 25.0 Mbits/sec [ 3] Sent 510637 datagrams [ 3] Server Report: [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 3] 0.0-240.0 sec 505 MBytes 17.6 Mbits/sec 0.918 ms 150584/510637 (29%) NPX6: iperf -s -p 7788 -u -i 30 ------------------------------------------------------------ Server listening on UDP port 7788 Receiving 1470 byte datagrams UDP buffer size: 41.1 KByte (default) ------------------------------------------------------------ [ 4] local 172.16.13.60 port 7788 connected with 172.16.38.80 port 17106 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 4] 0.0-30.0 sec 63.1 MBytes 17.6 Mbits/sec 0.735 ms 18774/63782 (29%) [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 4] 30.0-60.0 sec 63.1 MBytes 17.6 Mbits/sec 0.745 ms 18830/63830 (30%) [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 4] 60.0-90.0 sec 63.1 MBytes 17.6 Mbits/sec 0.756 ms 18831/63831 (30%) [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 4] 90.0-120.0 sec 63.1 MBytes 17.6 Mbits/sec 0.741 ms 18831/63831 (30%) [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 4] 120.0-150.0 sec 63.1 MBytes 17.6 Mbits/sec 0.738 ms 18830/63831 (29%) [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 4] 150.0-180.0 sec 63.1 MBytes 17.6 Mbits/sec 0.733 ms 18830/63830 (30%) [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 4] 180.0-210.0 sec 63.1 MBytes 17.6 Mbits/sec 0.718 ms 18828/63829 (29%) [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 4] 210.0-240.0 sec 63.1 MBytes 17.6 Mbits/sec 0.730 ms 18830/63830 (30%) [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 4] 0.0-240.0 sec 505 MBytes 17.6 Mbits/sec 0.918 ms 150584/510637 (29%)