From owner-freebsd-ipfw@FreeBSD.ORG Thu Mar 18 10:18:56 2004 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0DE0916A4CF for ; Thu, 18 Mar 2004 10:18:56 -0800 (PST) Received: from oahu.WURLDLINK.NET (oahu.wurldlink.net [66.193.144.7]) by mx1.FreeBSD.org (Postfix) with ESMTP id A162443D2D for ; Thu, 18 Mar 2004 10:18:55 -0800 (PST) (envelope-from vince@oahu.WURLDLINK.NET) Received: from oahu.WURLDLINK.NET (vince@localhost.WURLDLINK.NET [127.0.0.1]) by oahu.WURLDLINK.NET (8.12.9/8.12.9) with ESMTP id i2IIHmqQ061493; Thu, 18 Mar 2004 08:18:03 -1000 (HST) Received: from localhost (vince@localhost)i2IIHjBe061489; Thu, 18 Mar 2004 08:17:47 -1000 (HST) Date: Thu, 18 Mar 2004 08:17:45 -1000 (HST) From: Vincent Poy To: James In-Reply-To: <20040317021928.GA26065@scylla.towardex.com> Message-ID: <20040318072629.T8264-100000@oahu.WURLDLINK.NET> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-ipfw@freebsd.org cc: Nicolas DEFFAYET Subject: Re: Latency problem with traffic shaping X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Mar 2004 18:18:56 -0000 On Tue, 16 Mar 2004, James wrote: > ohhh... you are concerned about simple 1ms difference due to > granuality (sp) in 35 meg pipe, right? > > this is a simple problem to fix: > > ipfw add 1 allow icmp from any to any icmptypes 11,0,8 > ipfw add 1 allow udp from any to any 33434-33534 > < then insert your pipe rules > > > and also, you realize that you are putting people on vlan3 > to a half duplex pipe right? > > -J While on this subject, I have one of my own... I have a 6.016Mbps/608kbps ADSL connection with 8 static IP's from my ISP. I'm using the FreeBSD box to basically limit my upstream bandwidth to 480kbps so that the downloads would work while uploading. In my kernel, I do have the following options: options IPFIREWALL #firewall options IPDIVERT #divert sockets options DUMMYNET options BRIDGE options HZ=1000 options NMBCLUSTERS=65536 The 8 IP's I'm using is 208.204.244.224-231 on a /24 block with the gateway on the other side at my ISP being 208.204.244.1. The FreeBSD machine is 208.204.244.224 and I do have gateway ip forwarding enabled. My problem is that while as far as speeds are concerned, it's working correctly on both the .224 (FreeBSD box) as well as the .225-.231 boxes behind it. The issue is that tracerouting from any box other than the FreeBSD box shows latencies of 1000+ms after the FreeBSD router beginning with hop 2 when the upstream pipe is being used while the FreeBSD box shows the latency at 40-50ms which is correct under traffic load. Anyone knows what's causing this or is this the way it's supposed to work? All the machines are pointing to .224 (FreeBSD box) as the gateway. All local traffic doesn't go through dummynet's queues. This is how I have ipfw configured. setup_loopback # Traffic Shaping for DSL connection 6.016Mbps/608Kbps # Make packets exiting dummynet not continue down the chain # If this is not enabled, then packets leaving an early # queue might enter a later queue if the conditions for # the later queue are met, which would be completely # devastating to all the prioritizing we're doing ${fwcmd} enable one_pass # Add rules so that local routable IP LAN traffic does not use natd ${fwcmd} add 39 divert natd all from 10.0.0.0/8 to any via ${natd_interface} ${fwcmd} add 40 divert natd all from 172.16.0.0/12 to any via ${natd_interface} ${fwcmd} add 41 divert natd all from 192.168.0.0/16 to any via ${natd_interface} ${fwcmd} add 42 divert natd all from 208.201.244.224/29 to 10.0.0.0/8 via ${natd_interface} ${fwcmd} add 43 divert natd all from 208.201.244.224/29 to 172.16.0.0/12 via ${natd_interface} ${fwcmd} add 44 divert natd all from 208.201.244.224/29 to 192.168.0.0/16 via ${natd_interface} ${fwcmd} add 45 divert natd all from any to 10.0.0.0/8 via ${natd_interface} ${fwcmd} add 46 divert natd all from any to 172.16.0.0/12 via ${natd_interface} ${fwcmd} add 47 divert natd all from any to 192.168.0.0/16 via ${natd_interface} ${fwcmd} add 48 divert natd all from any to 208.201.244.224/29 via ${natd_interface} ${fwcmd} add 49 skipto 100 ip from 208.201.244.224/29 to any ${fwcmd} add 50 divert natd all from any to any via ${natd_interface} ${fwcmd} add 100 pass all from any to any via lo0 ${fwcmd} add 200 deny all from any to 127.0.0.0/8 ${fwcmd} add 300 deny ip from 127.0.0.0/8 to any # Route LAN and RFC1918 networks without Traffic Shaping ${fwcmd} add 63000 allow all from any to 10.0.0.0/8 out ${fwcmd} add 63001 allow all from any to 172.16.0.0/12 out ${fwcmd} add 63002 allow all from any to 192.168.0.0/16 out ${fwcmd} add 63003 allow all from any to 208.201.244.224/29 out # Define our upload pipe ${fwcmd} pipe 1 config bw 480Kbit/s # Define a high-priority queue ${fwcmd} queue 1 config pipe 1 weight 100 # Define a medium-high-priority queue ${fwcmd} queue 2 config pipe 1 weight 99 # Define a medium-low-priority queue ${fwcmd} queue 3 config pipe 1 weight 98 # Define a low-priority queue ${fwcmd} queue 4 config pipe 1 weight 97 # Assign outgoing empty/small ACK packets to the high-priority queue ${fwcmd} add 63004 set 0 queue 1 tcp from any to any tcpflags ack iplen 0-80 out # Assign outgoing UDP (DNS/gaming) and SSH traffic to the medium-high-priority queue ${fwcmd} add 63005 set 0 queue 2 tcp from any to any 22,23 out ${fwcmd} add 63006 set 0 queue 2 udp from any to any not 80,443 out # Assign outgoing HTTP/HTTPS WEB traffic to the medium-low-priority queue ${fwcmd} add 63007 set 0 queue 3 all from any to any 80,443 out # Assign all other outgoing traffic to the low-priority queue ${fwcmd} add 63008 set 0 queue 4 all from any to any out # End of Traffic Shaping ${fwcmd} add 65000 pass all from any to any This is what the latencies look like on the machines behind the FreeBSD router when there is a upload: Tracing route to wurldlink.net [66.193.144.22] over a maximum of 30 hops: 1 <1 ms <1 ms <1 ms adsl-208-201-244-224.sonic.net [208.201.244.224] 2 915 ms 933 ms 1025 ms adsl-208-201-244-1.sonic.net [208.201.244.1] 3 1082 ms 1015 ms 1089 ms fast1-0-0.border.sr.sonic.net [208.201.224.194] 4 1206 ms 816 ms 869 ms fast0-0.gw.equinix-sj.sonic.net [64.142.0.14] 5 943 ms 1022 ms 1091 ms bpr1-t3-7-2-0.SanJoseEquinix.cw.net [208.173.54.45] 6 1095 ms 1044 ms 1112 ms cable-and-wireless-peering.SanJoseEquinix.cw.net [208.173.54.70] 7 1160 ms 1070 ms 1115 ms sl-bb25-sj-10-0.sprintlink.net [144.232.20.62] 8 891 ms 962 ms 1049 ms sl-bb20-sj-13-0.sprintlink.net [144.232.3.197] 9 960 ms 891 ms 1005 ms sl-bb20-stk-12-0.sprintlink.net [144.232.20.98] 10 1218 ms 1101 ms 1189 ms sl-bb20-prl-9-0.sprintlink.net [144.232.8.25] 11 811 ms 889 ms 979 ms sl-gw2-prl-0-0.sprintlink.net [144.232.30.22] 12 1002 ms 1070 ms 1164 ms sl-timewarner-12-0.sprintlink.net [160.81.200.214] 13 1065 ms 1062 ms 1080 ms 64-132-26-250.gen.twtelecom.net [64.132.26.250] 14 1173 ms 1098 ms 1155 ms kpext.ksbe.edu [216.136.57.178] 15 1092 ms 1108 ms 1209 ms www.onenet-usa.net [66.193.144.22] This is a traceroute directly from the FreeBSD box... traceroute to wurldlink.net (66.193.144.22), 64 hops max, 40 byte packets 1 adsl-208-201-244-1.sonic.net (208.201.244.1) 58.235 ms 57.779 ms 76.804 ms 2 fast1-0-0.border.sr.sonic.net (208.201.224.194) 38.449 ms 48.158 ms 48.871 ms 3 fast0-0.gw.equinix-sj.sonic.net (64.142.0.14) 60.951 ms 56.486 ms 49.452 ms 4 bpr1-t3-7-2-0.SanJoseEquinix.cw.net (208.173.54.45) 53.794 ms 52.463 ms 68.045 ms 5 cable-and-wireless-peering.SanJoseEquinix.cw.net (208.173.54.70) 78.437 ms 50.674 ms 46.528 ms 6 sl-bb25-sj-10-0.sprintlink.net (144.232.20.62) 52.491 ms 81.473 ms 54.669 ms 7 sl-bb20-sj-13-0.sprintlink.net (144.232.3.197) 67.872 ms 53.260 ms 65.417 ms 8 sl-bb20-stk-12-0.sprintlink.net (144.232.20.98) 81.940 ms 48.695 ms 59.650 ms 9 sl-bb20-prl-9-0.sprintlink.net (144.232.8.25) 118.604 ms 107.292 ms 136.087 ms 10 sl-gw2-prl-0-0.sprintlink.net (144.232.30.22) 124.988 ms 128.812 ms 129.594 ms 11 sl-timewarner-12-0.sprintlink.net (160.81.200.214) 126.898 ms 149.349 ms 114.960 ms 12 64-132-26-250.gen.twtelecom.net (64.132.26.250) 116.782 ms 140.489 ms 123.899 ms 13 kpext.ksbe.edu (216.136.57.178) 165.563 ms 131.212 ms 118.557 ms 14 www.onenet-usa.net (66.193.144.22) 155.675 ms 140.607 ms 175.878 ms Any ideas why the machines behind the FreeBSD box shows the 1000+ms latency after it reaches the FreeBSD box when the upstream pipe is being used but the speeds are working correctly? Thanks! Cheers, Vince - vince@WURLDLINK.NET - Vice President ________ __ ____ Unix Networking Operations - FreeBSD-Real Unix for Free / / / / | / |[__ ] WurldLink Corporation / / / / | / | __] ] San Francisco - Honolulu - Hong Kong / / / / / |/ / | __] ] HongKong Stars/Gravis UltraSound Mailing Lists Admin /_/_/_/_/|___/|_|[____] Almighty1@IRC - oahu.DAL.NET Hawaii's DALnet IRC Network Server Admin