Date: Sat, 20 Mar 2004 01:14:38 -1000 (HST) From: Vincent Poy <vince@oahu.WURLDLINK.NET> To: freebsd-ipfw@freebsd.org, <questions@freebsd.org> Subject: Latency problem with traffic shaping (ipfw/dummynet) Message-ID: <20040320011336.C8264-100000@oahu.WURLDLINK.NET>
next in thread | raw e-mail | index | archive | help
On this subject, I have one of my own... I have a
6.016Mbps/608kbps ADSL connection with 8 static IP's from my ISP. I'm
using the FreeBSD box to basically limit my upstream bandwidth to 480kbps
so that the downloads would work while uploading. In my kernel, I do have
the following options:
options IPFIREWALL #firewall
options IPDIVERT #divert sockets
options DUMMYNET
options BRIDGE
options HZ=1000
options NMBCLUSTERS=65536
The 8 IP's I'm using is 208.204.244.224-231 on a /24 block with
the gateway on the other side at my ISP being 208.204.244.1. The FreeBSD
machine is 208.204.244.224 and I do have gateway ip forwarding enabled. My
problem is that while as far as speeds are concerned, it's working
correctly on both the .224 (FreeBSD box) as well as the .225-.231 boxes
behind it. The issue is that tracerouting from any box other than the
FreeBSD box shows latencies of 1000+ms after the FreeBSD router beginning
with hop 2 when the upstream pipe is being used while the FreeBSD box
shows the latency at 40-50ms which is correct under traffic load. Anyone
knows what's causing this or is this the way it's supposed to work? All
the machines are pointing to .224 (FreeBSD box) as the gateway. All local
traffic doesn't go through dummynet's queues. This is how I have ipfw
configured.
setup_loopback
# Traffic Shaping for DSL connection 6.016Mbps/608Kbps
# Make packets exiting dummynet not continue down the chain
# If this is not enabled, then packets leaving an early
# queue might enter a later queue if the conditions for
# the later queue are met, which would be completely
# devastating to all the prioritizing we're doing
${fwcmd} enable one_pass
# Add rules so that local routable IP LAN traffic does not use natd
${fwcmd} add 39 divert natd all from 10.0.0.0/8 to any via ${natd_interface}
${fwcmd} add 40 divert natd all from 172.16.0.0/12 to any via ${natd_interface}
${fwcmd} add 41 divert natd all from 192.168.0.0/16 to any via ${natd_interface}
${fwcmd} add 42 divert natd all from 208.201.244.224/29 to 10.0.0.0/8 via ${natd_interface}
${fwcmd} add 43 divert natd all from 208.201.244.224/29 to 172.16.0.0/12 via ${natd_interface}
${fwcmd} add 44 divert natd all from 208.201.244.224/29 to 192.168.0.0/16 via ${natd_interface}
${fwcmd} add 45 divert natd all from any to 10.0.0.0/8 via ${natd_interface}
${fwcmd} add 46 divert natd all from any to 172.16.0.0/12 via ${natd_interface}
${fwcmd} add 47 divert natd all from any to 192.168.0.0/16 via ${natd_interface}
${fwcmd} add 48 divert natd all from any to 208.201.244.224/29 via ${natd_interface}
${fwcmd} add 49 skipto 100 ip from 208.201.244.224/29 to any
${fwcmd} add 50 divert natd all from any to any via ${natd_interface}
${fwcmd} add 100 pass all from any to any via lo0
${fwcmd} add 200 deny all from any to 127.0.0.0/8
${fwcmd} add 300 deny ip from 127.0.0.0/8 to any
# Route LAN and RFC1918 networks without Traffic Shaping
${fwcmd} add 63000 allow all from any to 10.0.0.0/8 out
${fwcmd} add 63001 allow all from any to 172.16.0.0/12 out
${fwcmd} add 63002 allow all from any to 192.168.0.0/16 out
${fwcmd} add 63003 allow all from any to 208.201.244.224/29 out
# Define our upload pipe
${fwcmd} pipe 1 config bw 480Kbit/s
# Define a high-priority queue
${fwcmd} queue 1 config pipe 1 weight 100
# Define a medium-high-priority queue
${fwcmd} queue 2 config pipe 1 weight 99
# Define a medium-low-priority queue
${fwcmd} queue 3 config pipe 1 weight 98
# Define a low-priority queue
${fwcmd} queue 4 config pipe 1 weight 97
# Assign outgoing empty/small ACK packets to the high-priority queue
${fwcmd} add 63004 set 0 queue 1 tcp from any to any tcpflags ack iplen 0-80 out
# Assign outgoing UDP (DNS/gaming) and SSH traffic to the medium-high-priority queue
${fwcmd} add 63005 set 0 queue 2 tcp from any to any 22,23 out
${fwcmd} add 63006 set 0 queue 2 udp from any to any not 80,443 out
# Assign outgoing HTTP/HTTPS WEB traffic to the medium-low-priority queue
${fwcmd} add 63007 set 0 queue 3 all from any to any 80,443 out
# Assign all other outgoing traffic to the low-priority queue
${fwcmd} add 63008 set 0 queue 4 all from any to any out
# End of Traffic Shaping
${fwcmd} add 65000 pass all from any to any
This is what the latencies look like on the machines behind the
FreeBSD router when there is a upload:
Tracing route to wurldlink.net [66.193.144.22]
over a maximum of 30 hops:
1 <1 ms <1 ms <1 ms adsl-208-201-244-224.sonic.net [208.201.244.224]
2 915 ms 933 ms 1025 ms adsl-208-201-244-1.sonic.net [208.201.244.1]
3 1082 ms 1015 ms 1089 ms fast1-0-0.border.sr.sonic.net [208.201.224.194]
4 1206 ms 816 ms 869 ms fast0-0.gw.equinix-sj.sonic.net [64.142.0.14]
5 943 ms 1022 ms 1091 ms bpr1-t3-7-2-0.SanJoseEquinix.cw.net [208.173.54.45]
6 1095 ms 1044 ms 1112 ms cable-and-wireless-peering.SanJoseEquinix.cw.net [208.173.54.70]
7 1160 ms 1070 ms 1115 ms sl-bb25-sj-10-0.sprintlink.net [144.232.20.62]
8 891 ms 962 ms 1049 ms sl-bb20-sj-13-0.sprintlink.net [144.232.3.197]
9 960 ms 891 ms 1005 ms sl-bb20-stk-12-0.sprintlink.net [144.232.20.98]
10 1218 ms 1101 ms 1189 ms sl-bb20-prl-9-0.sprintlink.net [144.232.8.25]
11 811 ms 889 ms 979 ms sl-gw2-prl-0-0.sprintlink.net [144.232.30.22]
12 1002 ms 1070 ms 1164 ms sl-timewarner-12-0.sprintlink.net [160.81.200.214]
13 1065 ms 1062 ms 1080 ms 64-132-26-250.gen.twtelecom.net [64.132.26.250]
14 1173 ms 1098 ms 1155 ms kpext.ksbe.edu [216.136.57.178]
15 1092 ms 1108 ms 1209 ms www.onenet-usa.net [66.193.144.22]
This is a traceroute directly from the FreeBSD box...
traceroute to wurldlink.net (66.193.144.22), 64 hops max, 40 byte packets
1 adsl-208-201-244-1.sonic.net (208.201.244.1) 58.235 ms 57.779 ms
76.804 ms
2 fast1-0-0.border.sr.sonic.net (208.201.224.194) 38.449 ms 48.158 ms
48.871 ms
3 fast0-0.gw.equinix-sj.sonic.net (64.142.0.14) 60.951 ms 56.486 ms
49.452 ms
4 bpr1-t3-7-2-0.SanJoseEquinix.cw.net (208.173.54.45) 53.794 ms 52.463
ms 68.045 ms
5 cable-and-wireless-peering.SanJoseEquinix.cw.net (208.173.54.70)
78.437 ms 50.674 ms 46.528 ms
6 sl-bb25-sj-10-0.sprintlink.net (144.232.20.62) 52.491 ms 81.473 ms
54.669 ms
7 sl-bb20-sj-13-0.sprintlink.net (144.232.3.197) 67.872 ms 53.260 ms
65.417 ms
8 sl-bb20-stk-12-0.sprintlink.net (144.232.20.98) 81.940 ms 48.695 ms
59.650 ms
9 sl-bb20-prl-9-0.sprintlink.net (144.232.8.25) 118.604 ms 107.292 ms
136.087 ms
10 sl-gw2-prl-0-0.sprintlink.net (144.232.30.22) 124.988 ms 128.812 ms
129.594 ms
11 sl-timewarner-12-0.sprintlink.net (160.81.200.214) 126.898 ms
149.349 ms 114.960 ms
12 64-132-26-250.gen.twtelecom.net (64.132.26.250) 116.782 ms 140.489
ms 123.899 ms
13 kpext.ksbe.edu (216.136.57.178) 165.563 ms 131.212 ms 118.557 ms
14 www.onenet-usa.net (66.193.144.22) 155.675 ms 140.607 ms 175.878 ms
Any ideas why the machines behind the FreeBSD box shows the
1000+ms latency after it reaches the FreeBSD box when the upstream pipe is
being used but the speeds are working correctly? Thanks!
Cheers,
Vince - vince@WURLDLINK.NET - Vice President ________ __ ____
Unix Networking Operations - FreeBSD-Real Unix for Free / / / / | / |[__ ]
WurldLink Corporation / / / / | / | __] ]
San Francisco - Honolulu - Hong Kong / / / / / |/ / | __] ]
HongKong Stars/Gravis UltraSound Mailing Lists Admin /_/_/_/_/|___/|_|[____]
Almighty1@IRC - oahu.DAL.NET Hawaii's DALnet IRC Network Server Admin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20040320011336.C8264-100000>