Skip site navigation (1)Skip section navigation (2)
Date:      Sat, 20 Mar 2004 01:14:38 -1000 (HST)
From:      Vincent Poy <vince@oahu.WURLDLINK.NET>
To:        freebsd-ipfw@freebsd.org, <questions@freebsd.org>
Subject:   Latency problem with traffic shaping (ipfw/dummynet)
Message-ID:  <20040320011336.C8264-100000@oahu.WURLDLINK.NET>

next in thread | raw e-mail | index | archive | help
	On this subject, I have one of my own...  I have a
6.016Mbps/608kbps ADSL connection with 8 static IP's from my ISP.  I'm
using the FreeBSD box to basically limit my upstream bandwidth to 480kbps
so that the downloads would work while uploading.  In my kernel, I do have
the following options:

options         IPFIREWALL              #firewall
options         IPDIVERT                #divert sockets
options         DUMMYNET
options         BRIDGE
options         HZ=1000
options         NMBCLUSTERS=65536

	The 8 IP's I'm using is 208.204.244.224-231 on a /24 block with
the gateway on the other side at my ISP being 208.204.244.1.  The FreeBSD
machine is 208.204.244.224 and I do have gateway ip forwarding enabled. My
problem is that while as far as speeds are concerned, it's working
correctly on both the .224 (FreeBSD box) as well as the .225-.231 boxes
behind it.  The issue is that tracerouting from any box other than the
FreeBSD box shows latencies of 1000+ms after the FreeBSD router beginning
with hop 2 when the upstream pipe is being used while the FreeBSD box
shows the latency at 40-50ms which is correct under traffic load.  Anyone
knows what's causing this or is this the way it's supposed to work?  All
the machines are pointing to .224 (FreeBSD box) as the gateway.  All local
traffic doesn't go through dummynet's queues.  This is how I have ipfw
configured.

	setup_loopback
# Traffic Shaping for DSL connection 6.016Mbps/608Kbps
# Make packets exiting dummynet not continue down the chain
# If this is not enabled, then packets leaving an early
# queue might enter a later queue if the conditions for
# the later queue are met, which would be completely
# devastating to all the prioritizing we're doing
        ${fwcmd} enable one_pass
# Add rules so that local routable IP LAN traffic does not use natd
        ${fwcmd} add 39 divert natd all from 10.0.0.0/8 to any via ${natd_interface}
        ${fwcmd} add 40 divert natd all from 172.16.0.0/12 to any via ${natd_interface}
        ${fwcmd} add 41 divert natd all from 192.168.0.0/16 to any via ${natd_interface}
        ${fwcmd} add 42 divert natd all from 208.201.244.224/29 to 10.0.0.0/8 via ${natd_interface}
        ${fwcmd} add 43 divert natd all from 208.201.244.224/29 to 172.16.0.0/12 via ${natd_interface}
        ${fwcmd} add 44 divert natd all from 208.201.244.224/29 to 192.168.0.0/16 via ${natd_interface}
        ${fwcmd} add 45 divert natd all from any to 10.0.0.0/8 via ${natd_interface}
        ${fwcmd} add 46 divert natd all from any to 172.16.0.0/12 via ${natd_interface}
        ${fwcmd} add 47 divert natd all from any to 192.168.0.0/16 via ${natd_interface}
        ${fwcmd} add 48 divert natd all from any to 208.201.244.224/29 via ${natd_interface}
        ${fwcmd} add 49 skipto 100 ip from 208.201.244.224/29 to any
	${fwcmd} add 50 divert natd all from any to any via ${natd_interface}
        ${fwcmd} add 100 pass all from any to any via lo0
        ${fwcmd} add 200 deny all from any to 127.0.0.0/8
        ${fwcmd} add 300 deny ip from 127.0.0.0/8 to any
# Route LAN and RFC1918 networks without Traffic Shaping
        ${fwcmd} add 63000 allow all from any to 10.0.0.0/8 out
        ${fwcmd} add 63001 allow all from any to 172.16.0.0/12 out
        ${fwcmd} add 63002 allow all from any to 192.168.0.0/16 out
        ${fwcmd} add 63003 allow all from any to 208.201.244.224/29 out
# Define our upload pipe
        ${fwcmd} pipe 1 config bw 480Kbit/s
# Define a high-priority queue
        ${fwcmd} queue 1 config pipe 1 weight 100
# Define a medium-high-priority queue
        ${fwcmd} queue 2 config pipe 1 weight 99
# Define a medium-low-priority queue
        ${fwcmd} queue 3 config pipe 1 weight 98
# Define a low-priority queue
        ${fwcmd} queue 4 config pipe 1 weight 97
# Assign outgoing empty/small ACK packets to the high-priority queue
        ${fwcmd} add 63004 set 0 queue 1 tcp from any to any tcpflags ack iplen 0-80 out
# Assign outgoing UDP (DNS/gaming) and SSH traffic to the medium-high-priority queue
        ${fwcmd} add 63005 set 0 queue 2 tcp from any to any 22,23 out
        ${fwcmd} add 63006 set 0 queue 2 udp from any to any not 80,443 out
# Assign outgoing HTTP/HTTPS WEB traffic to the medium-low-priority queue
        ${fwcmd} add 63007 set 0 queue 3 all from any to any 80,443 out
# Assign all other outgoing traffic to the low-priority queue
        ${fwcmd} add 63008 set 0 queue 4 all from any to any out
# End of Traffic Shaping
        ${fwcmd} add 65000 pass all from any to any

	This is what the latencies look like on the machines behind the
FreeBSD router when there is a upload:

Tracing route to wurldlink.net [66.193.144.22]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  adsl-208-201-244-224.sonic.net [208.201.244.224]
  2   915 ms   933 ms  1025 ms  adsl-208-201-244-1.sonic.net [208.201.244.1]
  3  1082 ms  1015 ms  1089 ms  fast1-0-0.border.sr.sonic.net [208.201.224.194]
  4  1206 ms   816 ms   869 ms  fast0-0.gw.equinix-sj.sonic.net [64.142.0.14]
  5   943 ms  1022 ms  1091 ms  bpr1-t3-7-2-0.SanJoseEquinix.cw.net [208.173.54.45]
  6  1095 ms  1044 ms  1112 ms  cable-and-wireless-peering.SanJoseEquinix.cw.net [208.173.54.70]
  7  1160 ms  1070 ms  1115 ms  sl-bb25-sj-10-0.sprintlink.net [144.232.20.62]
  8   891 ms   962 ms  1049 ms  sl-bb20-sj-13-0.sprintlink.net [144.232.3.197]
  9   960 ms   891 ms  1005 ms  sl-bb20-stk-12-0.sprintlink.net [144.232.20.98]
 10  1218 ms  1101 ms  1189 ms  sl-bb20-prl-9-0.sprintlink.net [144.232.8.25]
 11   811 ms   889 ms   979 ms  sl-gw2-prl-0-0.sprintlink.net [144.232.30.22]
 12  1002 ms  1070 ms  1164 ms  sl-timewarner-12-0.sprintlink.net [160.81.200.214]
 13  1065 ms  1062 ms  1080 ms  64-132-26-250.gen.twtelecom.net [64.132.26.250]
 14  1173 ms  1098 ms  1155 ms  kpext.ksbe.edu [216.136.57.178]
 15  1092 ms  1108 ms  1209 ms  www.onenet-usa.net [66.193.144.22]

This is a traceroute directly from the FreeBSD box...

traceroute to wurldlink.net (66.193.144.22), 64 hops max, 40 byte packets
 1  adsl-208-201-244-1.sonic.net (208.201.244.1)  58.235 ms  57.779 ms
76.804 ms
 2  fast1-0-0.border.sr.sonic.net (208.201.224.194)  38.449 ms  48.158 ms
48.871 ms
 3  fast0-0.gw.equinix-sj.sonic.net (64.142.0.14)  60.951 ms  56.486 ms
49.452 ms
 4  bpr1-t3-7-2-0.SanJoseEquinix.cw.net (208.173.54.45)  53.794 ms  52.463
ms  68.045 ms
 5  cable-and-wireless-peering.SanJoseEquinix.cw.net (208.173.54.70)
78.437 ms  50.674 ms  46.528 ms
 6  sl-bb25-sj-10-0.sprintlink.net (144.232.20.62)  52.491 ms  81.473 ms
54.669 ms
 7  sl-bb20-sj-13-0.sprintlink.net (144.232.3.197)  67.872 ms  53.260 ms
65.417 ms
 8  sl-bb20-stk-12-0.sprintlink.net (144.232.20.98)  81.940 ms  48.695 ms
59.650 ms
 9  sl-bb20-prl-9-0.sprintlink.net (144.232.8.25)  118.604 ms  107.292 ms
136.087 ms
10  sl-gw2-prl-0-0.sprintlink.net (144.232.30.22)  124.988 ms  128.812 ms
129.594 ms
11  sl-timewarner-12-0.sprintlink.net (160.81.200.214)  126.898 ms
149.349 ms  114.960 ms
12  64-132-26-250.gen.twtelecom.net (64.132.26.250)  116.782 ms  140.489
ms  123.899 ms
13  kpext.ksbe.edu (216.136.57.178)  165.563 ms  131.212 ms  118.557 ms
14  www.onenet-usa.net (66.193.144.22)  155.675 ms  140.607 ms  175.878 ms

	Any ideas why the machines behind the FreeBSD box shows the
1000+ms latency after it reaches the FreeBSD box when the upstream pipe is
being used but the speeds are working correctly?  Thanks!


Cheers,
Vince - vince@WURLDLINK.NET - Vice President             ________   __ ____
Unix Networking Operations - FreeBSD-Real Unix for Free / / / / |  / |[__  ]
WurldLink Corporation                                  / / / /  | /  | __] ]
San Francisco - Honolulu - Hong Kong                  / / / / / |/ / | __] ]
HongKong Stars/Gravis UltraSound Mailing Lists Admin /_/_/_/_/|___/|_|[____]
Almighty1@IRC - oahu.DAL.NET Hawaii's DALnet IRC Network Server Admin




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20040320011336.C8264-100000>