Date: Thu, 28 Oct 2004 17:19:43 -0700 From: Vincent Poy <vincepoy@gmail.com> To: Brian Fundakowski Feldman <green@freebsd.org> Cc: Andre Oppermann <andre@freebsd.org> Subject: Re: Traffic Shaping not working correctly after ipfw coverted to use pfil_hooks API Message-ID: <429af92e041028171979d4bf42@mail.gmail.com> In-Reply-To: <20041026231745.GE93831@green.homeunix.org> References: <429af92e041020205510c66168@mail.gmail.com> <4177B899.5EC32F5F@freebsd.org> <429af92e04102114472add0e51@mail.gmail.com> <417835C7.7060808@freebsd.org> <429af92e04102404115bc7bc80@mail.gmail.com> <20041026133043.A24138@xorpc.icir.org> <20041026231745.GE93831@green.homeunix.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, 26 Oct 2004 19:17:46 -0400, Brian Fundakowski Feldman
<green@freebsd.org> wrote:
> On Tue, Oct 26, 2004 at 01:30:43PM -0700, Luigi Rizzo wrote:
>
>
> > On Sun, Oct 24, 2004 at 04:37:32PM +0200, Andre Oppermann wrote:
> > > [bouncing over to Luigi]
> > >
> > > Luigi, do you have any idea what might be going wrong here?
> >
> > no, sorry... I have to say the ipfw/natd/dummynet configuration is rather
> > convoluted here so it is a bit hard to tell whether the
> > problem is in dummynet calls or divert sockets.
> >
> > I am also confused by the numbers in the initial report:
> >
> > > > > >>Vincent Poy wrote:
> > > > > >>
> > > > > >>>However, after the latest -CURRENT upgrade, it will do 200KB/sec down
> > > > > >>>and 52KB/sec up. If I only download only, then it does show
> > > > > >>>650KB/sec. Normally, when I change the bandwidth to a number lower
> > > > > >>>than 480Kbps for the pipe, the download speeds would go up when
> > > > > >>>downloading. However, I have tried in 10kbps steps down to 350kbps
> > > > > >>>but it still did not top 200KB/sec in downloading.
> >
> > there is a mix of two different notations, Kbps and KB/sec, and
> > i cannot make sense of them.
> > Finally, I am curious as to why one would mix the upload and download
> > traffic, i believe *DSL data rates are independent in the two
> > directions unlike analog modems...
>
> In ip_divert.c the behavior may have changed when I implemented diverted
> rules, in divert_output(): the cookie (ipfw rule number) can be added
> and set to sin->sin_port if the sockaddr_in used for hinting the
> interface has a cleared sin->sin_addr field, but has not cleared the
> sin->sin_port. This specifically means for diverted output, and not
> loopback, packets could be skipping rules when they shouldn't be.
>
> Could you try modifying that function in src/sys/netinet/ip_divert.c
> so that "dt->cookie = sin->sin_port;" changes to
> if (sin->sin_addr.s_addr != 0)
> dt->cookie = sin->sin_port;
> and see if that changes things?
Before I did the modifying of the function, I updated to the latest
-CURRENT of 1AM PDT October 27, 2004 and this is what I did:
*** /usr/src/sys/netinet/ip_divert.c.old Wed Oct 27 00:30:59 2004
--- /usr/src/sys/netinet/ip_divert.c Thu Oct 28 15:46:25 2004
***************
*** 297,303 ****
if (sin) {
int i;
! dt->cookie = sin->sin_port;
/*
* Find receive interface with the given name, stuffed
* (if it exists) in the sin_zero[] field.
--- 297,304 ----
if (sin) {
int i;
! if (sin->sin_addr.s_addr != 0)
! dt->cookie = sin->sin_port;
/*
* Find receive interface with the given name, stuffed
* (if it exists) in the sin_zero[] field.
Recompiled kernel and rebooted.
As I mentioned before, the ADSL circuit is 6016Kbps/608Kbps on SBC's
ATM Network so there is a 13% overhead due to ATM making it
5000Kbps/520Kbps effective.
ipfw rules are as follows as it's been simplified, I am only using
ipfw2/dummynet so that ACK packets will have priority if I am
downloading while uploading. I'm using 400Kbit/s as the upload limit
so that downloads will be at their maximum while uploading.
${fwcmd} enable one_pass
# Define our upload pipe
${fwcmd} pipe 1 config bw 400Kbit/s
${fwcmd} add 49 skipto 100 ip from 208.201.244.224/29 to any
${fwcmd} add 50 divert natd all from any to any via ${natd_interface}
${fwcmd} add 100 pass all from any to any via lo0
${fwcmd} add 200 deny all from any to 127.0.0.0/8
${fwcmd} add 300 deny ip from 127.0.0.0/8 to any
# Route LAN and RFC1918 networks without Traffic Shaping
${fwcmd} add 63000 allow all from any to 10.0.0.0/8 out
${fwcmd} add 63001 allow all from any to 172.16.0.0/12 out
${fwcmd} add 63002 allow all from any to 192.168.0.0/16 out
${fwcmd} add 63003 allow all from any to 208.201.244.224/29 out
# Assign outgoing empty/small ACK packets to the high-priority queue
${fwcmd} add 63004 set 0 queue 1 tcp from any to any tcpflags ack out
# Assign outgoing UDP (DNS/gaming) and SSH traffic to the
medium-high-priority queue
${fwcmd} add 63005 set 0 queue 2 tcp from any to any 22,23 out
${fwcmd} add 63006 set 0 queue 2 udp from any to any not 80,443 out
# Assign outgoing HTTP/HTTPS WEB traffic to the medium-low-priority queue
${fwcmd} add 63007 set 0 queue 3 all from any to any 80,443 out
# Assign all other outgoing traffic to the low-priority queue
${fwcmd} add 63008 set 0 queue 4 all from any to any out
And ofcourse the default open rules of:
${fwcmd} add 65000 allow ip from any to any
${fwcmd} add 65535 deny ip from any to any
The ftp tests I am doing as follows is directly between me and my
ISP's ftp server:
traceroute to ftp.sonic.net (208.201.249.234), 64 hops max, 40 byte packets
1 adsl-208-201-244-1.sonic.net (208.201.244.1) 8.201 ms 8.162 ms 8.260 ms
2 fast0-1.dist2-1.sr.sonic.net (208.201.224.160) 8.873 ms 8.749 ms 9.525 ms
3 eth0.a.ftp.sonic.net (208.201.249.234) 8.625 ms 9.287 ms 8.104 ms
The results were the same with and without the change that Brian had
mentioned above.
This is what the maximum speed of the download/upload pipes. The upload pipe
is limited at 400Kbit/sec due to the ipfw pipe config.
Upload only:
10485760 bytes sent in 03:40 (46.37 KB/s)
Download only:
10485760 bytes received in 00:16 (616.48 KB/s)
Here is where the problem is if I do a upload and download at the same time:
Upload:
10485760 bytes sent in 04:01 (42.41 KB/s)
Download:
10485760 bytes received in 00:59 (172.91 KB/s)
It appears that somehow the ACKs were not sent with priority on the
upload pipe while downloading which is slowing the download side to
less than 33% speed of the pipe.
Here are the ipfw pipe and queue's:
root@bigbang [5:11pm][/home/vince] >> ipfw pipe show
00001: 400.000 Kbit/s 0 ms 50 sl. 0 queues (1 buckets) droptail
mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
q00001: weight 100 pipe 1 50 sl. 1 queues (1 buckets) droptail
mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
0 tcp 208.201.244.226/1449 64.12.185.119/80 96816 57368081 39 13603 311
q00002: weight 66 pipe 1 50 sl. 1 queues (1 buckets) droptail
mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
0 udp 208.201.244.225/1026 192.35.51.30/53 1092 98426 0 0 0
q00003: weight 33 pipe 1 50 sl. 1 queues (1 buckets) droptail
mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
0 tcp 208.201.244.226/1449 64.12.185.119/80 396 20592 0 0 0
q00004: weight 1 pipe 1 50 sl. 1 queues (1 buckets) droptail
mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
0 icmp 208.201.244.225/0 80.138.73.42/0 235 15900 0 0 0
root@bigbang [5:14pm][/home/vince] >> ipfw queue show
q00001: weight 100 pipe 1 50 sl. 1 queues (1 buckets) droptail
mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
0 tcp 208.201.244.226/1449 64.12.185.119/80 98271 58076455 29 35976 324
q00002: weight 66 pipe 1 50 sl. 1 queues (1 buckets) droptail
mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
0 udp 208.201.244.225/1026 192.35.51.30/53 1105 99511 0 0 0
q00003: weight 33 pipe 1 50 sl. 1 queues (1 buckets) droptail
mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
0 tcp 208.201.244.226/1449 64.12.185.119/80 396 20592 0 0 0
q00004: weight 1 pipe 1 50 sl. 1 queues (1 buckets) droptail
mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
0 icmp 208.201.244.225/0 80.138.73.42/0 237 16045 0 0 0
Which seems like it's sending everything through queue 1 when
uploading/downloading instead of ACKs only on queue 1 and the other
data on queue 4.
Hope this explains things better for Luigi.
Cheers,
Vince
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?429af92e041028171979d4bf42>