From owner-freebsd-current@FreeBSD.ORG Fri Oct 29 02:07:19 2004 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id F31F916A4CF for ; Fri, 29 Oct 2004 02:07:18 +0000 (GMT) Received: from wproxy.gmail.com (wproxy.gmail.com [64.233.184.199]) by mx1.FreeBSD.org (Postfix) with ESMTP id B58E443D49 for ; Fri, 29 Oct 2004 02:07:17 +0000 (GMT) (envelope-from vincepoy@gmail.com) Received: by wproxy.gmail.com with SMTP id 64so576984wri for ; Thu, 28 Oct 2004 19:07:07 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:reply-to:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:references; b=obwLEmvKokhvDCUmjbKriNySAe2578vjMJ0NBMLzi1AaYjtV7o1ABsovS0cRuRRyIwoedCSVWOqM9EJSWGi9BCa3/AxH9SItj353FyWEtzjHzQ7fbrqAJMAgA7ifxSN12DiVFbAL8WrqFqka+jN6zEb/FhBv3uXTgN7a4OKQx64= Received: by 10.38.163.52 with SMTP id l52mr2409363rne; Thu, 28 Oct 2004 19:07:06 -0700 (PDT) Received: by 10.38.14.49 with HTTP; Thu, 28 Oct 2004 19:07:05 -0700 (PDT) Message-ID: <429af92e04102819074e7b1311@mail.gmail.com> Date: Thu, 28 Oct 2004 19:07:05 -0700 From: Vincent Poy To: Brian Fundakowski Feldman In-Reply-To: <429af92e041028171979d4bf42@mail.gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit References: <429af92e041020205510c66168@mail.gmail.com> <4177B899.5EC32F5F@freebsd.org> <429af92e04102114472add0e51@mail.gmail.com> <417835C7.7060808@freebsd.org> <429af92e04102404115bc7bc80@mail.gmail.com> <20041026133043.A24138@xorpc.icir.org> <20041026231745.GE93831@green.homeunix.org> <429af92e041028171979d4bf42@mail.gmail.com> cc: Luigi Rizzo cc: freebsd-current@freebsd.org cc: Andre Oppermann Subject: Re: Traffic Shaping not working correctly after ipfw coverted to use pfil_hooks API X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: Vincent Poy List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Oct 2004 02:07:19 -0000 On Thu, 28 Oct 2004 17:19:43 -0700, Vincent Poy wrote: > On Tue, 26 Oct 2004 19:17:46 -0400, Brian Fundakowski Feldman > wrote: > > On Tue, Oct 26, 2004 at 01:30:43PM -0700, Luigi Rizzo wrote: > > > > > > > On Sun, Oct 24, 2004 at 04:37:32PM +0200, Andre Oppermann wrote: > > > > [bouncing over to Luigi] > > > > > > > > Luigi, do you have any idea what might be going wrong here? > > > > > > no, sorry... I have to say the ipfw/natd/dummynet configuration is rather > > > convoluted here so it is a bit hard to tell whether the > > > problem is in dummynet calls or divert sockets. > > > > > > I am also confused by the numbers in the initial report: > > > > > > > > > >>Vincent Poy wrote: > > > > > > >> > > > > > > >>>However, after the latest -CURRENT upgrade, it will do 200KB/sec down > > > > > > >>>and 52KB/sec up. If I only download only, then it does show > > > > > > >>>650KB/sec. Normally, when I change the bandwidth to a number lower > > > > > > >>>than 480Kbps for the pipe, the download speeds would go up when > > > > > > >>>downloading. However, I have tried in 10kbps steps down to 350kbps > > > > > > >>>but it still did not top 200KB/sec in downloading. > > > > > > there is a mix of two different notations, Kbps and KB/sec, and > > > i cannot make sense of them. > > > Finally, I am curious as to why one would mix the upload and download > > > traffic, i believe *DSL data rates are independent in the two > > > directions unlike analog modems... > > > > In ip_divert.c the behavior may have changed when I implemented diverted > > rules, in divert_output(): the cookie (ipfw rule number) can be added > > and set to sin->sin_port if the sockaddr_in used for hinting the > > interface has a cleared sin->sin_addr field, but has not cleared the > > sin->sin_port. This specifically means for diverted output, and not > > loopback, packets could be skipping rules when they shouldn't be. > > > > Could you try modifying that function in src/sys/netinet/ip_divert.c > > so that "dt->cookie = sin->sin_port;" changes to > > if (sin->sin_addr.s_addr != 0) > > dt->cookie = sin->sin_port; > > and see if that changes things? > > Before I did the modifying of the function, I updated to the latest > -CURRENT of 1AM PDT October 27, 2004 and this is what I did: > > *** /usr/src/sys/netinet/ip_divert.c.old Wed Oct 27 00:30:59 2004 > --- /usr/src/sys/netinet/ip_divert.c Thu Oct 28 15:46:25 2004 > *************** > *** 297,303 **** > if (sin) { > int i; > > ! dt->cookie = sin->sin_port; > /* > * Find receive interface with the given name, stuffed > * (if it exists) in the sin_zero[] field. > --- 297,304 ---- > if (sin) { > int i; > > ! if (sin->sin_addr.s_addr != 0) > ! dt->cookie = sin->sin_port; > /* > * Find receive interface with the given name, stuffed > * (if it exists) in the sin_zero[] field. > > Recompiled kernel and rebooted. > > As I mentioned before, the ADSL circuit is 6016Kbps/608Kbps on SBC's > ATM Network so there is a 13% overhead due to ATM making it > 5000Kbps/520Kbps effective. > > ipfw rules are as follows as it's been simplified, I am only using > ipfw2/dummynet so that ACK packets will have priority if I am > downloading while uploading. I'm using 400Kbit/s as the upload limit > so that downloads will be at their maximum while uploading. > > ${fwcmd} enable one_pass > # Define our upload pipe > ${fwcmd} pipe 1 config bw 400Kbit/s > ${fwcmd} add 49 skipto 100 ip from 208.201.244.224/29 to any > ${fwcmd} add 50 divert natd all from any to any via ${natd_interface} > ${fwcmd} add 100 pass all from any to any via lo0 > ${fwcmd} add 200 deny all from any to 127.0.0.0/8 > ${fwcmd} add 300 deny ip from 127.0.0.0/8 to any > # Route LAN and RFC1918 networks without Traffic Shaping > ${fwcmd} add 63000 allow all from any to 10.0.0.0/8 out > ${fwcmd} add 63001 allow all from any to 172.16.0.0/12 out > ${fwcmd} add 63002 allow all from any to 192.168.0.0/16 out > ${fwcmd} add 63003 allow all from any to 208.201.244.224/29 out > # Assign outgoing empty/small ACK packets to the high-priority queue > ${fwcmd} add 63004 set 0 queue 1 tcp from any to any tcpflags ack out > # Assign outgoing UDP (DNS/gaming) and SSH traffic to the > medium-high-priority queue > ${fwcmd} add 63005 set 0 queue 2 tcp from any to any 22,23 out > ${fwcmd} add 63006 set 0 queue 2 udp from any to any not 80,443 out > # Assign outgoing HTTP/HTTPS WEB traffic to the medium-low-priority queue > ${fwcmd} add 63007 set 0 queue 3 all from any to any 80,443 out > # Assign all other outgoing traffic to the low-priority queue > ${fwcmd} add 63008 set 0 queue 4 all from any to any out > And ofcourse the default open rules of: > ${fwcmd} add 65000 allow ip from any to any > ${fwcmd} add 65535 deny ip from any to any > > The ftp tests I am doing as follows is directly between me and my > ISP's ftp server: > > traceroute to ftp.sonic.net (208.201.249.234), 64 hops max, 40 byte packets > 1 adsl-208-201-244-1.sonic.net (208.201.244.1) 8.201 ms 8.162 ms 8.260 ms > 2 fast0-1.dist2-1.sr.sonic.net (208.201.224.160) 8.873 ms 8.749 ms 9.525 ms > 3 eth0.a.ftp.sonic.net (208.201.249.234) 8.625 ms 9.287 ms 8.104 ms > > The results were the same with and without the change that Brian had > mentioned above. > > This is what the maximum speed of the download/upload pipes. The upload pipe > is limited at 400Kbit/sec due to the ipfw pipe config. > > Upload only: > 10485760 bytes sent in 03:40 (46.37 KB/s) > Download only: > 10485760 bytes received in 00:16 (616.48 KB/s) > > Here is where the problem is if I do a upload and download at the same time: > > Upload: > 10485760 bytes sent in 04:01 (42.41 KB/s) > Download: > 10485760 bytes received in 00:59 (172.91 KB/s) > > It appears that somehow the ACKs were not sent with priority on the > upload pipe while downloading which is slowing the download side to > less than 33% speed of the pipe. > > Here are the ipfw pipe and queue's: > > root@bigbang [5:11pm][/home/vince] >> ipfw pipe show > 00001: 400.000 Kbit/s 0 ms 50 sl. 0 queues (1 buckets) droptail > mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000 > q00001: weight 100 pipe 1 50 sl. 1 queues (1 buckets) droptail > mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000 > BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp > 0 tcp 208.201.244.226/1449 64.12.185.119/80 96816 57368081 39 13603 311 > q00002: weight 66 pipe 1 50 sl. 1 queues (1 buckets) droptail > mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000 > BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp > 0 udp 208.201.244.225/1026 192.35.51.30/53 1092 98426 0 0 0 > q00003: weight 33 pipe 1 50 sl. 1 queues (1 buckets) droptail > mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000 > BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp > 0 tcp 208.201.244.226/1449 64.12.185.119/80 396 20592 0 0 0 > q00004: weight 1 pipe 1 50 sl. 1 queues (1 buckets) droptail > mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000 > BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp > 0 icmp 208.201.244.225/0 80.138.73.42/0 235 15900 0 0 0 > > root@bigbang [5:14pm][/home/vince] >> ipfw queue show > q00001: weight 100 pipe 1 50 sl. 1 queues (1 buckets) droptail > mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000 > BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp > 0 tcp 208.201.244.226/1449 64.12.185.119/80 98271 58076455 29 35976 324 > q00002: weight 66 pipe 1 50 sl. 1 queues (1 buckets) droptail > mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000 > BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp > 0 udp 208.201.244.225/1026 192.35.51.30/53 1105 99511 0 0 0 > q00003: weight 33 pipe 1 50 sl. 1 queues (1 buckets) droptail > mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000 > BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp > 0 tcp 208.201.244.226/1449 64.12.185.119/80 396 20592 0 0 0 > q00004: weight 1 pipe 1 50 sl. 1 queues (1 buckets) droptail > mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000 > BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp > 0 icmp 208.201.244.225/0 80.138.73.42/0 237 16045 0 0 0 > > Which seems like it's sending everything through queue 1 when > uploading/downloading instead of ACKs only on queue 1 and the other > data on queue 4. > > Hope this explains things better for Luigi. I just tried this again.... and it seems like the queues are not working at all while it did on the March 6, 2004 -CURRENT since the queue show output appears to not update the source and destination IPs for all of the queues. root@bigbang [6:44pm][/home/vince] >> ipfw queue show q00001: weight 100 pipe 1 50 sl. 1 queues (1 buckets) droptail mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 0 tcp 208.201.244.226/1449 64.12.185.119/80 181047 91968211 0 0 421 q00002: weight 66 pipe 1 50 sl. 1 queues (1 buckets) droptail mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 0 udp 208.201.244.225/1026 192.35.51.30/53 2727 238253 0 0 0 q00003: weight 33 pipe 1 50 sl. 1 queues (1 buckets) droptail mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 0 tcp 208.201.244.226/1449 64.12.185.119/80 2211 114696 0 0 0 q00004: weight 1 pipe 1 50 sl. 1 queues (1 buckets) droptail mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 0 icmp 208.201.244.225/0 80.138.73.42/0 1847 148609 0 0 0 Queue 1 I tested by doing a ftp receive from 208.201.249.234 to 208.201.244.224 which is updating the total pkt/bytes counter but the Source IP and Destination IP didn't change. Queue 2 I tested by querying my DNS server from an outside source. The results were similar with 1 as the total pkt/bytes counter did update but the Source IP and Destination IP didn't change. Queue 3 I tested by downloading a file via http from an outside source. None of the information changed at all as the Source IP and Destination IP didn't change. The total pkt/bytes counter didn't change at all. Queue 4 I tested by sending a file via ftp. The counters change but the protocol shows icmp and with the same Source IP and Destination IP as shown above. Hope this provides more information. Cheers, Vince