From owner-freebsd-ipfw@freebsd.org Fri Jul 31 10:23:12 2015 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4A4519AF81A for ; Fri, 31 Jul 2015 10:23:12 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from sola.nimnet.asn.au (paqi.nimnet.asn.au [115.70.110.159]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 59EE81DC8; Fri, 31 Jul 2015 10:23:10 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from localhost (localhost [127.0.0.1]) by sola.nimnet.asn.au (8.14.2/8.14.2) with ESMTP id t6VAN7bO065578; Fri, 31 Jul 2015 20:23:07 +1000 (EST) (envelope-from smithi@nimnet.asn.au) Date: Fri, 31 Jul 2015 20:23:07 +1000 (EST) From: Ian Smith To: hiren panchasara cc: freebsd-ipfw@FreeBSD.org Subject: Re: Traffic not going through dummynet In-Reply-To: <20150730182551.GF39365@strugglingcoder.info> Message-ID: <20150731171616.J17327@sola.nimnet.asn.au> References: <20150730182551.GF39365@strugglingcoder.info> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 Jul 2015 10:23:12 -0000 On Thu, 30 Jul 2015 11:25:51 -0700, hiren panchasara wrote: > (For various reason's I didn't get/see Ian's message. Trying to do the > right thing by setting "In-Reply-To".) No problem, thanks. > On 07/27/15 at 01:07P, Ian Smith wrote: > > On Sun, 19 Jul 2015 21:05:53 -0700, hiren panchasara wrote: > > > Bah. > > > > > > So I removed ipfw and dummynet from kernconf and loaded them manually > > > after machine came up and it worked as expected. > > > > In your previous post, you'd said you were using 11-current, and: > > > > > And GENERIC has: > > > options IPFIREWALL > > > options DUMMYNET > > > options HZ=1000 > > > > Are you sure this was a 11 GENERIC kernconf? Those options haven't > > been in GENERIC for ages (if ever?), though they haven't needed to be > > since (perhaps) 8.0. I guess people just follow the handbook :( > > I modified GENERIC and added those options. I should have been more > clear here. Yeah, the handbook tends to lead one that way. I 'include GENERIC' and use no{options,device} as needed; saves me getting even more confused :) > > > Looks like some ordering issue between ipfw and dummynet. Fwiw, for > > > working setup, kldstat shows: > > > > > > 13 2 0xffffffff81e21000 21490 ipfw.ko > > > 14 1 0xffffffff81e43000 d0f6 dummynet.ko > > > > Indeed. If you load ipfw and dummynet by the usual means, being > > firewall_enable=YES and dummynet_enable=YES in rc.conf, you'll notice > > that /etc/rc.d/ipfw, in ipfw_prestart, loads dummynet if enabled, and > > natd and/or firewall_nat if enabled, in that order. > > > > The downside to doing that is that you have to have specified a type for > > rc.firewall or pointed to a custom ruleset so it's sane on startup. > > Didn't know the usual mean of rc.conf modifications. Checking /etc/defaults/rc.conf /firewall_, all you'd need for that is: firewall_enable=YES firewall_type=OPEN # permit all, regardless of default_to_accept dummynet_anable=YES which would at least load those modules in the right order, however as shown below, you didn't need to be so fussy, up to 9.3 anyway. > > Regarding the related(?) Bug 201488 - dummynet appears broken in > > 10.0-RELEASE and onwards (can't traffic shape on bridges) > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=201488 > > it does seem likely to be the same issue as you noted. > > > > Did you ever hear back from James Rice (for whom I seem to have seen no > > other messages for an email address) as to whether your advice about > > loading these in the other order helped there? > > I haven't heard back yet. > > > > As to whether this is a regression, or it would have ever worked loading > > dummynet and then ipfw, I don't know, but I have a vague feeling that > > I've seen other issues regarding loading a module that's already in > > kernel in recent times .. sorry I can't be any more exact. Actually the other one was someone loading urtwn with it already in kernel getting bad results; I suggested trying without it in kernel, but haven't heard back either. > Yeah, good point about whether this is a regression or not. I am not > aware of any such loading issues wrt modules. In the olden days, attempting to load a module already in kernel would fail gracefully with a message early on in dmesg. I'm still wondering if there's some issue there. I'll try loading dummynet before ipfw on my 9.3 system using a similar setup to yours such as noticeable delay. > > Maybe dummynet needs a check that ipfw is loaded before starting? > > That'd be logical, imo. Will I ever learn to not suggest things I can't actually fix myself? :) Ok, here's a near-duplicate on 9.3-R of your procedure on 11.0 root@x200:~ # kldstat -v | egrep 'ipfw|dummynet' root@x200:~ # kldload dummynet root@x200:~ # kldstat -v | egrep 'ipfw|dummynet' 9 1 0xffffffff81a3b000 d910 dummynet.ko (/boot/kernel/dummynet.ko) 511 dummynet 10 1 0xffffffff81a49000 12200 ipfw.ko (/boot/kernel/ipfw.ko) 510 ipfw [ note that loading dummynet also loads ipfw - as expected I guess - and that they're here listed in that reverse order numerically, FWIW ] root@x200:~ # ipfw -t show 65535 0 0 deny ip from any to any root@x200:~ # ipfw add 65000 allow ip from any to any 65000 allow ip from any to any root@x200:~ # ipfw -t show 65000 0 0 allow ip from any to any 65535 2 560 Fri Jul 31 19:28:47 2015 deny ip from any to any [ I didn't reboot, so added the allow-all rule. I'd normally use the method of running 'kldload ipfw && ipfw add 32000 ip from any to any' from a remote ssh session, but 'kldload dummynet', since it auto-loads ipfw with default-deny rule, naturally killed my login ] root@x200:~ # ipfw add 32000 pipe 1 ip from any to any 32000 pipe 1 ip from any to any root@x200:~ # ipfw pipe 1 config delay 100ms root@x200:~ # ipfw -t show 32000 0 0 pipe 1 ip from any to any 65000 1 112 Fri Jul 31 19:28:59 2015 allow ip from any to any 65535 2 560 Fri Jul 31 19:28:47 2015 deny ip from any to any root@x200:~ # ipfw pipe show 00001: unlimited 100 ms burst 0 q131073 50 sl. 0 flows (1 buckets) sched 65537 weight 0 lmax 0 pri 0 droptail sched 65537 type FIFO flags 0x0 0 buckets 0 active root@x200:~ # sysctl kern.features | grep ipfw # not on 9.3 root@x200:~ # sysctl net.inet.ip.dummynet | grep io # if relevant net.inet.ip.dummynet.io_pkt_drop: 0 net.inet.ip.dummynet.io_pkt_fast: 0 net.inet.ip.dummynet.io_pkt: 7 net.inet.ip.dummynet.io_fast: 0 root@x200:~ # ping t23 PING t23 (192.168.7.7): 56 data bytes 64 bytes from 192.168.7.7: icmp_seq=0 ttl=64 time=198.792 ms 64 bytes from 192.168.7.7: icmp_seq=1 ttl=64 time=199.997 ms 64 bytes from 192.168.7.7: icmp_seq=2 ttl=64 time=199.884 ms ^C --- t23 ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 198.792/199.558/199.997/0.543 ms root@x200:~ # ipfw disable one_pass # no diff in this context root@x200:~ # ping t23 PING t23 (192.168.7.7): 56 data bytes 64 bytes from 192.168.7.7: icmp_seq=0 ttl=64 time=198.273 ms 64 bytes from 192.168.7.7: icmp_seq=1 ttl=64 time=200.964 ms 64 bytes from 192.168.7.7: icmp_seq=2 ttl=64 time=200.041 ms ^C --- t23 ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 198.273/199.759/200.964/1.117 ms root@x200:~ # ipfw pipe show 00001: unlimited 100 ms burst 0 q131073 50 sl. 0 flows (1 buckets) sched 65537 weight 0 lmax 0 pri 0 droptail sched 65537 type FIFO flags 0x0 0 buckets 0 active root@x200:~ # ipfw -t show 32000 25 4060 Fri Jul 31 19:37:59 2015 pipe 1 ip from any to any 65000 8 728 Fri Jul 31 19:37:59 2015 allow ip from any to any 65535 2 560 Fri Jul 31 19:28:47 2015 deny ip from any to any root@x200:~ # ipfw add 2000 skipto 40000 ip from any to any 02000 skipto 40000 ip from any to any root@x200:~ # ping t23 PING t23 (192.168.7.7): 56 data bytes 64 bytes from 192.168.7.7: icmp_seq=0 ttl=64 time=0.447 ms 64 bytes from 192.168.7.7: icmp_seq=1 ttl=64 time=0.392 ms 64 bytes from 192.168.7.7: icmp_seq=2 ttl=64 time=0.549 ms 64 bytes from 192.168.7.7: icmp_seq=3 ttl=64 time=0.607 ms ^C --- t23 ping statistics --- 4 packets transmitted, 4 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 0.392/0.499/0.607/0.084 ms [ note here I'd bypassed the pipe rule, and unloading dummynet is ok, ie it doesn't just then unload ipfw too ] root@x200:~ # kldunload dummynet root@x200:~ # ping t23 PING t23 (192.168.7.7): 56 data bytes 64 bytes from 192.168.7.7: icmp_seq=0 ttl=64 time=0.512 ms 64 bytes from 192.168.7.7: icmp_seq=1 ttl=64 time=0.392 ms ^C --- t23 ping statistics --- 2 packets transmitted, 2 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 0.392/0.452/0.512/0.060 ms [ but as soon as I delete that bypass rule, it's all over .. ] root@x200:~ # ipfw delete 2000 ipfw: rule 2000: setsockopt(IP_FW_DEL): Protocol not available root@x200:~ # ipfw -t show ipfw: getsockopt(IP_FW_GET): Protocol not available root@x200:~ # kldstat -v | egrep 'ipfw|dummynet' root@x200:~ # ======= So in summary: the behaviour you demonstrated on 11.0 does not occur on 9.3 - assuming I did things in the same order that you did - so it does indeed seem to be a regression. I have no 10.x system to test. cheers, Ian