From owner-freebsd-ipfw@FreeBSD.ORG Wed Mar 12 18:00:02 2008 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 970561065673 for ; Wed, 12 Mar 2008 18:00:02 +0000 (UTC) (envelope-from wadeklaver@itiva.com) Received: from mail.crossflux.com (a2.9.1243.static.theplanet.com [67.18.9.162]) by mx1.freebsd.org (Postfix) with SMTP id 5EBA88FC14 for ; Wed, 12 Mar 2008 18:00:02 +0000 (UTC) (envelope-from wadeklaver@itiva.com) Received: (qmail 25955 invoked by uid 522); 12 Mar 2008 12:22:25 -0500 Received: from 69.10.147.2 by mog (envelope-from , uid 508) with qmail-scanner-1.25 (clamdscan: 0.87/2614. spamassassin: 3.1.8. Clear:RC:0(69.10.147.2):SA:0(-2.6/5.0):. Processed in 1.198514 secs); 12 Mar 2008 17:22:25 -0000 X-Spam-Status: No, hits=-2.6 required=5.0 Received: from unknown (HELO ?192.168.7.50?) (wadeklaver@itiva.com@69.10.147.2) by mail.crossflux.com with SMTP; 12 Mar 2008 12:22:24 -0500 From: Wade Klaver To: freebsd-hackers@freebsd.org Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="=-L/S4LNxbzLpvKH3MrrIG" Date: Wed, 12 Mar 2008 10:33:04 -0700 Message-Id: <1205343184.4032.44.camel@wade-linux.itiva.com> Mime-Version: 1.0 X-Mailer: Evolution 2.8.3 (2.8.3-2.fc6) Cc: freebsd-ipfw@freebsd.org Subject: On the trail of a dummynet/bridge/ipfw bug. X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Mar 2008 18:00:02 -0000 --=-L/S4LNxbzLpvKH3MrrIG Content-Type: text/plain Content-Transfer-Encoding: quoted-printable PROBLEM DESCRIPTION I have a bridge set up on a 7.0 box and am attempting to use it to limit HTTP connections outgoing from a box behind it to 192Kbit/s for testing. During this testing I ran into some problems. At first, I found that the number of simultaneous pipes was limited to 1024, allowing only 1024 192Kbit/s clients. Additional clients were simply blocked. I am using a very simple firewall config: ipfw pipe 1 config bw 192Kbits/s mask all ipfw add 00051 skipto 99 ip from 192.168.0.0/16 to 192.168.0.0/16 ipfw add 00052 skipto 1000 ip from any to any ipfw add 00100 pipe 1 ip from 192.168.10.88 80 to any via bridge0 ipfw add 00200 pipe 1 ip from any 25111 to any via bridge Regardless of how many clients I threw at the box, I had the limit: [root@ibm3550b ~]# ipfw pipe show | wc -l 1028 We managed to track this down to a problem in the ipfw2 userland app. The following patch to /usr/src/sbin/ipfw/ipfw2.c allowed this limit to be surpassed. It would appear that ipfw does not dynamically resize the pipe array beyond the initial 1024 elements allocated. # diff ipfw2.c ipfw2.c.orig=20 2507c2507 < int nalloc =3D 8192; /* start somewhere... */ --- > int nalloc =3D 1024; /* start somewhere... */ However, this just revealed a bigger problem, potentially do to the above patch, potentially due to something worse. Now the bridge will allow more connections, up to around 2300 where the bridge just dies. and no more traffic passes. It is worth noting that I can still connect to the bridge itself if it has an IP assigned to it, but traffic through the bridge ceases. It is also remedied by a /etc/rc.d/netif restart. ADDITIONAL NOTES Please let me know if there is any additional information I can provide. In the kernel options below, HZ=3D2000 was just something I was trying. The problem exhibits itself with HZ=3D1000 as well. I posted this to -hackers and to -ipfw. Please direct me and future corresp= ondence=20 on this issue to the most appropriate list. I just felt it was not solid enough to go to -bugs yet. SYSTEM INFORMATION IBM 3550 XEON 5345 4GB Memory [root@ibm3550b /usr/src/sys]# uname -a FreeBSD ibm3550b.itivalabs.net 7.0-STABLE FreeBSD 7.0-STABLE #13: Wed Mar 12 03:26:08 PDT 2008 root@ibm3550b.itivalabs.net:/usr/obj/usr/src/sys/WADE amd64 Bridge members: bce0: mem 0xc8000000-0xc9ffffff irq 18 at device 0.0 on pci4 bce1: mem 0xce000000-0xcfffffff irq 16 at device 0.0 on pci6 Kernel options: # Make an SMP-capable kernel by default options SMP # Symmetric MultiProcessor Kernel options LIBALIAS options IPFIREWALL #firewall options IPFIREWALL_VERBOSE #enable logging to syslogd(8) options IPFIREWALL_VERBOSE_LIMIT=3D100 #limit verbosity options IPFIREWALL_DEFAULT_TO_ACCEPT #allow everything by default options IPFIREWALL_FORWARD #packet destination changes options IPFIREWALL_NAT #ipfw kernel nat support options IPDIVERT #divert sockets #options IPFILTER #ipfilter support #options IPFILTER_LOG #ipfilter logging #options IPFILTER_LOOKUP #ipfilter pools #options IPFILTER_DEFAULT_BLOCK #block all packets by default options IPSTEALTH #support for stealth forwarding options MBUF_STRESS_TEST options DUMMYNET options HZ=3D2000 options EXT2FS /etc/sysctl.conf net.inet.ip.fw.dyn_max=3D8192 net.inet.ip.dummynet.hash_size=3D4096 net.inet.ip.fw.dyn_buckets=3D1024 net.inet.ip.dummynet.max_chain_len=3D64 kern.ipc.nmbclusters=3D32768 --=-L/S4LNxbzLpvKH3MrrIG Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (GNU/Linux) iD8DBQBH2BPKne3UhGESRwURAkMVAKCVjfXweg6+gXn1c9kAM2v07o4+JgCfXZga Ebsj07bJ4wcROQhzdqVcBOQ= =B200 -----END PGP SIGNATURE----- --=-L/S4LNxbzLpvKH3MrrIG--