From owner-freebsd-current@FreeBSD.ORG Wed Oct 5 15:11:56 2011 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6243D106576E for ; Wed, 5 Oct 2011 15:11:56 +0000 (UTC) (envelope-from ianf@clue.co.za) Received: from brane.freislich.nom.za (www.freislich.nom.za [41.154.0.9]) by mx1.freebsd.org (Postfix) with ESMTP id 7E7928FC17 for ; Wed, 5 Oct 2011 15:11:55 +0000 (UTC) Received: from [10.0.2.227] (helo=clue.co.za) by brane.freislich.nom.za with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.76 (FreeBSD)) (envelope-from ) id 1RBSUQ-000MTZ-66 for current@freebsd.org; Wed, 05 Oct 2011 16:30:30 +0200 Received: from localhost ([127.0.0.1] helo=clue.co.za) by clue.co.za with esmtp (Exim 4.76 (FreeBSD)) (envelope-from ) id 1RBSUJ-0008x8-PC for current@freebsd.org; Wed, 05 Oct 2011 16:30:23 +0200 To: current@freebsd.org From: "Ian FREISLICH" X-Attribution: BOFH Date: Wed, 05 Oct 2011 16:30:23 +0200 Message-Id: X-Missing-rDNS: 10.0.2.227 Cc: Subject: 3 show-stopper issues with 9-BETA3 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Oct 2011 15:11:56 -0000 Hi In no particular order: 1. bce(4) transmit and recieve ring buffer overruns On a moderately busy router with a full BGP table and aggregate throughput of between 200mbps and 800mbps, I get these buffer overruns at an average rate of 28 per second on the busiest interface. [firewall1.jnb1] ~ # sysctl dev.bce |grep com_no_buffers dev.bce.0.com_no_buffers: 101 dev.bce.1.com_no_buffers: 0 dev.bce.2.com_no_buffers: 32547 dev.bce.3.com_no_buffers: 444 I've tried increasing the TX_PAGES and RX_PAGES in sys/dev/bce/if_bcereg.h as I've done in the past (to 64) which is what resolved this problem on 8.2-STABLE to no avail. It appears that there is a hard limit of 8 according to bce_set_tunables() in if_bce.c. But no values to hw.bce.tx_pages and hw.bce.rx_pages makes the slightest difference. 2. carp(4) on my backup router randomly takes over MASTER on the standby host, but when ifconfig claims the carp interface is master tcpdump shows that it's not broadcasting its advertisement. The actual master still broadcasts and no setting of advskew or advbase changes the 9-BETA host's idea of who is actually master. I have to reboot the host to reset the carp interfaces. destroying and re-creating them just brings them up as backup for about a second and then they regress to master. 3. PF doesn't expire state. The state table on my older host (pre OpenBSD-4.5) has the following stats: Status: Enabled for 0 days 00:37:17 Debug: Urgent State Table Total Rate current entries 169546 searches 94387451 42193.8/s inserts 4012389 1793.6/s removals 3842843 1717.9/s The 9-BETA3 host's current entries exactly match the number of inserts until it hits the hard limit of 1.5M entries and can add no more. It takes about 10 minutes to fill up and then no new flows are routed. We're in a quiet period at the moment, so I can keep a 9-X host around for a few days. I'll be able to try things until I have to downgrade the other host at the end of the week. Incompatibility between pf on 8.2-STABLE and 9-X after 2011-06-28 makes testing a little difficult though because I'm not able to synchronise state. FWIW, the tuning that has been done eliminates the issue on 8.2-STABLE: [firewall1.jnb1] ~ # cat /boot/loader.conf net.isr.maxthreads="8" net.isr.defaultqlimit="4096" net.isr.maxqlimit="81920" net.isr.direct="1" kern.ipc.nmbclusters="262144" kern.maxusers="1024" [firewall1.jnb1] ~ # cat /etc/sysctl.conf net.inet.tcp.blackhole=2 net.inet.udp.blackhole=1 net.inet.ip.fastforwarding=1 net.inet.carp.preempt=1 net.inet.icmp.icmplim_output=0 net.inet.icmp.icmplim=0 kern.random.sys.harvest.interrupt=0 kern.random.sys.harvest.ethernet=0 kern.random.sys.harvest.point_to_point=0 net.route.netisr_maxqlen=8192 diff -u -d -r1.26.2.7 if_bcereg.h --- if_bcereg.h 15 Aug 2010 23:56:57 -0000 1.26.2.7 +++ if_bcereg.h 5 Oct 2011 14:29:15 -0000 @@ -6150,7 +6150,7 @@ * Page count must remain a power of 2 for all * of the math to work correctly. */ -#define TX_PAGES 2 +#define TX_PAGES 64 #define TOTAL_TX_BD_PER_PAGE (BCM_PAGE_SIZE / sizeof(struct tx_bd)) #define USABLE_TX_BD_PER_PAGE (TOTAL_TX_BD_PER_PAGE - 1) #define TOTAL_TX_BD (TOTAL_TX_BD_PER_PAGE * TX_PAGES) @@ -6170,7 +6170,7 @@ * Page count must remain a power of 2 for all * of the math to work correctly. */ -#define RX_PAGES 2 +#define RX_PAGES 64 #define TOTAL_RX_BD_PER_PAGE (BCM_PAGE_SIZE / sizeof(struct rx_bd)) #define USABLE_RX_BD_PER_PAGE (TOTAL_RX_BD_PER_PAGE - 1) #define TOTAL_RX_BD (TOTAL_RX_BD_PER_PAGE * RX_PAGES) Ian -- Ian Freislich