From owner-freebsd-net@FreeBSD.ORG Sat Mar 12 17:35:50 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1984B106564A for ; Sat, 12 Mar 2011 17:35:50 +0000 (UTC) (envelope-from dudu@dudu.ro) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id E07B88FC0C for ; Sat, 12 Mar 2011 17:35:49 +0000 (UTC) Received: by iyj12 with SMTP id 12so4354547iyj.13 for ; Sat, 12 Mar 2011 09:35:49 -0800 (PST) Received: by 10.43.63.66 with SMTP id xd2mr11723774icb.76.1299951349149; Sat, 12 Mar 2011 09:35:49 -0800 (PST) MIME-Version: 1.0 Received: by 10.42.241.3 with HTTP; Sat, 12 Mar 2011 09:35:09 -0800 (PST) In-Reply-To: References: From: Vlad Galu Date: Sat, 12 Mar 2011 18:35:09 +0100 Message-ID: To: freebsd-net@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Re: bge(4) on RELENG_8 mbuf cluster starvation X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Mar 2011 17:35:50 -0000 On Sat, Mar 12, 2011 at 10:03 AM, Vlad Galu wrote: > Hi folks, > > On a fairly busy recent (r219010) RELENG_8 machine I keep getting > -- cut here -- > 1096/1454/2550 mbufs in use (current/cache/total) > 1035/731/1766/262144 mbuf clusters in use (current/cache/total/max) > 1035/202 mbuf+clusters out of packet secondary zone in use (current/cache) > 0/117/117/12800 4k (page size) jumbo clusters in use > (current/cache/total/max) > 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) > 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) > 2344K/2293K/4637K bytes allocated to network (current/cache/total) > 0/70128196/37726935 requests for mbufs denied > (mbufs/clusters/mbuf+clusters) > ^^^^^^^^^^^^^^^^^^^^^ > -- and here -- > > The interfaces are detected as BCM5750: > -- cut here -- > bge0: > mem 0xc0200000-0xc020ffff irq 16 at device 0.0 on pci4 > bge0: CHIP ID 0x00004101; ASIC REV 0x04; CHIP REV 0x41; PCI-E > miibus0: on bge0 > brgphy0: PHY 1 on miibus0 > brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, > 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow > bge0: Ethernet address: 00:11:25:22:0d:ec > bge1: > mem 0xc0300000-0xc030ffff irq 17 at device 0.0 on pci5 > bge1: CHIP ID 0x00004101; ASIC REV 0x04; CHIP REV 0x41; PCI-E > miibus1: on bge1 > brgphy1: PHY 1 on miibus1 > brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, > 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow > bge1: Ethernet address: 00:11:25:22:0d:ed > -- and here -- > > but pciconf says otherwise: > -- cut here -- > bge0@pci0:4:0:0: class=0x020000 card=0x02c61014 chip=0x165914e4 > rev=0x11 hdr=0x00 > vendor = 'Broadcom Corporation' > device = 'NetXtreme Gigabit Ethernet PCI Express (BCM5721)' > class = network > subclass = ethernet > bge1@pci0:5:0:0: class=0x020000 card=0x02c61014 chip=0x165914e4 > rev=0x11 hdr=0x00 > vendor = 'Broadcom Corporation' > device = 'NetXtreme Gigabit Ethernet PCI Express (BCM5721)' > class = network > subclass = ethernet > -- and here -- > > kern.ipc.nmbclusters is set to 131072. Other settings: > -- cut here -- > net.inet.tcp.rfc1323: 1 > net.inet.tcp.mssdflt: 512 > net.inet.tcp.keepidle: 7200000 > net.inet.tcp.keepintvl: 75000 > net.inet.tcp.sendspace: 262144 > net.inet.tcp.recvspace: 262144 > net.inet.tcp.keepinit: 75000 > net.inet.tcp.delacktime: 100 > net.inet.tcp.hostcache.purge: 0 > net.inet.tcp.hostcache.prune: 300 > net.inet.tcp.hostcache.expire: 3600 > net.inet.tcp.hostcache.count: 389 > net.inet.tcp.hostcache.bucketlimit: 30 > net.inet.tcp.hostcache.hashsize: 512 > net.inet.tcp.hostcache.cachelimit: 15360 > net.inet.tcp.read_locking: 1 > net.inet.tcp.recvbuf_max: 262144 > net.inet.tcp.recvbuf_inc: 16384 > net.inet.tcp.recvbuf_auto: 1 > net.inet.tcp.insecure_rst: 0 > net.inet.tcp.ecn.maxretries: 1 > net.inet.tcp.ecn.enable: 0 > net.inet.tcp.abc_l_var: 2 > net.inet.tcp.rfc3465: 1 > net.inet.tcp.rfc3390: 0 > net.inet.tcp.rfc3042: 0 > net.inet.tcp.drop_synfin: 1 > net.inet.tcp.delayed_ack: 1 > net.inet.tcp.blackhole: 2 > net.inet.tcp.log_in_vain: 0 > net.inet.tcp.sendbuf_max: 262144 > net.inet.tcp.sendbuf_inc: 8192 > net.inet.tcp.sendbuf_auto: 1 > net.inet.tcp.tso: 1 > net.inet.tcp.newreno: 1 > net.inet.tcp.local_slowstart_flightsize: 4 > net.inet.tcp.slowstart_flightsize: 4 > net.inet.tcp.path_mtu_discovery: 1 > net.inet.tcp.reass.overflows: 958 > net.inet.tcp.reass.cursegments: 0 > net.inet.tcp.reass.maxsegments: 16464 > net.inet.tcp.sack.globalholes: 0 > net.inet.tcp.sack.globalmaxholes: 65536 > net.inet.tcp.sack.maxholes: 128 > net.inet.tcp.sack.enable: 1 > net.inet.tcp.inflight.stab: 20 > net.inet.tcp.inflight.max: 1073725440 > net.inet.tcp.inflight.min: 6144 > net.inet.tcp.inflight.rttthresh: 10 > net.inet.tcp.inflight.debug: 0 > net.inet.tcp.inflight.enable: 0 > net.inet.tcp.isn_reseed_interval: 0 > net.inet.tcp.icmp_may_rst: 1 > net.inet.tcp.pcbcount: 924 > net.inet.tcp.do_tcpdrain: 1 > net.inet.tcp.tcbhashsize: 512 > net.inet.tcp.log_debug: 0 > net.inet.tcp.minmss: 216 > net.inet.tcp.syncache.rst_on_sock_fail: 0 > net.inet.tcp.syncache.rexmtlimit: 3 > net.inet.tcp.syncache.hashsize: 512 > net.inet.tcp.syncache.count: 0 > net.inet.tcp.syncache.cachelimit: 15360 > net.inet.tcp.syncache.bucketlimit: 30 > net.inet.tcp.syncookies_only: 0 > net.inet.tcp.syncookies: 1 > net.inet.tcp.timer_race: 0 > net.inet.tcp.finwait2_timeout: 60000 > net.inet.tcp.fast_finwait2_recycle: 1 > net.inet.tcp.always_keepalive: 0 > net.inet.tcp.rexmit_slop: 200 > net.inet.tcp.rexmit_min: 30 > net.inet.tcp.msl: 30000 > net.inet.tcp.nolocaltimewait: 0 > net.inet.tcp.maxtcptw: 13107 > net.inet.udp.checksum: 1 > net.inet.udp.maxdgram: 9216 > net.inet.udp.recvspace: 41600 > net.inet.udp.blackhole: 1 > net.inet.udp.log_in_vain: 0 > net.isr.numthreads: 1 > net.isr.maxprot: 16 > net.isr.defaultqlimit: 256 > net.isr.maxqlimit: 10240 > net.isr.bindthreads: 0 > net.isr.maxthreads: 1 > net.isr.direct: 1 > net.isr.direct_force: 0 > -- and here -- > > The machine normally uses PF so I disabled it for a while, but that didn't > remove the symptom, which didn't occur before - this machine initially ran > RELENG_6, then it was upgraded to RELENG_7 and finally to RELENG_8. Any > insight would be greatly appreciated. > > Thanks, > Vlad > > -- > Good, fast & cheap. Pick any two. > Just wanted to add that this box is not really busy in the traditional meaning. It only handles about 500Kbps of traffic in each direction, but connections are intermittent and shortlived, with about 2000 active at any given time. So my guess is that there's no way for those to max out the nmbclusters. -- Good, fast & cheap. Pick any two.