From owner-freebsd-net@FreeBSD.ORG Wed Mar 7 03:29:25 2012 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C0C21106566B; Wed, 7 Mar 2012 03:29:25 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-pw0-f54.google.com (mail-pw0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id 834E98FC18; Wed, 7 Mar 2012 03:29:25 +0000 (UTC) Received: by pbcwz17 with SMTP id wz17so147217pbc.13 for ; Tue, 06 Mar 2012 19:29:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=sV0+F5hLPGrlTaYd6HCfeyhi4RWYXMSIVREOzXuHTko=; b=iGRPvG0nHq0YMsJ8RpLTgHYa1QP9wke/oueBA2K1P+hz39uzuiBABKrfUn1SvQJuy5 UlAcK8fJpwChbwG1YZCeUJvqdAhZyJeQ2vUvShJBbiBhld8go4lWKf20yZTZQbHnFRNP WBDWE5hH2fe1aBsekXofN954HeRw1TpL358Wh01KGyaIOrZUdxmn0KJYcLH66obxwdoF T/AmmUvfqJitkNrVhymN2Q2gyWmdAr6vcSkQTeKf5tUyDfkmoa3P/lM9lk9qBmjWXlgo bseJzInYKnswVcvgE5cuupKNTCBMgGmUXXUIA39ddog4yuyKAvg6u8Vf0Yzr1qoBXIvZ k7vQ== Received: by 10.68.130.72 with SMTP id oc8mr1374668pbb.115.1331090959407; Tue, 06 Mar 2012 19:29:19 -0800 (PST) Received: from pyunyh@gmail.com ([114.111.62.249]) by mx.google.com with ESMTPS id j3sm16046pbb.29.2012.03.06.19.29.16 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 06 Mar 2012 19:29:18 -0800 (PST) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Wed, 07 Mar 2012 12:29:14 -0800 From: YongHyeon PYUN Date: Wed, 7 Mar 2012 12:29:14 -0800 To: Eugene Grosbein Message-ID: <20120307202914.GB9436@michelle.cdnetworks.com> References: <4F5608EA.6080705@rdtc.ru> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4F5608EA.6080705@rdtc.ru> User-Agent: Mutt/1.4.2.3i Cc: marius@freebsd.org, yongari@freebsd.org, "net@freebsd.org" Subject: Re: suboptimal bge(4) BCM5704 performance in RELENG_8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Mar 2012 03:29:25 -0000 On Tue, Mar 06, 2012 at 07:54:02PM +0700, Eugene Grosbein wrote: > Hi! > > Yesterday I've updated old HP ProLiant DL360 G4p to 8.3-PRELELEASE/amd64 > running busy icecast2 server in hope it can saturate 1G bge(4) link. > > This server has PCI-X connected HP NC7782 Gigabit Server Adapter (BCM5704). Would you show me the output of dmesg(bge(4) and brgphy(4) related ones)? > Is it supposed to emit more than 540Mbit/s with average packet size equal to 1300? > Yes. > netstat shows increasing "Drop" counters: > > # netstat -id > Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll Drop > bge0 1500 00:18:fe:86:ab:f4 635577217 0 0 1047897138 0 0 454509 > > vmstat shows relatively high but pretty sane interrupt rate: > > # vmstat -i | grep bge > irq25: bge0 232949783 10149 > > Input traffic is less than 20Mbit/s meantime, output does not go above 540Mbit/s. > Its mbufs usage is normal (server has 2G RAM): > > # netstat -m > 13459/15986/29445 mbufs in use (current/cache/total) > 13322/13112/26434/262144 mbuf clusters in use (current/cache/total/max) > 13322/13046 mbuf+clusters out of packet secondary zone in use (current/cache) > 0/104/104/131072 4k (page size) jumbo clusters in use (current/cache/total/max) > 0/0/0/65536 9k jumbo clusters in use (current/cache/total/max) > 0/0/0/32768 16k jumbo clusters in use (current/cache/total/max) > 30008K/30636K/60645K bytes allocated to network (current/cache/total) > 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > 0/0/0 requests for jumbo clusters denied (4k/9k/16k) > 0/0/0 sfbufs in use (current/peak/max) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 0 requests for I/O initiated by sendfile > 0 calls to protocol drain routines > > I've glanced at bge(4) code and see it uses lots of hardcoded constants > for rx/tx descriptor rings, for interrupt moderation and for interface FIFO queue. Firmware has a fixed number of descriptors so increasing them wouldn't give better numbers. > And no loader tunnables/sysctls like em/igb have. > Show me the output of "sysctl dev.bge.0.stats". > Should I try to play with constants in the code and if so, what are limits of this chip? > Or it will never be capable of utilizing full gigabit speed? BCM5704 is old controller but it should have no problems to saturate the link. I suspect it could be related with DMA configuration but needs more information. > > Eugene Grosbein