From owner-freebsd-current@FreeBSD.ORG Sat Mar 13 17:06:21 2010 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B8F23106566C for ; Sat, 13 Mar 2010 17:06:21 +0000 (UTC) (envelope-from ianf@clue.co.za) Received: from inbound01.jnb1.gp-online.net (inbound01.jnb1.gp-online.net [41.161.16.135]) by mx1.freebsd.org (Postfix) with ESMTP id D93888FC14 for ; Sat, 13 Mar 2010 17:06:20 +0000 (UTC) Received: from [41.154.88.19] (helo=clue.co.za) by inbound01.jnb1.gp-online.net with esmtpsa (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.63) (envelope-from ) id 1NqUn3-00033C-EK; Sat, 13 Mar 2010 19:06:17 +0200 Received: from localhost ([127.0.0.1] helo=clue.co.za) by clue.co.za with esmtp (Exim 4.69 (FreeBSD)) (envelope-from ) id 1NqUmz-000BCm-Fn; Sat, 13 Mar 2010 19:06:13 +0200 To: "David Christensen" From: Ian FREISLICH In-Reply-To: <5D267A3F22FD854F8F48B3D2B52381933B0A21DB97@IRVEXCHCCR01.corp.ad.broadcom.com> References: <5D267A3F22FD854F8F48B3D2B52381933B0A21DB97@IRVEXCHCCR01.corp.ad.broadcom.com> <20100310230220.GI10657@michelle.cdnetworks.com> <20100309212139.GO1311@michelle.cdnetworks.com> <5D267A3F22FD854F8F48B3D2B52381933AF90EED69@IRVEXCHCCR01.corp.ad.broadcom.com> <20100309214012.GQ1311@michelle.cdnetworks.com> <5D267A3F22FD854F8F48B3D2B52381933AF90EF16F@IRVEXCHCCR01.corp.ad.broadcom.com> <20100310195206.GB10657@michelle.cdnetworks.com> <5D267A3F22FD854F8F48B3D2B52381933AF90EF25A@IRVEXCHCCR01.corp.ad.broadcom.com> X-Attribution: BOFH Date: Sat, 13 Mar 2010 19:06:13 +0200 Message-Id: Cc: "pyunyh@gmail.com" , "current@freebsd.org" Subject: Re: dev.bce.X.com_no_buffers increasing and packet loss X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Mar 2010 17:06:21 -0000 "David Christensen" wrote: > > Pyun YongHyeon wrote: > > > On Wed, Mar 10, 2010 at 02:45:47PM -0800, David Christensen wrote: > > > > The bce(4) hardware supports a linked list of pages for RX buffer=20 > > > > descriptors. The stock build supports 2 pages (RX_PAGES) with a=20 > > > > total of 511 BD's per page. The hardware can support a=20 > > maximum of=20 > > > > 64K BD's but that would be an unnecessarily large amount of mbufs=20 > > > > for an infrequent problem. > >=20 > > I think that depends on how you define infrequent. Our use=20 > > case is a largish core router. It's highly likely that we'll=20 > > see this again and again in various packet storms on our network. > >=20 > > Are the packet storms always always from the same host or do they come > from multiple hosts? The hardware supports RSS which can spread the > network load across multiple receive queues and multiple CPU cores, but > only when the traffic is spread across several hosts. (The current > bce(4) driver doesn't include support for RSS.) If a storm of > small frames comes from a single host then almost all adapters will be > challenged to handle the flow. In this case the storm only involved 2 hosts. While it's an exceptional circumstance it isn't unusual in our environment (core router in a datacenter). Fortuately we controlled both machines in this instance. Perhaps if the load is spread across more CPUs, then perhaps only those flows unlucky to hash to the CPU handling the storm will be degraded. That is a marginally better situation than all flows being degraded. From the sounds of it RSS isn't the cure for this particular situation, but may improve performance in general. It does sound like reworking the buffer will solve the problem. Perhaps having a 2 recieve rings so that once one ring is available for processing, the other ready filled and clear ring can be used for recieving frames. Ian -- Ian Freislich