From owner-freebsd-net@FreeBSD.ORG Thu Sep 23 20:31:37 2010
Return-Path: <owner-freebsd-net@FreeBSD.ORG>
Delivered-To: freebsd-net@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
by hub.freebsd.org (Postfix) with ESMTP id 43E081065675;
Thu, 23 Sep 2010 20:31:37 +0000 (UTC)
(envelope-from davidch@broadcom.com)
Received: from mms2.broadcom.com (mms2.broadcom.com [216.31.210.18])
by mx1.freebsd.org (Postfix) with ESMTP id 0D57E8FC13;
Thu, 23 Sep 2010 20:31:36 +0000 (UTC)
Received: from [10.9.200.131] by mms2.broadcom.com with ESMTP (Broadcom
SMTP Relay (Email Firewall v6.3.2)); Thu, 23 Sep 2010 13:30:52 -0700
X-Server-Uuid: D3C04415-6FA8-4F2C-93C1-920E106A2031
Received: from IRVEXCHCCR01.corp.ad.broadcom.com ([10.252.49.30]) by
IRVEXCHHUB01.corp.ad.broadcom.com ([10.9.200.131]) with mapi; Thu, 23
Sep 2010 13:30:52 -0700
From: "David Christensen" <davidch@broadcom.com>
To: "Tom Judge" <tom@tomjudge.com>
Date: Thu, 23 Sep 2010 13:30:42 -0700
Thread-Topic: bce(4) - com_no_buffers (Again)
Thread-Index: ActbVVsV3eDkFvsDQ6a67+cfUoq2/QAB+jGw
Message-ID: <5D267A3F22FD854F8F48B3D2B52381933B5A78B544@IRVEXCHCCR01.corp.ad.broadcom.com>
References: <4C894A76.5040200@tomjudge.com>
<20100910002439.GO7203@michelle.cdnetworks.com>
<4C8E3D79.6090102@tomjudge.com>
<20100913184833.GF1229@michelle.cdnetworks.com>
<4C8E768E.7000003@tomjudge.com>
<20100913193322.GG1229@michelle.cdnetworks.com>
<4C8E8BD1.5090007@tomjudge.com>
<20100913205348.GJ1229@michelle.cdnetworks.com>
<4C9B6CBD.2030408@tomjudge.com>
<5D267A3F22FD854F8F48B3D2B52381933B5A78B484@IRVEXCHCCR01.corp.ad.broadcom.com>
<4C9BA9FD.50406@tomjudge.com>
In-Reply-To: <4C9BA9FD.50406@tomjudge.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-cr-hashedpuzzle: EUL4 EbMy GzIr JtLp J9+c LNK1 NJpr Ru1W T1Ye U+JH
a1H7 bcnD e81K hy77 mprW nJkm; 4;
ZgByAGUAZQBiAHMAZAAtAG4AZQB0AEAAZgByAGUAZQBiAHMAZAAuAG8AcgBnADsAcAB5AHUAbgB5AGgAQABnAG0AYQBpAGwALgBjAG8AbQA7AHQAbwBtAEAAdABvAG0AagB1AGQAZwBlAC4AYwBvAG0AOwB5AG8AbgBnAGEAcgBpAEAAZgByAGUAZQBiAHMAZAAuAG8AcgBnAA==;
Sosha1_v1; 7; {A5949329-902B-45B6-A2A3-C6334B20CA88};
ZABhAHYAaQBkAGMAaABAAGIAcgBvAGEAZABjAG8AbQAuAGMAbwBtAA==; Thu,
23 Sep 2010 20:30:42 GMT;
UgBFADoAIABiAGMAZQAoADQAKQAgAC0AIABjAG8AbQBfAG4AbwBfAGIAdQBmAGYAZQByAHMAIAAoAEEAZwBhAGkAbgApAA==
x-cr-puzzleid: {A5949329-902B-45B6-A2A3-C6334B20CA88}
acceptlanguage: en-US
MIME-Version: 1.0
X-WSS-ID: 608567763N04836252-01-01
Content-Type: text/plain;
charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Cc: "pyunyh@gmail.com" <pyunyh@gmail.com>,
"freebsd-net@freebsd.org" <freebsd-net@freebsd.org>,
"yongari@freebsd.org" <yongari@freebsd.org>
Subject: RE: bce(4) - com_no_buffers (Again)
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
<mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
<mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 23 Sep 2010 20:31:37 -0000
> > Failure to allocate a new buffer should cause the driver to
> > drop the received frame and reuse the buffer, not lock up the
> > system. Are you seeing the lockup come from bce(4) or does
> > it come from somewhere else due to the dropped data?
> >
> >
>=20
> The lockup is not from the NIC as such, the systems have the appearance
> of locking up as home directories are on NFS and the user information
> is
> stored in a remote LDAP server. When the system starts to drop frames
> due to lack of 9k memory regions it tends to last for a few minutes
> (when it is really bad) and stop all traffic into the system. This
> appears to the average user as a complete system pause.
>=20
Ack.
>
> I am under the impression that the best solution is to tune the RX ring
> so that flow control can be disabled but I not sure I could do this.
>=20
Are small frames common in the actual failing case or are they=20
restricted to your test case? At least one of the Broadcom
Linux drivers addresses this issue by copying frames < 128 bytes
to a standard size buffer, saving the jumbo buffer allocation=20
and DMA mapping operation. If small frames are at the root of=20
your problem then I think this would be a better solution than=20
allocating more 9KB blocks by increasing the ring size.=20
Dave