From owner-freebsd-net@FreeBSD.ORG Mon Sep 13 19:34:15 2010 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DAE26106566B; Mon, 13 Sep 2010 19:34:14 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-pw0-f54.google.com (mail-pw0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id A1CBB8FC18; Mon, 13 Sep 2010 19:34:14 +0000 (UTC) Received: by pwi8 with SMTP id 8so2669836pwi.13 for ; Mon, 13 Sep 2010 12:34:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:received:from:date:to:cc :subject:message-id:reply-to:references:mime-version:content-type :content-disposition:in-reply-to:user-agent; bh=LATBn5ZRPy2nWQUZPfmnp+Qmp0iw0KjHziaRGc5SRaY=; b=d6jVpjeVhZNNTGyKMkYMdtrVqsjX1CXQiPdmYO7B/DRjBJMd+Vm/4cc+XvBJ7ILsjI 5ceVEQKEKUHFo5pRt8lGsyV9ybk7vTynIBOKa4ILQn9ZjBETD3LGDqAvKPCdJo66nr8w dXGMHGuQtnHZYVx5Ie6krTdT9KayNQ1tOKJic= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=iVQM3zozpv3hqFXmGzL59xN3HfrqgIZTzhE4HMGmlpYudz1aUHzvKyzB2ZgtugkGj0 eGoWi/I/PJ3pH2JWyXH0WDIEeUUWK8k2P+zvC7YC73QPh8ws0dWKLbIU30QsXO+tbZt6 HK+hkcQY1FfA1e8VOo988li2brEfwxZpVDArM= Received: by 10.114.79.10 with SMTP id c10mr494541wab.220.1284406453859; Mon, 13 Sep 2010 12:34:13 -0700 (PDT) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id d39sm12612405wam.4.2010.09.13.12.34.11 (version=TLSv1/SSLv3 cipher=RC4-MD5); Mon, 13 Sep 2010 12:34:12 -0700 (PDT) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Mon, 13 Sep 2010 12:33:22 -0700 From: Pyun YongHyeon Date: Mon, 13 Sep 2010 12:33:22 -0700 To: Tom Judge Message-ID: <20100913193322.GG1229@michelle.cdnetworks.com> References: <4C894A76.5040200@tomjudge.com> <20100910002439.GO7203@michelle.cdnetworks.com> <4C8E3D79.6090102@tomjudge.com> <20100913184833.GF1229@michelle.cdnetworks.com> <4C8E768E.7000003@tomjudge.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4C8E768E.7000003@tomjudge.com> User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@freebsd.org, davidch@broadcom.com, yongari@freebsd.org Subject: Re: bce(4) - com_no_buffers (Again) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Sep 2010 19:34:15 -0000 On Mon, Sep 13, 2010 at 02:07:58PM -0500, Tom Judge wrote: > On 09/13/2010 01:48 PM, Pyun YongHyeon wrote: > > On Mon, Sep 13, 2010 at 10:04:25AM -0500, Tom Judge wrote: > > > >> > > >> Does this mean that these cards are going to perform badly? This is was > >> what I gathered from the previous thread. > >> > >> > > I mean there are still many rooms to be done in driver for better > > performance. bce(4) controllers are one of best controllers for > > servers and driver didn't take full advantage of it. > > > > > > So far our experiences with bce(4) on FreeBSD have been very > disappointing. Starting with when Dell switched to bce(4) based NIC's > (around the time 6.2 was released and with the introduction of the Power > Edge X9XX hardware) we have always had problems with the driver in every > release we have used: 6.2, 7.0 and 7.1. Luckily David has been helpful > and helped us fix the issues. > > > > > >> Without BCE_JUMBO_HDRSPLIT then we see no errors. With it we see number > >> of errors, however the rate seems to be reduced compaired to the > >> previous version of the driver. > >> > >> > > It seems there are issues in header splitting and it was disabled > > by default. Header splitting reduces packet processing overhead in > > upper layer so it's normal to see better performance with header > > splitting. > > > > The reason that we have had header splitting enabled in the past is that > historically there have been issues with memory fragmentation when using > 8k jumbo frames (resulting in 9k mbuf's). > Yes, if you use jumbo frames, header splitting would help to reduce memory fragmentation as header splitting wouldn't allocate jumbo clusters. > I have a kernel with the following configuration in testing right now: > > * Flow control enabled. > * Jumbo header splitting turned off. > > > Is there any way that we can fix flow control with jumbo header > splitting turned on? > Flow control has nothing to do with header splitting(i.e. flow control is always enabled). > Thanks > > Tom > > PS. The following test was more than enough to trigger buffer shortages > with header splitting on: > > ( while true; do ldapsearch -h ldap-server1 -b "ou=Some,o=Base" dn; done > ) & > ( while true; do ldapsearch -h ldap-server1 -b "ou=Some,o=Base" dn; done > ) & > ( while true; do ldapsearch -h ldap-server1 -b "ou=Some,o=Base" dn; done > ) & > > The search in question returned about 1700 entries. > I can trigger this kind of buffer shortage with benchmark tools. Actually fixing header splitting is on my TODO list as well as other things but I don't know how long it would take.