From owner-freebsd-net@FreeBSD.ORG Mon Sep 13 19:18:41 2010 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 09088106564A; Mon, 13 Sep 2010 19:18:41 +0000 (UTC) (envelope-from tom@tomjudge.com) Received: from eu1sys200aog105.obsmtp.com (eu1sys200aog105.obsmtp.com [207.126.144.119]) by mx1.freebsd.org (Postfix) with SMTP id A836A8FC12; Mon, 13 Sep 2010 19:18:38 +0000 (UTC) Received: from source ([63.174.175.251]) by eu1sys200aob105.postini.com ([207.126.147.11]) with SMTP ID DSNKTI55DHGMEIj6SpVR2ib3iyHkDuwz8+7A@postini.com; Mon, 13 Sep 2010 19:18:40 UTC Received: from [172.17.10.53] (unknown [172.17.10.53]) by bbbx3.usdmm.com (Postfix) with ESMTP id 735E8FD01C; Mon, 13 Sep 2010 19:18:36 +0000 (UTC) Message-ID: <4C8E7904.9090004@tomjudge.com> Date: Mon, 13 Sep 2010 14:18:28 -0500 From: Tom Judge User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.12) Gecko/20100826 Lightning/1.0b1 Thunderbird/3.0.7 MIME-Version: 1.0 To: Andre Oppermann References: <4C894A76.5040200@tomjudge.com> <20100910002439.GO7203@michelle.cdnetworks.com> <4C8E3D79.6090102@tomjudge.com> <20100913184833.GF1229@michelle.cdnetworks.com> <4C8E775D.8070202@freebsd.org> In-Reply-To: <4C8E775D.8070202@freebsd.org> X-Enigmail-Version: 1.0.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: pyunyh@gmail.com, freebsd-net@freebsd.org, davidch@broadcom.com, yongari@freebsd.org Subject: Re: bce(4) - com_no_buffers (Again) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Sep 2010 19:18:41 -0000 On 09/13/2010 02:11 PM, Andre Oppermann wrote: > On 13.09.2010 20:48, Pyun YongHyeon wrote: >> On Mon, Sep 13, 2010 at 10:04:25AM -0500, Tom Judge wrote: >>> Without BCE_JUMBO_HDRSPLIT then we see no errors. With it we see >>> number >>> of errors, however the rate seems to be reduced compaired to the >>> previous version of the driver. Please note that 'rate' here relates to the rate at which dev.bce.X.com_no_buffers is increasing not to PPS or bandwidth. However the discussion is still interesting. >>> >> >> It seems there are issues in header splitting and it was disabled >> by default. Header splitting reduces packet processing overhead in >> upper layer so it's normal to see better performance with header >> splitting. > > I'm not sure that header splitting really helps much at least for TCP. > The only place where it could make a difference is at socket buffer > append time. There the header get 'thrown away'. With header splitting > the first mbuf in the chain containing the header can be returned to the > free pool. Without header splitting it's just a offset change in the > mbuf. > > IIRC header splitting was introduced with the Tigeon cards which were > the first programmable network cards and the first to support putting > the header in a different mbuf. Header splitting, in theory, could > make a difference with zero copy sockets where the data portion in a > separate mbuf is flipped by VM magic into userspace. The trouble is > that no driver fully supports the semantics required for page flipping > and the zero copy code, if compiled in, is less much less optimized for > the non-flipping case than the standard code path. With the many dozen > gigabyte per second memory copy bandwidth of current CPU's it remains > questionable whether the page-flipping VM magic is actually faster than > a plain kernel/userspace copy as in the standard code path. I generally > recommend not to use ZERO_COPY_SOCKETS. > > I suspect in the case of the bce(4) driver the change in header splitting > is probably not the cause of the performance difference. > -- TJU13-ARIN