From owner-freebsd-net@FreeBSD.ORG Mon Sep 13 19:40:39 2010 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4E300106566C; Mon, 13 Sep 2010 19:40:39 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-pw0-f54.google.com (mail-pw0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id 0E6908FC1D; Mon, 13 Sep 2010 19:40:38 +0000 (UTC) Received: by pwi8 with SMTP id 8so2671848pwi.13 for ; Mon, 13 Sep 2010 12:40:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:received:from:date:to:cc :subject:message-id:reply-to:references:mime-version:content-type :content-disposition:in-reply-to:user-agent; bh=Q2ONQY9VWzkHZhs9qdtLXa33CVmJ53ls0jGrGb4zWUE=; b=lfcDXcZUnsrQcvM+g2o0eF0OcK6GR7TjD3pxxUJgbc9uZ+3VM/v+Ael5RVmQ52u0GS 745Y9yM3/wfx1BJeRqKbYV34BPRYvzV6AbA/+hJqFVPLf+TGbONg9ybN65zhIY3qJcGn Ezl0r/0Jh5gm1r3Sy/yKPd2R5fPv3y9TQ3Mb4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=rrV5T38mgION+CvqYfZme5FkKda4mubWnikIFcRfrNPzZK0MUJC2Oaeeuwb2fFHiW3 Ieo8vsvZ1edOFgmfJmvVKmzvfKIkJCQat2v0W1ZhPvuW9R+9O/2zKR0Qi3ZHv6tMOUks Kf0DBXotSGbeoJeugTXytqLqkPgicrLDtJOQc= Received: by 10.114.131.8 with SMTP id e8mr167380wad.43.1284406838439; Mon, 13 Sep 2010 12:40:38 -0700 (PDT) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id 33sm12623187wad.6.2010.09.13.12.40.36 (version=TLSv1/SSLv3 cipher=RC4-MD5); Mon, 13 Sep 2010 12:40:37 -0700 (PDT) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Mon, 13 Sep 2010 12:39:47 -0700 From: Pyun YongHyeon Date: Mon, 13 Sep 2010 12:39:47 -0700 To: Andre Oppermann Message-ID: <20100913193947.GH1229@michelle.cdnetworks.com> References: <4C894A76.5040200@tomjudge.com> <20100910002439.GO7203@michelle.cdnetworks.com> <4C8E3D79.6090102@tomjudge.com> <20100913184833.GF1229@michelle.cdnetworks.com> <4C8E775D.8070202@freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4C8E775D.8070202@freebsd.org> User-Agent: Mutt/1.4.2.3i Cc: Tom Judge , freebsd-net@freebsd.org, davidch@broadcom.com, yongari@freebsd.org Subject: Re: bce(4) - com_no_buffers (Again) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Sep 2010 19:40:39 -0000 On Mon, Sep 13, 2010 at 09:11:25PM +0200, Andre Oppermann wrote: > On 13.09.2010 20:48, Pyun YongHyeon wrote: > >On Mon, Sep 13, 2010 at 10:04:25AM -0500, Tom Judge wrote: > >>Without BCE_JUMBO_HDRSPLIT then we see no errors. With it we see number > >>of errors, however the rate seems to be reduced compaired to the > >>previous version of the driver. > >> > > > >It seems there are issues in header splitting and it was disabled > >by default. Header splitting reduces packet processing overhead in > >upper layer so it's normal to see better performance with header > >splitting. > > I'm not sure that header splitting really helps much at least for TCP. > The only place where it could make a difference is at socket buffer > append time. There the header get 'thrown away'. With header splitting > the first mbuf in the chain containing the header can be returned to the > free pool. Without header splitting it's just a offset change in the > mbuf. > > IIRC header splitting was introduced with the Tigeon cards which were > the first programmable network cards and the first to support putting > the header in a different mbuf. Header splitting, in theory, could > make a difference with zero copy sockets where the data portion in a > separate mbuf is flipped by VM magic into userspace. The trouble is > that no driver fully supports the semantics required for page flipping > and the zero copy code, if compiled in, is less much less optimized for > the non-flipping case than the standard code path. With the many dozen > gigabyte per second memory copy bandwidth of current CPU's it remains > questionable whether the page-flipping VM magic is actually faster than > a plain kernel/userspace copy as in the standard code path. I generally > recommend not to use ZERO_COPY_SOCKETS. > > I suspect in the case of the bce(4) driver the change in header splitting > is probably not the cause of the performance difference. > I'm under the impression the header splitting in bce(4) is for LRO(opposite of TSO), not for VM magic to enable page flipping tricks. > -- > Andre