From owner-svn-src-head@freebsd.org Mon Sep 14 15:53:56 2015 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7DF65A04AA7; Mon, 14 Sep 2015 15:53:56 +0000 (UTC) (envelope-from royger@gmail.com) Received: from mail-wi0-x22f.google.com (mail-wi0-x22f.google.com [IPv6:2a00:1450:400c:c05::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0EB3E120C; Mon, 14 Sep 2015 15:53:56 +0000 (UTC) (envelope-from royger@gmail.com) Received: by wicfx3 with SMTP id fx3so146459983wic.1; Mon, 14 Sep 2015 08:53:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:subject:to:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-type:content-transfer-encoding; bh=kb41wCOunkJq1jcwo92d/Ti7g889K91mTBe4udgGpBA=; b=MUcFHwZoJ2MkWQZ1RdHgOWROVvuaCmLmAJaBSimqVpu4sFQUAgpCxlhCZOYnwfDpbe JwZKJozJn0TvfVTIzmWiLNpph1HS4UUl6ETmKEX7Nnwiw1YeAxVooyuVrOmUGdYvCVSk eJZHOH+U1Klwm6wMayzkBPhfnfevOhqIFO+EXIj7j4z1shYM1Ag1VtIjRnnuP4i11MGa LgoIg7U5CfEnIGp9IHXTvOByEYArTb7AOcctlUX9WMb1hsTRNZ/GXfj3UsTSLbp7pvwh zJGpKlr7Cc90gBT3Bfq1WOxnegQ2/hetsAARpLAD3xc8RCuZBEIqpDj3QIywOwEngUrb yJZw== X-Received: by 10.180.87.198 with SMTP id ba6mr28119366wib.39.1442246034558; Mon, 14 Sep 2015 08:53:54 -0700 (PDT) Received: from [172.16.1.30] (195.Red-83-39-7.dynamicIP.rima-tde.net. [83.39.7.195]) by smtp.gmail.com with ESMTPSA id ry1sm16218913wjb.24.2015.09.14.08.53.53 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 14 Sep 2015 08:53:53 -0700 (PDT) Sender: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= Subject: Re: svn commit: r271946 - in head/sys: dev/oce dev/vmware/vmxnet3 dev/xen/netfront kern net netinet ofed/drivers/net/mlx4 sys To: Hans Petter Selasky , src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org References: <201409220827.s8M8RRHB031526@svn.freebsd.org> <55F69093.5050807@FreeBSD.org> <55F6935C.9000000@selasky.org> <55F6A694.7020404@FreeBSD.org> <55F6A914.6050109@selasky.org> From: =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= X-Enigmail-Draft-Status: N1110 Message-ID: <55F6ED8F.5030402@FreeBSD.org> Date: Mon, 14 Sep 2015 17:53:51 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: <55F6A914.6050109@selasky.org> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Sep 2015 15:53:56 -0000 El 14/09/15 a les 13.01, Hans Petter Selasky ha escrit: > On 09/14/15 12:51, Roger Pau Monné wrote: >> El 14/09/15 a les 11.29, Hans Petter Selasky ha escrit: >>> On 09/14/15 11:17, Roger Pau Monné wrote: >>>> El 22/09/14 a les 10.27, Hans Petter Selasky ha escrit: >>>>> Author: hselasky >>>>> Date: Mon Sep 22 08:27:27 2014 >>>>> New Revision: 271946 >>>>> URL: http://svnweb.freebsd.org/changeset/base/271946 >>>>> >>>>> Log: >>>>> Improve transmit sending offload, TSO, algorithm in general. >>>>> >>>>> The current TSO limitation feature only takes the total number of >>>>> bytes in an mbuf chain into account and does not limit by the >>>>> number >>>>> of mbufs in a chain. Some kinds of hardware is limited by two >>>>> factors. One is the fragment length and the second is the fragment >>>>> count. Both of these limits need to be taken into account when >>>>> doing >>>>> TSO. Else some kinds of hardware might have to drop completely >>>>> valid >>>>> mbuf chains because they cannot loaded into the given >>>>> hardware's DMA >>>>> engine. The new way of doing TSO limitation has been made >>>>> backwards >>>>> compatible as input from other FreeBSD developers and will use >>>>> defaults for values not set. >>>>> >>>>> Reviewed by: adrian, rmacklem >>>>> Sponsored by: Mellanox Technologies >>>> >>>> This commit makes xen-netfront tx performance drop from ~5Gbits/sec >>>> (with debug options enabled) to 446 Mbits/sec. I'm currently looking, >>>> but if anyone has ideas they are welcome. >>>> >>> >>> Hi Roger, >>> >>> Looking at the netfront code you should subtract 1 from tsomaxsegcount >>> prior to r287775. The reason might simply be that 2K clusters are used >>> instead of 4K clusters, causing m_defrag() to be called. >>> >>>> ifp->if_hw_tsomax = 65536 - (ETHER_HDR_LEN + >>>> ETHER_VLAN_ENCAP_LEN); >>>> ifp->if_hw_tsomaxsegcount = MAX_TX_REQ_FRAGS; >>>> ifp->if_hw_tsomaxsegsize = PAGE_SIZE; >>> >>> After r287775 can you try these settings: >>> >>> ifp->if_hw_tsomax = 65536; >>> ifp->if_hw_tsomaxsegcount = MAX_TX_REQ_FRAGS; >>> ifp->if_hw_tsomaxsegsize = PAGE_SIZE; >>> >>> And see if the performance is the same like before? >> > > Hi Roger, > >> Yes, performance seems to be fine after setting if_hw_tsomax to 65536. >> Is there some documentation about the usage of if_hw_tsomax? Does the >> network subsystem already takes care of subtracting the space for ether >> header and the vlan encapsulation, so it's no longer needed to specify >> them in if_hw_tsomax? > > In the past only the TCP and IP layers were accounted for by the TSO > parameters. A the present all layers are accounted for. This might fit > the kind of adapter you are using better, because it appears to me it is > DMA'ing all of the mbuf chain. Some other network adapters only DMA the > TCP payload data and copy the ETH/TCP/IP headers into a special DMA'able > memory area. Thanks for the hint, I'm not sure where that DMA tag is coming from, xen-netfront doesn't define any DMA tag at all, and AFAICT none of it's parents do: nexus0 [...] xenpv0 granttable0 xen_et0 xenstore0 xenballoon0 xctrl0 xs_dev0 xenbusb_front0 xbd0 xn0 So I don't see where this bouncing requirement is coming from, although I'm sure I'm missing something... >> >> Also, this commit was MFC'ed to stable/10 and 10.2 suffers from the same >> problem. Can we issue and EN to get this fixed in 10.2? > > When this patch has been given some time to settle, and more people have > tested it, I can submit a request for re @ to do that. Please remind me > if I forget. No problem, will do so if needed :). Roger.