From owner-freebsd-stable@freebsd.org Wed Aug 19 12:14:02 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B4A109BC02B; Wed, 19 Aug 2015 12:14:02 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 2C9B0979; Wed, 19 Aug 2015 12:14:01 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) IronPort-PHdr: 9a23:GkC7YBd9aSdqXt8Roqe2oc6zlGMj4u6mDksu8pMizoh2WeGdxc6/Yx7h7PlgxGXEQZ/co6odzbGG6Oa8Bydcvt6oizMrTt9lb1c9k8IYnggtUoauKHbQC7rUVRE8B9lIT1R//nu2YgB/Ecf6YEDO8DXptWZBUiv2OQc9HOnpAIma153xjLDpvcGNKFkXzBOGIppMbzyO5T3LsccXhYYwYo0Q8TDu5kVyRuJN2GlzLkiSlRuvru25/Zpk7jgC86l5r50IeezAcq85Vb1VCig9eyBwvZWz9EqLcQza/moBVHQWuhVNCgnBqhr9W8TfqCz/49B80yrSGMT9TrQ5XHz29aJiQxzshSIvKjk27WzTksw2h6sN80HpnAB234OBONLdD/F5ZK6IOIpCHWc= X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2AnAgA0cdRV/61jaINdg29pBoMfuiQBCYFtCoUxSgKBehQBAQEBAQEBAYEJgh2CBwEBBAEBASArIAsQAgEIGAICDRkCAicBCSYCDAcEARwEiA0NuUuWGwEBAQEBAQEDAQEBAQEZBIEiijGEMQEGAQEcNAeCaYFDBZUjhQSFB4QskDeESINmAiaCDhyBbyIzB34BCBcjgQQBAQE X-IronPort-AV: E=Sophos;i="5.15,709,1432612800"; d="scan'208";a="233342498" Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-annu.net.uoguelph.ca with ESMTP; 19 Aug 2015 08:14:00 -0400 Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 75F1715F574; Wed, 19 Aug 2015 08:14:00 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id 9c6wl2qMDweJ; Wed, 19 Aug 2015 08:13:59 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id A102F15F577; Wed, 19 Aug 2015 08:13:59 -0400 (EDT) X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id p_9FsjkfQATs; Wed, 19 Aug 2015 08:13:59 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 7EE9215F574; Wed, 19 Aug 2015 08:13:59 -0400 (EDT) Date: Wed, 19 Aug 2015 08:13:59 -0400 (EDT) From: Rick Macklem To: pyunyh@gmail.com Cc: Hans Petter Selasky , FreeBSD stable , FreeBSD Net , Slawa Olhovchenkov , Christopher Forgeron , Daniel Braniss Message-ID: <1154739904.25677089.1439986439408.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <20150819081308.GC964@michelle.fasterthan.com> References: <473274181.23263108.1439814072514.JavaMail.zimbra@uoguelph.ca> <1721122651.24481798.1439902381663.JavaMail.zimbra@uoguelph.ca> <55D333D6.5040102@selasky.org> <1325951625.25292515.1439934848268.JavaMail.zimbra@uoguelph.ca> <55D429A4.3010407@selasky.org> <20150819074212.GB964@michelle.fasterthan.com> <55D43590.8050508@selasky.org> <20150819081308.GC964@michelle.fasterthan.com> Subject: Re: ix(intel) vs mlxen(mellanox) 10Gb performance MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.95.11] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - FF34 (Win)/8.0.9_GA_6191) Thread-Topic: ix(intel) vs mlxen(mellanox) 10Gb performance Thread-Index: wv7zo8RPDc9ayJBYkipemnefFyp/cg== X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Aug 2015 12:14:02 -0000 Yonghyeon PYUN wrote: > On Wed, Aug 19, 2015 at 09:51:44AM +0200, Hans Petter Selasky wrote: > > On 08/19/15 09:42, Yonghyeon PYUN wrote: > > >On Wed, Aug 19, 2015 at 09:00:52AM +0200, Hans Petter Selasky wrote: > > >>On 08/18/15 23:54, Rick Macklem wrote: > > >>>Ouch! Yes, I now see that the code that counts the # of mbufs is before > > >>>the > > >>>code that adds the tcp/ip header mbuf. > > >>> > > >>>In my opinion, this should be fixed by setting if_hw_tsomaxsegcount to > > >>>whatever > > >>>the driver provides - 1. It is not the driver's responsibility to know > > >>>if > > >>>a tcp/ip > > >>>header mbuf will be added and is a lot less confusing that expecting the > > >>>driver > > >>>author to know to subtract one. (I had mistakenly thought that > > >>>tcp_output() had > > >>>added the tc/ip header mbuf before the loop that counts mbufs in the > > >>>list. > > >>>Btw, > > >>>this tcp/ip header mbuf also has leading space for the MAC layer > > >>>header.) > > >>> > > >> > > >>Hi Rick, > > >> > > >>Your question is good. With the Mellanox hardware we have separate > > >>so-called inline data space for the TCP/IP headers, so if the TCP stack > > >>subtracts something, then we would need to add something to the limit, > > >>because then the scatter gather list is only used for the data part. > > >> > > > > > >I think all drivers in tree don't subtract 1 for > > >if_hw_tsomaxsegcount. Probably touching Mellanox driver would be > > >simpler than fixing all other drivers in tree. > > > > Hi, > > > > If you change the behaviour don't forget to update and/or add comments > > describing it. Maybe the amount of subtraction could be defined by some > > macro? Then drivers which inline the headers can subtract it? > > > > I'm also ok with your suggestion. > > > Your suggestion is fine by me. > > > > > The initial TSO limits were tried to be preserved, and I believe that > > TSO limits never accounted for IP/TCP/ETHERNET/VLAN headers! > > > > I guess FreeBSD used to follow MS LSOv1 specification with minor > exception in pseudo checksum computation. If I recall correctly the > specification says upper stack can generate up to IP_MAXPACKET sized > packet. Other L2 headers like ethernet/vlan header size is not > included in the packet and it's drivers responsibility to allocate > additional DMA buffers/segments for L2 headers. > Yep. The default for if_hw_tsomax was reduced from IP_MAXPACKET to 32 * MCLBYTES - max_ethernet_header_size as a workaround/hack so that devices limited to 32 transmit segments would work (ie. the entire packet, including MAC header would fit in 32 MCLBYTE clusters). This implied that many drivers did end up using m_defrag() to copy the mbuf list to one made up of 32 MCLBYTE clusters. If a driver sets if_hw_tsomaxsegcount correctly, then it can set if_hw_tsomax to whatever it can handle as the largest TSO packet (without MAC header) the hardware can handle. If it can handle > IP_MAXPACKET, then it can set it to that. rick > > > > > >>Maybe it can be controlled by some kind of flag, if all the three TSO > > >>limits should include the TCP/IP/ethernet headers too. I'm pretty sure > > >>we want both versions. > > >> > > > > > >Hmm, I'm afraid it's already complex. Drivers have to tell almost > > >the same information to both bus_dma(9) and network stack. > > > > You're right it's complicated. Not sure if bus_dma can provide an API > > for this though. > > > > --HPS > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >