From owner-freebsd-stable@freebsd.org Thu Oct 8 12:32:36 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 819889D16D5; Thu, 8 Oct 2015 12:32:36 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 269401EDC; Thu, 8 Oct 2015 12:32:35 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) IronPort-PHdr: 9a23:CZe7jhEZjK1HvBSkyYJXlZ1GYnF86YWxBRYc798ds5kLTJ75oMSwAkXT6L1XgUPTWs2DsrQf27aQ7P2rBDZIyK3CmU5BWaQEbwUCh8QSkl5oK+++Imq/EsTXaTcnFt9JTl5v8iLzG0FUHMHjew+a+SXqvnYsExnyfTB4Ov7yUtaLyZ/ni6btptaOOU1hv3mUX/BbFF2OtwLft80b08NJC50a7V/3mEZOYPlc3mhyJFiezF7W78a0+4N/oWwL46pyv+YJa6jxfrw5QLpEF3xmdjltvIy4/SXEGCuG4GBUamgKjhdSSzPI6BjhXYa55ivircJm1S2TJs7nC7cuVmLxwb1sTUrSiSwEfxsw+2LTh8k42LheqRmioxF665PTb5yYMOJ+OKjUK4BJDVFdV9pcAnQSSri3aJECWq9YZb5V X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2CyAgCNYRZW/61jaINeg3tuBr1AAQ2BWhcKgnKCCjVKAoF+FAEBAQEBAQEBgQmCH4IHAQEBAwEBAQEgKyALBQsCAQgYAgINGQICJwEJJgIECAIFBAEcBIgFCA2uTJQwAQEBAQEBAQEBAQEBAQEBAQEBFgSBIoVRhH6EOwEBHDQHgmmBRQWWCIUYhRiEPoQ5lWYCHwEBQoIRHYFwIjMHhiU6gQYBAQE X-IronPort-AV: E=Sophos;i="5.17,654,1437451200"; d="scan'208";a="243370407" Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-annu.net.uoguelph.ca with ESMTP; 08 Oct 2015 08:32:28 -0400 Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 803BE15F565; Thu, 8 Oct 2015 08:32:28 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id vosyeExQTXCM; Thu, 8 Oct 2015 08:32:27 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id CC86C15F56D; Thu, 8 Oct 2015 08:32:27 -0400 (EDT) X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id zY5zWHc1e1gS; Thu, 8 Oct 2015 08:32:27 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 9DC8315F565; Thu, 8 Oct 2015 08:32:27 -0400 (EDT) Date: Thu, 8 Oct 2015 08:32:27 -0400 (EDT) From: Rick Macklem To: Hans Petter Selasky Cc: Daniel Braniss , pyunyh@gmail.com, FreeBSD Net , FreeBSD stable Message-ID: <855415533.28142431.1444307547414.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <56163896.3020907@selasky.org> References: <1D52028A-B39F-4F9B-BD38-CB1D73BF5D56@cs.huji.ac.il> <49173B1F-7B5E-4D59-8651-63D97B0CB5AC@cs.huji.ac.il> <1815942485.29539597.1440370972998.JavaMail.zimbra@uoguelph.ca> <55DAC623.60006@selasky.org> <62C7B1A3-CC6B-41A1-B254-6399F19F8FF7@cs.huji.ac.il> <2112273205.29795512.1440419111720.JavaMail.zimbra@uoguelph.ca> <1E679659-BA50-42C3-B569-03579E322685@cs.huji.ac.il> <56163896.3020907@selasky.org> Subject: Re: ix(intel) vs mlxen(mellanox) 10Gb performance MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.95.11] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - FF34 (Win)/8.0.9_GA_6191) Thread-Topic: ix(intel) vs mlxen(mellanox) 10Gb performance Thread-Index: eT4NUGq9p4YEyXbCx7sOzdsvj3/VsQ== X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Oct 2015 12:32:36 -0000 Hans Petter Selasky wrote: > Hi, > > I've now MFC'ed r287775 to 10-stable and 9-stable. I hope this will > resolve the issues with m_defrag() being called on too long mbuf chains > due to an off-by-one in the driver TSO parameters and that it will be > easier to maintain these parameters in the future. > > Some comments were made that we might want to have an option to select > if the IP-header should be counted or not. Certain network drivers > require copying of the whole ETH/TCP/IP-header into separate memory > areas, and can then handle one more data payload mbuf for TSO. Others > required DMA-ing of the whole mbuf TSO chain. I think it is acceptable > to have one TX-DMA segment slot free, in case of 2K mbuf clusters being > used for TSO. From my experience the limitation typically kicks in when > 2K mbuf clusters are used for TSO instead of 4K mbuf clusters. 65536 / > 4096 = 16, whereas 65536 / 2048 = 32. If an ethernet hardware driver has > a limitation of 24 data segments (mlxen), and assuming that each mbuf > represent a single segment, then iff the majority of mbufs being > transmitted are 2K clusters we may have a small, 1/24 = 4.2%, loss of TX > capability per TSO packet. From what I've seen using iperf, which in > turn calls m_uiotombuf() which in turn calls m_getm2(), MJUMPPAGESIZE'ed > mbuf clusters are preferred for large data transfers, so this issue > might only happen in case of NODELAY being used on the socket and if the > writes are small from the application point of view. If an application > is writing small amounts of data per send() system call, it is expected > to degrade the system performance. > Btw, last year I did some testing with NFS generating chains of 4K (page size) clusters instead of 2K (MCLBYTES). Although not easily reproduced, I was able to fragment the KVM used for the cluster enough that allocations would fail. (I could only get it to happen when the code used 4K clusters for large NFS requests/replies and 2K clusters otherwise, resulting in a mix of allocations of both sizes.) As such, I never committed the changes to head. Any kernel change that does 4K cluster allocations needs to be carefully tested carefully (a small i386 like I have), imho. > Please file a PR if it becomes an issue. > > Someone asked me to MFC r287775 to 10.X release aswell. Is this still > required? > > --HPS Thanks for doing this, rick > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >