From owner-svn-src-head@freebsd.org Tue Sep 15 17:13:03 2015 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 35F1DA030E8; Tue, 15 Sep 2015 17:13:03 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-ig0-x22b.google.com (mail-ig0-x22b.google.com [IPv6:2607:f8b0:4001:c05::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id F16451622; Tue, 15 Sep 2015 17:13:02 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by igcpb10 with SMTP id pb10so18782270igc.1; Tue, 15 Sep 2015 10:13:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=eH8D/0tQ9YvFQ7Tb4KPP4RzgkKTG3ptRr3uoAEA2RyI=; b=gEbJPFgIT2a+sKBKAkfIUispDnUtwxucM/faALtz3kexT/4in+/J7YZ7XlEr3h4UVD KexmUuh9HCydYtDD4GcpPb3RwZDaYb7cVoaPjMo1J5k0ciBr2E76PHCzolwHaek1MCU0 W3yOa9atvTMU5J8vSRSvg4zTQo3FEJLBKAdOUMkcsBwU388m052PE7rVaBl6WfqjG4dR oVr7ejsYUGTYYa+Ng+9gQkOk02T63dC2SJ5zoY7+O6baejp/IGn1PXE6xguBPLOIUxWj EF+gq606k+IRjjxXjwwU5R6xduKT6GBqTkj6HiomyIE6AeLUwsaBz2xpqcR7HXli+Ft9 7kfA== MIME-Version: 1.0 X-Received: by 10.50.45.33 with SMTP id j1mr7584940igm.61.1442337182163; Tue, 15 Sep 2015 10:13:02 -0700 (PDT) Received: by 10.36.28.208 with HTTP; Tue, 15 Sep 2015 10:13:02 -0700 (PDT) In-Reply-To: <55F7FE1A.3050500@selasky.org> References: <201409220827.s8M8RRHB031526@svn.freebsd.org> <55F69093.5050807@FreeBSD.org> <55F6935C.9000000@selasky.org> <55F6A694.7020404@FreeBSD.org> <55F6A914.6050109@selasky.org> <55F6ED8F.5030402@FreeBSD.org> <55F7FE1A.3050500@selasky.org> Date: Tue, 15 Sep 2015 10:13:02 -0700 Message-ID: Subject: Re: svn commit: r271946 - in head/sys: dev/oce dev/vmware/vmxnet3 dev/xen/netfront kern net netinet ofed/drivers/net/mlx4 sys From: Adrian Chadd To: Hans Petter Selasky Cc: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= , "src-committers@freebsd.org" , "svn-src-all@freebsd.org" , "svn-src-head@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Sep 2015 17:13:03 -0000 On 15 September 2015 at 04:16, Hans Petter Selasky wrote: > On 09/14/15 20:35, Adrian Chadd wrote: >> >> Hi, >> >> So what's the actual behaviour of the new tso logic before and after >> the above change in tsomax? > > > Hi, > > The behaviour is the same, only the limits have changed a bit. > >> like, what are the actual packet sizes >> >> being sent up to the hardware? > > > It is not about the packet sizes, it is about the number of packets we > transmit per TSO block. > >> Is TSO or the TCP stack so fragile that >> a slight change in how packets are broken up results in ridiculously >> less throughput? It's only a few bytes. > > > Network adapters which support TSO typically has a hard limit on the number > of mbufs it can load. When we exceed this limit, m_defrag() or packet drop > is next. It is the responsibility of the TCP stack to generated mbuf chains > which are within the network adapter given limits. Previously only the > length was accounted for. Now we also account for the number of segments, > because there are many ways a 64K bytes long mbuf chain can be generated. I know all of this. What I'm asking is different - what about the change(s) that broke/fixed the Xen network performance actually caused the change in behaviour? I know things are sensitive to how the mbufs are broken up - I'd like to see exactly this. Whilst futzing around at Netflix on TCP behaviour, I found with everything lined up correctly I'd get 40gbit TCP over a lot of sockets on sandy bridge hardware - but only as long as the TCP send buffer size resulted in nicely 4k segments. Some (larger) send buffer sizes resulted in the stack settling on non-4k segments for whatever reason, which drastically changed TSO performance. So, subtle changes had much bigger effects. I'm asking what specifically is going on here. +1 on the "going to make things more correct" path, but -2 on the "subtle changes and big unintended, un-described effects" results :( -adrian