From owner-freebsd-net@freebsd.org Thu Feb 20 21:58:09 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6385E247A2E for ; Thu, 20 Feb 2020 21:58:09 +0000 (UTC) (envelope-from jpaetzel@FreeBSD.org) Received: from wout1-smtp.messagingengine.com (wout1-smtp.messagingengine.com [64.147.123.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48NpPC4kLbz4L9R for ; Thu, 20 Feb 2020 21:58:07 +0000 (UTC) (envelope-from jpaetzel@FreeBSD.org) Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailout.west.internal (Postfix) with ESMTP id 1A1CB513 for ; Thu, 20 Feb 2020 16:58:05 -0500 (EST) Received: from imap2 ([10.202.2.52]) by compute2.internal (MEProxy); Thu, 20 Feb 2020 16:58:05 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=XwxS5P IHYyFCADFoO8/3llQbxU2mfgRijfb0i6vzrr4=; b=Br97/fL8PKJObOpch/cr2P T3isMQ9EBan/wrZOMleGcP2myok0YIMZFQXG8jaKUkZ9ZYTtvUSHs7T8PbteyDWq jTF2/3w41rmUrec7NExvq/YU6EUKlMpEZbU4YTC9k38GWSDxsiURn2scJd6Me/e5 f87dcbe6mQ3Iyi2Tmdb1d9Ijf39sW4HN1pz/9XB48/TiaadKXHI27DHjSMsI/rdS 6PrkcXNZvcHK3XJNPaCn5BQeIJPXThUT/Uka1UxmqeRO7dlTAsBojXKrCg2Tttif ngfRMHEuYROwuj8gjFwZU4DsiebgP+nXQcsK7kIp+zWyMfYdL6Vu5V5/NB1Oqo7A == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedugedrkedvgdduheekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefofgggkfgjfhffhffvufgtsehttd ertderredtnecuhfhrohhmpedflfhoshhhucfrrggvthiivghlfdcuoehjphgrvghtiigv lheshfhrvggvuefuffdrohhrgheqnecuffhomhgrihhnpehfrhgvvggsshgurdhorhhgne cuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepjhhprggv thiivghlsefhrhgvvgeuufffrdhorhhg X-ME-Proxy: Received: by mailuser.nyi.internal (Postfix, from userid 501) id 8BF08E00A2; Thu, 20 Feb 2020 16:58:04 -0500 (EST) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.1.7-802-g7a41c81-fmstable-20200203v1 Mime-Version: 1.0 Message-Id: In-Reply-To: <5e5d423b-0711-7454-626a-cc9cb4b004cd@FreeBSD.org> References: <40c4a4df-3df6-d95d-53c2-eef905ff45b1@FreeBSD.org> <5e5d423b-0711-7454-626a-cc9cb4b004cd@FreeBSD.org> Date: Thu, 20 Feb 2020 15:57:38 -0600 From: "Josh Paetzel" To: freebsd-net@freebsd.org Subject: Re: terrible if_vmx / vmxnet3 rx performance with lro (post iflib) Content-Type: text/plain X-Rspamd-Queue-Id: 48NpPC4kLbz4L9R X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-1.99 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-0.99)[-0.987,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; ASN(0.00)[asn:11403, ipnet:64.147.123.0/24, country:US] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Feb 2020 21:58:09 -0000 On Wed, Feb 19, 2020, at 7:17 AM, Andriy Gapon wrote: > On 18/02/2020 16:09, Andriy Gapon wrote: > > My general experience with post-iflib vmxnet3 is that vmxnet3 has some > > peculiarities that result in a certain "impedance mismatch" with iflib. > > Although we now have a bit less code and it is a bit more regular, there are a > > few significant (for us, at least) problems: > > - https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=243126 > > - https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=240608 > > By the way, we (Panzura) use these changes to fix or work around the above two > problems: https://people.freebsd.org/~avg/iflib-vmx.pz.diff > > Questions / comments are welcome. > Especially from people who worked on iflib. > > > - https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=243392 > > - the problem described above > > - a couple of issues that we already fixed or worked around > > > > We are contemplating locally reverting to the pre-iflib vmxnet3 and we are > > wondering if the conversion was really worth it in general. > > > -- > Andriy Gapon > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > I'd like to follow this up just to make it 100% clear. The problem is a ~4x regression in RX performance. It affects stock FreeBSD, including 12.1-RELEASE. In my 40Gbps connected lab single thread iperf receive went from 9Gbps to 2.5Gbps. If this can't be fixed or looked at I'd heavily suggest looking at reverting "iflib"ing change in stock FreeBSD. -- Thanks, Josh Paetzel