From owner-freebsd-current@freebsd.org Tue Apr 21 12:51:18 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 29EBC2CC4FE for ; Tue, 21 Apr 2020 12:51:18 +0000 (UTC) (envelope-from crest@rlwinm.de) Received: from mail.rlwinm.de (mail.rlwinm.de [IPv6:2a01:4f8:171:f902::5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4963N46rVnz4GYk for ; Tue, 21 Apr 2020 12:51:16 +0000 (UTC) (envelope-from crest@rlwinm.de) Received: from crest-mbp.local (200116b864f3ff00201c96d4c844ede6.dip.versatel-1u1.de [IPv6:2001:16b8:64f3:ff00:201c:96d4:c844:ede6]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.rlwinm.de (Postfix) with ESMTPSA id CDA4A31D80 for ; Tue, 21 Apr 2020 12:51:08 +0000 (UTC) Subject: Re: CFT: if_bridge performance improvements To: freebsd-current@freebsd.org References: <5377E42E-4C01-4BCC-B934-011AC3448B54@FreeBSD.org> From: Jan Bramkamp Message-ID: Date: Tue, 21 Apr 2020 14:51:07 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:68.0) Gecko/20100101 Thunderbird/68.7.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-GB X-Rspamd-Queue-Id: 4963N46rVnz4GYk X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of crest@rlwinm.de designates 2a01:4f8:171:f902::5 as permitted sender) smtp.mailfrom=crest@rlwinm.de X-Spamd-Result: default: False [-5.02 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[6.e.d.e.4.4.8.c.4.d.6.9.c.1.0.2.0.0.f.f.3.f.4.6.8.b.6.1.1.0.0.2.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.11]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; DMARC_NA(0.00)[rlwinm.de]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; IP_SCORE(-2.72)[ip: (-9.38), ipnet: 2a01:4f8::/29(-2.64), asn: 24940(-1.54), country: DE(-0.02)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Apr 2020 12:51:18 -0000 On 16.04.20 08:34, Pavel Timofeev wrote: > вт, 14 апр. 2020 г., 12:51 Kristof Provost : > >> Hi, >> >> Thanks to support from The FreeBSD Foundation I’ve been able to work >> on improving the throughput of if_bridge. >> It changes the (data path) locking to use the NET_EPOCH infrastructure. >> Benchmarking shows substantial improvements (x5 in test setups). >> >> This work is ready for wider testing now. >> >> It’s under review here: https://reviews.freebsd.org/D24250 >> >> Patch for CURRENT: https://reviews.freebsd.org/D24250?download=true >> Patches for stable/12: >> https://people.freebsd.org/~kp/if_bridge/stable_12/ >> >> I’m not currently aware of any panics or issues resulting from these >> patches. >> >> Do note that if you run a Bhyve + tap on bridges setup the tap code >> suffers from a similar bottleneck and you will likely not see major >> improvements in single VM to host throughput. I would expect, but have >> not tested, improvements in overall throughput (i.e. when multiple VMs >> send traffic at the same time). >> >> Best regards, >> Kristof >> > Hi! > Thank you for your work! > Do you know if epair suffers from the same issue as tap? As Kirstof Provost said if_epair locks has per CPU locks, but a problem exists a layer about the epair driver. At leas on FreeBSD 12.0 and 12.1 all the packet processing happens in a single netisr thread that becomes CPU bound and limits how fast useful traffic can move through epair interfaces. Afaik TCP doesn't benifit from multiple netisr threads, but unorderer protocols (e.g. UDP) could profit from multiple threads. I have only tested with iperf (using multiple connections) between the FreeBSD 12.x host and a vnet enabled jail connected via an epair interface and maxed out at about 1-2Gb/s depending on the CPUs single threaded throughput.