From nobody Sun Mar 13 16:33:28 2022 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id BF8281A09BBF for ; Sun, 13 Mar 2022 16:33:43 +0000 (UTC) (envelope-from grembo@freebsd.org) Received: from mail.evolve.de (mail.evolve.de [213.239.217.29]) (using TLSv1.3 with cipher TLS_CHACHA20_POLY1305_SHA256 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA512 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mail.evolve.de", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KGlbn6NCNz3lxk; Sun, 13 Mar 2022 16:33:41 +0000 (UTC) (envelope-from grembo@freebsd.org) Received: by mail.evolve.de (OpenSMTPD) with ESMTP id 56f53a4e; Sun, 13 Mar 2022 16:33:39 +0000 (UTC) Received: by mail.evolve.de (OpenSMTPD) with ESMTPSA id 0ea39fea (TLSv1.3:AEAD-CHACHA20-POLY1305-SHA256:256:NO); Sun, 13 Mar 2022 16:33:35 +0000 (UTC) Date: Sun, 13 Mar 2022 17:33:28 +0100 From: Michael Gmelin To: Johan Hendriks , Kristof Provost Cc: "Patrick M. Hausen" , Michael Gmelin , freeBSD-net Subject: Re: epair and vnet jail loose connection. Message-ID: <20220313173328.78511501.grembo@freebsd.org> In-Reply-To: <728d2993-7813-cf67-0184-8af56044c114@gmail.com> References: <94B8885D-F63F-40C3-9E7E-158CC252FF9A@FreeBSD.org> <95793CDF-6E72-4FAB-8BF5-F2E67D3F69CD@freebsd.org> <810820a6-e319-fa78-72a3-3d1cb43f3af3@gmail.com> <7DD42D89-7706-47C2-B8B6-82A29DE9D351@punkt.de> <728d2993-7813-cf67-0184-8af56044c114@gmail.com> X-Face: $wrgCtfdVw_H9WAY?S&9+/F"!41z'L$uo*WzT8miX?kZ~W~Lr5W7v?j0Sde\mwB&/ypo^}> +a'4xMc^^KroE~+v^&^#[B">soBo1y6(TW6#UZiC]o>C6`ej+i Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAJFBMVEWJBwe5BQDl LASZU0/LTEWEfHbyj0Txi32+sKrp1Mv944X8/fm1rS+cAAAACXBIWXMAAAsTAAAL EwEAmpwYAAAAB3RJTUUH3wESCxwC7OBhbgAAACFpVFh0Q29tbWVudAAAAAAAQ3Jl YXRlZCB3aXRoIFRoZSBHSU1QbbCXAAAAAghJREFUOMu11DFvEzEUAGCfEhBVFzuq AKkLd0O6VrIQsLXVSZXoWE5N1K3DobBBA9fQpRWc8OkWouaIjedWKiyREOKs+3PY fvalCNjgLVHeF7/3bMtBzV8C/VsQ8tecEgCcDgrzjekwKZ7TwsJZd/ywEKwwP+ZM 8P3drTsAwWn2mpWuDDuYiK1bFs6De0KUUFw0tWxm+D4AIhuuvZqtyWYeO7jQ4Aea 7jUqI+ixhQoHex4WshEvSXdood7stlv4oSuFOC4tqGcr0NjEqXgV4mMJO38nld4+ xKNxRDon7khyKVqY7YR4d+Cg0OMrkWXZOM7YDkEfKiilCn1qYv4mighZiynuHHOA Wq9QJq+BIES7lMFUtcikMnkDGHUoncA+uHgrP0ctIEqfwLHzeSo+eUA66AqzwN6n 2ZHJhw6Qh/PoyC/QENyEyC/AyNjq74Bs+3UH0xYwzDUC4B97HgLocg1QLYgDDO1v f3UX9Y307Ew4AHh67YAFFsxEpkXwpXY3eIgMhAAE3R19L919nNnuD2wlPcDE3UeT L2ytEICQib9BXgS2fU8PrD82ToYO1OEmMSnYTjSqSv9wdC0tPYC+rQRQD9ESnldF CyqfmiYW+tlALt8gH2xrMdC/youbjzPXEun+/ReXsMCDyve3dZc09fn2Oas8oXGc Jj6/fOeK5UmSMPmf/jL+GD8BEj0k/Fn6IO4AAAAASUVORK5CYII= List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4KGlbn6NCNz3lxk X-Spamd-Bar: + Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=softfail (mx1.freebsd.org: 213.239.217.29 is neither permitted nor denied by domain of grembo@freebsd.org) smtp.mailfrom=grembo@freebsd.org X-Spamd-Result: default: False [1.39 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FREEFALL_USER(0.00)[grembo]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_MEDIUM(-0.88)[-0.878]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[freebsd.org]; R_SPF_SOFTFAIL(0.00)[~all:c]; RCPT_COUNT_FIVE(0.00)[5]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-0.996]; NEURAL_SPAM_LONG(0.86)[0.861]; MID_CONTAINS_FROM(1.00)[]; MLMMJ_DEST(0.00)[freebsd-net]; FREEMAIL_TO(0.00)[gmail.com,FreeBSD.org]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:213.239.192.0/18, country:DE]; RCVD_TLS_ALL(0.00)[]; SUSPICIOUS_RECIPS(1.50)[] X-ThisMailContainsUnwantedMimeParts: N On Sun, 13 Mar 2022 14:32:50 +0100 Johan Hendriks wrote: > On 13/03/2022 14:06, Patrick M. Hausen wrote: > > Hi all, > > > > i was a bit puzzled by Michael using bhyve trying to reproduce. > > Up until now I thought bhyve uses tap and not epair? > > > > Anyway ... > > > >> Am 13.03.2022 um 14:01 schrieb Johan Hendriks > >> : I have no idea why it does not work on > >> my setup, which is nothing out of the ordinary i think, basic full > >> jails connected to a bridge interface and one of them exposed to > >> the world wide web using pf binat. > > What we do is full exposed VNET jails connected to the bridge > > on the external interface of the host. ipfw kernel module loaded > > but not used in this case, i.e. only the "default to accept" rule > > active in the jails. > > > > I will probably downgrade the production host from 13.1-PRERELEASE > > to 13.0-pX tomorrow and see if that changes anything. > > > > Kind regards, > > Patrick > Downgrading to 13.0-p7 worked for me, it even works on 13.0-STABLE > till this commit 18 days ago. > https://freshbsd.org/freebsd/src/commit/2e0bee4c7f8176e0f8396c9389275745bac1e263 > > After that commit my setup stops working. > @all Johan gave me access to a test system where I could see the problem in action. There's nothing wrong with his config in respect to the issue at hand. I tried a few times more on my smaller test setup and I could reproduce the issue there now as well (with ncpu=2). I created a reduced test case that triggers the issue every time. It's assumed to be run on a dedicated vm or host. It doesn't require pf, bridges, tuning sysctl.conf, or any other special considerations. /etc/rc.conf is very basic/vanilla: hostname="johan" ifconfig_vtnet0="10.1.1.16/24" defaultrouter="10.1.1.1" gateway_enable="YES" sshd_enable="YES" dumpdev="NO" zfs_enable="YES" sendmail_enable="NONE" Script to test/reproduce: #!/bin/sh export PATH=/usr/local/bin:"$PATH" jname="tj" ename="epair_$jname" set -e echo "====> Install packages" pkg install -y haproxy hey echo "====> Remove some leftovers" ( killall hey || true jail -r "$jname" || true ifconfig "$ename" destroy || true ) 2>/dev/null sleep 1 echo "====> Create interfaces" intf=$(ifconfig epair create) jintf=$(echo "$intf" | sed "s|a$|b|") ifconfig "$intf" name "$ename" ifconfig "$ename" 10.233.185.1/24 echo "====> Create and start jail" jail -c vnet name="$jname" persist path=/ \ host.hostname="$jname" vnet.interface="$jintf" jexec "$jname" ifconfig lo0 127.0.0.1/8 jexec "$jname" ifconfig "$jintf" 10.233.185.2/24 up jexec "$jname" route add default 10.233.185.1 cat >/tmp/haproxy.conf< Start hey instances" hey -h2 -n 10 -c 10 -z 300s http://10.233.185.2& hey -h2 -n 10 -c 10 -z 300s http://10.233.185.2& hey -h2 -n 10 -c 10 -z 300s http://10.233.185.2& echo "====> Ping jail" ping 10.233.185.2 # EOF This script can be called multiple times in a row (it tears down what it created in previous runs). Now, testing with this script, I get: ====> Install packages Updating FreeBSD repository catalogue... FreeBSD repository is up to date. All repositories are up to date. Checking integrity... done (0 conflicting) The most recent versions of packages are already installed ====> Remove some leftovers tj: removed ====> Create interfaces epair_tj ====> Create and start jail add net default: gateway 10.233.185.1 ====> Start hey instances ====> Ping jail PING 10.233.185.2 (10.233.185.2): 56 data bytes 64 bytes from 10.233.185.2: icmp_seq=0 ttl=64 time=0.076 ms 64 bytes from 10.233.185.2: icmp_seq=1 ttl=64 time=0.138 ms 64 bytes from 10.233.185.2: icmp_seq=2 ttl=64 time=0.086 ms 64 bytes from 10.233.185.2: icmp_seq=3 ttl=64 time=0.158 ms 64 bytes from 10.233.185.2: icmp_seq=4 ttl=64 time=0.081 ms 64 bytes from 10.233.185.2: icmp_seq=5 ttl=64 time=0.093 ms At which point it gets stuck. The exact moment when this happens differs between runs, but it happens every time on my test host and always within a couple of seconds. It's important to point out that this only happens with kern.ncpu>1. With kern.ncpu==1 nothing gets stuck. This perfectly fits into the picture, since, as pointed out by Johan, the first commit that is affected[0] is about multicore support. Cheers Michael [0] https://cgit.freebsd.org/src/commit/?id=24f0bfbad57b9c3cb9b543a60b2ba00e4812c286 -- Michael Gmelin