From owner-freebsd-current@freebsd.org Mon Aug 3 20:12:28 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id DD59C3A6463 for ; Mon, 3 Aug 2020 20:12:28 +0000 (UTC) (envelope-from ronald-lists@klop.ws) Received: from smarthost1.greenhost.nl (smarthost1.greenhost.nl [195.190.28.88]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4BL8F66p3vz4Ly5 for ; Mon, 3 Aug 2020 20:12:26 +0000 (UTC) (envelope-from ronald-lists@klop.ws) Received: from smtp.greenhost.nl ([213.108.110.112]) by smarthost1.greenhost.nl with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1k2gop-0005U8-Gj; Mon, 03 Aug 2020 22:12:24 +0200 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=klop.ws; s=mail; h=In-Reply-To:Message-ID:From:Content-Transfer-Encoding:MIME-Version: Date:References:Subject:Cc:To:Content-Type:Sender:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=3pKrqkewy5Dq0loaEIBU4N9V84c4eWIgAD2JEPCm5pc=; b=YjO99PaG6r1jfDyX/McxabdSra 4uQ1NcG8Z7ggOQCJTj2ubG8+S8iu2Ca6I6iE2kvbxmBGObPi8dcQnUFZu8Tpfovx9RuFpigILklis Zfncd5MkNCu8si35pCTF6OVrHn6kju6ao3S9QZZuio3ajuUzvyh5YhKJ0obcy3rV7DAY=; Content-Type: text/plain; charset=utf-8; format=flowed; delsp=yes To: "Ernie Luzar" Cc: freebsd-current@freebsd.org Subject: Re: vnet/jail crashdump References: <5F2856FB.1010305@gmail.com> Date: Mon, 03 Aug 2020 22:12:19 +0200 MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: "Ronald Klop" Message-ID: In-Reply-To: <5F2856FB.1010305@gmail.com> User-Agent: Opera Mail/12.16 (FreeBSD) X-Authenticated-As-Hash: 398f5522cb258ce43cb679602f8cfe8b62a256d1 X-Virus-Scanned: by clamav at smarthost1.samage.net X-Spam-Level: / X-Spam-Score: -0.4 X-Spam-Status: No, score=-0.4 required=5.0 tests=ALL_TRUSTED, BAYES_50, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF autolearn=disabled version=3.4.2 X-Scan-Signature: 1621a07189ffd89c587b37d28d532eca X-Rspamd-Queue-Id: 4BL8F66p3vz4Ly5 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=klop.ws header.s=mail header.b=YjO99PaG; dmarc=none; spf=pass (mx1.freebsd.org: domain of ronald-lists@klop.ws designates 195.190.28.88 as permitted sender) smtp.mailfrom=ronald-lists@klop.ws X-Spamd-Result: default: False [-2.35 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.99)[-0.989]; R_DKIM_ALLOW(-0.20)[klop.ws:s=mail]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; RWL_MAILSPIKE_GOOD(0.00)[195.190.28.88:from]; R_SPF_ALLOW(-0.20)[+ip4:195.190.28.64/27]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[klop.ws]; NEURAL_HAM_LONG(-1.02)[-1.021]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[klop.ws:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[195.190.28.88:from]; NEURAL_HAM_SHORT(-0.34)[-0.336]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; MID_RHS_NOT_FQDN(0.50)[]; ASN(0.00)[asn:47172, ipnet:195.190.28.0/24, country:NL]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Aug 2020 20:12:28 -0000 On Mon, 03 Aug 2020 20:27:07 +0200, Ernie Luzar wrote: > Ronald Klop wrote: >> Hi, >> After stopping a jail I get a crashdump. >> core.txt: >> https://www.klop.ws/core_2eef39c581f90f2f0c4921e43f1998c1/core.txt.0 >> Jail.conf: >> ---------- >> exec.stop = "/bin/sh /etc/rc.shutdown"; >> exec.clean; >> exec.prestart = "ifconfig bridge0 > /dev/null 2> /dev/null || ( >> ifconfig bridge0 create && ifconfig bridge0 addm vtnet0 && ifconfig >> bridge0 up)"; >> exec.consolelog = "/var/log/jail_${name}_console.log"; >> mount.devfs; >> path = "/data/jails/$name"; >> host.hostname = "$name"; >> mount.fstab = "/data/jails/fstab.$name"; >> vnet; >> allow.mlock; >> devfs_ruleset="110"; >> freebsd12 { >> osrelease = 12.1-RELEASE-p4; >> osreldate = 1201000; >> vnet.interface = "epair0b"; >> # make sure the exec.prestart has a "+=" as we de it in the global >> definition >> # when checking for the bridge >> exec.prestart += "ifconfig epair0 create up"; >> exec.prestart += "ifconfig bridge0 addm epair0a"; >> exec.prestart += "ifconfig epair0b link 02:xxxxxx:0c"; >> exec.start = "dhclient epair0b"; >> exec.start += "/bin/sh /etc/rc"; >> exec.poststop = "ifconfig bridge0 deletem epair0a"; >> exec.poststop += "ifconfig epair0a destroy"; >> } >> freebsd13 { >> vnet.interface = "epair1b"; >> # make sure the exec.prestart has a "+=" as we de it in the global >> definition >> # when checking for the bridge >> exec.prestart += "ifconfig epair1 create up"; >> exec.prestart += "ifconfig bridge0 addm epair1a"; >> exec.prestart += "ifconfig epair1b link 02:xxxxxx:0d"; >> exec.start = "dhclient epair1b"; >> exec.start += "/bin/sh /etc/rc"; >> exec.poststop = "ifconfig bridge0 deletem epair1a"; >> exec.poststop += "ifconfig epair1a destroy"; >> } >> ---------- >> What can I do to help debug? >> > > > Don't understand why you have these 2 statements > > exec.prestart += "ifconfig epair1b link 02:xxxxxx:0d"; > exec.start = "dhclient epair1b"; Using dhcp on a fixed MAC is much faster in my network. This might be written in a better way. Please enlighten me. After a lot of twiddling with settings this worked. > There is a well known bug with bridge vnet tear down since release 9.0. > Their is a rewrite of if_bridge going on right now to fix the problem > and increase the performance of if_bridge. As of today this fix is not > in 12.2 stable or 13.0 current. Ah ok, so it is a known issue. > There also looks like a bug in jail(8) when you have both vnet jails and > non-vnet jails being started on the same host at the same time. In most > cases the host just loses internet access until all the jails are > stopped. Some times you will get a system crash. Ok. Not my use case, but good to know. > This jail.conf def seems to work around the bridge tear down problem > > # vnet jail using the bridge/epair method on 12.1 > v0jail1 { > host.hostname = "v0jail1"; > path = "/usr/jails/v0jail1"; > mount.fstab = "/usr/local/etc/fstab/v0jail1"; > exec.consolelog = "/var/log/v0jail1.console.log"; > mount.devfs; > devfs_ruleset = "4"; > vnet = "new"; > vnet.interface = "epair55b"; > exec.prestart = "ifconfig epair55 create up"; > exec.prestart += "ifconfig bridge0 addm epair55a"; > exec.prestart += "ifconfig epair55a descr vnet-v0jail1"; > exec.prestart += "ifconfig bridge0 inet 10.0.48.2 netmask 255.255.255.0 > alias"; > exec.start = "/bin/sh /etc/rc"; > exec.start += "ifconfig epair55b inet 10.0.48.1 netmask > 255.255.255.0"; > exec.start += "route add default 10.0.48.2"; > exec.prestop = "ifconfig epair55b -vnet v0jail1"; > exec.stop = "/bin/sh /etc/rc.shutdown"; > exec.poststop = "ifconfig bridge0 deletem epair55a"; > exec.poststop += "sleep 2"; > exec.poststop += "ifconfig epair55a destroy"; > exec.poststop += "ifconfig bridge0 inet 10.0.48.2 -alias"; > } > > Remember that your host firewall processes all traffic in & out of the > host including any vnet jail traffic. Yes a vnet jail has its own stack > and can have its own firewall, but the host firewall still has the last > say. The host must NAT any private ip addresses used by the vnet jails. > > jail.conf jail definitions are based on hard codded ip addresses. You > can not use the host dhcp to assign local lan private ip addresses to a > jail. > > You may find this helpful > > https://forums.freebsd.org/threads/vnet-jail-with-public-internet-access-using-the-bridge-epair-method.76071/ > Thanks for all the info. Ronald.