From owner-freebsd-current@freebsd.org Mon Aug 3 18:27:09 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id EA5D43A4753 for ; Mon, 3 Aug 2020 18:27:09 +0000 (UTC) (envelope-from luzar722@gmail.com) Received: from mail-qk1-x72f.google.com (mail-qk1-x72f.google.com [IPv6:2607:f8b0:4864:20::72f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BL5vd18KTz4FNk for ; Mon, 3 Aug 2020 18:27:08 +0000 (UTC) (envelope-from luzar722@gmail.com) Received: by mail-qk1-x72f.google.com with SMTP id l6so36041335qkc.6 for ; Mon, 03 Aug 2020 11:27:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-transfer-encoding; bh=JAjfI37UCwO0ReSoLF+q44InJD20sgpF4bYdQI+Hjsg=; b=KFt0KQ3IKVt6APGWYjmUdgmuAnfIVNutepzFcsxXSq34Y2kEh05Q9++lDpcWzhGHr+ SEsMT0tejA6JfWqeccwjFoSn+7e4jVurt51ctit34wCEtTTnWFsKPbMNZCd+5FAwOT6d w4qDBt0acGyxrlwjBTt0GypYsstB/dlpHXx3O+Um5VKjdgdCkpBlQMISukHBA497YuY5 MZ4rqEj/RquUceynDzWFrbqIqOBgj4c138z9CyjQiaKOZkiUQ+88txzve3uimPhRAyxh WcKajW2jPgnqdS2RR2AtDb2Z3TeFoYB7wfSvUtoSOeKOy7vE/Fa+zjChcjoTpZDUpAla QMiQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-transfer-encoding; bh=JAjfI37UCwO0ReSoLF+q44InJD20sgpF4bYdQI+Hjsg=; b=XLA5YDmBhl/qJizSndzQv0L+0YFzk3mttW2SbRQcR9H3cnvAM7m0JfmyNSXsQP6M1l FqHAni74ykW+lXT2ToQ478lmJeNcT0BISSFHJtYeRHXs6MvCazf4MhZJqE5K3b8GkkLV cHlEvZpG6KfTmD8JDNob5HfgLJaZlp6WlHRAosc90fFyo+6cRsvyZA2vZqrdQ1XweOaW 3xlQOJCwhHD1X9JuAsgwGv5XH69ilpqgbGVOJa+82e8dyydj/Pd8KBgo9xdxzyGz1lB9 LcpTkFP6yg5NaAskyPmo8mQrLpmjcE5Lado4eugSBVUsp7+utc/XYwnjGiXK0/RMz+d0 mEKQ== X-Gm-Message-State: AOAM532xSX+llh5wri3UZqbxhxxDhY4aNiIjCoA2ULnhX1EWju6dDOAi tn3jq/X40PVEFLptG9oZAj4OTb0h X-Google-Smtp-Source: ABdhPJzYTQk62H6VUG+98RpmcYdkkCkwDb27HcC5tAwKW0UXECL4e7Dt6t1rSq/TFO9syN3daD6qsg== X-Received: by 2002:a05:620a:62f:: with SMTP id 15mr16828822qkv.483.1596479228084; Mon, 03 Aug 2020 11:27:08 -0700 (PDT) Received: from [10.0.10.8] (cpe-65-25-51-0.neo.res.rr.com. [65.25.51.0]) by smtp.googlemail.com with ESMTPSA id a67sm13960429qkd.40.2020.08.03.11.27.06 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 03 Aug 2020 11:27:07 -0700 (PDT) Message-ID: <5F2856FB.1010305@gmail.com> Date: Mon, 03 Aug 2020 14:27:07 -0400 From: Ernie Luzar User-Agent: Thunderbird 2.0.0.24 (Windows/20100228) MIME-Version: 1.0 To: Ronald Klop CC: freebsd-current@freebsd.org Subject: Re: vnet/jail crashdump References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4BL5vd18KTz4FNk X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=KFt0KQ3I; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of luzar722@gmail.com designates 2607:f8b0:4864:20::72f as permitted sender) smtp.mailfrom=luzar722@gmail.com X-Spamd-Result: default: False [-2.81 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RECEIVED_SPAMHAUS_PBL(0.00)[65.25.51.0:received]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.02)[-1.019]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_SPAM_SHORT(0.18)[0.182]; NEURAL_HAM_LONG(-0.97)[-0.974]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::72f:from]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Aug 2020 18:27:10 -0000 Ronald Klop wrote: > Hi, > > After stopping a jail I get a crashdump. > core.txt: > https://www.klop.ws/core_2eef39c581f90f2f0c4921e43f1998c1/core.txt.0 > > Jail.conf: > ---------- > exec.stop = "/bin/sh /etc/rc.shutdown"; > exec.clean; > > exec.prestart = "ifconfig bridge0 > /dev/null 2> /dev/null || ( ifconfig > bridge0 create && ifconfig bridge0 addm vtnet0 && ifconfig bridge0 up)"; > > exec.consolelog = "/var/log/jail_${name}_console.log"; > > mount.devfs; > path = "/data/jails/$name"; > host.hostname = "$name"; > mount.fstab = "/data/jails/fstab.$name"; > vnet; > allow.mlock; > devfs_ruleset="110"; > > freebsd12 { > osrelease = 12.1-RELEASE-p4; > osreldate = 1201000; > vnet.interface = "epair0b"; > # make sure the exec.prestart has a "+=" as we de it in the global > definition > # when checking for the bridge > exec.prestart += "ifconfig epair0 create up"; > exec.prestart += "ifconfig bridge0 addm epair0a"; > exec.prestart += "ifconfig epair0b link 02:xxxxxx:0c"; > exec.start = "dhclient epair0b"; > exec.start += "/bin/sh /etc/rc"; > exec.poststop = "ifconfig bridge0 deletem epair0a"; > exec.poststop += "ifconfig epair0a destroy"; > > } > freebsd13 { > vnet.interface = "epair1b"; > # make sure the exec.prestart has a "+=" as we de it in the global > definition > # when checking for the bridge > exec.prestart += "ifconfig epair1 create up"; > exec.prestart += "ifconfig bridge0 addm epair1a"; > exec.prestart += "ifconfig epair1b link 02:xxxxxx:0d"; > exec.start = "dhclient epair1b"; > exec.start += "/bin/sh /etc/rc"; > exec.poststop = "ifconfig bridge0 deletem epair1a"; > exec.poststop += "ifconfig epair1a destroy"; > } > ---------- > > What can I do to help debug? > Don't understand why you have these 2 statements exec.prestart += "ifconfig epair1b link 02:xxxxxx:0d"; exec.start = "dhclient epair1b"; There is a well known bug with bridge vnet tear down since release 9.0. Their is a rewrite of if_bridge going on right now to fix the problem and increase the performance of if_bridge. As of today this fix is not in 12.2 stable or 13.0 current. There also looks like a bug in jail(8) when you have both vnet jails and non-vnet jails being started on the same host at the same time. In most cases the host just loses internet access until all the jails are stopped. Some times you will get a system crash. This jail.conf def seems to work around the bridge tear down problem # vnet jail using the bridge/epair method on 12.1 v0jail1 { host.hostname = "v0jail1"; path = "/usr/jails/v0jail1"; mount.fstab = "/usr/local/etc/fstab/v0jail1"; exec.consolelog = "/var/log/v0jail1.console.log"; mount.devfs; devfs_ruleset = "4"; vnet = "new"; vnet.interface = "epair55b"; exec.prestart = "ifconfig epair55 create up"; exec.prestart += "ifconfig bridge0 addm epair55a"; exec.prestart += "ifconfig epair55a descr vnet-v0jail1"; exec.prestart += "ifconfig bridge0 inet 10.0.48.2 netmask 255.255.255.0 alias"; exec.start = "/bin/sh /etc/rc"; exec.start += "ifconfig epair55b inet 10.0.48.1 netmask 255.255.255.0"; exec.start += "route add default 10.0.48.2"; exec.prestop = "ifconfig epair55b -vnet v0jail1"; exec.stop = "/bin/sh /etc/rc.shutdown"; exec.poststop = "ifconfig bridge0 deletem epair55a"; exec.poststop += "sleep 2"; exec.poststop += "ifconfig epair55a destroy"; exec.poststop += "ifconfig bridge0 inet 10.0.48.2 -alias"; } Remember that your host firewall processes all traffic in & out of the host including any vnet jail traffic. Yes a vnet jail has its own stack and can have its own firewall, but the host firewall still has the last say. The host must NAT any private ip addresses used by the vnet jails. jail.conf jail definitions are based on hard codded ip addresses. You can not use the host dhcp to assign local lan private ip addresses to a jail. You may find this helpful https://forums.freebsd.org/threads/vnet-jail-with-public-internet-access-using-the-bridge-epair-method.76071/