From owner-freebsd-jail@freebsd.org Sun Aug 2 18:49:59 2020 Return-Path: Delivered-To: freebsd-jail@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 46F353A4DBD; Sun, 2 Aug 2020 18:49:59 +0000 (UTC) (envelope-from luzar722@gmail.com) Received: from mail-qt1-x832.google.com (mail-qt1-x832.google.com [IPv6:2607:f8b0:4864:20::832]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BKVSQ36nvz3YFw; Sun, 2 Aug 2020 18:49:58 +0000 (UTC) (envelope-from luzar722@gmail.com) Received: by mail-qt1-x832.google.com with SMTP id o22so26522124qtt.13; Sun, 02 Aug 2020 11:49:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-transfer-encoding; bh=Ve9f5UlQyqCWclV7CQngHZ6H5aIprjMzlJuvaTmaQlQ=; b=NWQNJrUwaxnOCScKatUa/jX7dBL8jQgqsVXo62J4WCI0bP6OrkkAHrsVJXujhqpFPq dM16X9qEh6ZuBa0hybgRdIaYrdEriz591gkb7OgQjpOjiadKyXtyEJuN37nOe+X8LAf1 MDB8LT2nIOcmciMX5HfEO4ihROmgGychFEaf7JVtHz8x8b6TU87olBaL0S5IxcnlHsH1 yZ+hBPVXitKNNht2RfvMbk0URvWJe+Q8bf3Kl8/68X0kTDekB2JCVALhYfmY7hfYFOgw k7aNwo2+yvJNrnpvSuo8NWdZy7KKKY2RZ2pPztCDcStKM1mWOa8dwjUZgGK1UZMgnGjp ZCmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-transfer-encoding; bh=Ve9f5UlQyqCWclV7CQngHZ6H5aIprjMzlJuvaTmaQlQ=; b=XEgu+tsGHKxsfPrv+CmzNSeKxCjmpP0EeMTJaVuLbAb7RGjSk94WWlFP4SNoxYl/2d 23LThYBjMRtr0RDF0qdQohKlKO1J+7tSsFapuNZTBEe37OgrcSmhpNBVvz+MXZyqc3+E HsHOPvMftpIBabmnsp0uU+b3c/gejaQQ30Zy0LwA0zs8g9p4dydjF6GhG+r/qR3wemEp QLNber0sXWkYf/JL7++jQPRFkboQM6Nasw4ReoRCs6FO38m69z4JFiViiNqs/o/mYsur z/WCgMaJg6DyG19Ck78n+egK6sgoqMwi71g0Z/a6HQL9a8QtFJ6WCp1PlQgoEWRsIYvR GilA== X-Gm-Message-State: AOAM5304xvLHlvWCazaUV2ZqTqG5yjQyMbuvFLdphLPYC53TvC+V3E/c N5jfuq5dh4D+P70/divsNXBJKdnQ X-Google-Smtp-Source: ABdhPJzjq10UVHqSphdNALvJPaFegfKKNaaJkvPaj2V2kdGQGIennpQqVbGVQGuU7SNdauFT/45QzA== X-Received: by 2002:ac8:7c8d:: with SMTP id y13mr13409740qtv.387.1596394197304; Sun, 02 Aug 2020 11:49:57 -0700 (PDT) Received: from [10.0.10.8] (cpe-65-25-51-0.neo.res.rr.com. [65.25.51.0]) by smtp.googlemail.com with ESMTPSA id c33sm19653756qtk.40.2020.08.02.11.49.56 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sun, 02 Aug 2020 11:49:56 -0700 (PDT) Message-ID: <5F270AD4.8080001@gmail.com> Date: Sun, 02 Aug 2020 14:49:56 -0400 From: Ernie Luzar User-Agent: Thunderbird 2.0.0.24 (Windows/20100228) MIME-Version: 1.0 To: Dan Langille CC: "freebsd-questions@freebsd.org" , "freebsd-jail@freebsd.org" Subject: Re: jail(8) bug with vnet & non-vnet jails running at same time? References: <5F26FC5B.6030706@gmail.com> In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4BKVSQ36nvz3YFw X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=NWQNJrUw; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of luzar722@gmail.com designates 2607:f8b0:4864:20::832 as permitted sender) smtp.mailfrom=luzar722@gmail.com X-Spamd-Result: default: False [-2.46 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.52)[-0.521]; RECEIVED_SPAMHAUS_PBL(0.00)[65.25.51.0:received]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.98)[-0.983]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-0.95)[-0.952]; MIME_GOOD(-0.10)[text/plain]; SUBJECT_ENDS_QUESTION(1.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::832:from]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Aug 2020 18:49:59 -0000 Dan Langille wrote: >> On Aug 2, 2020, at 1:48 PM, Ernie Luzar wrote: >> >> Hello list; >> Please review configuration looking for something I may have missed. Hopping someone can suggest something that will change the behavior eliminating the problem. >> >> >> Equipment. Real hardware, 12.1 release, amd64 dual cpu. >> >> Description; >> non-vnet jails and vnet jails using the bridge/epair method can ping the public internet when only non-vnet jails are started at a time or when only vnet jails are started at a time. But when both non-vnet jails and vnet jails are started together then neither one can ping the public internet. The order of the jails definitions in the jail.conf file has no effect on changing what is happening. >> >> Bug description: >> When non-vnet jails are started their ip addresses are added to the NIC facing the public AFTER the public ip address and the non-vnet jail has access to the public internet. But when both non-vnet jails and vnet jails are started at the same time then the non-vnet jails ip addresses gets added before the public ip address of the NIC facing the public internet causing the host to lose all access to the public internet. This seems to be a jail(8) bug. >> >> It makes no difference which command method is used to start and stop the jails. >> Service jail onestart jailname or jail –cv jailname > > This may be related to my twitter rant about vnet problems in my own jails: > > https://twitter.com/DLangille/status/1289944047763693569 > > The symptoms you describe to similar to my own. I cannot access ports on jails on the same host, but I can access ports on other hosts. > Your twitter posts are all pf firewall related. From what I can tell you are using local only vnet jails and want to talk between them. Do you have any non-vnet jails running on the host where the 2 vnet jails are running? Do you have any local only vnet jails working on any other systems? To me knowledge there is only 1 way to have local only vnet jails to talk to each other. Do not assign ip address to epairXa or to the bridge. Only assign an ip address to epairXb the interface in the vnet jail. All the vnet jails you want to be local only have to be members on the same bridge.