From owner-freebsd-jail@freebsd.org Sun Aug 2 17:48:16 2020 Return-Path: Delivered-To: freebsd-jail@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0E86F3A37C1; Sun, 2 Aug 2020 17:48:16 +0000 (UTC) (envelope-from luzar722@gmail.com) Received: from mail-qt1-x844.google.com (mail-qt1-x844.google.com [IPv6:2607:f8b0:4864:20::844]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BKT5B1s5nz3Tw7; Sun, 2 Aug 2020 17:48:13 +0000 (UTC) (envelope-from luzar722@gmail.com) Received: by mail-qt1-x844.google.com with SMTP id v22so20780226qtq.8; Sun, 02 Aug 2020 10:48:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:date:from:user-agent:mime-version:to:subject :content-transfer-encoding; bh=xwFe9guNwA1dGgL48tMXyF8rwowlvw2p6UvPFVpArMM=; b=QfMdyleWvQx1Jf8Rk9m88SCW2i4o1t1QUkCuwLVQRr45DyHs7iLAZg/ExJZPetu3nq TEQdu6PxM+ym9RplV9rR4Bgs5DBLFv1ahHJjcFSTYSKSuzucLYymUpSjHT/JYLYGga9K YGLsBCUQLbHe+nHLTDxbrVr6Kakesy+uIFaGvIQlUCRTYconMeqii4G+PV74moJ2fAz4 bl8AIralwVK/KcFJGjMw42nWobVE/pC/avwxfAWB+B7lyskUPsDtZnNKV+R1zJOxTy5V +hl9FhwZK22eWU/2hY3g7AoHTiJey8Y/cV10F3iaCGmZtY/YT1lAwdgIaGixoc+n34qB DZFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :subject:content-transfer-encoding; bh=xwFe9guNwA1dGgL48tMXyF8rwowlvw2p6UvPFVpArMM=; b=ZnRxHF3QBd/WNv5Zlju/rcZKQ/ujMx7/RS1bDdp4qlDLpInGA/K3n9T8U1XQ8ZfZyt TEkA3n5nRglrLumQ+CSe9FdUKfLo2L80mrzNDssg+iVhO8ZyoN0MoaZgkRemPNlBLH19 tsPkMvz1qBVJMzIcjXlTY8WZeSddaln34aXS4CPk/MAMmubGZo/m1HIu0Nter1PZ8SVy zKU/kzLBQBAW0HRru0G/xWDxhB6RLbrA257fcPRYaMaTh2eiZqRxOpl4MzxuJcTYQa7W EHlqByWn5BEBd5EaJHdzCfhMNZnPBQJsoYwBNrzQccxHiWnS4VkFc+mMykddmqj7XKJE Z0aQ== X-Gm-Message-State: AOAM530IbLQwSg0Z7vW7ecuPO7T4v7X99Lr5OSudo11X2nbR51y+sUtM Nhb3ODOVngIb4lLMatw3vYAGR5MH X-Google-Smtp-Source: ABdhPJz4MyCz1Pk/dMBtybrKZqzAzVVeWYA9UcJzmdHh8K+hYYa9Ri+LKyRlEOA+h56oqkBS2hkziw== X-Received: by 2002:ac8:7383:: with SMTP id t3mr13020575qtp.160.1596390492663; Sun, 02 Aug 2020 10:48:12 -0700 (PDT) Received: from [10.0.10.8] (cpe-65-25-51-0.neo.res.rr.com. [65.25.51.0]) by smtp.googlemail.com with ESMTPSA id c205sm18018744qkg.98.2020.08.02.10.48.11 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sun, 02 Aug 2020 10:48:11 -0700 (PDT) Message-ID: <5F26FC5B.6030706@gmail.com> Date: Sun, 02 Aug 2020 13:48:11 -0400 From: Ernie Luzar User-Agent: Thunderbird 2.0.0.24 (Windows/20100228) MIME-Version: 1.0 To: "freebsd-questions@freebsd.org" , freebsd-jail@FreeBSD.org Subject: jail(8) bug with vnet & non-vnet jails running at same time? Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4BKT5B1s5nz3Tw7 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=QfMdyleW; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of luzar722@gmail.com designates 2607:f8b0:4864:20::844 as permitted sender) smtp.mailfrom=luzar722@gmail.com X-Spamd-Result: default: False [-2.46 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.53)[-0.534]; RECEIVED_SPAMHAUS_PBL(0.00)[65.25.51.0:received]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.97)[-0.971]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-0.96)[-0.959]; MIME_GOOD(-0.10)[text/plain]; SUBJECT_ENDS_QUESTION(1.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::844:from]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Aug 2020 17:48:16 -0000 Hello list; Please review configuration looking for something I may have missed. Hopping someone can suggest something that will change the behavior eliminating the problem. Equipment. Real hardware, 12.1 release, amd64 dual cpu. Description; non-vnet jails and vnet jails using the bridge/epair method can ping the public internet when only non-vnet jails are started at a time or when only vnet jails are started at a time. But when both non-vnet jails and vnet jails are started together then neither one can ping the public internet. The order of the jails definitions in the jail.conf file has no effect on changing what is happening. Bug description: When non-vnet jails are started their ip addresses are added to the NIC facing the public AFTER the public ip address and the non-vnet jail has access to the public internet. But when both non-vnet jails and vnet jails are started at the same time then the non-vnet jails ip addresses gets added before the public ip address of the NIC facing the public internet causing the host to lose all access to the public internet. This seems to be a jail(8) bug. It makes no difference which command method is used to start and stop the jails. Service jail onestart jailname or jail –cv jailname The following is a capture of the command sequence showing this bug. Follow the re0 NIC public ip address xx.25.51.0 in the ifconfig -a listing. Before any jails are started. /root >ifconfig -a snip ... re0: flags=8943 metric 0 mtu 1500 options=8209b ether 50:3e:aa:06:11:22 inet xx.25.51.0 netmask 0xfffff000 broadcast 255.255.255.255 media: Ethernet autoselect (1000baseT ) status: active nd6 options=29 bridge0: flags=8843 metric 0 mtu 1500 ether 02:3e:ba:a7:58:00 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: re0 flags=143 ifmaxaddr 0 port 2 priority 128 path cost 20000 groups: bridge nd6 options=1 /root >cat /etc/jail.conf # non-vnet jail zdir20 { host.hostname = "zdir20"; path = "/usr/jails/zdir20"; mount.fstab = "/usr/local/etc/fstab/zdir20"; exec.consolelog = "/var/log/zdir20.console.log"; mount.devfs; ip4.addr = 10.0.22.5; interface = "re0"; allow.raw_sockets; devfs_ruleset = "4"; exec.start = "/bin/sh /etc/rc"; exec.stop = "/bin/sh /etc/rc.shutdown"; } # vnet jail using the bridge/epair method v0jail1 { host.hostname = "v0jail1"; path = "/usr/jails/v0jail1"; mount.fstab = "/usr/local/etc/fstab/v0jail1"; exec.consolelog = "/var/log/v0jail1.console.log"; mount.devfs; devfs_ruleset = "4"; vnet = "new"; vnet.interface = "epair55b"; exec.prestart = "ifconfig epair55 create up"; exec.prestart += "ifconfig bridge0 addm epair55a"; exec.prestart += "ifconfig epair55a descr vnet-v0jail1"; exec.prestart += "ifconfig bridge0 inet 10.0.48.2 netmask 255.255.255.0 alias"; exec.start = "/bin/sh /etc/rc"; exec.start += "ifconfig epair55b inet 10.0.48.1 netmask 255.255.255.0"; exec.start += "route add default 10.0.48.2"; exec.prestop = "ifconfig epair55b -vnet v0jail1"; exec.stop = "/bin/sh /etc/rc.shutdown"; exec.poststop = "ifconfig bridge0 deletem epair55a"; exec.poststop += "sleep 2"; exec.poststop += "ifconfig epair55a destroy"; exec.poststop += "ifconfig bridge0 inet 10.0.48.2 -alias"; } /root >jls JID IP Address Hostname Path # start only the non-vnet jail /root >service jail onestart zdir20 Starting jails: zdir20. /root >jls JID IP Address Hostname Path 18 10.0.22.5 zdir20 /usr/jails/zdir20 # Take notice that the non-vnet jails ip address follows the nic’s # public ip address. /root >ifconfig -a re0: flags=8943 metric 0 mtu 1500 options=8209b ether 50:3e:aa:06:11:22 inet xx.25.51.0 netmask 0xfffff000 broadcast 255.255.255.255 inet 10.0.22.5 netmask 0xffffffff broadcast 10.0.22.5 media: Ethernet autoselect (1000baseT ) status: active nd6 options=29 bridge0: flags=8843 metric 0 mtu 1500 ether 02:3e:ba:a7:58:00 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: re0 flags=143 ifmaxaddr 0 port 2 priority 128 path cost 20000 groups: bridge nd6 options=1 # login to the non-vnet jail and ping the public /root >jexec zdir20 login -f root Last login: Sun Aug 2 11:30:40 on pts/0 FreeBSD 12.1-RELEASE-p6 GENERIC Welcome to your FreeBSD jail. zdir20 /root > zdir20 /root >ping -c 2 freebsd.org PING freebsd.org (96.47.72.84): 56 data bytes 64 bytes from 96.47.72.84: icmp_seq=0 ttl=48 time=44.426 ms 64 bytes from 96.47.72.84: icmp_seq=1 ttl=48 time=44.481 ms --- freebsd.org ping statistics --- 2 packets transmitted, 2 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 44.426/44.453/44.481/0.027 ms zdir20 /root >exit logout # stop the non-vnet jail and show that the network is back to # starting condition. /root >service jail onestop zdir20 Stopping jails: zdir20. /root >jls JID IP Address Hostname Path /root >ifconfig -a re0: flags=8943 metric 0 mtu 1500 options=8209b ether 50:3e:aa:06:11:22 inet xx.25.51.0 netmask 0xfffff000 broadcast 255.255.255.255 media: Ethernet autoselect (1000baseT ) status: active nd6 options=29 bridge0: flags=8843 metric 0 mtu 1500 ether 02:3e:ba:a7:58:00 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: re0 flags=143 ifmaxaddr 0 port 2 priority 128 path cost 20000 groups: bridge nd6 options=1 # start only the vnet jail and see the bridge0 /root >service jail onestart v0jail1 Starting jails: v0jail1. /root >jls JID IP Address Hostname Path 19 v0jail1 /usr/jails/v0jail1 /root >ifconfig -a re0: flags=8943 metric 0 mtu 1500 options=82099 ether 50:3e:aa:06:11:22 inet xx.25.51.0 netmask 0xfffff000 broadcast 255.255.255.255 media: Ethernet autoselect (1000baseT ) status: active nd6 options=29 bridge0: flags=8843 metric 0 mtu 1500 ether 02:3e:ba:a7:58:00 inet 10.0.48.2 netmask 0xffffff00 broadcast 10.0.48.255 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: epair55a flags=143 ifmaxaddr 0 port 5 priority 128 path cost 2000 member: re0 flags=143 ifmaxaddr 0 port 2 priority 128 path cost 20000 groups: bridge nd6 options=1 epair55a: flags=8943 metric 0 mtu 1500 description: vnet-v0jail1 options=8 ether 02:eb:be:f5:15:0a inet6 fe80::eb:beff:fef5:150a%epair55a prefixlen 64 scopeid 0x5 groups: epair media: Ethernet 10Gbase-T (10Gbase-T ) status: active nd6 options=21 # login to the vnet jail and ping the public internet. /root >jexec v0jail1 login -f root Last login: Sun Aug 2 11:29:41 on pts/0 FreeBSD 12.1-RELEASE-p6 GENERIC Welcome to your FreeBSD jail. v0jail1 /root >ping -c 2 freebsd.org PING freebsd.org (96.47.72.84): 56 data bytes 64 bytes from 96.47.72.84: icmp_seq=0 ttl=47 time=46.745 ms 64 bytes from 96.47.72.84: icmp_seq=1 ttl=47 time=43.930 ms --- freebsd.org ping statistics --- 2 packets transmitted, 2 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 43.930/45.337/46.745/1.407 ms v0jail1 /root >exit logout # close the vnet jail and return to starting condition. /root >service jail onestop v0jail1 Stopping jails: v0jail1. /root >jls JID IP Address Hostname Path /root >ifconfig -a re0: flags=8943 metric 0 mtu 1500 options=8209b ether 50:3e:aa:06:11:22 inet xx.25.51.0 netmask 0xfffff000 broadcast 255.255.255.255 media: Ethernet autoselect (1000baseT ) status: active nd6 options=29 bridge0: flags=8843 metric 0 mtu 1500 ether 02:3e:ba:a7:58:00 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: re0 flags=143 ifmaxaddr 0 port 2 priority 128 path cost 20000 groups: bridge nd6 options=1 # Start both the non-vnet jail and the vnet jail together. /root >service jail onestart Starting jails: zdir20 v0jail1. # login to the non-vnet jail and it has no public access. /root >jexec zdir20 login -f root [K Last login: Sun Aug 2 11:36:34 on pts/0 FreeBSD 12.1-RELEASE-p6 GENERIC Welcome to your FreeBSD jail. zdir20 /root >ping -c 2 freebsd.org ping: cannot resolve freebsd.org: Host name lookup failure zdir20 /root >exit logout # login to the vnet jail and it has no public access. /root >jexec v0jail1 login -f root Last login: Sun Aug 2 11:38:56 on pts/0 FreeBSD 12.1-RELEASE-p6 GENERIC Welcome to your FreeBSD jail. v0jail1 /root >ping -c 2 freebsd.org ping: cannot resolve freebsd.org: Host name lookup failure v0jail1 /root >exit logout /root >jls JID IP Address Hostname Path 20 10.0.22.5 zdir20 /usr/jails/zdir20 21 v0jail1 /usr/jails/v0jail1 # Here is the bug. See that the non-vnet jail ip address comes before the # public address causing the host to lose access to the public internet. /root >ifconfig -a re0: flags=8943 metric 0 mtu 1500 options=82099 ether 50:3e:aa:06:11:22 inet 10.0.22.5 netmask 0xffffffff broadcast 10.0.22.5 inet xx.25.51.0 netmask 0xfffff000 broadcast 255.255.255.255 media: Ethernet autoselect (1000baseT ) status: active nd6 options=29 bridge0: flags=8843 metric 0 mtu 1500 ether 02:3e:ba:a7:58:00 inet 10.0.48.2 netmask 0xffffff00 broadcast 10.0.48.255 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: epair55a flags=143 ifmaxaddr 0 port 5 priority 128 path cost 2000 member: re0 flags=143 ifmaxaddr 0 port 2 priority 128 path cost 20000 groups: bridge nd6 options=1 epair55a: flags=8943 metric 0 mtu 1500 description: vnet-v0jail1 options=8 ether 02:77:b8:5f:e4:0a inet6 fe80::77:b8ff:fe5f:e40a%epair55a prefixlen 64 scopeid 0x5 groups: epair media: Ethernet 10Gbase-T (10Gbase-T ) status: active nd6 options=21 # stop both jails and return to starting condition. /root >service jail onestop Stopping jails: zdir20 v0jail1. /root >jls JID IP Address Hostname Path /root >ifconfig -a re0: flags=8943 metric 0 mtu 1500 options=8209b ether 50:3e:aa:06:11:22 inet xx.25.51.0 netmask 0xfffff000 broadcast 255.255.255.255 media: Ethernet autoselect (1000baseT ) status: active nd6 options=29 bridge0: flags=8843 metric 0 mtu 1500 ether 02:3e:ba:a7:58:00 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: re0 flags=143 ifmaxaddr 0 port 2 priority 128 path cost 20000 groups: bridge nd6 options=1 From owner-freebsd-jail@freebsd.org Sun Aug 2 17:55:37 2020 Return-Path: Delivered-To: freebsd-jail@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C75BF3A36F5; Sun, 2 Aug 2020 17:55:37 +0000 (UTC) (envelope-from dan@langille.org) Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4BKTFh1JDpz3V1Y; Sun, 2 Aug 2020 17:55:32 +0000 (UTC) (envelope-from dan@langille.org) Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailout.nyi.internal (Postfix) with ESMTP id 8F16F5C00B9; Sun, 2 Aug 2020 13:55:32 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Sun, 02 Aug 2020 13:55:32 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=langille.org; h= content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; s=fm1; bh=0 +fFF6R0a9eR9HEYhklm8fqCysHrqAfHpioJdqOLj4Y=; b=UD4OsdWupSbNPX6SQ rY6hwmsUAwaErZU1gsEavpNGMo7/N3o60cEFm4gG3nlH8zCZMgwouAPlP613cM/o 33D1bZiY/R7SkAzfScDiPZkpyYlCvviYSf+s4qFGwwOOxsjnbAB5Y9LoTUezldon BF6QwfvbFmpGyoxJ+32Y60NIBXIlR+rXOLoX0YzaBkCQ49Z3ny6IOSZhK4nGJDT1 3twNBWL+Ov5pPN4vsYwtTMWKfazvoTbiBBA7MridncnuIkJ5MrUejYQNU3erzQVc ZBPcj2lW6l8DnkrVeYa4KlhL4QlaqKoZlUO124WrUUM3AxuBARhhROQNbWttWg8s Mj2DQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; bh=0+fFF6R0a9eR9HEYhklm8fqCysHrqAfHpioJdqOLj 4Y=; b=tVMzG8+zJw8RouBHeMBvZOPb3QUIgCY4gPNAGzCsrw+TV3HacX66DSGeX cY8O7/Sxx3s8LpRngZ4OUVwz+Xs1XgGFmpPpIi0BtZu9enYV3NKZmEIRPSfJ7iKN Sk8yts/xecd67q6Gksu/uCOJR/JBx7FRs+FwQ3zlue6P8EVa4YoeS2Iu/FlsevPo KB361To0o3iXcxQILFeWEbbSaGTb0z8DMUDHzTgK0QOpSj9nDoJhwATPofgZgerW hhIYge/P33u2UEelHZvwWxlIUSp1SB5Al9wUmjUBR+DLmuuSdWnt1ytPpO8iVCn8 u8CUp1EGVcBj8JtwrEY88IhRLE9xA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduiedrjedvgdduvdduucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurheptggguffhjgffgffkfhfvofesthhqmhdthhdtjeenucfhrhhomhepffgrnhcu nfgrnhhgihhllhgvuceouggrnheslhgrnhhgihhllhgvrdhorhhgqeenucggtffrrghtth gvrhhnpeevvdfgveeugeffieduhfetffduvdffkedtleejteegleduffevhfefkeehgedv veenucffohhmrghinhepthifihhtthgvrhdrtghomhenucfkphepuddtkedrfeeirdelhe druddtnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhep uggrnheslhgrnhhgihhllhgvrdhorhhg X-ME-Proxy: Received: from air01.wifi.int.unixathome.org (pool-108-36-95-10.phlapa.fios.verizon.net [108.36.95.10]) by mail.messagingengine.com (Postfix) with ESMTPA id 2604D3280059; Sun, 2 Aug 2020 13:55:32 -0400 (EDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.1\)) Subject: Re: jail(8) bug with vnet & non-vnet jails running at same time? From: Dan Langille In-Reply-To: <5F26FC5B.6030706@gmail.com> Date: Sun, 2 Aug 2020 13:55:31 -0400 Cc: "freebsd-questions@freebsd.org" , "freebsd-jail@freebsd.org" Content-Transfer-Encoding: quoted-printable Message-Id: References: <5F26FC5B.6030706@gmail.com> To: Ernie Luzar X-Mailer: Apple Mail (2.3608.120.23.2.1) X-Rspamd-Queue-Id: 4BKTFh1JDpz3V1Y X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=langille.org header.s=fm1 header.b=UD4OsdWu; dkim=pass header.d=messagingengine.com header.s=fm3 header.b=tVMzG8+z; dmarc=pass (policy=none) header.from=langille.org; spf=pass (mx1.freebsd.org: domain of dan@langille.org designates 66.111.4.25 as permitted sender) smtp.mailfrom=dan@langille.org X-Spamd-Result: default: False [-2.97 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ip4:66.111.4.25]; RWL_MAILSPIKE_GOOD(0.00)[66.111.4.25:from]; RCVD_COUNT_THREE(0.00)[4]; DKIM_TRACE(0.00)[langille.org:+,messagingengine.com:+]; DMARC_POLICY_ALLOW(-0.50)[langille.org,none]; NEURAL_HAM_SHORT(-1.45)[-1.449]; FREEMAIL_TO(0.00)[gmail.com]; RECEIVED_SPAMHAUS_PBL(0.00)[108.36.95.10:received]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11403, ipnet:66.111.4.0/24, country:US]; SUBJECT_ENDS_QUESTION(1.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_IN_DNSWL_LOW(-0.10)[66.111.4.25:from]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.002]; R_DKIM_ALLOW(-0.20)[langille.org:s=fm1,messagingengine.com:s=fm3]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-0.92)[-0.922]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[] X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Aug 2020 17:55:37 -0000 > On Aug 2, 2020, at 1:48 PM, Ernie Luzar wrote: >=20 > Hello list; > Please review configuration looking for something I may have missed. = Hopping someone can suggest something that will change the behavior = eliminating the problem. >=20 >=20 > Equipment. Real hardware, 12.1 release, amd64 dual cpu. >=20 > Description; > non-vnet jails and vnet jails using the bridge/epair method can ping = the public internet when only non-vnet jails are started at a time or = when only vnet jails are started at a time. But when both non-vnet jails = and vnet jails are started together then neither one can ping the public = internet. The order of the jails definitions in the jail.conf file has = no effect on changing what is happening. >=20 > Bug description: > When non-vnet jails are started their ip addresses are added to the = NIC facing the public AFTER the public ip address and the non-vnet jail = has access to the public internet. But when both non-vnet jails and vnet = jails are started at the same time then the non-vnet jails ip addresses = gets added before the public ip address of the NIC facing the public = internet causing the host to lose all access to the public internet. = This seems to be a jail(8) bug. >=20 > It makes no difference which command method is used to start and stop = the jails. > Service jail onestart jailname or jail =E2=80=93cv jailname This may be related to my twitter rant about vnet problems in my own = jails: https://twitter.com/DLangille/status/1289944047763693569 The symptoms you describe to similar to my own. I cannot access ports = on jails on the same host, but I can access ports on other hosts. --=20 Dan Langille - BSDCan / PGCon dan@langille.org From owner-freebsd-jail@freebsd.org Sun Aug 2 18:49:59 2020 Return-Path: Delivered-To: freebsd-jail@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 46F353A4DBD; Sun, 2 Aug 2020 18:49:59 +0000 (UTC) (envelope-from luzar722@gmail.com) Received: from mail-qt1-x832.google.com (mail-qt1-x832.google.com [IPv6:2607:f8b0:4864:20::832]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BKVSQ36nvz3YFw; Sun, 2 Aug 2020 18:49:58 +0000 (UTC) (envelope-from luzar722@gmail.com) Received: by mail-qt1-x832.google.com with SMTP id o22so26522124qtt.13; Sun, 02 Aug 2020 11:49:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-transfer-encoding; bh=Ve9f5UlQyqCWclV7CQngHZ6H5aIprjMzlJuvaTmaQlQ=; b=NWQNJrUwaxnOCScKatUa/jX7dBL8jQgqsVXo62J4WCI0bP6OrkkAHrsVJXujhqpFPq dM16X9qEh6ZuBa0hybgRdIaYrdEriz591gkb7OgQjpOjiadKyXtyEJuN37nOe+X8LAf1 MDB8LT2nIOcmciMX5HfEO4ihROmgGychFEaf7JVtHz8x8b6TU87olBaL0S5IxcnlHsH1 yZ+hBPVXitKNNht2RfvMbk0URvWJe+Q8bf3Kl8/68X0kTDekB2JCVALhYfmY7hfYFOgw k7aNwo2+yvJNrnpvSuo8NWdZy7KKKY2RZ2pPztCDcStKM1mWOa8dwjUZgGK1UZMgnGjp ZCmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-transfer-encoding; bh=Ve9f5UlQyqCWclV7CQngHZ6H5aIprjMzlJuvaTmaQlQ=; b=XEgu+tsGHKxsfPrv+CmzNSeKxCjmpP0EeMTJaVuLbAb7RGjSk94WWlFP4SNoxYl/2d 23LThYBjMRtr0RDF0qdQohKlKO1J+7tSsFapuNZTBEe37OgrcSmhpNBVvz+MXZyqc3+E HsHOPvMftpIBabmnsp0uU+b3c/gejaQQ30Zy0LwA0zs8g9p4dydjF6GhG+r/qR3wemEp QLNber0sXWkYf/JL7++jQPRFkboQM6Nasw4ReoRCs6FO38m69z4JFiViiNqs/o/mYsur z/WCgMaJg6DyG19Ck78n+egK6sgoqMwi71g0Z/a6HQL9a8QtFJ6WCp1PlQgoEWRsIYvR GilA== X-Gm-Message-State: AOAM5304xvLHlvWCazaUV2ZqTqG5yjQyMbuvFLdphLPYC53TvC+V3E/c N5jfuq5dh4D+P70/divsNXBJKdnQ X-Google-Smtp-Source: ABdhPJzjq10UVHqSphdNALvJPaFegfKKNaaJkvPaj2V2kdGQGIennpQqVbGVQGuU7SNdauFT/45QzA== X-Received: by 2002:ac8:7c8d:: with SMTP id y13mr13409740qtv.387.1596394197304; Sun, 02 Aug 2020 11:49:57 -0700 (PDT) Received: from [10.0.10.8] (cpe-65-25-51-0.neo.res.rr.com. [65.25.51.0]) by smtp.googlemail.com with ESMTPSA id c33sm19653756qtk.40.2020.08.02.11.49.56 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sun, 02 Aug 2020 11:49:56 -0700 (PDT) Message-ID: <5F270AD4.8080001@gmail.com> Date: Sun, 02 Aug 2020 14:49:56 -0400 From: Ernie Luzar User-Agent: Thunderbird 2.0.0.24 (Windows/20100228) MIME-Version: 1.0 To: Dan Langille CC: "freebsd-questions@freebsd.org" , "freebsd-jail@freebsd.org" Subject: Re: jail(8) bug with vnet & non-vnet jails running at same time? References: <5F26FC5B.6030706@gmail.com> In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4BKVSQ36nvz3YFw X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=NWQNJrUw; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of luzar722@gmail.com designates 2607:f8b0:4864:20::832 as permitted sender) smtp.mailfrom=luzar722@gmail.com X-Spamd-Result: default: False [-2.46 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.52)[-0.521]; RECEIVED_SPAMHAUS_PBL(0.00)[65.25.51.0:received]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.98)[-0.983]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-0.95)[-0.952]; MIME_GOOD(-0.10)[text/plain]; SUBJECT_ENDS_QUESTION(1.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::832:from]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Aug 2020 18:49:59 -0000 Dan Langille wrote: >> On Aug 2, 2020, at 1:48 PM, Ernie Luzar wrote: >> >> Hello list; >> Please review configuration looking for something I may have missed. Hopping someone can suggest something that will change the behavior eliminating the problem. >> >> >> Equipment. Real hardware, 12.1 release, amd64 dual cpu. >> >> Description; >> non-vnet jails and vnet jails using the bridge/epair method can ping the public internet when only non-vnet jails are started at a time or when only vnet jails are started at a time. But when both non-vnet jails and vnet jails are started together then neither one can ping the public internet. The order of the jails definitions in the jail.conf file has no effect on changing what is happening. >> >> Bug description: >> When non-vnet jails are started their ip addresses are added to the NIC facing the public AFTER the public ip address and the non-vnet jail has access to the public internet. But when both non-vnet jails and vnet jails are started at the same time then the non-vnet jails ip addresses gets added before the public ip address of the NIC facing the public internet causing the host to lose all access to the public internet. This seems to be a jail(8) bug. >> >> It makes no difference which command method is used to start and stop the jails. >> Service jail onestart jailname or jail –cv jailname > > This may be related to my twitter rant about vnet problems in my own jails: > > https://twitter.com/DLangille/status/1289944047763693569 > > The symptoms you describe to similar to my own. I cannot access ports on jails on the same host, but I can access ports on other hosts. > Your twitter posts are all pf firewall related. From what I can tell you are using local only vnet jails and want to talk between them. Do you have any non-vnet jails running on the host where the 2 vnet jails are running? Do you have any local only vnet jails working on any other systems? To me knowledge there is only 1 way to have local only vnet jails to talk to each other. Do not assign ip address to epairXa or to the bridge. Only assign an ip address to epairXb the interface in the vnet jail. All the vnet jails you want to be local only have to be members on the same bridge. From owner-freebsd-jail@freebsd.org Sun Aug 2 19:19:01 2020 Return-Path: Delivered-To: freebsd-jail@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 345383A518E; Sun, 2 Aug 2020 19:19:01 +0000 (UTC) (envelope-from dan@langille.org) Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4BKW5w3FVfz3ZlF; Sun, 2 Aug 2020 19:19:00 +0000 (UTC) (envelope-from dan@langille.org) Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailout.nyi.internal (Postfix) with ESMTP id C54F55C0113; Sun, 2 Aug 2020 15:18:59 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute2.internal (MEProxy); Sun, 02 Aug 2020 15:18:59 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=langille.org; h= content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; s=fm1; bh=6 0tPHlhGipLqhlriBCG12YoArhP0+M3aQnPe0rKBmFU=; b=n6VoBWM6hEq4j6HXX ZrXEnlzQ0xn6utnuthaRNkPQDZzrn7kJ2OfDN9Vgj3mxcxnhhNEnQp4SSXz2vE7T x256XlUFJ0JgHz5t25YOhwxt5gb9W7sJwmgkBvwCcLHwEJZZu1oU+EczdEJgnAH/ ZC/fWgkc9lNMvUAIrM9pO7k9BtwDrmJ05LO4mdAhPPSprT2WMbOWLVX2wpQsIl3N 2IM3TS1CLH8/xYjElTsfhgJoIG+IduDDIesNK6rgQq3LK4+SW4QEw9nWdA/Ll5m9 fKc6tBUBefpYDpZfz3EfjNdDFmZL0v43Y4qq1mSL06KF99OBIF3pVjtvAxY2ELtr +mxHA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; bh=60tPHlhGipLqhlriBCG12YoArhP0+M3aQnPe0rKBm FU=; b=QlEiGkan5Qh9n+Mf4DBoTJbAY6/7yLmvMZrRwrxH0x0WxtZfUnuCYqTWg 4TTzunvmCmC6wKk2jREw3o7UvIe0+DKIr6ZMdJx+WVxBLDdMXZYO8jYITrmiJZh+ WxMd2fjVplzEJMhg/Ut/hX9/6lI1EpDqosh8X8RROPrcjMsW7E2PgnMMeY2AtBs0 +pN2xo1RycvftUK5kYWZ4aiPKggf/WtE5PGDYE9Q+8EskZYDvgkd5OtkBlEMp1jy 9ltN3KDTXGVHh+5yYsgQdJR5/WY3MdM3eNOYZkcEIRnLjqa0B+i8XFYSCZ5qFot4 +KfhB+Lqf8oVrDLpH2oJKVFU4loYg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduiedrjedvgddufeelucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurheptggguffhjgffgffkfhfvofesthhqmhdthhdtjeenucfhrhhomhepffgrnhcu nfgrnhhgihhllhgvuceouggrnheslhgrnhhgihhllhgvrdhorhhgqeenucggtffrrghtth gvrhhnpeevvdfgveeugeffieduhfetffduvdffkedtleejteegleduffevhfefkeehgedv veenucffohhmrghinhepthifihhtthgvrhdrtghomhenucfkphepuddtkedrfeeirdelhe druddtnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhep uggrnheslhgrnhhgihhllhgvrdhorhhg X-ME-Proxy: Received: from air01.wifi.int.unixathome.org (pool-108-36-95-10.phlapa.fios.verizon.net [108.36.95.10]) by mail.messagingengine.com (Postfix) with ESMTPA id 73EE13060067; Sun, 2 Aug 2020 15:18:59 -0400 (EDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.1\)) Subject: Re: jail(8) bug with vnet & non-vnet jails running at same time? From: Dan Langille In-Reply-To: <5F270AD4.8080001@gmail.com> Date: Sun, 2 Aug 2020 15:18:58 -0400 Cc: "freebsd-questions@freebsd.org" , "freebsd-jail@freebsd.org" Content-Transfer-Encoding: quoted-printable Message-Id: <01D7BB67-FCC8-4896-8E02-0C26CF6036CC@langille.org> References: <5F26FC5B.6030706@gmail.com> <5F270AD4.8080001@gmail.com> To: Ernie Luzar X-Mailer: Apple Mail (2.3608.120.23.2.1) X-Rspamd-Queue-Id: 4BKW5w3FVfz3ZlF X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=langille.org header.s=fm1 header.b=n6VoBWM6; dkim=pass header.d=messagingengine.com header.s=fm3 header.b=QlEiGkan; dmarc=pass (policy=none) header.from=langille.org; spf=pass (mx1.freebsd.org: domain of dan@langille.org designates 66.111.4.25 as permitted sender) smtp.mailfrom=dan@langille.org X-Spamd-Result: default: False [-2.87 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ip4:66.111.4.25]; RWL_MAILSPIKE_GOOD(0.00)[66.111.4.25:from]; RCVD_COUNT_THREE(0.00)[4]; DKIM_TRACE(0.00)[langille.org:+,messagingengine.com:+]; DMARC_POLICY_ALLOW(-0.50)[langille.org,none]; NEURAL_HAM_SHORT(-1.35)[-1.350]; FREEMAIL_TO(0.00)[gmail.com]; RECEIVED_SPAMHAUS_PBL(0.00)[108.36.95.10:received]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11403, ipnet:66.111.4.0/24, country:US]; SUBJECT_ENDS_QUESTION(1.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_IN_DNSWL_LOW(-0.10)[66.111.4.25:from]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[langille.org:s=fm1,messagingengine.com:s=fm3]; NEURAL_HAM_MEDIUM(-1.00)[-1.002]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-0.92)[-0.922]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[] X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Aug 2020 19:19:01 -0000 > On Aug 2, 2020, at 2:49 PM, Ernie Luzar wrote: >=20 > Dan Langille wrote: >>> On Aug 2, 2020, at 1:48 PM, Ernie Luzar wrote: >>>=20 >>> Hello list; >>> Please review configuration looking for something I may have missed. = Hopping someone can suggest something that will change the behavior = eliminating the problem. >>>=20 >>>=20 >>> Equipment. Real hardware, 12.1 release, amd64 dual cpu. >>>=20 >>> Description; >>> non-vnet jails and vnet jails using the bridge/epair method can ping = the public internet when only non-vnet jails are started at a time or = when only vnet jails are started at a time. But when both non-vnet jails = and vnet jails are started together then neither one can ping the public = internet. The order of the jails definitions in the jail.conf file has = no effect on changing what is happening. >>>=20 >>> Bug description: >>> When non-vnet jails are started their ip addresses are added to the = NIC facing the public AFTER the public ip address and the non-vnet jail = has access to the public internet. But when both non-vnet jails and vnet = jails are started at the same time then the non-vnet jails ip addresses = gets added before the public ip address of the NIC facing the public = internet causing the host to lose all access to the public internet. = This seems to be a jail(8) bug. >>>=20 >>> It makes no difference which command method is used to start and = stop the jails. >>> Service jail onestart jailname or jail =C3=A2=E2=82=AC=E2=80=9Ccv = jailname >> This may be related to my twitter rant about vnet problems in my own = jails: >> https://twitter.com/DLangille/status/1289944047763693569 >> The symptoms you describe to similar to my own. I cannot access = ports on jails on the same host, but I can access ports on other hosts. >=20 > Your twitter posts are all pf firewall related. =46rom what I can = tell you are using local only vnet jails and want to talk between them. >=20 > Do you have any non-vnet jails running on the host where the 2 vnet = jails are running? >=20 > Do you have any local only vnet jails working on any other systems? One of those two jails in question is vnet, the other is not. There are = many non-vnet jails on this host, only one vnet. > To me knowledge there is only 1 way to have local only vnet jails to = talk to each other. Do not assign ip address to epairXa or to the = bridge. Only assign an ip address to epairXb the interface in the vnet = jail. All the vnet jails you want to be local only have to be members on = the same bridge. I will look at that for this jail. Thank you. --=20 Dan Langille - BSDCan / PGCon dan@langille.org From owner-freebsd-jail@freebsd.org Tue Aug 4 19:55:05 2020 Return-Path: Delivered-To: freebsd-jail@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6358D37DCA5 for ; Tue, 4 Aug 2020 19:55:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 4BLlpd0wCkz3Y8B for ; Tue, 4 Aug 2020 19:55:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 1C04137DE1C; Tue, 4 Aug 2020 19:55:05 +0000 (UTC) Delivered-To: jail@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1A99F37DE1B for ; Tue, 4 Aug 2020 19:55:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BLlpc5lfYz3Y3X for ; Tue, 4 Aug 2020 19:55:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id A52C2AB96 for ; Tue, 4 Aug 2020 19:55:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 074Jt4nD079846 for ; Tue, 4 Aug 2020 19:55:04 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 074Jt4CU079845 for jail@FreeBSD.org; Tue, 4 Aug 2020 19:55:04 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: jail@FreeBSD.org Subject: [Bug 248468] jail(8) host has no internet access when vnet & non-vnet jails running at same time Date: Tue, 04 Aug 2020 19:55:04 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: jail@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Aug 2020 19:55:05 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D248468 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bugs@FreeBSD.org |jail@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-jail@freebsd.org Wed Aug 5 01:02:24 2020 Return-Path: Delivered-To: freebsd-jail@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7F4583A4ADB; Wed, 5 Aug 2020 01:02:24 +0000 (UTC) (envelope-from luzar722@gmail.com) Received: from mail-qv1-xf2e.google.com (mail-qv1-xf2e.google.com [IPv6:2607:f8b0:4864:20::f2e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BLtdC55ntz45vT; Wed, 5 Aug 2020 01:02:23 +0000 (UTC) (envelope-from luzar722@gmail.com) Received: by mail-qv1-xf2e.google.com with SMTP id j10so13324367qvo.13; Tue, 04 Aug 2020 18:02:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:date:from:user-agent:mime-version:to:subject :content-transfer-encoding; bh=nst0EJwDkoD6V2/Aq9C5nUqpkmpFpDLE/sveIAJ42M0=; b=k8GszHBP6Mk6BwzADf61nhseqoH7HCHvVc11A7u7ny4pTOH3QEXQJGTRe0X4eRrhoa 0njvqSlfTyHAO3nBzTRqEnuLUIi1RvSvDNa7mKC6eY/n4QjiAxugtc7YAjjfZGVjJKPi zvZYQOHMHTWlh9XSWOdBTQcRhHDi/ZrMqwxitnoqJH5J9l7KK9xsU27Lw8ODN08KFXWR BfLlUwMC3bkX/Yu9REh5TDlVX5IT+9ED5k+thubmhTLDP9G20r0AK+9nCR33H6GMpyn+ VRXbMlN3IA7R2rElkCto1wWmQ1jDdm2HxJPta5mZXUaLbg3dhXuaKfuIl4si2tG+KMV4 7ffQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :subject:content-transfer-encoding; bh=nst0EJwDkoD6V2/Aq9C5nUqpkmpFpDLE/sveIAJ42M0=; b=XECV74mPLObVZZKUJYL9jGjg1G6srxrEzxF1/ukYPYGfKROsorSWpqdEiHPobQSGoQ B0vZOI4MYZU/mQvlzOO82kNzd5JjGBWmtaBTPm5oJx4N2cE47DvXk3RPpGyjjuGVuA6A UrYJrGtx1FqTJ3e2NQCjct5iDQf1STvjaCeKYF9HdujQO23RhH0tBJTGk/r62C0LGg7V V17GeIG4pug8xfrqWiSrOdDJkcqiO/o2ZW9/1u6yDbE6saHreawZmWdW7PgvpnzTcS69 z22KgeTmam7Otvtu37X7sWJmbNEFFHoFdcTRyeTIMLIlQ17bpa1jH5TAEJp+E88aBncy MPaQ== X-Gm-Message-State: AOAM530k3ESdzhP42/BH2Kl5oHf4flxgq2e1cK+m20iUCfGCpkdhlYFf GE2Xd7hsoLh4pLXRoAZTo6ZHyVb7 X-Google-Smtp-Source: ABdhPJyXgbQliEsZYMNqBu2k5BSNczNqKaYWcEJv5m6pV5T9caSsIMhiCi6NawBthV3lqOoyOhA20A== X-Received: by 2002:a0c:c409:: with SMTP id r9mr1152431qvi.123.1596589342511; Tue, 04 Aug 2020 18:02:22 -0700 (PDT) Received: from [10.0.10.8] (cpe-65-25-51-0.neo.res.rr.com. [65.25.51.0]) by smtp.googlemail.com with ESMTPSA id x198sm438513qka.37.2020.08.04.18.02.21 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 04 Aug 2020 18:02:21 -0700 (PDT) Message-ID: <5F2A051D.4030604@gmail.com> Date: Tue, 04 Aug 2020 21:02:21 -0400 From: Ernie Luzar User-Agent: Thunderbird 2.0.0.24 (Windows/20100228) MIME-Version: 1.0 To: "freebsd-questions@freebsd.org" , "freebsd-jail@freebsd.org" Subject: how to make a non-vnet jail local only? Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4BLtdC55ntz45vT X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=k8GszHBP; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of luzar722@gmail.com designates 2607:f8b0:4864:20::f2e as permitted sender) smtp.mailfrom=luzar722@gmail.com X-Spamd-Result: default: False [-1.88 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RECEIVED_SPAMHAUS_PBL(0.00)[65.25.51.0:received]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.97)[-0.972]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-0.95)[-0.954]; MIME_GOOD(-0.10)[text/plain]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_SPAM_SHORT(0.04)[0.042]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::f2e:from]; TO_DN_EQ_ADDR_ALL(0.00)[]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Aug 2020 01:02:24 -0000 I have non-vnet jails working that can reach the public internet. But now I would like to make some local only non-vnet jails that can only access other local only non-vnet jails. BY local meaning have no access to the public internet. How do I make this happen? Thanks for any pointers. From owner-freebsd-jail@freebsd.org Wed Aug 5 01:08:07 2020 Return-Path: Delivered-To: freebsd-jail@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3BF033A5402; Wed, 5 Aug 2020 01:08:07 +0000 (UTC) (envelope-from reshadpatuck1@gmail.com) Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BLtlp1h4fz46FY; Wed, 5 Aug 2020 01:08:06 +0000 (UTC) (envelope-from reshadpatuck1@gmail.com) Received: by mail-wm1-x32d.google.com with SMTP id q76so4639315wme.4; Tue, 04 Aug 2020 18:08:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=WQ6gCJODMr4SH6zxmf8uKWTRN83QGbqFunHTCZgmedA=; b=fWhc5KEBuyXhaooIUmWGtitpo5uICGjKmz2sFcbtrcpi/hmDMwI3auJNJwqDpIsUL5 kxvxgWXzBKjgXpaigWQCHHkXlsyfjmmzGyb7GnzTvPosZTO9nXfaei9+trdFeX0yyz0H o/lLxiNgrzv21CKZGo73MkOptNPczR98njAKAXfZlTDoqWXM3Ak9F4yacIttbeHwnFGq RXYUiUaZoY6/1Vz3nvPJW3GkxPXsZ4ldABW7IKKQUmjEKoRxTj0YiHWqxaCMuV/i7yJz ifp2MFxMyxAtgsxqWgNxkcvtwwAEts/fMtuJ9hkiKgxE9ULoghxfary9GahTsxRrDNET naOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=WQ6gCJODMr4SH6zxmf8uKWTRN83QGbqFunHTCZgmedA=; b=j7n5XQI08jWDW56co2H8d7rlQ6LjqQPfJ3Z1aHhS1eAsVnWwJ80ZP/3vACa1PK6cW7 ytfTkevnso2k4ql6cVpmFEZAsGdFPTuLva3vRLEkZS8Mql8vx/Je/xZrV1BAynTgf2Rp whjFRUDw/5QH1Lq9B93ZPIt2JPkb7U97tkeGtqk8pA9noXD3btfGqrudPo6HCUAbYoLx AVLA4Tgd9/QjVrgqCT2LCocP/xlTmQw2RTszep0HVn4MUQeS7Y5YaMFQ0c+BnK5eAwRX 8p7jkKF5kN88JAhdmV5QIvWT1/Uu2fbkbIK2+7S5laMSYelb90/9hQ063LZA8a5HTa/l hFfg== X-Gm-Message-State: AOAM533tkGimc5wLBABgLfA6tSlfnWJ8yPu6oPJ1T35q7HDLJ5e3nLRk +Jsr1c3kWzwvFzcQWBfsLlzkLvLBoKMoC4HKCgY= X-Google-Smtp-Source: ABdhPJxk+bPtFuDW2jLe9C+cwDILWH0goSsd5XHu6btCaYsKTKtFEhsvTx925osvdBVPK5HIQ+Z+Pz325L/jjfw7sFo= X-Received: by 2002:a1c:2646:: with SMTP id m67mr936609wmm.137.1596589683821; Tue, 04 Aug 2020 18:08:03 -0700 (PDT) MIME-Version: 1.0 References: <5F2A051D.4030604@gmail.com> In-Reply-To: <5F2A051D.4030604@gmail.com> From: Reshad Patuck Date: Wed, 5 Aug 2020 06:37:52 +0530 Message-ID: Subject: Re: how to make a non-vnet jail local only? To: Ernie Luzar Cc: freebsd-questions@freebsd.org, freebsd-jail@freebsd.org X-Rspamd-Queue-Id: 4BLtlp1h4fz46FY X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=fWhc5KEB; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of reshadpatuck1@gmail.com designates 2a00:1450:4864:20::32d as permitted sender) smtp.mailfrom=reshadpatuck1@gmail.com X-Spamd-Result: default: False [-2.09 / 15.00]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; NEURAL_HAM_MEDIUM(-0.99)[-0.986]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; NEURAL_HAM_LONG(-0.99)[-0.987]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::32d:from]; NEURAL_HAM_SHORT(-0.12)[-0.115]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; SUBJECT_ENDS_QUESTION(1.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; FREEMAIL_ENVFROM(0.00)[gmail.com]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.33 X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Aug 2020 01:08:07 -0000 Hi Ernie, For local system only access you can use 127.0.0.1 as the jail IP. You could use a pf rdr rule to allow only local access to the port running your jailed service. Best, Reshad On Wed, 5 Aug, 2020, 06:32 Ernie Luzar, wrote: > I have non-vnet jails working that can reach the public internet. > But now I would like to make some local only non-vnet jails that can > only access other local only non-vnet jails. BY local meaning have no > access to the public internet. > > How do I make this happen? > > Thanks for any pointers. > _______________________________________________ > freebsd-jail@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-jail > To unsubscribe, send any mail to "freebsd-jail-unsubscribe@freebsd.org" > From owner-freebsd-jail@freebsd.org Wed Aug 5 07:17:13 2020 Return-Path: Delivered-To: freebsd-jail@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4C7883AC9BB; Wed, 5 Aug 2020 07:17:13 +0000 (UTC) (envelope-from freebsd@qeng-ho.org) Received: from bede.qeng-ho.org (bede.qeng-ho.org [217.155.128.241]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4BM2xg6KGGz4RSH; Wed, 5 Aug 2020 07:17:11 +0000 (UTC) (envelope-from freebsd@qeng-ho.org) Received: from arthur.home.qeng-ho.org (arthur.home.qeng-ho.org [172.23.1.2]) by bede.qeng-ho.org (Postfix) with ESMTP id D2CF310195; Wed, 5 Aug 2020 08:17:03 +0100 (BST) Subject: Re: how to make a non-vnet jail local only? To: Ernie Luzar , "freebsd-questions@freebsd.org" , "freebsd-jail@freebsd.org" References: <5F2A051D.4030604@gmail.com> From: Arthur Chance Message-ID: <77719bef-6c53-21a7-ca17-3ebac05427b9@qeng-ho.org> Date: Wed, 5 Aug 2020 08:17:03 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.11.0 MIME-Version: 1.0 In-Reply-To: <5F2A051D.4030604@gmail.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-GB Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4BM2xg6KGGz4RSH X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of freebsd@qeng-ho.org designates 217.155.128.241 as permitted sender) smtp.mailfrom=freebsd@qeng-ho.org X-Spamd-Result: default: False [-1.74 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.95)[-0.952]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:217.155.128.240/29]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[qeng-ho.org]; NEURAL_HAM_LONG(-1.01)[-1.014]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.47)[-0.474]; FREEMAIL_TO(0.00)[gmail.com,freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:13037, ipnet:217.155.0.0/16, country:GB]; SUBJECT_ENDS_QUESTION(1.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Aug 2020 07:17:13 -0000 On 05/08/2020 02:02, Ernie Luzar wrote: > I have non-vnet jails working that can reach the public internet. > But now I would like to make some local only non-vnet jails that can > only access other local only non-vnet jails. BY local meaning have no > access to the public internet. > > How do I make this happen? > > Thanks for any pointers. Create a second loopback interface (cloned_interfaces="lo1" in /etc/rc.conf or ifconfig lo1 create for manual control) and put the local jails on lo1 without access to any other interface. -- The number of people predicting the demise of Moore's Law doubles every 18 months. From owner-freebsd-jail@freebsd.org Wed Aug 5 14:17:39 2020 Return-Path: Delivered-To: freebsd-jail@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5F3B23776F7; Wed, 5 Aug 2020 14:17:39 +0000 (UTC) (envelope-from luzar722@gmail.com) Received: from mail-qk1-x72b.google.com (mail-qk1-x72b.google.com [IPv6:2607:f8b0:4864:20::72b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BMDGp1VcSz4slq; Wed, 5 Aug 2020 14:17:38 +0000 (UTC) (envelope-from luzar722@gmail.com) Received: by mail-qk1-x72b.google.com with SMTP id 77so9466476qkm.5; Wed, 05 Aug 2020 07:17:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-transfer-encoding; bh=HMQ1s5gwHDSBqmKQ8Tf4HdQdNk2BA9ZYT9SJyi6XanI=; b=UoOkNQdFdvxVpfBNMuzkS0G7Y+xv2oAhn7sXCoqaADYLpQi8HTlHxjM3FcF3FMatkF TJOnC2Ur+5yZ1GPxqZaYVya11DaBOjx7tDKeYKVHYsVWWFS3lw5VHmtX7GOyH6BFyA3H 0fGPefX93xdCL89er1NvLg1UFlrYlIFvcQAJ+EHq5c8xelX/bmE8pXRpMe36zcAMapjg e5/cK2GwPNPlW6MFpPd9v2f8y4NunoFX+eiju+o9rPzXE5sUNIKxjVa+5zu3PUAWDWfB eeRRJfv5pIAtuTfwbc5NCRLtX6K+ve7KnJBKWrrenkGpagSNDSBw+Gxxb1j8083jFIrY xTQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-transfer-encoding; bh=HMQ1s5gwHDSBqmKQ8Tf4HdQdNk2BA9ZYT9SJyi6XanI=; b=spS2gktWIS/qBRJDrTgV7BE814H2vkqqTrk7DNbfFcchjBkVYN3NNmbILFvtpWYWmy H2tBNv4G0YZp6aLrc0MEUhJCCt6ebEr5j83xY4WctTcH+GO2DozRBVxZmnqKcZEbJlbQ V241nwXCF0S8Qru8erBDcls/N/1ZKvS9EJhuGwCm75X/3ox4sQfGl6d0+H/HtxPZwkht fGRwiC1ooS4+uN0sghv/XgqFgaXTo1SrYZxUjaiKeOQ/ce3viIlZ6fzOAaG9d6bWpPuA wyBjD+e3xjv7Dg4lAIZpo9lD35dpKy+97RoUXNaqwSES504HCKRxaSiNj3HXZzfnp6b3 jh7A== X-Gm-Message-State: AOAM533/Y6TKfqVIudP+1XC/Sf9vqCHbnHywtD8lk18kHFgcKLux55wT 3YLrenIS/3++b1Vy655gDqR7KExM X-Google-Smtp-Source: ABdhPJzj656MtDZU4ySZ7Htu6vLUCbkzMMu06H7832tSwREbX6haHLOMP7tOqArSabdDwEO9WqBQgw== X-Received: by 2002:a37:9e09:: with SMTP id h9mr3414807qke.361.1596637057391; Wed, 05 Aug 2020 07:17:37 -0700 (PDT) Received: from [10.0.10.8] (cpe-65-25-51-0.neo.res.rr.com. [65.25.51.0]) by smtp.googlemail.com with ESMTPSA id s184sm1715842qkf.50.2020.08.05.07.17.36 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 05 Aug 2020 07:17:36 -0700 (PDT) Message-ID: <5F2ABF80.4080208@gmail.com> Date: Wed, 05 Aug 2020 10:17:36 -0400 From: Ernie Luzar User-Agent: Thunderbird 2.0.0.24 (Windows/20100228) MIME-Version: 1.0 To: Arthur Chance CC: "freebsd-questions@freebsd.org" , "freebsd-jail@freebsd.org" Subject: Re: how to make a non-vnet jail local only? References: <5F2A051D.4030604@gmail.com> <77719bef-6c53-21a7-ca17-3ebac05427b9@qeng-ho.org> In-Reply-To: <77719bef-6c53-21a7-ca17-3ebac05427b9@qeng-ho.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4BMDGp1VcSz4slq X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=UoOkNQdF; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of luzar722@gmail.com designates 2607:f8b0:4864:20::72b as permitted sender) smtp.mailfrom=luzar722@gmail.com X-Spamd-Result: default: False [-2.33 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.39)[-0.389]; RECEIVED_SPAMHAUS_PBL(0.00)[65.25.51.0:received]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.96)[-0.962]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-0.98)[-0.983]; MIME_GOOD(-0.10)[text/plain]; SUBJECT_ENDS_QUESTION(1.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::72b:from]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Aug 2020 14:17:39 -0000 Arthur Chance wrote: > On 05/08/2020 02:02, Ernie Luzar wrote: >> I have non-vnet jails working that can reach the public internet. >> But now I would like to make some local only non-vnet jails that can >> only access other local only non-vnet jails. BY local meaning have no >> access to the public internet. >> >> How do I make this happen? >> >> Thanks for any pointers. > > Create a second loopback interface (cloned_interfaces="lo1" in > /etc/rc.conf or ifconfig lo1 create for manual control) and put the > local jails on lo1 without access to any other interface. > I tested this already and it doesn't work. non-vnet jail with lo99 for the nic and ip address of 10.0.28.5 can still reach the public internet. Also tested a non-vnet jail with re0 for the nic and ip address of 127.0.10.10 and it can NOT reach the public internet. Created a second non-vnet jail with re0 for the nic and ip address of 127.0.10.11 and it can NOT reach the public internet. But these 2 jails can ping each other. So the nic loX has nothing to do with limiting the non-vnet jail to local host access only. Based on the above 2 tests it looks like the 127.0.0.2 through 127.255.255.254 ip address range is the local host controlling factor. Just to cover all the bases. The host firewall allows the lo0 interface to pass without any rules. The lo99 interface has no firewall rules at all or any NAT rules for 127.0.0.0/8. 10.0.0.0/8 is the only ip address range being NATed. To see if 127.0.0.0/8 has some special internal limiting factor on it or if because the firewall does not NAT 127.0.0.0/8 is the cause of non-vnet jails not being able to reach the public internet. So I created a 3rd non-vnet jail with re0 for the nic and ip address of 192.168.10.10 and made no changes to the firewall or NAT. This jail can NOT reach the public internet, but can ping the other 2 local only jails 127.0.10.10 and 127.0.10.11. So the conclusion is that loX or 127.0.0.0/8 has nothing to do with being the controlling factor between local or public non-vnet jails. The real controlling factor is in the jails ip address being NATed or not. Can this conclusion be disputed? From owner-freebsd-jail@freebsd.org Wed Aug 5 16:09:33 2020 Return-Path: Delivered-To: freebsd-jail@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6B23D37AA96; Wed, 5 Aug 2020 16:09:33 +0000 (UTC) (envelope-from freebsd@qeng-ho.org) Received: from bede.qeng-ho.org (bede.qeng-ho.org [217.155.128.241]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4BMGlw4Y51z3WqJ; Wed, 5 Aug 2020 16:09:32 +0000 (UTC) (envelope-from freebsd@qeng-ho.org) Received: from arthur.home.qeng-ho.org (arthur.home.qeng-ho.org [172.23.1.2]) by bede.qeng-ho.org (Postfix) with ESMTP id 73ED910640; Wed, 5 Aug 2020 17:09:30 +0100 (BST) Subject: Re: how to make a non-vnet jail local only? To: Ernie Luzar Cc: "freebsd-questions@freebsd.org" , "freebsd-jail@freebsd.org" References: <5F2A051D.4030604@gmail.com> <77719bef-6c53-21a7-ca17-3ebac05427b9@qeng-ho.org> <5F2ABF80.4080208@gmail.com> From: Arthur Chance Message-ID: <15ab4539-afaf-df6e-8c36-bf8056723999@qeng-ho.org> Date: Wed, 5 Aug 2020 17:09:30 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.11.0 MIME-Version: 1.0 In-Reply-To: <5F2ABF80.4080208@gmail.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-GB Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4BMGlw4Y51z3WqJ X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of freebsd@qeng-ho.org designates 217.155.128.241 as permitted sender) smtp.mailfrom=freebsd@qeng-ho.org X-Spamd-Result: default: False [-1.66 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.90)[-0.901]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:217.155.128.240/29]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[qeng-ho.org]; NEURAL_HAM_LONG(-1.01)[-1.014]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.44)[-0.443]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:13037, ipnet:217.155.0.0/16, country:GB]; SUBJECT_ENDS_QUESTION(1.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Aug 2020 16:09:33 -0000 On 05/08/2020 15:17, Ernie Luzar wrote: > Arthur Chance wrote: >> On 05/08/2020 02:02, Ernie Luzar wrote: >>> I have non-vnet jails working that can reach the public internet. >>> But now I would like to make some local only non-vnet jails that can >>> only access other local only non-vnet jails. BY local meaning have no >>> access to the public internet. >>> >>> How do I make this happen? >>> >>> Thanks for any pointers. >> >> Create a second loopback interface (cloned_interfaces="lo1" in >> /etc/rc.conf or ifconfig lo1 create for manual control) and put the >> local jails on lo1 without access to any other interface. >> > > I tested this already and it doesn't work. > > non-vnet jail with lo99 for the nic and ip address of 10.0.28.5 can > still reach the public internet. This surprises me. It's a while since I looked at the network handling code, but I was under the impression any packet, whatever its address, on an interface with the LOOPBACK flag set was not routed off the machine. But see below. > Also tested a non-vnet jail with re0 for the nic and ip address of > 127.0.10.10 and it can NOT reach the public internet. > > Created a second non-vnet jail with re0 for the nic and ip address of > 127.0.10.11 and it can NOT reach the public internet. Also in the network code, any packet with a loopback address as either source or destination is not routed off machine. This behaviour is mandated by RFC 1122. It obviously works, no matter what the interface. However, using a loopback address on a non-loopback interface is logged as a bad address. You're not supposed to put loopback addresses on non loopback interfaces. > But these 2 jails can ping each other. This does not involve off machine routing. Given that you wanted non-vnet jails to talk to each other, this would seem to be exactly what you need. > So the nic loX has nothing to do with limiting the non-vnet jail to > local host access only. Based on the above 2 tests it looks like the > 127.0.0.2 through 127.255.255.254 ip address range is the local host > controlling factor. > > Just to cover all the bases. The host firewall allows the lo0 interface > to pass without any rules. The lo99 interface has no firewall rules at > all or any NAT rules for 127.0.0.0/8. 10.0.0.0/8 is the only ip address > range being NATed. I suspect (well, guess) the NATing is what is allowing the lo99 packets to be routed off machine. Try reading the relevant bits of the source code (in /usr/src/sys/netinet, probably ip_{in,out}put.c and maybe ip_fastfwd.c), plus the relevant firewall code if you're really interested. > To see if 127.0.0.0/8 has some special internal limiting factor on it or > if because the firewall does not NAT 127.0.0.0/8 is the cause of > non-vnet jails not being able to reach the public internet. As I said above, this is mandatory behaviour. Even if a packet with a loopback address was emitted by your machine, your ISP should drop it as part of their bogon filtering. (Sadly should isn't the same as will.) > So I created a 3rd non-vnet jail with re0 for the nic and ip address of > 192.168.10.10 and made no changes to the firewall or NAT. This jail can > NOT reach the public internet, but can ping the other 2 local only jails > 127.0.10.10 and 127.0.10.11. > > So the conclusion is that loX or 127.0.0.0/8 has nothing to do with > being the controlling factor between local or public non-vnet jails. The > real controlling factor is in the jails ip address being NATed or not. > > Can this conclusion be disputed? I'm sure you can find someone on the net to dispute with you if you really want. :-) Personally I can't be bothered. The firewall rules also have an effect on routing (duh!), but as you've not said which firewall you're using I can't address that. However, by putting all jails you want not to access the outside world on their own lo interface with their own subnet means you could simply firewall that interface from talking to the outside which is nice and simple. The approach I'm using these days is to use IPv6 for almost everything that's purely in house, and avoid NAT like the horrible hack it is. If you want a set of jails to only be able to speak to each other and the host then using a cloned lo interface and pure link local addresses is nice and simple and doesn't involve a firewall at all. -- The number of people predicting the demise of Moore's Law doubles every 18 months.