From owner-freebsd-bugs@freebsd.org Tue Aug 4 15:06:22 2020 Return-Path: Delivered-To: freebsd-bugs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A4EBB37683B for ; Tue, 4 Aug 2020 15:06:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4BLdPV46V2z4Svc for ; Tue, 4 Aug 2020 15:06:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 8AB72376993; Tue, 4 Aug 2020 15:06:22 +0000 (UTC) Delivered-To: bugs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8A78637683A for ; Tue, 4 Aug 2020 15:06:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BLdPV3Nkfz4Sby for ; Tue, 4 Aug 2020 15:06:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 543152706F for ; Tue, 4 Aug 2020 15:06:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 074F6Mv6018246 for ; Tue, 4 Aug 2020 15:06:22 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 074F6M8Y018245 for bugs@FreeBSD.org; Tue, 4 Aug 2020 15:06:22 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 248468] jail(8) host has no internet access when vnet & non-vnet jails running at same time Date: Tue, 04 Aug 2020 15:06:22 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: joeb1@a1poweruser.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Aug 2020 15:06:22 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D248468 Bug ID: 248468 Summary: jail(8) host has no internet access when vnet & non-vnet jails running at same time Product: Base System Version: 12.1-RELEASE Hardware: Any OS: Any Status: New Severity: Affects Many People Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: joeb1@a1poweruser.com Equipment. Real hardware, 12.1 release, amd64 dual cpu. Summery Description; non-vnet jails and vnet jails using the bridge/epair method can ping the pu= blic internet when only non-vnet jails are started at a time or when only vnet j= ails are started at a time. But when both non-vnet jails and vnet jails are star= ted together then neither one can ping the public internet. The order of the ja= ils definitions in the jail.conf file has no effect on changing what is happeni= ng. The vnet jail doesn't run a firewall inside of it so the hosts firewall is doing the NATing of the private addresses. Bug description: When non-vnet jails are started their ip addresses are added to the NIC fac= ing the public AFTER the public ip address and the non-vnet jail has access to = the public internet. But when both non-vnet jails and vnet jails are started at= the same time then the non-vnet jails ip addresses gets added BEFORE the public= ip address of the NIC facing the public internet causing the host to lose all access to the public internet.=20 It makes no difference which command method is used to start and stop the jails. Service jail start jailname or jail =E2=80=93cv jailname The following is a capture of the command sequence showing this bug. Follow= the re0 NIC public ip address xx.25.51.0 in the ifconfig -a listing. Before any jails are started. /root >ifconfig -a snip ... re0: flags=3D8943 metric 0 = mtu 1500 =20=20=20 options=3D8209b ether 50:3e:aa:06:11:22 inet xx.25.51.0 netmask 0xfffff000 broadcast 255.255.255.255 media: Ethernet autoselect (1000baseT ) status: active nd6 options=3D29 bridge0: flags=3D8843 metric 0 mtu = 1500 ether 02:3e:ba:a7:58:00 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: re0 flags=3D143 ifmaxaddr 0 port 2 priority 128 path cost 20000 groups: bridge nd6 options=3D1 /root >cat /etc/jail.conf # non-vnet jail zdir20 { host.hostname =3D "zdir20"; path =3D "/usr/jails/zdir20"; mount.fstab =3D "/usr/local/etc/fstab/zdir20"; exec.consolelog =3D "/var/log/zdir20.console.log"; mount.devfs; ip4.addr =3D 10.0.22.5; interface =3D "re0"; allow.raw_sockets; devfs_ruleset =3D "4"; exec.start =3D "/bin/sh /etc/rc"; exec.stop =3D "/bin/sh /etc/rc.shutdown"; } # vnet jail using the bridge/epair method v0jail1 { host.hostname =3D "v0jail1"; path =3D "/usr/jails/v0jail1"; mount.fstab =3D "/usr/local/etc/fstab/v0jail1"; exec.consolelog =3D "/var/log/v0jail1.console.log"; mount.devfs; devfs_ruleset =3D "4"; vnet =3D "new"; vnet.interface =3D "epair55b"; exec.prestart =3D "ifconfig epair55 create up"; exec.prestart +=3D "ifconfig bridge0 addm epair55a"; exec.prestart +=3D "ifconfig epair55a descr vnet-v0jail1"; exec.prestart +=3D "ifconfig bridge0 inet 10.0.48.2 netmask 255.255.255.0 alias"; exec.start =3D "/bin/sh /etc/rc"; exec.start +=3D "ifconfig epair55b inet 10.0.48.1 netmask 255.255.255.0= "; exec.start +=3D "route add default 10.0.48.2"; exec.prestop =3D "ifconfig epair55b -vnet v0jail1"; exec.stop =3D "/bin/sh /etc/rc.shutdown"; exec.poststop =3D "ifconfig bridge0 deletem epair55a"; exec.poststop +=3D "sleep 2"; exec.poststop +=3D "ifconfig epair55a destroy"; exec.poststop +=3D "ifconfig bridge0 inet 10.0.48.2 -alias"; } /root >jls JID IP Address Hostname Path # start only the non-vnet jail /root >service jail start zdir20 Starting jails: zdir20. /root >jls JID IP Address Hostname Path 18 10.0.22.5 zdir20 /usr/jails/zdir20 # Take notice that the non-vnet jails ip address follows the nic=E2=80=99s # public ip address. /root >ifconfig -a re0: flags=3D8943 metric 0 = mtu 1500 =20=20=20 options=3D8209b ether 50:3e:aa:06:11:22 inet xx.25.51.0 netmask 0xfffff000 broadcast 255.255.255.255 inet 10.0.22.5 netmask 0xffffffff broadcast 10.0.22.5 media: Ethernet autoselect (1000baseT ) status: active nd6 options=3D29 bridge0: flags=3D8843 metric 0 mtu = 1500 ether 02:3e:ba:a7:58:00 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: re0 flags=3D143 ifmaxaddr 0 port 2 priority 128 path cost 20000 groups: bridge nd6 options=3D1 # login to the non-vnet jail and ping the public /root >jexec zdir20 login -f root Last login: Sun Aug 2 11:30:40 on pts/0 FreeBSD 12.1-RELEASE-p6 GENERIC Welcome to your FreeBSD jail. zdir20 /root > zdir20 /root >ping -c 2 freebsd.org PING freebsd.org (96.47.72.84): 56 data bytes 64 bytes from 96.47.72.84: icmp_seq=3D0 ttl=3D48 time=3D44.426 ms 64 bytes from 96.47.72.84: icmp_seq=3D1 ttl=3D48 time=3D44.481 ms --- freebsd.org ping statistics --- 2 packets transmitted, 2 packets received, 0.0% packet loss round-trip min/avg/max/stddev =3D 44.426/44.453/44.481/0.027 ms zdir20 /root >exit logout # stop the non-vnet jail and show that the network is back to # starting condition. /root >service jail stop zdir20 Stopping jails: zdir20. /root >jls JID IP Address Hostname Path /root >ifconfig -a re0: flags=3D8943 metric 0 = mtu 1500 =20=20=20 options=3D8209b ether 50:3e:aa:06:11:22 inet xx.25.51.0 netmask 0xfffff000 broadcast 255.255.255.255 media: Ethernet autoselect (1000baseT ) status: active nd6 options=3D29 bridge0: flags=3D8843 metric 0 mtu = 1500 ether 02:3e:ba:a7:58:00 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: re0 flags=3D143 ifmaxaddr 0 port 2 priority 128 path cost 20000 groups: bridge nd6 options=3D1 # start only the vnet jail and see the bridge0 /root >service jail start v0jail1 Starting jails: v0jail1. /root >jls JID IP Address Hostname Path 19 v0jail1 /usr/jails/v0jail1 /root >ifconfig -a re0: flags=3D8943 metric 0 = mtu 1500 =20=20=20 options=3D82099 ether 50:3e:aa:06:11:22 inet xx.25.51.0 netmask 0xfffff000 broadcast 255.255.255.255 media: Ethernet autoselect (1000baseT ) status: active nd6 options=3D29 bridge0: flags=3D8843 metric 0 mtu = 1500 ether 02:3e:ba:a7:58:00 inet 10.0.48.2 netmask 0xffffff00 broadcast 10.0.48.255 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: epair55a flags=3D143 ifmaxaddr 0 port 5 priority 128 path cost 2000 member: re0 flags=3D143 ifmaxaddr 0 port 2 priority 128 path cost 20000 groups: bridge nd6 options=3D1 epair55a: flags=3D8943 metr= ic 0 mtu 1500 description: vnet-v0jail1 options=3D8 ether 02:eb:be:f5:15:0a inet6 fe80::eb:beff:fef5:150a%epair55a prefixlen 64 scopeid 0x5 groups: epair media: Ethernet 10Gbase-T (10Gbase-T ) status: active nd6 options=3D21 # login to the vnet jail and ping the public internet. /root >jexec v0jail1 login -f root Last login: Sun Aug 2 11:29:41 on pts/0 FreeBSD 12.1-RELEASE-p6 GENERIC Welcome to your FreeBSD jail. v0jail1 /root >ping -c 2 freebsd.org PING freebsd.org (96.47.72.84): 56 data bytes 64 bytes from 96.47.72.84: icmp_seq=3D0 ttl=3D47 time=3D46.745 ms 64 bytes from 96.47.72.84: icmp_seq=3D1 ttl=3D47 time=3D43.930 ms --- freebsd.org ping statistics --- 2 packets transmitted, 2 packets received, 0.0% packet loss round-trip min/avg/max/stddev =3D 43.930/45.337/46.745/1.407 ms v0jail1 /root >exit logout # close the vnet jail and return to starting condition. /root >service jail stop v0jail1 Stopping jails: v0jail1. /root >jls JID IP Address Hostname Path /root >ifconfig -a re0: flags=3D8943 metric 0 = mtu 1500 =20=20=20 options=3D8209b ether 50:3e:aa:06:11:22 inet xx.25.51.0 netmask 0xfffff000 broadcast 255.255.255.255 media: Ethernet autoselect (1000baseT ) status: active nd6 options=3D29 bridge0: flags=3D8843 metric 0 mtu = 1500 ether 02:3e:ba:a7:58:00 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: re0 flags=3D143 ifmaxaddr 0 port 2 priority 128 path cost 20000 groups: bridge nd6 options=3D1 # Start both the non-vnet jail and the vnet jail together. /root >service jail start Starting jails: zdir20 v0jail1. # login to the non-vnet jail and it has no public access. /root >jexec zdir20 login -f root [K Last login: Sun Aug 2 11:36:34 on pts/0 FreeBSD 12.1-RELEASE-p6 GENERIC Welcome to your FreeBSD jail. zdir20 /root >ping -c 2 freebsd.org ping: cannot resolve freebsd.org: Host name lookup failure zdir20 /root >exit logout # login to the vnet jail and it has no public access. /root >jexec v0jail1 login -f root Last login: Sun Aug 2 11:38:56 on pts/0 FreeBSD 12.1-RELEASE-p6 GENERIC Welcome to your FreeBSD jail. v0jail1 /root >ping -c 2 freebsd.org ping: cannot resolve freebsd.org: Host name lookup failure v0jail1 /root >exit logout /root >jls JID IP Address Hostname Path 20 10.0.22.5 zdir20 /usr/jails/zdir20 21 v0jail1 /usr/jails/v0jail1 # Here is the bug. See that the non-vnet jail ip address comes before the # public address causing the host to lose access to the public internet. /root >ifconfig -a re0: flags=3D8943 metric 0 = mtu 1500 =20=20=20 options=3D82099 ether 50:3e:aa:06:11:22 inet 10.0.22.5 netmask 0xffffffff broadcast 10.0.22.5 inet xx.25.51.0 netmask 0xfffff000 broadcast 255.255.255.255 media: Ethernet autoselect (1000baseT ) status: active nd6 options=3D29 bridge0: flags=3D8843 metric 0 mtu = 1500 ether 02:3e:ba:a7:58:00 inet 10.0.48.2 netmask 0xffffff00 broadcast 10.0.48.255 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: epair55a flags=3D143 ifmaxaddr 0 port 5 priority 128 path cost 2000 member: re0 flags=3D143 ifmaxaddr 0 port 2 priority 128 path cost 20000 groups: bridge nd6 options=3D1 epair55a: flags=3D8943 metr= ic 0 mtu 1500 description: vnet-v0jail1 options=3D8 ether 02:77:b8:5f:e4:0a inet6 fe80::77:b8ff:fe5f:e40a%epair55a prefixlen 64 scopeid 0x5 groups: epair media: Ethernet 10Gbase-T (10Gbase-T ) status: active nd6 options=3D21 # stop both jails and return to starting condition. /root >service jail stop Stopping jails: zdir20 v0jail1. /root >jls JID IP Address Hostname Path /root >ifconfig -a re0: flags=3D8943 metric 0 = mtu 1500 =20=20=20 options=3D8209b ether 50:3e:aa:06:11:22 inet xx.25.51.0 netmask 0xfffff000 broadcast 255.255.255.255 media: Ethernet autoselect (1000baseT ) status: active nd6 options=3D29 bridge0: flags=3D8843 metric 0 mtu = 1500 ether 02:3e:ba:a7:58:00 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: re0 flags=3D143 ifmaxaddr 0 port 2 priority 128 path cost 20000 groups: bridge nd6 options=3D1 --=20 You are receiving this mail because: You are the assignee for the bug.=