From owner-freebsd-jail@freebsd.org Mon Jan 28 12:44:16 2019 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AF9B314B27F5 for ; Mon, 28 Jan 2019 12:44:16 +0000 (UTC) (envelope-from o.hartmann@walstatt.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 1C18F8A07A for ; Mon, 28 Jan 2019 12:44:16 +0000 (UTC) (envelope-from o.hartmann@walstatt.org) Received: by mailman.ysv.freebsd.org (Postfix) id CEA7614B27E2; Mon, 28 Jan 2019 12:44:15 +0000 (UTC) Delivered-To: jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A951B14B27E1; Mon, 28 Jan 2019 12:44:15 +0000 (UTC) (envelope-from o.hartmann@walstatt.org) Received: from mout.gmx.net (mout.gmx.net [212.227.15.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 198938A04C; Mon, 28 Jan 2019 12:44:13 +0000 (UTC) (envelope-from o.hartmann@walstatt.org) Received: from freyja ([79.192.173.163]) by mail.gmx.com (mrgmx001 [212.227.17.190]) with ESMTPSA (Nemesis) id 0Lm34j-1hNR1E3FUP-00ZggB; Mon, 28 Jan 2019 13:44:01 +0100 Date: Mon, 28 Jan 2019 13:44:00 +0100 From: "O. Hartmann" To: freebsd-current , jail@freebsd.org Subject: icmp (IPv4) issues with VIMAGE JAILs and IPv6 Message-ID: <20190128134356.23a41e81@freyja> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Provags-ID: V03:K1:Ihr+Lb+c9mg2/VNa7sS8MESGp0Jb7DR1nbuI7sZQGENxdk9o1EP q6fC+pxX47eGmWdn1Q1TvtYgXXiKnhVHzN9qEbGBKo10CU/rOCFLcCMc48088cazSXrApcl 07xBRin2O36XLW4wWoJeTD5Q1oKVav76anRvv6+9lO9PFdC1jP1ipKCfzuvcI5dW5F96AZi chdY2x23BKpNCA+M+djCg== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:OH3CTtksNSs=:B+NnD1Iwf5U5m+rvIHs1NF nAz+NDfwEAbS0ZjutZNIeFO0A1IoL0jkYP7p9I4Uz8x5DLcxDHQN50vIY2DuIn/286EFTQgDL GUkrpak6xp2gI4j826cRN4MW6qS+DFSmUD05uIMLX6s3nk+e1YlrbiyNEj84UE6nIIKKZS5Ky kGg8PCKW/lqqZxIVV2OyJLuU9lYndbfVIQ5N7wNui1ZzLxXaKruSd+Bj+W2smu3NTD3s7Kw5A 7syCL1D4iiT3oRoMzWzEXasQep/k0XmJO9KbfUe18cCcZew0gO0JgGL/GoA+uMWN+CKU/8zs/ eYRU2eSeqY07dYnKC4mf+p8fm2cUYTmG1A/Fv8H9dtUWojJ697DtsrDOX3U9mqxhF4ssvFNcz enzpOwPCdrvJj9zGCTONqt8TrIEF4DOH5+oN66ndQGC9+GlJUCDnQrk/CEh6/CS8b2057dKWA eTLDxE7qGhiGMlVLKJFmu2lglwr2+9i9WYKNP/T3CDf9dxW6uDbBxlkK0wNtQdh8XSW436WYr eNZhuxWb9R9qNbEvQJJSya9/yji8ceq27tVc9PASDyoZKlP8W0aG+qa0SHAWWKQNMwLuTks68 zck8oFBdsUZXA4wEtCdXErr+QsCD9MHousW2c8yt/xD4V2uKt8dfmBAT+ErznZHPB0w2pK6vp dKMBdIE6KUn/EfuM12onPlJcQzE+M/otyTBUCbeZdT58lQr49O3WOVovdnQnTVXo+gBGPtO6N GJHfjyCKUF7bK9jfEcy/yVDbQNkMtUW8roNPnripOc7xw73hfcpaLx78lATHFbntoF/fJZp1k 8/VSxxRHgtFo3eHUdqmeBAHw8ISAja/5mzRt7fvYXsYxLXezm3UVhNSuzHI9OHEOn2WCTN04S Sd/sKCFjteUjRviLPyvkrpJPopNx1TVl+NcWgDssWCOPjGwnIeD1ofH9gWbplyOHfCi5N+6RH 6qQsnL68Atg== X-Rspamd-Queue-Id: 198938A04C X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-1.05 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; RECEIVED_SPAMHAUS_PBL(0.00)[163.173.192.79.zen.spamhaus.org : 127.0.0.10]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-0.997,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[walstatt.org]; AUTH_NA(1.00)[]; NEURAL_SPAM_SHORT(0.17)[0.166,0]; IP_SCORE(-0.66)[ip: (-4.79), ipnet: 212.227.0.0/16(-0.67), asn: 8560(2.16), country: DE(-0.01)]; MX_GOOD(-0.01)[mx00.gmx.net,mx01.gmx.net]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_MEDIUM(-0.84)[-0.843,0]; R_SPF_NA(0.00)[]; RCVD_IN_DNSWL_LOW(-0.10)[18.15.227.212.list.dnswl.org : 127.0.3.1]; R_DKIM_NA(0.00)[]; MID_RHS_NOT_FQDN(0.50)[]; ASN(0.00)[asn:8560, ipnet:212.227.0.0/16, country:DE]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[] X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Jan 2019 12:44:17 -0000 I ran into severe problems on CURRENT ( FreeBSD 13.0-CURRENT #193 r343521: Mon Jan 28 10:26:36 CET 2019 amd64), VIMAGE enabled host with jails utilizing IPv6. Scenario: The main host has two Braodcom (bce0|1) NICs. bce0 is the physical NIC attached to a routed/switched network for the main host. bce1 is also attached to the same network, but via another port on the switch (Cisco). Gatewaying is not allowed on the main host. bce1 is also member of bridge0. The main host hosts a bunch of vnet/VIMAGE jails (~12): each jail has its "epair" pseudo NIC, of which the a-part (epairXXa) is owned by the jail and the b-part is member of the bridge0. NIC bce1 ensures the connection to the physical network. On all hosts IPV6 is enabled. All host use an ULA IPV6 address. All hosts and jails use FreeBSD's native IPFW as their IP filter. bridge0 is configured to not filter on Level 2 (ethernet). IPFW is configured on each jail via rc.conf and script "WORKSTATION". For example, services are allowed by the rc.conf-line: (main host) firewall_type="WORKSTATION" firewall_myservices="22/tcp 53/udp 80/tcp 443/tcp ..." firewall_allowservices="192.168.255.0/24 fdff:dead:beef::/48" firewall_trusted="192.168.255.2 fdff:dead:beef::34 ..." and similar for the jails. Problem: I can not ping (icmp IPv4) any jail from the main host, any host on the regular internet (i.e. google.de/google.com and so on) or any jail, nor can I ping from inside a jail any host or other jail. Since we use some ICINGA2 facilities, pinging is essential. The weird part: ping6 is working perfectly! Alos, any non-ICMPv4 connection is performed well (ssh, http-80, http-443, NFS via 2049 and so on). Disabling IPFW or switch to "OPEN" on a jail or the main host makes things work again. A very similar setup on a host without jails, using also rc.conf for configuring the IPFW paketfilter doesn't reveal such a misbehaviour. The ruleset on a JAIL with ipfw script "WORKSTATION" configured, which is NOT working (icmp doesn't work as expected), looks like this: [...] service ipfw restart Flushed all rules. 00100 allow ip from any to any via lo0 00200 deny ip from any to 127.0.0.0/8 00300 deny ip from 127.0.0.0/8 to any 00400 deny ip from any to ::1 00500 deny ip from ::1 to any 00600 allow ipv6-icmp from :: to ff02::/16 00700 allow ipv6-icmp from fe80::/10 to fe80::/10 00800 allow ipv6-icmp from fe80::/10 to ff02::/16 00900 allow ipv6-icmp from any to any icmp6types 1 01000 allow ipv6-icmp from any to any icmp6types 2,135,136 00000 check-state :default 01200 allow tcp from me to any established 00000 allow tcp from me to any setup keep-state :default 00000 allow udp from me to any keep-state :default 00000 allow icmp from me to any keep-state :default 00000 allow ipv6-icmp from me to any keep-state :default 01700 allow udp from 0.0.0.0 68 to 255.255.255.255 67 out 01800 allow udp from any 67 to me 68 in 01900 allow udp from any 67 to 255.255.255.255 68 in 02000 allow udp from fe80::/10 to me 546 in 02100 allow icmp from any to any icmptypes 8 02200 allow ipv6-icmp from any to any icmp6types 128,129 02300 allow icmp from any to any icmptypes 3,4,11 02400 allow ipv6-icmp from any to any icmp6types 3 02500 allow tcp from 192.168.255.0/24 to me 22 02600 allow tcp from 192.168.255.0/24 to me 80 02700 allow tcp from 192.168.255.0/24 to me 443 02800 allow tcp from fdff:dead:beef::/48 to me 22 02900 allow tcp from fdff:dead:beef::/48 to me 80 03000 allow tcp from fdff:dead:beef::/48 to me 443 65000 count ip from any to any 65100 deny { tcp or udp } from any to any 135-139,445 in 65200 deny { tcp or udp } from any to any 1026,1027 in 65300 deny { tcp or udp } from any to any 1433,1434 in 65400 deny ip from any to 255.255.255.255 65500 deny ip from any to 224.0.0.0/24 in 65500 deny udp from any to any 520 in 65500 deny tcp from any 80,443 to any 1024-65535 in 65500 deny ip from any to any Firewall rules loaded. [...] I can not see the problem here in the configuration :-( On the main host (owner of bce1 and bridge0), net.link.bridge looks like: # sysctl net.link.bridge net.link.bridge.ipfw: 0 net.link.bridge.allow_llz_overlap: 0 net.link.bridge.inherit_mac: 1 net.link.bridge.log_stp: 1 net.link.bridge.pfil_local_phys: 0 net.link.bridge.pfil_member: 0 net.link.bridge.ipfw_arp: 0 net.link.bridge.pfil_bridge: 0 net.link.bridge.pfil_onlyip: 0 Stopping all jails, destroying all epairs and bridge0 doesn't change anything. The problems occured when IPv6 came into play on the specific host in question. Does anyone have any ideas? I'm out of ideas. Thanks in advance, Oliver