Date: Mon, 28 Jan 2019 13:44:00 +0100 From: "O. Hartmann" <o.hartmann@walstatt.org> To: freebsd-current <freebsd-current@freebsd.org>, jail@freebsd.org Subject: icmp (IPv4) issues with VIMAGE JAILs and IPv6 Message-ID: <20190128134356.23a41e81@freyja>
next in thread | raw e-mail | index | archive | help
I ran into severe problems on CURRENT ( FreeBSD 13.0-CURRENT #193
r343521: Mon Jan 28 10:26:36 CET 2019 amd64), VIMAGE enabled host with jails
utilizing IPv6.
Scenario:
The main host has two Braodcom (bce0|1) NICs. bce0 is the physical NIC attached
to a routed/switched network for the main host. bce1 is also attached
to the same network, but via another port on the switch (Cisco). Gatewaying is
not allowed on the main host. bce1 is also member of bridge0. The main host
hosts a bunch of vnet/VIMAGE jails (~12): each jail has its "epair" pseudo
NIC, of which the a-part (epairXXa) is owned by the jail and the b-part is
member of the bridge0. NIC bce1 ensures the connection to the physical network.
On all hosts IPV6 is enabled. All host use an ULA IPV6 address. All hosts and
jails use FreeBSD's native IPFW as their IP filter. bridge0 is configured to
not filter on Level 2 (ethernet). IPFW is configured on each jail via rc.conf
and script "WORKSTATION". For example, services are allowed by the
rc.conf-line:
(main host)
firewall_type="WORKSTATION"
firewall_myservices="22/tcp 53/udp 80/tcp 443/tcp ..."
firewall_allowservices="192.168.255.0/24 fdff:dead:beef::/48"
firewall_trusted="192.168.255.2 fdff:dead:beef::34 ..."
and similar for the jails.
Problem:
I can not ping (icmp IPv4) any jail from the main host, any host on the regular
internet (i.e. google.de/google.com and so on) or any jail, nor can I ping from
inside a jail any host or other jail. Since we use some ICINGA2 facilities,
pinging is essential.
The weird part: ping6 is working perfectly! Alos, any non-ICMPv4 connection is
performed well (ssh, http-80, http-443, NFS via 2049 and so on).
Disabling IPFW or switch to "OPEN" on a jail or the main host makes things work
again.
A very similar setup on a host without jails, using also rc.conf for
configuring the IPFW paketfilter doesn't reveal such a misbehaviour.
The ruleset on a JAIL with ipfw script "WORKSTATION" configured, which is NOT
working (icmp doesn't work as expected), looks like this:
[...]
service ipfw restart
Flushed all rules.
00100 allow ip from any to any via lo0
00200 deny ip from any to 127.0.0.0/8
00300 deny ip from 127.0.0.0/8 to any
00400 deny ip from any to ::1
00500 deny ip from ::1 to any
00600 allow ipv6-icmp from :: to ff02::/16
00700 allow ipv6-icmp from fe80::/10 to fe80::/10
00800 allow ipv6-icmp from fe80::/10 to ff02::/16
00900 allow ipv6-icmp from any to any icmp6types 1
01000 allow ipv6-icmp from any to any icmp6types 2,135,136
00000 check-state :default
01200 allow tcp from me to any established
00000 allow tcp from me to any setup keep-state :default
00000 allow udp from me to any keep-state :default
00000 allow icmp from me to any keep-state :default
00000 allow ipv6-icmp from me to any keep-state :default
01700 allow udp from 0.0.0.0 68 to 255.255.255.255 67 out
01800 allow udp from any 67 to me 68 in
01900 allow udp from any 67 to 255.255.255.255 68 in
02000 allow udp from fe80::/10 to me 546 in
02100 allow icmp from any to any icmptypes 8
02200 allow ipv6-icmp from any to any icmp6types 128,129
02300 allow icmp from any to any icmptypes 3,4,11
02400 allow ipv6-icmp from any to any icmp6types 3
02500 allow tcp from 192.168.255.0/24 to me 22
02600 allow tcp from 192.168.255.0/24 to me 80
02700 allow tcp from 192.168.255.0/24 to me 443
02800 allow tcp from fdff:dead:beef::/48 to me 22
02900 allow tcp from fdff:dead:beef::/48 to me 80
03000 allow tcp from fdff:dead:beef::/48 to me 443
65000 count ip from any to any
65100 deny { tcp or udp } from any to any 135-139,445 in
65200 deny { tcp or udp } from any to any 1026,1027 in
65300 deny { tcp or udp } from any to any 1433,1434 in
65400 deny ip from any to 255.255.255.255
65500 deny ip from any to 224.0.0.0/24 in
65500 deny udp from any to any 520 in
65500 deny tcp from any 80,443 to any 1024-65535 in
65500 deny ip from any to any
Firewall rules loaded.
[...]
I can not see the problem here in the configuration :-(
On the main host (owner of bce1 and bridge0), net.link.bridge looks like:
# sysctl net.link.bridge
net.link.bridge.ipfw: 0
net.link.bridge.allow_llz_overlap: 0
net.link.bridge.inherit_mac: 1
net.link.bridge.log_stp: 1
net.link.bridge.pfil_local_phys: 0
net.link.bridge.pfil_member: 0
net.link.bridge.ipfw_arp: 0
net.link.bridge.pfil_bridge: 0
net.link.bridge.pfil_onlyip: 0
Stopping all jails, destroying all epairs and bridge0 doesn't change anything.
The problems occured when IPv6 came into play on the specific host in question.
Does anyone have any ideas? I'm out of ideas.
Thanks in advance,
Oliver
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20190128134356.23a41e81>