Date: Wed, 11 May 2022 20:47:55 +0200 From: FreeBSD User <freebsd@walstatt-de.de> To: freebsd-jail@freebsd.org, freebsd-net@freebsd.org Subject: Re: FreeBSD 12.3-p5: problems vnet on if_bridge Message-ID: <20220511204755.2028dce9@hermann> In-Reply-To: <20220510212129.35041f02@hermann> References: <20220510212129.35041f02@hermann>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, 10 May 2022 21:21:29 +0200
FreeBSD User <freebsd@walstatt-de.de> wrote:
> Hello,
>
> I ran into serious trouble setting up a FreeBSD 12.3-RELEASE-p5 host having a second NIC
> and vnt jails attached to that second NIC (basically, the host is a recent Xigmanas with
> Bastille jails, but the issue also occurs on a vanilla FreeBSD 12.3).
>
> The host is compromised of two NICs, em0 (management only) and igb0 (service/jails).
> Both, the server and the jails as well as the igb0 interface are residing on the same
> network, but both NICs are connected to two different ports on a switch, to which we do
> not have access (part of the campus infrastructure).
>
> Both NICs are attached with a IPv4 of the same network, the host is listening on both
> NICs for services, i.e. port 22 for ssh. No problem to connect to both(!) addresses via
> ssh. igb0 is member of an if_bridge. The box also hosts a bunch of vnet jails, each jail
> does have an if_epair created via "jib" and these vnet epairs are members of the bridge,
> to which ifb0 is also member.
>
> Problem: while any service bound to NIC igb0/IPv4 residing on igb0 is accessible
> flawlessly, accessing an jail is almost impossible. Pinging a jail does work after a
> while the ping initiating host has been waiting, in ery rare situations someone can
> access the sshd of the jail, but any access of that kind is highly erratic. From 5
> jails, at most two are responding to pings, the other don't and it is non-deterministic
> which host will respond.
>
> Following some advices found on the web, the following sysctl settings are provided to
> if_bridge:
>
> device if_bridge
> net.link.bridge.ipfw: 0
> net.link.bridge.allow_llz_overlap: 0
> net.link.bridge.inherit_mac: 0
> net.link.bridge.log_stp: 0
> net.link.bridge.pfil_local_phys: 0
> net.link.bridge.pfil_member: 0
> net.link.bridge.ipfw_arp: 0
> net.link.bridge.pfil_bridge: 0
> net.link.bridge.pfil_onlyip: 0
>
> We do not have access to the switch the box is connected to, so I don't have access to
> any logs revealing a problem either to a conceptual misunderstanding of networking of
> mine and so a misconfiguration or a probelm with Layer 2 or the switches themselfes.
>
> I'd like to ask whether someone has a similar setup up and running and could report this
> - or give a hint of the problem I possibly made (igb0 is attached to an IPv4 AND is
> member of an if_brige on which IPv4 attached vnet jails are residing).
>
> We have also already setup another "similar" scenarion with the same FreeBSD 12.3-p5
> version and also two NICs, but our "service/jail" NIC is part of a different IPv4
> network and the NIC is attached to a different switch (to which we have full access).
>
> Thanks in advance,
>
> O. Hartmann
>
On FreeBSD 12.3-p5, em0 seems to suffer from a bug regarding hardware chesum support, I
see a lot of :
[...]
Flags [.], cksum 0xe826 (incorrect -> 0x606b), seq 101269476:101270000, ack 5077, win
257, options [nop,nop,TS val 2618589801 ecr 3610923914], length 524
Disabling TXCSUM via "ifconfig em0 -txcsum" renders incorrect -> correct.
em0 is:
em0@pci0:0:25:0: class=0x020000 card=0x20528086 chip=0x153b8086 rev=0x04 hdr=0x00
vendor = 'Intel Corporation'
device = 'Ethernet Connection I217-V'
class = network
subclass = ethernet
bar [10] = type Memory, range 32, base 0xf7d00000, size 131072, enabled
bar [14] = type Memory, range 32, base 0xf7d35000, size 4096, enabled
bar [18] = type I/O Port, range 32, base 0xf080, size 32, enabled
cap 01[c8] = powerspec 2 supports D0 D3 current D0
cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
cap 13[e0] = PCI Advanced Features: FLR TP
I remember faintly that there was an issue when I used to use FBSD 12
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20220511204755.2028dce9>