Date: Tue, 24 May 2022 17:17:19 +0200 From: FreeBSD User <freebsd@walstatt-de.de> To: Ole <mail.ole@posteo.de> Cc: freebsd-jail@freebsd.org, freebsd-net@freebsd.org Subject: Re: FreeBSD 12.3-p5: problems vnet on if_bridge Message-ID: <20220524171654.705f67b2@hermann> In-Reply-To: <20220524115246.0ef1ff59@lenp43s> References: <20220510212129.35041f02@hermann> <20220511204755.2028dce9@hermann> <20220524115246.0ef1ff59@lenp43s>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, 24 May 2022 09:52:46 +0000 Ole <mail.ole@posteo.de> wrote: Hello, > Hello, > > could you solve the problem? I think I ran into the same problem. > I opened a Ticket. I couldn't solve the problem. > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=264198 > > I seems to be related to IPFW and effects vnet-Jails and also bhyve VMs. There is also a PR regarding vnet/if_bridge/routing issues, at https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=240106 but I can not guarantee this PR is in any way similaror adjacent to the problem of mine (and probably yours). For reasons not to be discussed here I'm bound to FreeBSD 12.3-RELENG (XigmaNAS base) and running several boxes of different hardware grades with the very same XigmaNAS version gives different results. The host in question in my case has Intel NICs, the box with a very similar setup and vnet jails also on the same NIC the LAN resides on, has a Realtek NIC (it is a very small home NAS). Another has a dedicated NIC into another network (diffrent IP of the NIC and the vnet jails bound to that NIC compared to the LAN/host itself - Intel i350 NICs all over the place, no problems. Another case, our build platform, is running CURRENT and hosts about almost 15 jails on a NIC, which is part of the same network as the main host's NIC - no problem. All of our FreeBSD boxes uses IPFW as their IP filtering facility. Have you checked the MAC addresses of you vnet jails / vnet hosts? I have had on 12-CURRENT a serious issue with the very same MAC address on all jail-belonging parts of the epair vnet interface. I haven't checked this on our 12.3 (XigmaNAS) installations, I remember this issue as I writet this email, it could be a hint to look after ... > > regards > Ole Kind regarads, O. Hartmann > > > Wed, 11 May 2022 20:47:55 +0200 - FreeBSD User <freebsd@walstatt-de.de>: > > > On Tue, 10 May 2022 21:21:29 +0200 > > FreeBSD User <freebsd@walstatt-de.de> wrote: > > > > > Hello, > > > > > > I ran into serious trouble setting up a FreeBSD 12.3-RELEASE-p5 > > > host having a second NIC and vnt jails attached to that second NIC > > > (basically, the host is a recent Xigmanas with Bastille jails, but > > > the issue also occurs on a vanilla FreeBSD 12.3). > > > > > > The host is compromised of two NICs, em0 (management only) and igb0 > > > (service/jails). Both, the server and the jails as well as the igb0 > > > interface are residing on the same network, but both NICs are > > > connected to two different ports on a switch, to which we do not > > > have access (part of the campus infrastructure). > > > > > > Both NICs are attached with a IPv4 of the same network, the host is > > > listening on both NICs for services, i.e. port 22 for ssh. No > > > problem to connect to both(!) addresses via ssh. igb0 is member of > > > an if_bridge. The box also hosts a bunch of vnet jails, each jail > > > does have an if_epair created via "jib" and these vnet epairs are > > > members of the bridge, to which ifb0 is also member. > > > > > > Problem: while any service bound to NIC igb0/IPv4 residing on igb0 > > > is accessible flawlessly, accessing an jail is almost impossible. > > > Pinging a jail does work after a while the ping initiating host has > > > been waiting, in ery rare situations someone can access the sshd of > > > the jail, but any access of that kind is highly erratic. From 5 > > > jails, at most two are responding to pings, the other don't and it > > > is non-deterministic which host will respond. > > > > > > Following some advices found on the web, the following sysctl > > > settings are provided to if_bridge: > > > > > > device if_bridge > > > net.link.bridge.ipfw: 0 > > > net.link.bridge.allow_llz_overlap: 0 > > > net.link.bridge.inherit_mac: 0 > > > net.link.bridge.log_stp: 0 > > > net.link.bridge.pfil_local_phys: 0 > > > net.link.bridge.pfil_member: 0 > > > net.link.bridge.ipfw_arp: 0 > > > net.link.bridge.pfil_bridge: 0 > > > net.link.bridge.pfil_onlyip: 0 > > > > > > We do not have access to the switch the box is connected to, so I > > > don't have access to any logs revealing a problem either to a > > > conceptual misunderstanding of networking of mine and so a > > > misconfiguration or a probelm with Layer 2 or the switches > > > themselfes. > > > > > > I'd like to ask whether someone has a similar setup up and running > > > and could report this > > > - or give a hint of the problem I possibly made (igb0 is attached > > > to an IPv4 AND is member of an if_brige on which IPv4 attached vnet > > > jails are residing). > > > > > > We have also already setup another "similar" scenarion with the > > > same FreeBSD 12.3-p5 version and also two NICs, but our > > > "service/jail" NIC is part of a different IPv4 network and the NIC > > > is attached to a different switch (to which we have full access). > > > > > > Thanks in advance, > > > > > > O. Hartmann > > > > > > > On FreeBSD 12.3-p5, em0 seems to suffer from a bug regarding hardware > > chesum support, I see a lot of : > > [...] > > Flags [.], cksum 0xe826 (incorrect -> 0x606b), seq > > 101269476:101270000, ack 5077, win 257, options [nop,nop,TS val > > 2618589801 ecr 3610923914], length 524 > > > > Disabling TXCSUM via "ifconfig em0 -txcsum" renders incorrect -> > > correct. > > > > em0 is: > > > > em0@pci0:0:25:0: class=0x020000 card=0x20528086 > > chip=0x153b8086 rev=0x04 hdr=0x00 vendor = 'Intel Corporation' > > device = 'Ethernet Connection I217-V' > > class = network > > subclass = ethernet > > bar [10] = type Memory, range 32, base 0xf7d00000, size 131072, > > enabled bar [14] = type Memory, range 32, base 0xf7d35000, size > > 4096, enabled bar [18] = type I/O Port, range 32, base 0xf080, size > > 32, enabled cap 01[c8] = powerspec 2 supports D0 D3 current D0 > > cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message > > cap 13[e0] = PCI Advanced Features: FLR TP > > > > > > I remember faintly that there was an issue when I used to use FBSD 12 > > > > > > > > > >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20220524171654.705f67b2>