Date: Fri, 5 Jan 2024 13:22:01 +0100 From: "Patrick M. Hausen" <hausen@punkt.de> To: FreeBSD Net <freebsd-net@freebsd.org> Subject: Odd performance problems with many vnet jail on a bridge and (possibly) ipfw Message-ID: <7ADA08DC-3EEA-415B-98BD-CA292584C55B@punkt.de>
next in thread | raw e-mail | index | archive | help
Hi all, we have that single host with the largest number of jails - about 250 = active. All jails are bridged to the external interfaces (lagg + vlan) and also have a private bridge not connected to any hardware or network. The jails run a CMS and the database server is centralised. Today the customer called in and told me that he could finally spot a problem they had noticed for quite some time but where never able to really diagnose. RTT across the private bridge - again, not connected to any = infrastructure - and with 250 hosts not really an overpopulated broadcast domain is way to high and highly fluctuating: root@vpro0742:~ # ping dbhost PING6(56=3D40+8+8 bytes) fd31:4159:96::742 --> fd31:4159:96::472 16 bytes from fd31:4159:96::472, icmp_seq=3D0 hlim=3D64 time=3D140.344 = ms 16 bytes from fd31:4159:96::472, icmp_seq=3D1 hlim=3D64 time=3D166.879 = ms 16 bytes from fd31:4159:96::472, icmp_seq=3D2 hlim=3D64 time=3D158.025 = ms 16 bytes from fd31:4159:96::472, icmp_seq=3D3 hlim=3D64 time=3D111.139 = ms 16 bytes from fd31:4159:96::472, icmp_seq=3D4 hlim=3D64 time=3D124.974 = ms 16 bytes from fd31:4159:96::472, icmp_seq=3D5 hlim=3D64 time=3D137.521 = ms ^C --- dbhost ping6 statistics --- 6 packets transmitted, 6 packets received, 0.0% packet loss round-trip min/avg/max/std-dev =3D 111.139/139.813/166.879/18.762 ms For the external bridge things look similar but due to other factors at = play (larger broadcast domain, real infrastructure etc.) I'll concentrate on = the private bridge first. When the host system is booted at first the performance is as expected. The more jails are brought online the worse it seems to get. I have found various discussions concerning pf NAT and LRO and similar problems with hardware offloading. Needless to say it's all disabled on the external network interfaces (igb), but the private bridge of course = does not even have any of this. Also we do not use pf anywhere but we do use ipfw in every single jail for our inbound SNI proxy connections. So for a quick test I changed these rules to firewall_type=3D"open" in = rc.conf in two of the jails but that seemed not to change much although the = numbers seem to be slightly lower: root@vpro0742:~ # ping dbhost PING6(56=3D40+8+8 bytes) fd31:4159:96::742 --> fd31:4159:96::472 16 bytes from fd31:4159:96::472, icmp_seq=3D0 hlim=3D64 time=3D67.275 ms 16 bytes from fd31:4159:96::472, icmp_seq=3D1 hlim=3D64 time=3D69.677 ms 16 bytes from fd31:4159:96::472, icmp_seq=3D2 hlim=3D64 time=3D65.679 ms 16 bytes from fd31:4159:96::472, icmp_seq=3D3 hlim=3D64 time=3D75.663 ms 16 bytes from fd31:4159:96::472, icmp_seq=3D4 hlim=3D64 time=3D66.416 ms 16 bytes from fd31:4159:96::472, icmp_seq=3D5 hlim=3D64 time=3D92.396 ms 16 bytes from fd31:4159:96::472, icmp_seq=3D6 hlim=3D64 time=3D103.968 = ms 16 bytes from fd31:4159:96::472, icmp_seq=3D7 hlim=3D64 time=3D84.832 ms 16 bytes from fd31:4159:96::472, icmp_seq=3D8 hlim=3D64 time=3D71.411 ms 16 bytes from fd31:4159:96::472, icmp_seq=3D9 hlim=3D64 time=3D73.255 ms 16 bytes from fd31:4159:96::472, icmp_seq=3D10 hlim=3D64 time=3D50.535 = ms 16 bytes from fd31:4159:96::472, icmp_seq=3D11 hlim=3D64 time=3D51.770 = ms The rules in action look like this: --------- add 2000 fwd ::1,57 tcp from 2a00:b580:8000:12:deaf:beef:dead:beef to = me6 443 in add 2100 fwd ::1,87 tcp from 2a00:b580:8000:12:deaf:beef:dead:beef to = me6 80 in --------- The IPv6 address is our SNI proxy. We have special vhosts configured in = the web servers listening on 57 and 87, respectively, with proxy protocol = enabled for the connections via IPv4 and proxy, while 80 and 443 are regular vhosts = listening on the public IPv6 address. Any ideas would be greatly appreciated. Thanks, Patrick --=20 punkt.de GmbH Patrick M. Hausen .infrastructure Sophienstr. 187 76185 Karlsruhe Tel. +49 721 9109500 https://infrastructure.punkt.de info@punkt.de AG Mannheim 108285 Gesch=C3=A4ftsf=C3=BChrer: Daniel Lienert, Fabian Stein
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?7ADA08DC-3EEA-415B-98BD-CA292584C55B>