Date: Sat, 19 Oct 2019 18:35:20 +0200 From: Michael Tuexen <michael.tuexen@lurchi.franken.de> To: Paul <devgs@ukr.net> Cc: freebsd-net@freebsd.org, freebsd-stable@freebsd.org Subject: Re: Network anomalies after update from 11.2 STABLE to 12.1 STABLE Message-ID: <F80C784B-1653-4CEB-B131-E7FAC5F55675@lurchi.franken.de> In-Reply-To: <1571499556.409350000.a1ewtyar@frv39.fwdcdn.com> References: <1571499556.409350000.a1ewtyar@frv39.fwdcdn.com>
next in thread | previous in thread | raw e-mail | index | archive | help
> On 19. Oct 2019, at 18:09, Paul <devgs@ukr.net> wrote: >=20 > Hi Michael, >=20 > Thank you, for taking your time! >=20 > We use physical machines. We don not have any special `pf` rules.=20 > Both sides ran `pfctl -d` before testing. Hi Paul, OK. How are the physical machines connected to each other? What happens when you don't use a lagg interface, but the physical ones? (Trying to localise the problem...) Best regards Michael >=20 >=20 > `nginx` config is primitive, no secrets there: >=20 > ------------------------------------------------------------------- > user www; > worker_processes auto; >=20 > error_log /var/log/nginx/error.log warn; >=20 > events { > worker_connections 81920; > kqueue_changes 4096; > use kqueue; > } >=20 > http { > include mime.types; > default_type application/octet-stream; >=20 > sendfile off; > keepalive_timeout 65; > tcp_nopush on; > tcp_nodelay on; >=20 > # Logging > log_format main '$remote_addr - $remote_user = [$time_local] "$request" ' > '$status $request_length = $body_bytes_sent "$http_referer" ' > '"$http_user_agent" "$http_x_real_ip" = "$realip_remote_addr" "$request_completion" "$request_time" ' > '"$request_body"'; >=20 > access_log /var/log/nginx/access.log main; >=20 > server { > listen 80 default; >=20 > server_name localhost _; >=20 > location / { > return 404; > } > } > } > ------------------------------------------------------------------- >=20 >=20 > `wrk` is compiled with a default configuration. We test like this: >=20 > `wrk -c 10 --header "Connection: close" -d 10 -t 1 --latency = http://10.10.10.92:80/missing` >=20 >=20 > Also, it seems that our issue, and the one described in this thread, = are identical: >=20 > = https://lists.freebsd.org/pipermail/freebsd-net/2019-June/053667.html >=20 > We both have the Intel network cards, BTW. Our network cards are = these: >=20 > em0 at pci0:10:0:0: class=3D0x020000 card=3D0x000015d9 = chip=3D0x10d38086 rev=3D0x00 hdr=3D0x00 > vendor =3D 'Intel Corporation' > device =3D '82574L Gigabit Network Connection' >=20 > ixl0 at pci0:4:0:0: class=3D0x020000 card=3D0x00078086 = chip=3D0x15728086 rev=3D0x01 hdr=3D0x00 > vendor =3D 'Intel Corporation' > device =3D 'Ethernet Controller X710 for 10GbE SFP+' >=20 >=20 > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D >=20 > Additional info: >=20 > During the tests, we have bonded two interfaces into a lagg: >=20 > ixl0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 = mtu 1500 > = options=3Dc500b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILT= ER,VLAN_HWTSO,TXCSUM_IPV6> > ether 3c:fd:fe:aa:60:20 > media: Ethernet autoselect (10Gbase-SR <full-duplex>) > status: active > nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> > ixl1: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 = mtu 1500 > = options=3Dc500b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILT= ER,VLAN_HWTSO,TXCSUM_IPV6> > ether 3c:fd:fe:aa:60:20 > hwaddr 3c:fd:fe:aa:60:21 > media: Ethernet autoselect (10Gbase-SR <full-duplex>) > status: active > nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> >=20 >=20 > lagg0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 = mtu 1500 > = options=3Dc500b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILT= ER,VLAN_HWTSO,TXCSUM_IPV6> > ether 3c:fd:fe:aa:60:20 > inet 10.10.10.92 netmask 0xffff0000 broadcast 10.10.255.255 > laggproto failover lagghash l2,l3,l4 > laggport: ixl0 flags=3D5<MASTER,ACTIVE> > laggport: ixl1 flags=3D0<> > groups: lagg > media: Ethernet autoselect > status: active > nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> >=20 > using this config: >=20 > ifconfig_ixl0=3D"up -lro -tso -rxcsum -txcsum" (tried different = options - got the same outcome) > ifconfig_ixl1=3D"up -lro -tso -rxcsum -txcsum" > ifconfig_lagg0=3D"laggproto failover laggport ixl0 laggport ixl1 = 10.10.10.92/24" >=20 >=20 > We have randomly picked `ixl0` and restricted number of RX/TX queues = to 1: > /boot/loader.conf : > dev.ixl.0.iflib.override_ntxqs=3D1 > dev.ixl.0.iflib.override_nrxqs=3D1 >=20 > leaving `ixl1` with a default number, matching number of cores (6). >=20 >=20 > ixl0: <Intel(R) Ethernet Controller X710 for 10GbE SFP+ - 2.1.0-k> = mem 0xf8800000-0xf8ffffff,0xf9808000-0xf980ffff irq 40 at device 0.0 on = pci4 > ixl0: fw 5.0.40043 api 1.5 nvm 5.05 etid 80002927 oem 1.261.0 > ixl0: PF-ID[0]: VFs 64, MSI-X 129, VF MSI-X 5, QPs 768, I2C > ixl0: Using 1024 TX descriptors and 1024 RX descriptors > ixl0: Using 1 RX queues 1 TX queues > ixl0: Using MSI-X interrupts with 2 vectors > ixl0: Ethernet address: 3c:fd:fe:aa:60:20 > ixl0: Allocating 1 queues for PF LAN VSI; 1 queues active > ixl0: PCI Express Bus: Speed 8.0GT/s Width x4 > ixl0: SR-IOV ready > ixl0: netmap queues/slots: TX 1/1024, RX 1/1024 > ixl1: <Intel(R) Ethernet Controller X710 for 10GbE SFP+ - 2.1.0-k> = mem 0xf8000000-0xf87fffff,0xf9800000-0xf9807fff irq 40 at device 0.1 on = pci4 > ixl1: fw 5.0.40043 api 1.5 nvm 5.05 etid 80002927 oem 1.261.0 > ixl1: PF-ID[1]: VFs 64, MSI-X 129, VF MSI-X 5, QPs 768, I2C > ixl1: Using 1024 TX descriptors and 1024 RX descriptors > ixl1: Using 6 RX queues 6 TX queues > ixl1: Using MSI-X interrupts with 7 vectors > ixl1: Ethernet address: 3c:fd:fe:aa:60:21 > ixl1: Allocating 8 queues for PF LAN VSI; 6 queues active > ixl1: PCI Express Bus: Speed 8.0GT/s Width x4 > ixl1: SR-IOV ready > ixl1: netmap queues/slots: TX 6/1024, RX 6/1024 >=20 >=20 > This allowed us easy switch between different configurations without > the need to reboot, by simply shutting down one interface or the = other: >=20 > `ifconfig XXX down` >=20 > When testing `ixl0` that runs only a single queue: > ixl0: Using 1 RX queues 1 TX queues > ixl0: netmap queues/slots: TX 1/1024, RX 1/1024 >=20 > we've got these results: >=20 > `wrk -c 10 --header "Connection: close" -d 10 -t 1 --latency = http://10.10.10.92:80/missing` > Running 10s test @ http://10.10.10.92:80/missing > 1 threads and 10 connections > Thread Stats Avg Stdev Max +/- Stdev > Latency 281.31us 297.74us 22.66ms 99.70% > Req/Sec 19.91k 2.79k 21.25k 97.59% > Latency Distribution > 50% 266.00us > 75% 309.00us > 90% 374.00us > 99% 490.00us > 164440 requests in 10.02s, 47.52MB read > Socket errors: read 0, write 0, timeout 0 > Non-2xx or 3xx responses: 164440 > Requests/sec: 16412.09 > Transfer/sec: 4.74MB >=20 >=20 > When testing `ixl1` that runs 6 queues: > ixl1: Using 6 RX queues 6 TX queues > ixl1: netmap queues/slots: TX 6/1024, RX 6/1024 >=20 > we've got these results: >=20 > `wrk -c 10 --header "Connection: close" -d 10 -t 1 --latency = http://10.10.10.92:80/missing` > Running 10s test @ http://10.10.10.92:80/missing > 1 threads and 10 connections > Thread Stats Avg Stdev Max +/- Stdev > Latency 216.16us 71.97us 511.00us 47.56% > Req/Sec 4.34k 2.76k 15.44k 83.17% > Latency Distribution > 50% 216.00us > 75% 276.00us > 90% 312.00us > 99% 365.00us > 43616 requests in 10.10s, 12.60MB read > Socket errors: connect 0, read 24, write 8, timeout 0 > Non-2xx or 3xx responses: 43616 > Requests/sec: 4318.26 > Transfer/sec: 1.25MB >=20 > Do note, that, not only multiple queues cause issues they also = dramatically =20 > decrease the performance of the network.=20 >=20 > Using `sysctl -w net.inet.tcp.ts_offset_per_conn=3D0` didn't help at = all. >=20 > Best regards, > -Paul >=20 >=20
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?F80C784B-1653-4CEB-B131-E7FAC5F55675>