From owner-freebsd-net@freebsd.org Sat Oct 19 16:09:29 2019 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id EA8CD1501A6 for ; Sat, 19 Oct 2019 16:09:29 +0000 (UTC) (envelope-from devgs@ukr.net) Received: from frv190.fwdcdn.com (frv190.fwdcdn.com [212.42.77.190]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "*.ukr.net", Issuer "Thawte RSA CA 2018" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46wSX86Qtkz4Ngv for ; Sat, 19 Oct 2019 16:09:28 +0000 (UTC) (envelope-from devgs@ukr.net) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ukr.net; s=ffe; h=Content-Type:MIME-Version:Message-Id:Cc:To:Subject:From:Date:Sender: Reply-To:Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date :Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To: References:List-Id:List-Help:List-Unsubscribe:List-Subscribe:List-Post: List-Owner:List-Archive; bh=JON/9xKyVPBm8Y00LpL8wHxWH75ZHFo/KDtYa5UG/tA=; b=n o7lW8O9TbpnY+mfdDVTb+TON8y+Mm66DAUMtO4/G3oddnkMiRwalIZ98J1Fd5rsjCYrnO8/RUqKXy h1ke2FrjgtaCLuTgxula0JnZASslMDFKoY2+49JDuyjNi3lObFnHbK2KHg8mvsFggpIY2Yd2PFMph 6u/B6HxnBtWWV7Is=; Received: from [10.10.10.39] (helo=frv39.fwdcdn.com) by frv190.fwdcdn.com with smtp ID 1iLrI8-0006LJ-DD for freebsd-net@freebsd.org; Sat, 19 Oct 2019 19:09:20 +0300 Date: Sat, 19 Oct 2019 19:09:20 +0300 From: Paul Subject: Re[2]: Network anomalies after update from 11.2 STABLE to 12.1 STABLE To: michael.tuexen@lurchi.franken.de, freebsd-net@freebsd.org, freebsd-stable@freebsd.org Received: from devgs@ukr.net by frv39.fwdcdn.com; Sat, 19 Oct 2019 19:09:20 +0300 Message-Id: <1571499556.409350000.a1ewtyar@frv39.fwdcdn.com> X-Mailer: mail.ukr.net 5.0 MIME-Version: 1.0 X-Rspamd-Queue-Id: 46wSX86Qtkz4Ngv X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=ukr.net header.s=ffe header.b=n o7lW8O; dmarc=pass (policy=none) header.from=ukr.net; spf=pass (mx1.freebsd.org: domain of devgs@ukr.net designates 212.42.77.190 as permitted sender) smtp.mailfrom=devgs@ukr.net X-Spamd-Result: default: False [-3.90 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[ukr.net:s=ffe]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:212.42.77.0/24]; FREEMAIL_FROM(0.00)[ukr.net]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; TO_DN_NONE(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; IP_SCORE_FREEMAIL(0.00)[]; DWL_DNSWL_LOW(-1.00)[ukr.net.dwl.dnswl.org : 127.0.5.1]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[ukr.net:+]; MIME_BASE64_TEXT(0.10)[]; DMARC_POLICY_ALLOW(-0.50)[ukr.net,none]; IP_SCORE(0.00)[ipnet: 212.42.77.0/24(-4.81), asn: 8856(-3.87), country: UA(0.08)]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[ukr.net]; ASN(0.00)[asn:8856, ipnet:212.42.77.0/24, country:UA]; RCVD_TLS_LAST(0.00)[] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: binary X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 19 Oct 2019 16:09:30 -0000 Hi Michael, Thank you, for taking your time! We use physical machines. We don not have any special `pf` rules. Both sides ran `pfctl -d` before testing. `nginx` config is primitive, no secrets there: ------------------------------------------------------------------- user www; worker_processes auto; error_log /var/log/nginx/error.log warn; events { worker_connections 81920; kqueue_changes 4096; use kqueue; } http { include mime.types; default_type application/octet-stream; sendfile off; keepalive_timeout 65; tcp_nopush on; tcp_nodelay on; # Logging log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $request_length $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_real_ip" "$realip_remote_addr" "$request_completion" "$request_time" ' '"$request_body"'; access_log /var/log/nginx/access.log main; server { listen 80 default; server_name localhost _; location / { return 404; } } } ------------------------------------------------------------------- `wrk` is compiled with a default configuration. We test like this: `wrk -c 10 --header "Connection: close" -d 10 -t 1 --latency http://10.10.10.92:80/missing` Also, it seems that our issue, and the one described in this thread, are identical: https://lists.freebsd.org/pipermail/freebsd-net/2019-June/053667.html We both have the Intel network cards, BTW. Our network cards are these: em0 at pci0:10:0:0: class=0x020000 card=0x000015d9 chip=0x10d38086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '82574L Gigabit Network Connection' ixl0 at pci0:4:0:0: class=0x020000 card=0x00078086 chip=0x15728086 rev=0x01 hdr=0x00 vendor = 'Intel Corporation' device = 'Ethernet Controller X710 for 10GbE SFP+' ============================== Additional info: During the tests, we have bonded two interfaces into a lagg: ixl0: flags=8843 metric 0 mtu 1500 options=c500b8 ether 3c:fd:fe:aa:60:20 media: Ethernet autoselect (10Gbase-SR ) status: active nd6 options=29 ixl1: flags=8843 metric 0 mtu 1500 options=c500b8 ether 3c:fd:fe:aa:60:20 hwaddr 3c:fd:fe:aa:60:21 media: Ethernet autoselect (10Gbase-SR ) status: active nd6 options=29 lagg0: flags=8843 metric 0 mtu 1500 options=c500b8 ether 3c:fd:fe:aa:60:20 inet 10.10.10.92 netmask 0xffff0000 broadcast 10.10.255.255 laggproto failover lagghash l2,l3,l4 laggport: ixl0 flags=5 laggport: ixl1 flags=0<> groups: lagg media: Ethernet autoselect status: active nd6 options=29 using this config: ifconfig_ixl0="up -lro -tso -rxcsum -txcsum" (tried different options - got the same outcome) ifconfig_ixl1="up -lro -tso -rxcsum -txcsum" ifconfig_lagg0="laggproto failover laggport ixl0 laggport ixl1 10.10.10.92/24" We have randomly picked `ixl0` and restricted number of RX/TX queues to 1: /boot/loader.conf : dev.ixl.0.iflib.override_ntxqs=1 dev.ixl.0.iflib.override_nrxqs=1 leaving `ixl1` with a default number, matching number of cores (6). ixl0: mem 0xf8800000-0xf8ffffff,0xf9808000-0xf980ffff irq 40 at device 0.0 on pci4 ixl0: fw 5.0.40043 api 1.5 nvm 5.05 etid 80002927 oem 1.261.0 ixl0: PF-ID[0]: VFs 64, MSI-X 129, VF MSI-X 5, QPs 768, I2C ixl0: Using 1024 TX descriptors and 1024 RX descriptors ixl0: Using 1 RX queues 1 TX queues ixl0: Using MSI-X interrupts with 2 vectors ixl0: Ethernet address: 3c:fd:fe:aa:60:20 ixl0: Allocating 1 queues for PF LAN VSI; 1 queues active ixl0: PCI Express Bus: Speed 8.0GT/s Width x4 ixl0: SR-IOV ready ixl0: netmap queues/slots: TX 1/1024, RX 1/1024 ixl1: mem 0xf8000000-0xf87fffff,0xf9800000-0xf9807fff irq 40 at device 0.1 on pci4 ixl1: fw 5.0.40043 api 1.5 nvm 5.05 etid 80002927 oem 1.261.0 ixl1: PF-ID[1]: VFs 64, MSI-X 129, VF MSI-X 5, QPs 768, I2C ixl1: Using 1024 TX descriptors and 1024 RX descriptors ixl1: Using 6 RX queues 6 TX queues ixl1: Using MSI-X interrupts with 7 vectors ixl1: Ethernet address: 3c:fd:fe:aa:60:21 ixl1: Allocating 8 queues for PF LAN VSI; 6 queues active ixl1: PCI Express Bus: Speed 8.0GT/s Width x4 ixl1: SR-IOV ready ixl1: netmap queues/slots: TX 6/1024, RX 6/1024 This allowed us easy switch between different configurations without the need to reboot, by simply shutting down one interface or the other: `ifconfig XXX down` When testing `ixl0` that runs only a single queue: ixl0: Using 1 RX queues 1 TX queues ixl0: netmap queues/slots: TX 1/1024, RX 1/1024 we've got these results: `wrk -c 10 --header "Connection: close" -d 10 -t 1 --latency http://10.10.10.92:80/missing` Running 10s test @ http://10.10.10.92:80/missing 1 threads and 10 connections Thread Stats Avg Stdev Max +/- Stdev Latency 281.31us 297.74us 22.66ms 99.70% Req/Sec 19.91k 2.79k 21.25k 97.59% Latency Distribution 50% 266.00us 75% 309.00us 90% 374.00us 99% 490.00us 164440 requests in 10.02s, 47.52MB read Socket errors: read 0, write 0, timeout 0 Non-2xx or 3xx responses: 164440 Requests/sec: 16412.09 Transfer/sec: 4.74MB When testing `ixl1` that runs 6 queues: ixl1: Using 6 RX queues 6 TX queues ixl1: netmap queues/slots: TX 6/1024, RX 6/1024 we've got these results: `wrk -c 10 --header "Connection: close" -d 10 -t 1 --latency http://10.10.10.92:80/missing` Running 10s test @ http://10.10.10.92:80/missing 1 threads and 10 connections Thread Stats Avg Stdev Max +/- Stdev Latency 216.16us 71.97us 511.00us 47.56% Req/Sec 4.34k 2.76k 15.44k 83.17% Latency Distribution 50% 216.00us 75% 276.00us 90% 312.00us 99% 365.00us 43616 requests in 10.10s, 12.60MB read Socket errors: connect 0, read 24, write 8, timeout 0 Non-2xx or 3xx responses: 43616 Requests/sec: 4318.26 Transfer/sec: 1.25MB Do note, that, not only multiple queues cause issues they also dramatically decrease the performance of the network. Using `sysctl -w net.inet.tcp.ts_offset_per_conn=0` didn't help at all. Best regards, -Paul