From owner-freebsd-net@freebsd.org Sat Oct 19 16:24:42 2019 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C5848150AAB; Sat, 19 Oct 2019 16:24:42 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670089.outbound.protection.outlook.com [40.107.67.89]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46wSsj1tgRz4PQn; Sat, 19 Oct 2019 16:24:40 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=YlbsuPP6MAK89kPftkQR7+nbjOnHbGvl2T1gMrv5ZfULWVhY1cpmfiqbDMGEeWABljUAygZHA3w8rO6wLd1ZVWGXnCRZ0PM10tGuuaC+L9TKvcmNBX0mMvOwGBsKNuyNPxTUUtQt5YSaVaiCCZgMkoWN/X3j2JudSNrBCkpZ9rRbRLgVZz0uYtWgwr9Uqrtxa7u2/NHFn6UtESg/7DXZpNy+64vrYTs8n2LievhlNGEwrhbxJyj8IiV7bkGRzxr/Ik8x/Q2OHtM/jkzWcw9FY7eT93FxHUn7U6zVQPSJCpG0Xe7qjDS/x/qCwUlKZxBBLfzx5eujDk5lq3E6mE6Jpg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=nmDu2kTrZ/KZjgSAP/EI6XzLFuwxFtA05QPIF91BrgQ=; b=X6CyavNKzJRTKxDONZevhKe+MSIu5p9xx8HY5Tp0kl5mBAxn/xOxqo7zgocdCRGc09pkzc4G59fLVSug9eXrkJRcGzwkQdS/iM1oCin1BrHv8xBMnw8Qa7NTS6Ot7XxvwwM4iJtWpjSRbQPIFFfXU+py0KMo+eacLnaaG3XegHDJXpj6ogDYINysIGDC5n/vbjcdois4Ja3PIGLTYKNb5m96SPMI3A522ND3Lo5DrVUJAGwgL63ScwEE0LNHLN6tDNzSjz1a5NA52cnNH0uCqOk/80ZU1GgRYOyrNvgdsx6/OKQX6v9dNPzu74SvyQQipBuwJrshvQq3PjBFkooPgg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none Received: from YQBPR0101MB1652.CANPRD01.PROD.OUTLOOK.COM (52.132.66.144) by YQBPR0101MB0754.CANPRD01.PROD.OUTLOOK.COM (52.132.73.21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2347.16; Sat, 19 Oct 2019 16:24:38 +0000 Received: from YQBPR0101MB1652.CANPRD01.PROD.OUTLOOK.COM ([fe80::9051:db7:40f3:2ba3]) by YQBPR0101MB1652.CANPRD01.PROD.OUTLOOK.COM ([fe80::9051:db7:40f3:2ba3%7]) with mapi id 15.20.2347.028; Sat, 19 Oct 2019 16:24:38 +0000 From: Rick Macklem To: Paul , "michael.tuexen@lurchi.franken.de" , "freebsd-net@freebsd.org" , "freebsd-stable@freebsd.org" Subject: Re: Re[2]: Network anomalies after update from 11.2 STABLE to 12.1 STABLE Thread-Topic: Re[2]: Network anomalies after update from 11.2 STABLE to 12.1 STABLE Thread-Index: AQHVhpeb/qQpao8VP0Wq+sk7OrrSy6diJbjH Date: Sat, 19 Oct 2019 16:24:38 +0000 Message-ID: References: <1571499556.409350000.a1ewtyar@frv39.fwdcdn.com> In-Reply-To: <1571499556.409350000.a1ewtyar@frv39.fwdcdn.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 27b834bb-216c-42f1-9ac6-08d754b0d6b1 x-ms-traffictypediagnostic: YQBPR0101MB0754: x-ms-exchange-purlcount: 3 x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:3631; x-forefront-prvs: 01952C6E96 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6019001)(346002)(136003)(39850400004)(396003)(366004)(376002)(51234002)(189003)(199004)(256004)(110136005)(76116006)(99286004)(478600001)(86362001)(14454004)(966005)(14444005)(305945005)(7696005)(8936002)(476003)(2501003)(2420400007)(76176011)(11346002)(53546011)(15650500001)(446003)(102836004)(6506007)(52536014)(186003)(81166006)(81156014)(6306002)(8676002)(46003)(6246003)(55016002)(6436002)(25786009)(2906002)(66946007)(316002)(66476007)(2201001)(71190400001)(64756008)(66556008)(786003)(66446008)(71200400001)(74316002)(486006)(33656002)(7110500001)(9686003)(5660300002)(229853002)(21314003); DIR:OUT; SFP:1101; SCL:1; SRVR:YQBPR0101MB0754; H:YQBPR0101MB1652.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: mXvfZLPTclq2Mao7G2rP9zD/VpNiUDG8m4ZhFbkJhZRemA3c8pzz+gyu0dgloYvv6/gQThg0Q0Ig3Zglbj5cxuPb2AYTDJGCf8iojSpZx7L3Ztn/mcQA+s0NIyQAzCP9eI+56Ja+JRmntzQ+qZuCNOgVXbj95RzxxXoeMkETjYR+TkLKOB1vDNd80gNegWK3mkNqhX6WF24eM1flMCUYsxYmMheXIZEKDmijA+EbJn+3Tfv+OPVjNxyfPXpDsN1lo61j6QSLhW/Ex+ltKWyn6rQhhSP8vln1A4j2LS8LcJVLNzqr/iQlpJK6z+Gk6Yp0uj4tu/jo+OSycw2c+eelU0mYCEW7zHQLzcdmvUk6MllBra9gbB40/C0IVU3QPLqcJ1I5Ekx+BdOq9QJIi+IJO7+eVSHWxdLUnEhTg8fHf9FOjzHp3ds4U8nM+5N5GK0iNmDvO59E9YXQDYY2+QOFjA== x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: 27b834bb-216c-42f1-9ac6-08d754b0d6b1 X-MS-Exchange-CrossTenant-originalarrivaltime: 19 Oct 2019 16:24:38.8266 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: Re0lB8T9uWkb9SmCHtV7fAmHj4uA451AxB2XXrMWD/JyGY/uAulYZPmfh8+KBy4YAY2kmaBwtr4vGi04Rs2qBQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQBPR0101MB0754 X-Rspamd-Queue-Id: 46wSsj1tgRz4PQn X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.67.89 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-4.63 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[uoguelph.ca]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[89.67.107.40.list.dnswl.org : 127.0.3.0]; IP_SCORE(-1.33)[ipnet: 40.64.0.0/10(-3.92), asn: 8075(-2.68), country: US(-0.05)]; FREEMAIL_TO(0.00)[ukr.net]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:8075, ipnet:40.64.0.0/10, country:US]; ARC_ALLOW(-1.00)[i=1] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 19 Oct 2019 16:24:42 -0000 Btw, I once ran into a situation where "smart networking" was injecting RSTs into a TCP stream. The packet captures at the client and server machines were identical, except for the RSTs and the problem went away when I connected the two machines with a cable, bypassing the network. Might be worth a try, if you can do it? Good luck with it, rick ________________________________________ From: owner-freebsd-net@freebsd.org on beha= lf of Paul Sent: Saturday, October 19, 2019 12:09 PM To: michael.tuexen@lurchi.franken.de; freebsd-net@freebsd.org; freebsd-stab= le@freebsd.org Subject: Re[2]: Network anomalies after update from 11.2 STABLE to 12.1 STA= BLE Hi Michael, Thank you, for taking your time! We use physical machines. We don not have any special `pf` rules. Both sides ran `pfctl -d` before testing. `nginx` config is primitive, no secrets there: ------------------------------------------------------------------- user www; worker_processes auto; error_log /var/log/nginx/error.log warn; events { worker_connections 81920; kqueue_changes 4096; use kqueue; } http { include mime.types; default_type application/octet-stream; sendfile off; keepalive_timeout 65; tcp_nopush on; tcp_nodelay on; # Logging log_format main '$remote_addr - $remote_user [$time_local] = "$request" ' '$status $request_length $body_bytes_sent "= $http_referer" ' '"$http_user_agent" "$http_x_real_ip" "$rea= lip_remote_addr" "$request_completion" "$request_time" ' '"$request_body"'; access_log /var/log/nginx/access.log main; server { listen 80 default; server_name localhost _; location / { return 404; } } } ------------------------------------------------------------------- `wrk` is compiled with a default configuration. We test like this: `wrk -c 10 --header "Connection: close" -d 10 -t 1 --latency http://10.10.1= 0.92:80/missing` Also, it seems that our issue, and the one described in this thread, are id= entical: https://lists.freebsd.org/pipermail/freebsd-net/2019-June/053667.html We both have the Intel network cards, BTW. Our network cards are these: em0 at pci0:10:0:0: class=3D0x020000 card=3D0x000015d9 chip=3D0x10d3= 8086 rev=3D0x00 hdr=3D0x00 vendor =3D 'Intel Corporation' device =3D '82574L Gigabit Network Connection' ixl0 at pci0:4:0:0: class=3D0x020000 card=3D0x00078086 chip=3D0x1572= 8086 rev=3D0x01 hdr=3D0x00 vendor =3D 'Intel Corporation' device =3D 'Ethernet Controller X710 for 10GbE SFP+' =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D Additional info: During the tests, we have bonded two interfaces into a lagg: ixl0: flags=3D8843 metric 0 mtu 150= 0 options=3Dc500b8 ether 3c:fd:fe:aa:60:20 media: Ethernet autoselect (10Gbase-SR ) status: active nd6 options=3D29 ixl1: flags=3D8843 metric 0 mtu 150= 0 options=3Dc500b8 ether 3c:fd:fe:aa:60:20 hwaddr 3c:fd:fe:aa:60:21 media: Ethernet autoselect (10Gbase-SR ) status: active nd6 options=3D29 lagg0: flags=3D8843 metric 0 mtu 15= 00 options=3Dc500b8 ether 3c:fd:fe:aa:60:20 inet 10.10.10.92 netmask 0xffff0000 broadcast 10.10.255.255 laggproto failover lagghash l2,l3,l4 laggport: ixl0 flags=3D5 laggport: ixl1 flags=3D0<> groups: lagg media: Ethernet autoselect status: active nd6 options=3D29 using this config: ifconfig_ixl0=3D"up -lro -tso -rxcsum -txcsum" (tried different option= s - got the same outcome) ifconfig_ixl1=3D"up -lro -tso -rxcsum -txcsum" ifconfig_lagg0=3D"laggproto failover laggport ixl0 laggport ixl1 10.10.= 10.92/24" We have randomly picked `ixl0` and restricted number of RX/TX queues to 1: /boot/loader.conf : dev.ixl.0.iflib.override_ntxqs=3D1 dev.ixl.0.iflib.override_nrxqs=3D1 leaving `ixl1` with a default number, matching number of cores (6). ixl0: mem = 0xf8800000-0xf8ffffff,0xf9808000-0xf980ffff irq 40 at device 0.0 on pci4 ixl0: fw 5.0.40043 api 1.5 nvm 5.05 etid 80002927 oem 1.261.0 ixl0: PF-ID[0]: VFs 64, MSI-X 129, VF MSI-X 5, QPs 768, I2C ixl0: Using 1024 TX descriptors and 1024 RX descriptors ixl0: Using 1 RX queues 1 TX queues ixl0: Using MSI-X interrupts with 2 vectors ixl0: Ethernet address: 3c:fd:fe:aa:60:20 ixl0: Allocating 1 queues for PF LAN VSI; 1 queues active ixl0: PCI Express Bus: Speed 8.0GT/s Width x4 ixl0: SR-IOV ready ixl0: netmap queues/slots: TX 1/1024, RX 1/1024 ixl1: mem = 0xf8000000-0xf87fffff,0xf9800000-0xf9807fff irq 40 at device 0.1 on pci4 ixl1: fw 5.0.40043 api 1.5 nvm 5.05 etid 80002927 oem 1.261.0 ixl1: PF-ID[1]: VFs 64, MSI-X 129, VF MSI-X 5, QPs 768, I2C ixl1: Using 1024 TX descriptors and 1024 RX descriptors ixl1: Using 6 RX queues 6 TX queues ixl1: Using MSI-X interrupts with 7 vectors ixl1: Ethernet address: 3c:fd:fe:aa:60:21 ixl1: Allocating 8 queues for PF LAN VSI; 6 queues active ixl1: PCI Express Bus: Speed 8.0GT/s Width x4 ixl1: SR-IOV ready ixl1: netmap queues/slots: TX 6/1024, RX 6/1024 This allowed us easy switch between different configurations without the need to reboot, by simply shutting down one interface or the other: `ifconfig XXX down` When testing `ixl0` that runs only a single queue: ixl0: Using 1 RX queues 1 TX queues ixl0: netmap queues/slots: TX 1/1024, RX 1/1024 we've got these results: `wrk -c 10 --header "Connection: close" -d 10 -t 1 --latency http://10.10.1= 0.92:80/missing` Running 10s test @ http://10.10.10.92:80/missing 1 threads and 10 connections Thread Stats Avg Stdev Max +/- Stdev Latency 281.31us 297.74us 22.66ms 99.70% Req/Sec 19.91k 2.79k 21.25k 97.59% Latency Distribution 50% 266.00us 75% 309.00us 90% 374.00us 99% 490.00us 164440 requests in 10.02s, 47.52MB read Socket errors: read 0, write 0, timeout 0 Non-2xx or 3xx responses: 164440 Requests/sec: 16412.09 Transfer/sec: 4.74MB When testing `ixl1` that runs 6 queues: ixl1: Using 6 RX queues 6 TX queues ixl1: netmap queues/slots: TX 6/1024, RX 6/1024 we've got these results: `wrk -c 10 --header "Connection: close" -d 10 -t 1 --latency http://10.10.1= 0.92:80/missing` Running 10s test @ http://10.10.10.92:80/missing 1 threads and 10 connections Thread Stats Avg Stdev Max +/- Stdev Latency 216.16us 71.97us 511.00us 47.56% Req/Sec 4.34k 2.76k 15.44k 83.17% Latency Distribution 50% 216.00us 75% 276.00us 90% 312.00us 99% 365.00us 43616 requests in 10.10s, 12.60MB read Socket errors: connect 0, read 24, write 8, timeout 0 Non-2xx or 3xx responses: 43616 Requests/sec: 4318.26 Transfer/sec: 1.25MB Do note, that, not only multiple queues cause issues they also dramatically decrease the performance of the network. Using `sysctl -w net.inet.tcp.ts_offset_per_conn=3D0` didn't help at all. Best regards, -Paul _______________________________________________ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"