From owner-freebsd-net@freebsd.org Thu Jun 22 22:58:18 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BAAC7D96226 for ; Thu, 22 Jun 2017 22:58:18 +0000 (UTC) (envelope-from prvs=339ef6e64=youssef.ghorbal@pasteur.fr) Received: from mx0.pasteur.fr (mx0.pasteur.fr [157.99.45.50]) (using TLSv1.2 with cipher RC4-SHA (128/128 bits)) (Client CN "Cisco Appliance Demo Certificate", Issuer "Cisco Appliance Demo Certificate" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 1841A68D24 for ; Thu, 22 Jun 2017 22:58:17 +0000 (UTC) (envelope-from prvs=339ef6e64=youssef.ghorbal@pasteur.fr) Authentication-Results: mx0.pasteur.fr; spf=None smtp.pra=youssef.ghorbal@pasteur.fr; spf=None smtp.mailfrom=youssef.ghorbal@pasteur.fr; spf=None smtp.helo=postmaster@EXCHANGE01.corp.pasteur.fr Received-SPF: None (mx0.pasteur.fr: no sender authenticity information available from domain of youssef.ghorbal@pasteur.fr) identity=pra; client-ip=157.99.211.31; receiver=mx0.pasteur.fr; envelope-from="youssef.ghorbal@pasteur.fr"; x-sender="youssef.ghorbal@pasteur.fr"; x-conformance=sidf_compatible Received-SPF: None (mx0.pasteur.fr: no sender authenticity information available from domain of youssef.ghorbal@pasteur.fr) identity=mailfrom; client-ip=157.99.211.31; receiver=mx0.pasteur.fr; envelope-from="youssef.ghorbal@pasteur.fr"; x-sender="youssef.ghorbal@pasteur.fr"; x-conformance=sidf_compatible Received-SPF: None (mx0.pasteur.fr: no sender authenticity information available from domain of postmaster@EXCHANGE01.corp.pasteur.fr) identity=helo; client-ip=157.99.211.31; receiver=mx0.pasteur.fr; envelope-from="youssef.ghorbal@pasteur.fr"; x-sender="postmaster@EXCHANGE01.corp.pasteur.fr"; x-conformance=sidf_compatible X-IronPort-AV: E=Sophos;i="5.39,374,1493676000"; d="scan'208";a="1347628" Received: from exchange01.corp.pasteur.fr ([157.99.211.31]) by mx0.pasteur.fr with ESMTP/TLS/AES256-GCM-SHA384; 23 Jun 2017 00:57:03 +0200 Received: from exchange00.corp.pasteur.fr (157.99.211.253) by EXCHANGE01.corp.pasteur.fr (157.99.211.31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.1.845.34; Fri, 23 Jun 2017 00:57:03 +0200 Received: from EXCHANGE02.corp.pasteur.fr (157.99.211.32) by exchange00.corp.pasteur.fr (157.99.211.253) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.1.845.34; Fri, 23 Jun 2017 00:57:03 +0200 Received: from EXCHANGE02.corp.pasteur.fr ([fe80::a819:199f:2049:3d20]) by EXCHANGE02.corp.pasteur.fr ([fe80::a819:199f:2049:3d20%18]) with mapi id 15.01.0845.034; Fri, 23 Jun 2017 00:57:01 +0200 From: "Youssef GHORBAL" To: "freebsd-net@freebsd.org" Subject: Sporadic TCP/RST sent to client Thread-Topic: Sporadic TCP/RST sent to client Thread-Index: AQHS66rbXenakl+Z70SNEJ7M0BITJQ== Date: Thu, 22 Jun 2017 22:57:01 +0000 Message-ID: <84CB0795-B28E-46DF-9593-4C1BAAB7DDF5@pasteur.fr> Accept-Language: en-US, fr-FR Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [157.99.101.114] Content-Type: text/plain; charset="us-ascii" Content-ID: <05377CD832F6314B9DC9346320657E2B@corp.pasteur.fr> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Jun 2017 22:58:18 -0000 Hello, I'm having an issue with a FreeBSD 11 based system, sending sporadically T= CP/RST to clients after initial TCP session correctly initiated. The sequence goes this way : 1 Client -> Server : SYN 2 Server -> Client : SYN/ACK 3 Client -> Server : ACK 4 Client -> Server : PSH/ACK (upper protocol data sending starts here) 5 Server -> Client : RST =09 - The problem happens sporadically, same client and same server can commun= icate smoothely on the same service port. But from time to time (hours, som= etime days) the previous sequence happens. - The service running on server is not responsible for the RST sent. The s= ervice was deeply profiled and nothing happens to justify the RST. - tcpdump on the server side assures that packet arrives timely ordered. - the traffic is very light. Some TCP sessions per day. - the server is connected using a lagg enslaving two cxgb interfaces. In my effort to diagnose the problem (try to have a reproductible test cas= e) I noticed that the issue is triggered most likely when those two conditi= ons are met : - the ACK (in step 3) and the PSH/ACK (in step 4) arrive on different lagg= NICs. - the timing between those two packets is sub 10 microseconds. When searching the interwebs I came across a strangely similar issue repor= ted here 7 years ago : https://lists.freebsd.org/pipermail/freebsd-net/2010-August/026029.html (The OP seemed to have resolved his issue changing the netisr policy from = direct to hybrid. but no reference of laggs being used) I'm pretty sure that I'm hitting some race condition, a scenario where due= to multithreading the PSH/ACK is somehow handled before the ACK making the= kernel rising TCP/RST since the initial TCP handshake did'nt finish yet. I've read about netisr work and I was under the impression that even if it= 's SMP enabled it was made to keep prorocol ordering. What's the expected behaviour in this scenario on the netisr side ? How can I push the investigation further ? Youssef Ghorbal