From owner-freebsd-net@freebsd.org Tue Jun 27 09:15:40 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7601BDA5838 for ; Tue, 27 Jun 2017 09:15:40 +0000 (UTC) (envelope-from prvs=34432f271=youssef.ghorbal@pasteur.fr) Received: from mx0.pasteur.fr (mx0.pasteur.fr [157.99.45.50]) (using TLSv1.2 with cipher RC4-SHA (128/128 bits)) (Client CN "Cisco Appliance Demo Certificate", Issuer "Cisco Appliance Demo Certificate" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 9D45282E17 for ; Tue, 27 Jun 2017 09:15:39 +0000 (UTC) (envelope-from prvs=34432f271=youssef.ghorbal@pasteur.fr) Authentication-Results: mx0.pasteur.fr; spf=None smtp.pra=youssef.ghorbal@pasteur.fr; spf=None smtp.mailfrom=youssef.ghorbal@pasteur.fr; spf=None smtp.helo=postmaster@EXCHANGE02.corp.pasteur.fr Received-SPF: None (mx0.pasteur.fr: no sender authenticity information available from domain of youssef.ghorbal@pasteur.fr) identity=pra; client-ip=157.99.211.32; receiver=mx0.pasteur.fr; envelope-from="youssef.ghorbal@pasteur.fr"; x-sender="youssef.ghorbal@pasteur.fr"; x-conformance=sidf_compatible Received-SPF: None (mx0.pasteur.fr: no sender authenticity information available from domain of youssef.ghorbal@pasteur.fr) identity=mailfrom; client-ip=157.99.211.32; receiver=mx0.pasteur.fr; envelope-from="youssef.ghorbal@pasteur.fr"; x-sender="youssef.ghorbal@pasteur.fr"; x-conformance=sidf_compatible Received-SPF: None (mx0.pasteur.fr: no sender authenticity information available from domain of postmaster@EXCHANGE02.corp.pasteur.fr) identity=helo; client-ip=157.99.211.32; receiver=mx0.pasteur.fr; envelope-from="youssef.ghorbal@pasteur.fr"; x-sender="postmaster@EXCHANGE02.corp.pasteur.fr"; x-conformance=sidf_compatible X-IronPort-AV: E=Sophos;i="5.39,399,1493676000"; d="scan'208";a="1443271" Received: from exchange02.corp.pasteur.fr ([157.99.211.32]) by mx0.pasteur.fr with ESMTP/TLS/AES256-GCM-SHA384; 27 Jun 2017 11:15:37 +0200 Received: from EXCHANGE02.corp.pasteur.fr (2002:9d63:d320::9d63:d320) by EXCHANGE02.corp.pasteur.fr (2002:9d63:d320::9d63:d320) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.1.845.34; Tue, 27 Jun 2017 11:15:36 +0200 Received: from EXCHANGE02.corp.pasteur.fr ([fe80::a819:199f:2049:3d20]) by EXCHANGE02.corp.pasteur.fr ([fe80::a819:199f:2049:3d20%18]) with mapi id 15.01.0845.034; Tue, 27 Jun 2017 11:15:36 +0200 From: "Youssef GHORBAL" To: Matt Joras CC: Navdeep Parhar , "freebsd-net@freebsd.org" Subject: Re: Sporadic TCP/RST sent to client Thread-Topic: Sporadic TCP/RST sent to client Thread-Index: AQHS66rb2Qoho5P9iUa0cYMOeRPraqI3fpaAgAAbfoCAALjuAA== Date: Tue, 27 Jun 2017 09:15:36 +0000 Message-ID: <5ABA962E-A90A-4C25-A5A7-EE5CF66FFDD4@pasteur.fr> References: <84CB0795-B28E-46DF-9593-4C1BAAB7DDF5@pasteur.fr> In-Reply-To: Accept-Language: en-US, fr-FR Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [157.99.101.113] Content-Type: text/plain; charset="us-ascii" Content-ID: <4270729117D59544BAA4C524A37127EE@corp.pasteur.fr> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Jun 2017 09:15:40 -0000 Imagine this set up : freebsd host port0 <-> switch 1 <-> linux host port0 freebsd host port1 <-> switch 2 <-> linux host port1 On the linux box, port 0&1 are enslaved in a bond with a RR algorithm (Roun= d Robin) On the freebsd box, port 0&1 are enslaved in a lagg. switchs 1&2 are configured for doing MLAG. The Linux box disapatchs packets on both NICs (since the RR algo dictates t= hat) packets are dispatched in order. Packets outgoing on port0 gets handled by switch1 and hits the freebsd box = on port 0 Packets outgoing on port1 gets handled by switch2 and hits the freebsd box = on port 1 As I stated earlier, from the tcpdump traces I've done on the freebsd box (= both on the lagg interface and the actual ports) packets do arrive ordered = but on different NICs and it works great until the elapes times start to be= around microsecond. I don't really have control over the Linux box to make them use other hash = algo (but I'm stil trying) Youssef ------------------------ > On 27 Jun 2017, at 00:13, Matt Joras wrote: >=20 > Out of curiosity, what sort of lagg setup are you using that's causing > the TCP packets to be split across the two lagg interfaces? >=20 > Matt >=20 > On Mon, Jun 26, 2017 at 1:35 PM, Navdeep Parhar wrote= : >> On Thu, Jun 22, 2017 at 3:57 PM, Youssef GHORBAL >> wrote: >>> Hello, >>>=20 >>> I'm having an issue with a FreeBSD 11 based system, sending spor= adically TCP/RST to clients after initial TCP session correctly initiated. >>> The sequence goes this way : >>>=20 >>> 1 Client -> Server : SYN >>> 2 Server -> Client : SYN/ACK >>> 3 Client -> Server : ACK >>> 4 Client -> Server : PSH/ACK (upper protocol data sending starts= here) >>> 5 Server -> Client : RST >>>=20 >>> - The problem happens sporadically, same client and same server = can communicate smoothely on the same service port. But from time to time (= hours, sometime days) the previous sequence happens. >>> - The service running on server is not responsible for the RST s= ent. The service was deeply profiled and nothing happens to justify the RST= . >>> - tcpdump on the server side assures that packet arrives timely = ordered. >>> - the traffic is very light. Some TCP sessions per day. >>> - the server is connected using a lagg enslaving two cxgb interf= aces. >>>=20 >>> In my effort to diagnose the problem (try to have a reproductibl= e test case) I noticed that the issue is triggered most likely when those t= wo conditions are met : >>> - the ACK (in step 3) and the PSH/ACK (in step 4) arrive on diff= erent lagg NICs. >>> - the timing between those two packets is sub 10 microseconds. >>>=20 >>> When searching the interwebs I came across a strangely similar i= ssue reported here 7 years ago : >>> https://lists.freebsd.org/pipermail/freebsd-net/2010-August/0260= 29.html >>>=20 >>> (The OP seemed to have resolved his issue changing the netisr po= licy from direct to hybrid. but no reference of laggs being used) >>>=20 >>> I'm pretty sure that I'm hitting some race condition, a scenario= where due to multithreading the PSH/ACK is somehow handled before the ACK = making the kernel rising TCP/RST since the initial TCP handshake did'nt fin= ish yet. >>>=20 >>> I've read about netisr work and I was under the impression that = even if it's SMP enabled it was made to keep prorocol ordering. >>>=20 >>> What's the expected behaviour in this scenario on the netisr sid= e ? >>> How can I push the investigation further ? >>=20 >> I think you've already figured out the situation here -- the PSH/ACK is = likely >> being handled before the ACK for the SYN because they arrived on differe= nt >> interfaces. There is nothing in netisr dispatch that will maintain prot= ocol >> ordering in this case. >>=20 >> Regards, >> Navdeep >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"