From owner-freebsd-net@freebsd.org Mon Jun 26 22:13:46 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9684BD92D4F for ; Mon, 26 Jun 2017 22:13:46 +0000 (UTC) (envelope-from matt.joras@gmail.com) Received: from mail-wm0-x22c.google.com (mail-wm0-x22c.google.com [IPv6:2a00:1450:400c:c09::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2BCA0683B5 for ; Mon, 26 Jun 2017 22:13:46 +0000 (UTC) (envelope-from matt.joras@gmail.com) Received: by mail-wm0-x22c.google.com with SMTP id 62so9278170wmw.1 for ; Mon, 26 Jun 2017 15:13:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=27Btf0Fv07zuYCxQ9Uzw7YwBgpdaZeWPtbdR9DAsb9E=; b=lTkw8igVPe3cQUItDqlUYrVZjQrcrjJcwwMHH5JaEO0V/iwl+aflsAlYoVrv/1htC5 mmqpxtbc3S2WIQTk6dRf3tbp9j8B0K5CLxExs7lr5io01WnWkLOglls6+l7mZeeXT8o/ N+W096y4mdCsi5UxL9LpxmlaYgg5EmWpPrw8Jxl40iDxGSFJwxoYCyLcfNr+AxZbCvP1 GOrA3GOTtxqn1mOXAiQlB7mcEoM7KjWt1oP0Hjj0f3kmoZ1uRM+2+nFZ8Bel2pD9DBqo 8QZRY8CasYYCVaipFC2UJ6OdB93HnPjeCDEYepppqEMZFr43E0Y4wRITnOBMQw7WsRsc O3mg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=27Btf0Fv07zuYCxQ9Uzw7YwBgpdaZeWPtbdR9DAsb9E=; b=YO9FvGL8ie1UNWVNYHYUExNTGI0ygol2IYMaoSs8v8REGh46Xu9dlp4WZJ2dE0Cqsp f0PruACnSQf3AZZCdeXmE0GscAxNCEpaKEeOrM4+9N9nICo4Y653A3DvZCBy3FBEDpl9 ZBZyvEZRnEs8A/RF2HUfj6j+upEKgDSopgQ5ybsSa6JM/wX7fnpvfY4+anA+TKxzKkgK 2mMwdJwNUZ/VrzdM43SNfLvX74RChvggRKw8+LhV92UL1QL2hpq7ZWVdVddnpYPD47ej IWGGOk9C2E1Na9VcxAnHC8acQ/WFREhFZC5PbgN9gCuag3OtoRRkzrxG6VQur90D01sp 8ePA== X-Gm-Message-State: AKS2vOxKqOW6G7JlYRMkS/TV6WfwuYOHGf0O+IjiXxLx76OTTmQMQwsj 9GIgB41qeXVGFH/qCjQzTXkaudW7kQ== X-Received: by 10.28.113.21 with SMTP id m21mr1154663wmc.80.1498515224120; Mon, 26 Jun 2017 15:13:44 -0700 (PDT) MIME-Version: 1.0 Received: by 10.223.160.42 with HTTP; Mon, 26 Jun 2017 15:13:43 -0700 (PDT) In-Reply-To: References: <84CB0795-B28E-46DF-9593-4C1BAAB7DDF5@pasteur.fr> From: Matt Joras Date: Mon, 26 Jun 2017 15:13:43 -0700 Message-ID: Subject: Re: Sporadic TCP/RST sent to client To: Navdeep Parhar Cc: Youssef GHORBAL , "freebsd-net@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Jun 2017 22:13:46 -0000 Out of curiosity, what sort of lagg setup are you using that's causing the TCP packets to be split across the two lagg interfaces? Matt On Mon, Jun 26, 2017 at 1:35 PM, Navdeep Parhar wrote: > On Thu, Jun 22, 2017 at 3:57 PM, Youssef GHORBAL > wrote: >> Hello, >> >> I'm having an issue with a FreeBSD 11 based system, sending sporadically TCP/RST to clients after initial TCP session correctly initiated. >> The sequence goes this way : >> >> 1 Client -> Server : SYN >> 2 Server -> Client : SYN/ACK >> 3 Client -> Server : ACK >> 4 Client -> Server : PSH/ACK (upper protocol data sending starts here) >> 5 Server -> Client : RST >> >> - The problem happens sporadically, same client and same server can communicate smoothely on the same service port. But from time to time (hours, sometime days) the previous sequence happens. >> - The service running on server is not responsible for the RST sent. The service was deeply profiled and nothing happens to justify the RST. >> - tcpdump on the server side assures that packet arrives timely ordered. >> - the traffic is very light. Some TCP sessions per day. >> - the server is connected using a lagg enslaving two cxgb interfaces. >> >> In my effort to diagnose the problem (try to have a reproductible test case) I noticed that the issue is triggered most likely when those two conditions are met : >> - the ACK (in step 3) and the PSH/ACK (in step 4) arrive on different lagg NICs. >> - the timing between those two packets is sub 10 microseconds. >> >> When searching the interwebs I came across a strangely similar issue reported here 7 years ago : >> https://lists.freebsd.org/pipermail/freebsd-net/2010-August/026029.html >> >> (The OP seemed to have resolved his issue changing the netisr policy from direct to hybrid. but no reference of laggs being used) >> >> I'm pretty sure that I'm hitting some race condition, a scenario where due to multithreading the PSH/ACK is somehow handled before the ACK making the kernel rising TCP/RST since the initial TCP handshake did'nt finish yet. >> >> I've read about netisr work and I was under the impression that even if it's SMP enabled it was made to keep prorocol ordering. >> >> What's the expected behaviour in this scenario on the netisr side ? >> How can I push the investigation further ? > > I think you've already figured out the situation here -- the PSH/ACK is likely > being handled before the ACK for the SYN because they arrived on different > interfaces. There is nothing in netisr dispatch that will maintain protocol > ordering in this case. > > Regards, > Navdeep > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"