From owner-freebsd-net@freebsd.org Mon Jun 26 20:35:21 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1B134D91710 for ; Mon, 26 Jun 2017 20:35:21 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: from mail-pg0-x22d.google.com (mail-pg0-x22d.google.com [IPv6:2607:f8b0:400e:c05::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E09176556A for ; Mon, 26 Jun 2017 20:35:20 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: by mail-pg0-x22d.google.com with SMTP id u62so5252143pgb.3 for ; Mon, 26 Jun 2017 13:35:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=MBrkhB/nQ4rMVjBt8Psf127iUBJp70pGWThfI2v+Dxg=; b=loqTvy6iYvbq0xbCkZNWNRds8HIYVp5lCARoN3fWHC9DQcdqHgh/LYGVahRW+2Nwxv lXyGyoOW18YNDmDf6JDmoF4BtLRP/yU7AT5QVn8As31IzunolhGBBoZEJct7coXQeWUG 8LzQx3NgxsKdTVmkb9DjmIPE5FsLrcG4oDbsUSVvvUTiAKrygbkhM14R7BCNCETCRY5v KbZcj/KSEIFzPM5nlILhMuX2c1F0FP1ZlmX59t1y3SgOwxYI6l9nYa2lHzWAFUsdYH3o MOshueILLvextZ/aN9P9eG9Mq9qpQqPiAh6suS5xSDYQeFz/jwYHXu626fR5bF9NxyV1 AB7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=MBrkhB/nQ4rMVjBt8Psf127iUBJp70pGWThfI2v+Dxg=; b=bc85KWoeUoW/FsVLIQrkHcGIVIL0rGRWoqJeXTKixiRxtVqbddZFobCMkoYkUBinv7 oBIqCpvJ8TiTLW8HpilIrIutAyqF25rNNkXIkiqIas+JYscmc4izjYYAalKzbtpa94R5 42TVIrud6IljE6HrUPyt5Fll/jhZ9FBBvTM9bvLIGyahUF4XGN3fzI9tDcBnY0fQXuBv SfAdDhtC9VeYjyXQDJdfvhUiobwlLzbbDBGqXHJ6cu01QCG0cZHB5l7zDepAdh8nFHyJ CxvGnGlIAGx/BLZB5/f5xVo62EvXY3ANUhQqyXK5ZzoARLSmcPCjsmJZDe7O6NAtHhMK au3A== X-Gm-Message-State: AKS2vOyutVLHpsVTe0/nKMVewrfe7y1KPRZ2lFc+1Ol8FSq63b9n4ygH UA4+C4TuGjzN5otkkmgz1ZuN+LEj/Q== X-Received: by 10.84.171.193 with SMTP id l59mr2042574plb.139.1498509319998; Mon, 26 Jun 2017 13:35:19 -0700 (PDT) MIME-Version: 1.0 Received: by 10.100.135.80 with HTTP; Mon, 26 Jun 2017 13:35:19 -0700 (PDT) In-Reply-To: <84CB0795-B28E-46DF-9593-4C1BAAB7DDF5@pasteur.fr> References: <84CB0795-B28E-46DF-9593-4C1BAAB7DDF5@pasteur.fr> From: Navdeep Parhar Date: Mon, 26 Jun 2017 13:35:19 -0700 Message-ID: Subject: Re: Sporadic TCP/RST sent to client To: Youssef GHORBAL Cc: "freebsd-net@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Jun 2017 20:35:21 -0000 On Thu, Jun 22, 2017 at 3:57 PM, Youssef GHORBAL wrote: > Hello, > > I'm having an issue with a FreeBSD 11 based system, sending sporadically TCP/RST to clients after initial TCP session correctly initiated. > The sequence goes this way : > > 1 Client -> Server : SYN > 2 Server -> Client : SYN/ACK > 3 Client -> Server : ACK > 4 Client -> Server : PSH/ACK (upper protocol data sending starts here) > 5 Server -> Client : RST > > - The problem happens sporadically, same client and same server can communicate smoothely on the same service port. But from time to time (hours, sometime days) the previous sequence happens. > - The service running on server is not responsible for the RST sent. The service was deeply profiled and nothing happens to justify the RST. > - tcpdump on the server side assures that packet arrives timely ordered. > - the traffic is very light. Some TCP sessions per day. > - the server is connected using a lagg enslaving two cxgb interfaces. > > In my effort to diagnose the problem (try to have a reproductible test case) I noticed that the issue is triggered most likely when those two conditions are met : > - the ACK (in step 3) and the PSH/ACK (in step 4) arrive on different lagg NICs. > - the timing between those two packets is sub 10 microseconds. > > When searching the interwebs I came across a strangely similar issue reported here 7 years ago : > https://lists.freebsd.org/pipermail/freebsd-net/2010-August/026029.html > > (The OP seemed to have resolved his issue changing the netisr policy from direct to hybrid. but no reference of laggs being used) > > I'm pretty sure that I'm hitting some race condition, a scenario where due to multithreading the PSH/ACK is somehow handled before the ACK making the kernel rising TCP/RST since the initial TCP handshake did'nt finish yet. > > I've read about netisr work and I was under the impression that even if it's SMP enabled it was made to keep prorocol ordering. > > What's the expected behaviour in this scenario on the netisr side ? > How can I push the investigation further ? I think you've already figured out the situation here -- the PSH/ACK is likely being handled before the ACK for the SYN because they arrived on different interfaces. There is nothing in netisr dispatch that will maintain protocol ordering in this case. Regards, Navdeep