From owner-svn-src-all@freebsd.org Fri Jan 5 09:34:30 2018 Return-Path: Delivered-To: svn-src-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0F163EC0245 for ; Fri, 5 Jan 2018 09:34:30 +0000 (UTC) (envelope-from steven@multiplay.co.uk) Received: from mail-wm0-x232.google.com (mail-wm0-x232.google.com [IPv6:2a00:1450:400c:c09::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8D03F80125 for ; Fri, 5 Jan 2018 09:34:29 +0000 (UTC) (envelope-from steven@multiplay.co.uk) Received: by mail-wm0-x232.google.com with SMTP id r78so1294305wme.5 for ; Fri, 05 Jan 2018 01:34:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language; bh=by3a9B9WJ1DtuWHXPHBJknuwH1a6dKb3i8wqHZi8dB4=; b=cusSvwDZa+luuNSgxLNNfZxRTgZpzn9cCCMTv/mU62CIjo/CPzR6BOu8Zu5QyoPO0q VBu1DVUzQ307BbMknOyA0qx00seyn7hrnvHN1zwV6gfvVyJMym/w5s3Fm0L7W5rDQ4Uu 2J1RVqlPtDy1++M/M4pcsrKaXSsH9hTV9idp4Yx/fcq61Hi+iwyyAKnf5pcpVSJTJ+01 F3oZTb6NL/2NJXznU4zNEKIF50Y3U/oCrhOy0kzQNJSlSa2fQPNlM0SGnXYIiFCkPDb+ Cv6Fw3jqLID5hw80FvWOZoFpV+ij5/ZUvaQ6oYW69TzRa1xrRdaeujb4ggzlgzUDMIfc v3lA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language; bh=by3a9B9WJ1DtuWHXPHBJknuwH1a6dKb3i8wqHZi8dB4=; b=HeOHC+5CxBIA+brKdipFxXqTp8zZMVARS6jH76u41RiSr5SrS3/0wmUa1WcRCpLb3P aIXuFwc1EPYxdv3MtGfG0/SgVBOHQ+TdKEdQWIku7Q0Q+tBXa3K25d/3kYNlTY4S/TzE J3i41JJwRDQjasPnyIqP0mAaXSCs/l4Gwl8eb6LNlHayChpGSw++MziYdYcj7SPrb/IM +54b49TQaKthMjMx85zXMEHM5YHC1K3AyrcAJyBa8XuzR4BMGDvdpCy4vC3cP1rvRTzl E2OC/JGQFwHkIr8YqgL3wWxouoWFThc4xU2Kq/Ggkh+Nf0olEQrbY7UHzi4QOQZC0N/L v0dQ== X-Gm-Message-State: AKGB3mIDnlbd58rlGomOmbykrAkeDNJLHj4yjoX5mvsGl28Q69e2cBu0 LnBXMwvPVElqEvtaS9hb24QoQw== X-Google-Smtp-Source: ACJfBotM/fhnixqS/kM0n1vTDVjhVglYaqOb7t/cc3+7qQfB3/VpA41P5KWn5Nx4MFzQWgfbN7uilA== X-Received: by 10.80.136.2 with SMTP id b2mr3314101edb.239.1515144867610; Fri, 05 Jan 2018 01:34:27 -0800 (PST) Received: from [10.10.1.111] ([185.97.61.1]) by smtp.gmail.com with ESMTPSA id v20sm3319969edm.10.2018.01.05.01.34.26 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 05 Jan 2018 01:34:26 -0800 (PST) Subject: Re: svn commit: r327559 - in head: . sys/net To: Eugene Grosbein , hiren panchasara Cc: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org References: <201801042005.w04K5liB049411@repo.freebsd.org> <5A4E9397.9000308@grosbein.net> <20180104224214.GD18879@strugglingcoder.info> <63c3c450-aeaf-bdd5-5e16-414146c9bb3a@multiplay.co.uk> <5A4EDE65.1010201@grosbein.net> From: Steven Hartland Message-ID: <688c4d08-3dac-545f-1bed-b21270a03eca@multiplay.co.uk> Date: Fri, 5 Jan 2018 09:34:27 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.5.2 MIME-Version: 1.0 In-Reply-To: <5A4EDE65.1010201@grosbein.net> Content-Language: en-US Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Jan 2018 09:34:30 -0000 On 05/01/2018 02:09, Eugene Grosbein wrote: > 05.01.2018 6:37, Steven Hartland wrote: > >> Our TCP stack seems fragile during setup to out of order packets >> which this multipath behavior causes, we've seen this on our loadbalancers >> which is what triggered the investigation. The concrete result is many aborted TCP connections, >> over 300k ~2% on the machine I'm looking at. > This is another problem that needs to be fixed in general and not hidden under the carpet. > Meantime, practical problems you see can be solved locally with any settings you like. While it may seem like it, there's not denying that the problem is caused by fact that the packets for a single flow arrive on two different interfaces in normal (none failure) workflow, which contravenes 802.3ad which states: 43.2.4 Frame Distributor … This standard does not mandate any particular distribution algorithm(s); however, any distribution algorithm shall ensure that, when frames are received by a Frame Collector as specified in 43.2.3, the algorithm shall not cause a) Mis-ordering of frames that are part of any given conversation, or b) Duplication of frames. The above requirement to maintain frame ordering is met by *ensuring that all frames that compose a given conversation are transmitted on a single link in the order* that they are generated by the MAC Client; hence, this requirement does not involve the addition (or modification) of any information to the MAC frame, nor any buffering or processing on the part of the corresponding Frame Collector in order to re-order frames. > >> I hope there's some improvements that can be made, for example if we can determine >> the stream was instigated remotely then flowid would always be valid hence we can use it assuming it >> matches the requested spec or if we can make it clear to the user that laggproto is not the one they requested, I'm open to ideas? > We just need to clear flow id from incoming TCP segments and always generate new flow id for responses > keeping old flow id for IP forwarding case. Please back out your change to not degrade IP forwarding performance. Sorry I don't follow you. You seem to be inferring that we can easily generate a flowid without involving the sending hardware; I can't see how that is possible as that's chicken and egg i.e. you can't get the HW interface to generate the flowid without sending a packet and you can't send a packet until you have a the flowid to decide which interface to send it from.     Regards     Steve