From owner-svn-src-head@freebsd.org Fri Jan 5 09:44:28 2018 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1F660EC0AAB for ; Fri, 5 Jan 2018 09:44:28 +0000 (UTC) (envelope-from steven@multiplay.co.uk) Received: from mail-wr0-x229.google.com (mail-wr0-x229.google.com [IPv6:2a00:1450:400c:c0c::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9E86C8092E for ; Fri, 5 Jan 2018 09:44:27 +0000 (UTC) (envelope-from steven@multiplay.co.uk) Received: by mail-wr0-x229.google.com with SMTP id s13so1451146wra.10 for ; Fri, 05 Jan 2018 01:44:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language; bh=4qNXjTrJAhuAoFJQq8RJI4GSnA6eFBe1dn0FxXF4fBY=; b=EFxBUG22EgWbpfunoWo/NZ4n9A3Hjrf+h3db8nyj8Mhc+l9/+MjF0CS4URNqiUA4Y0 GJneEB46bSxJyJm0n5K0PcjKby78omvckcRjRlqeaqcdeLmHnR1FSK3Og2ewmS66KX3r GquRZGQ7lTchhtdZeZPKQNGESYVB9AvYpYnPpk93GYkPh3re1SdcR2TadagUhObik0QQ Bvmw2+TFV/X2gnxsf3j4/KEKFSlh3i10vLaa5YIeqhjr7/8hnRO3cuQE3gHSZ3BhRI3p qxLvAlSgk8XmAbS1KXMNUsXMPnh2GN0+8rAyriKKVN3Vwq26LhIUCQgOnDcETg6iqv/8 IRSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language; bh=4qNXjTrJAhuAoFJQq8RJI4GSnA6eFBe1dn0FxXF4fBY=; b=mH5cZjN2GMaBM4VFUlVoU5XBsQaPLVcwzBc0hS+77O/3nEbeJE2pmi+Ky9m5WiyaTV JE3ch3UUvJ7DgMKfcbrPDtt1Kpx8qJnVEYawmMzqeKEfxUp071W0/wZMQuzNYK97XoFi CLNLgiK1UVzUqybYE79z0TvF3WrVWEFu8YWTEUCyPfZMB4SchzZ1xAuqdUp1jGlJaZic Mg+m5vJJfgbSRzZOdpIe5xVi19wD73V+l7FgfwYn/sfq1AX6DDkNuM+iLOqbF5grlEle jaGsCKNh6449i8qoD8jv5uvKczi0n6iIHW0VeTNOic8l24qvL0AQMAUXbXuZF6wA3xcX c1nw== X-Gm-Message-State: AKGB3mI2fFbUEZKxA/XdxG1S5FZtwiGOV4CJ1SCv3Y75nzlplNELLPCi TkYqlr5r3Igx04+99z60bPmhcLgQJIE= X-Google-Smtp-Source: ACJfBosC96rEHd9AM20SPgIl1pTuzsUKvscPuHx67M804dhThvBIyS9aW6QyD1ZHUn+vknHtZipVMw== X-Received: by 10.223.188.78 with SMTP id a14mr2403847wrh.267.1515145465518; Fri, 05 Jan 2018 01:44:25 -0800 (PST) Received: from [10.10.1.111] ([185.97.61.1]) by smtp.gmail.com with ESMTPSA id q3sm260869wre.28.2018.01.05.01.44.24 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 05 Jan 2018 01:44:24 -0800 (PST) Subject: Re: svn commit: r327559 - in head: . sys/net To: Adrian Chadd Cc: hiren panchasara , Eugene Grosbein , src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org References: <201801042005.w04K5liB049411@repo.freebsd.org> <5A4E9397.9000308@grosbein.net> <20180104224214.GD18879@strugglingcoder.info> <63c3c450-aeaf-bdd5-5e16-414146c9bb3a@multiplay.co.uk> From: Steven Hartland Message-ID: <4b7b011a-9fe3-f0ec-04a0-c201a2cfcfda@multiplay.co.uk> Date: Fri, 5 Jan 2018 09:44:25 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.5.2 MIME-Version: 1.0 In-Reply-To: Content-Language: en-US Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Jan 2018 09:44:28 -0000 I found https://wiki.freebsd.org/NetworkRSS but I couldn't see any options mentioned, is there a sysctl or kernel option for that Adrian? For reference our current test is on a production LB running 11.0-RELEASE. We're in the process of updating our HEAD box for additional testing. On 05/01/2018 02:55, Adrian Chadd wrote: > does it also happen when you actually enable RSS in the kernel? Since > like I went through a whole lot of pain to assign a flowid at > connection setup time. > > > > -a > > > On 4 January 2018 at 15:37, Steven Hartland wrote: >> >> On 04/01/2018 22:42, hiren panchasara wrote: >> >> On 01/04/18 at 09:52P, Steven Hartland wrote: >> >> On 04/01/2018 20:50, Eugene Grosbein wrote: >> >> 05.01.2018 3:05, Steven Hartland wrote: >> >> Author: smh >> Date: Thu Jan 4 20:05:47 2018 >> New Revision: 327559 >> URL: https://svnweb.freebsd.org/changeset/base/327559 >> >> Log: >> Disabled the use of flowid for lagg by default >> >> Disabled the use of RSS hash from the network card aka flowid for >> lagg(4) interfaces by default as it's currently incompatible with >> the lacp and loadbalance protocols. >> >> The incompatibility is due to the fact that the flowid isn't know >> for the first packet of a new outbound stream which can result in >> the hash calculation method changing and hence a stream being >> incorrectly split across multiple interfaces during normal >> operation. >> >> This can be re-enabled by setting the following in loader.conf: >> net.link.lagg.default_use_flowid="1" >> >> Discussed with: kmacy >> Sponsored by: Multiplay >> >> RSS by definition has meaning to received stream. What is "outbound" stream >> in this context, why can the hash calculatiom method change and what exactly >> does it mean "a stream being incorrectly split"? >> >> Yes RSS is indeed a received stream but that is used by lagg for lacp >> and loadbalance protocols to decide which port of the lagg to "send" the >> packet out of. As the flowid is not known when a new "output" stream is >> instigated the current code falls back to manual hash calculation to >> determine which port to send the initial packet from. Once a response is >> received a tx then uses the flowid. This change of hash calculation >> method can result in the initial packet being sent from a different port >> than the rest of the stream; this is what I meant by "incorrectly split". >> >> For my understanding, is this just an issue for the first packet when we >> originate the flow? Once we have a response and if flowid is there, we'd >> use it, right? OR am I missing something? >> >> Initially yes, but that can cause a whole cascading set of problems. If the >> source machine sends from two different ports then flow can traverse across >> the network using different paths and hence arrive at the destination on >> different ports too, causing the corresponding issue on the other side. >> >> And with this change, we'd always go and do manual calculation even when >> we have a valid flowid (i.e. we didn't initiate a connection)? >> >> Correct, but there's potentially no easy way to correctly determine what the >> flowid and hence hash should be in this case, likely impossible if the lagg >> consists of different interface types. >> >> In addition if the hardware hash doesn't match the requested one as per >> laggproto then additional issues could also be triggered. >> >> Our TCP stack seems fragile during setup to out of order packets which this >> multipath behavior causes, we've seen this on our loadbalancers which is >> what triggered the investigation. The concrete result is many aborted TCP >> connections, over 300k ~2% on the machine I'm looking at. >> >> I hope there's some improvements that can be made, for example if we can >> determine the stream was instigated remotely then flowid would always be >> valid hence we can use it assuming it matches the requested spec or if we >> can make it clear to the user that laggproto is not the one they requested, >> I'm open to ideas? >> >> Regards >> Steve >>