From owner-svn-src-head@freebsd.org Fri Jan 5 02:55:24 2018 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 41C84EA9D10; Fri, 5 Jan 2018 02:55:24 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-wm0-x22c.google.com (mail-wm0-x22c.google.com [IPv6:2a00:1450:400c:c09::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C81826FDB6; Fri, 5 Jan 2018 02:55:23 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by mail-wm0-x22c.google.com with SMTP id b76so95488wmg.1; Thu, 04 Jan 2018 18:55:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=NeOBSvadnbFzqsm4VZz2k4qtsxgG9LwdusuVhtqb7u8=; b=JhnmHjfo42iihOM5vIXU22C8r2OfpAWmQ3tzXvltlp3DCXok3hOeSkL6GfCCUPMtAt 2XCKzQIsii177xdDJ2Nv4l5CMr4sA88b4QOrQOx83/1ISpW9qXVx+UQyQxUTyW4w0XlU mZAGiGIVIGZ7+b6vFwtq59sjbwJ8X4M5GjXns+Oca+EhnXzwIGfOrKpiVW00LsgXgJRM xg7itzBKaSyxPcLSQPngZ+6plTrfBo0xLo8vgQFaLvlmFjlIW7AsZ5NgeF+Bj19Wo0KA 5GxzkH0KWuEMFBHqQ76KupMQTxFweGOCJ5z0UbIa5ZZsqWL0l4qLJQzBOJ6TZ2tx0bBh qVXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=NeOBSvadnbFzqsm4VZz2k4qtsxgG9LwdusuVhtqb7u8=; b=uf+E6ew7zFjHatlJI+Lf2TLTgVn+SkHzW2aRqt4PUoXoByQr+s2j5XpmuvOUxmVf5g blOa5a8YmN7hig9umtM//x8fv4K6as86SZIJCek//P2F6tQ9nA6MGK8OFlTydjMq8rhs fCjAWD+95cHFnWvu6ZpdWHf86bQ4AS7+BYqB/vAVOE3nnM+DtjctO9VcO0nkZDyWVcpG 0flWjHauXX/gvS0os4KksF0XyEpI8nVYovkmuf6bVN6j178fl1LmbDzbt2yubEWpJraM dRO8DLHIACXl2PzBqmJiX+iODcPnjguB+Fw30Ih+b+dCsIWpo6feB3yHQ/B6RCIiET2l 43pA== X-Gm-Message-State: AKGB3mIcy5baSmN5rCVa89eADO65qrgxmAk2v2/8DP6EUYaosc+/sWce Abv47zdPDprJrlyzQeWq6X62zodeTAMMXE1vaJ7TWg== X-Google-Smtp-Source: ACJfBovls25K4zAi7HLVsm/oPyqRygUh8i5CpaO5WHyVQIOau5WIuW4HnlHRQ4FtPzeXgkRUghcvjB2Voq72qtzfcIE= X-Received: by 10.28.55.74 with SMTP id e71mr1102673wma.50.1515120922254; Thu, 04 Jan 2018 18:55:22 -0800 (PST) MIME-Version: 1.0 Received: by 10.28.213.11 with HTTP; Thu, 4 Jan 2018 18:55:20 -0800 (PST) In-Reply-To: <63c3c450-aeaf-bdd5-5e16-414146c9bb3a@multiplay.co.uk> References: <201801042005.w04K5liB049411@repo.freebsd.org> <5A4E9397.9000308@grosbein.net> <20180104224214.GD18879@strugglingcoder.info> <63c3c450-aeaf-bdd5-5e16-414146c9bb3a@multiplay.co.uk> From: Adrian Chadd Date: Thu, 4 Jan 2018 18:55:20 -0800 Message-ID: Subject: Re: svn commit: r327559 - in head: . sys/net To: Steven Hartland Cc: hiren panchasara , Eugene Grosbein , src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Content-Type: text/plain; charset="UTF-8" X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Jan 2018 02:55:24 -0000 does it also happen when you actually enable RSS in the kernel? Since like I went through a whole lot of pain to assign a flowid at connection setup time. -a On 4 January 2018 at 15:37, Steven Hartland wrote: > > > On 04/01/2018 22:42, hiren panchasara wrote: > > On 01/04/18 at 09:52P, Steven Hartland wrote: > > On 04/01/2018 20:50, Eugene Grosbein wrote: > > 05.01.2018 3:05, Steven Hartland wrote: > > Author: smh > Date: Thu Jan 4 20:05:47 2018 > New Revision: 327559 > URL: https://svnweb.freebsd.org/changeset/base/327559 > > Log: > Disabled the use of flowid for lagg by default > > Disabled the use of RSS hash from the network card aka flowid for > lagg(4) interfaces by default as it's currently incompatible with > the lacp and loadbalance protocols. > > The incompatibility is due to the fact that the flowid isn't know > for the first packet of a new outbound stream which can result in > the hash calculation method changing and hence a stream being > incorrectly split across multiple interfaces during normal > operation. > > This can be re-enabled by setting the following in loader.conf: > net.link.lagg.default_use_flowid="1" > > Discussed with: kmacy > Sponsored by: Multiplay > > RSS by definition has meaning to received stream. What is "outbound" stream > in this context, why can the hash calculatiom method change and what exactly > does it mean "a stream being incorrectly split"? > > Yes RSS is indeed a received stream but that is used by lagg for lacp > and loadbalance protocols to decide which port of the lagg to "send" the > packet out of. As the flowid is not known when a new "output" stream is > instigated the current code falls back to manual hash calculation to > determine which port to send the initial packet from. Once a response is > received a tx then uses the flowid. This change of hash calculation > method can result in the initial packet being sent from a different port > than the rest of the stream; this is what I meant by "incorrectly split". > > For my understanding, is this just an issue for the first packet when we > originate the flow? Once we have a response and if flowid is there, we'd > use it, right? OR am I missing something? > > Initially yes, but that can cause a whole cascading set of problems. If the > source machine sends from two different ports then flow can traverse across > the network using different paths and hence arrive at the destination on > different ports too, causing the corresponding issue on the other side. > > And with this change, we'd always go and do manual calculation even when > we have a valid flowid (i.e. we didn't initiate a connection)? > > Correct, but there's potentially no easy way to correctly determine what the > flowid and hence hash should be in this case, likely impossible if the lagg > consists of different interface types. > > In addition if the hardware hash doesn't match the requested one as per > laggproto then additional issues could also be triggered. > > Our TCP stack seems fragile during setup to out of order packets which this > multipath behavior causes, we've seen this on our loadbalancers which is > what triggered the investigation. The concrete result is many aborted TCP > connections, over 300k ~2% on the machine I'm looking at. > > I hope there's some improvements that can be made, for example if we can > determine the stream was instigated remotely then flowid would always be > valid hence we can use it assuming it matches the requested spec or if we > can make it clear to the user that laggproto is not the one they requested, > I'm open to ideas? > > Regards > Steve >