Date: Fri, 5 Jan 2018 01:41:26 -0800 From: hiren panchasara <hiren@strugglingcoder.info> To: Steven Hartland <steven@multiplay.co.uk> Cc: Eugene Grosbein <eugen@grosbein.net>, src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: Re: svn commit: r327559 - in head: . sys/net Message-ID: <20180105094126.GE18879@strugglingcoder.info> In-Reply-To: <63c3c450-aeaf-bdd5-5e16-414146c9bb3a@multiplay.co.uk> References: <201801042005.w04K5liB049411@repo.freebsd.org> <5A4E9397.9000308@grosbein.net> <f133b587-1f7e-4594-31d1-974775ad55be@freebsd.org> <20180104224214.GD18879@strugglingcoder.info> <63c3c450-aeaf-bdd5-5e16-414146c9bb3a@multiplay.co.uk>
next in thread | previous in thread | raw e-mail | index | archive | help
--XStn23h1fwudRqtG Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 01/04/18 at 11:37P, Steven Hartland wrote: >=20 >=20 > On 04/01/2018 22:42, hiren panchasara wrote: > > On 01/04/18 at 09:52P, Steven Hartland wrote: > >> On 04/01/2018 20:50, Eugene Grosbein wrote: > >>> 05.01.2018 3:05, Steven Hartland wrote: > >>> > >>>> Author: smh > >>>> Date: Thu Jan 4 20:05:47 2018 > >>>> New Revision: 327559 > >>>> URL: https://svnweb.freebsd.org/changeset/base/327559 > >>>> > >>>> Log: > >>>> Disabled the use of flowid for lagg by default > >>>> =20 > >>>> Disabled the use of RSS hash from the network card aka flowid for > >>>> lagg(4) interfaces by default as it's currently incompatible with > >>>> the lacp and loadbalance protocols. > >>>> =20 > >>>> The incompatibility is due to the fact that the flowid isn't know > >>>> for the first packet of a new outbound stream which can result in > >>>> the hash calculation method changing and hence a stream being > >>>> incorrectly split across multiple interfaces during normal > >>>> operation. > >>>> =20 > >>>> This can be re-enabled by setting the following in loader.conf: > >>>> net.link.lagg.default_use_flowid=3D"1" > >>>> =20 > >>>> Discussed with: kmacy > >>>> Sponsored by: Multiplay > >>> RSS by definition has meaning to received stream. What is "outbound" = stream > >>> in this context, why can the hash calculatiom method change and what = exactly > >>> does it mean "a stream being incorrectly split"? > >> Yes RSS is indeed a received stream but that is used by lagg for lacp > >> and loadbalance protocols to decide which port of the lagg to "send" t= he > >> packet out of. As the flowid is not known when a new "output" stream is > >> instigated the current code falls back to manual hash calculation to > >> determine which port to send the initial packet from. Once a response = is > >> received a tx then uses the flowid. This change of hash calculation > >> method can result in the initial packet being sent from a different po= rt > >> than the rest of the stream; this is what I meant by "incorrectly spli= t". > > For my understanding, is this just an issue for the first packet when we > > originate the flow? Once we have a response and if flowid is there, we'd > > use it, right? OR am I missing something? > Initially yes, but that can cause a whole cascading set of problems. If= =20 > the source machine sends from two different ports then flow can traverse= =20 > across the network using different paths and hence arrive at the=20 > destination on different ports too, causing the corresponding? issue on= =20 > the other side. > > And with this change, we'd always go and do manual calculation even when > > we have a valid flowid (i.e. we didn't initiate a connection)? > Correct, but there's potentially no easy way to correctly determine what= =20 > the flowid and hence hash should be in this case, likely impossible if=20 > the lagg consists of different interface types. >=20 > In addition if the hardware hash doesn't match the requested one as per= =20 > laggproto then additional issues could also be triggered. >=20 > Our TCP stack seems fragile during setup to out of order packets which=20 > this multipath behavior causes, we've seen this on our loadbalancers=20 > which is what triggered the investigation. The concrete result is many=20 > aborted TCP connections, over 300k ~2% on the machine I'm looking at. >=20 > I hope there's some improvements that can be made, for example if we can= =20 > determine the stream was instigated remotely then flowid would always be= =20 > valid hence we can use it assuming it matches the requested spec or if=20 > we can make it clear to the user that laggproto is not the one they=20 > requested, I'm open to ideas? IIRC, with 'RSS' in kernconf, most NIC drivers and stack should do the right thing. Look at drivers and also conn startup code in TCP as I recall it doing the flowid mapping correctly when stream originated from the other side and had flowid assigned to it by the NIC. I am mostly concerned about the overhead of manual calculation but my knowledge is a bit rusty right now and lagg has always been special so please try this out and see. Thank you. Hiren --XStn23h1fwudRqtG Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQF8BAABCgBmBQJaT0hDXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRBNEUyMEZBMUQ4Nzg4RjNGMTdFNjZGMDI4 QjkyNTBFMTU2M0VERkU1AAoJEIuSUOFWPt/lefcH/i4+CVA9iz+JvTFG+PpmSbXu k7zV65InItMtcSDrv+rse5CRKs7Aew7wblNYd0PB8qzCUD2esNEv4pvdGDf9SLqF a7i8ErkzexdpBI82YoFr/3DeaCaSnik/DazVDRDnlLrGtx02QeK0Ls4sS1ZxTqXU HU6N6vU95O55uzZ2q325TB6hR+5Y465ZNfAG74s/mIgrT3M1jooXLrGURvIX7JgN H1CM7Ladxq1CvC42q+eDHwWdnn+JGPffKkNfXltf3NYLxBh0OoRAvn6eGpsuQteh A9yjJgaBL002CvsbDwx0D5wmkE7WHpW7QYcf33N69t+Zv7iMVB/B1Ckx8kHm6ho= =oGtE -----END PGP SIGNATURE----- --XStn23h1fwudRqtG--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20180105094126.GE18879>