From owner-freebsd-net@freebsd.org Tue Jun 27 17:06:13 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 65BF4D8B0D0 for ; Tue, 27 Jun 2017 17:06:13 +0000 (UTC) (envelope-from matt.joras@gmail.com) Received: from mail-wr0-x22d.google.com (mail-wr0-x22d.google.com [IPv6:2a00:1450:400c:c0c::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EC3C7745E6 for ; Tue, 27 Jun 2017 17:06:12 +0000 (UTC) (envelope-from matt.joras@gmail.com) Received: by mail-wr0-x22d.google.com with SMTP id c11so162505359wrc.3 for ; Tue, 27 Jun 2017 10:06:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=VS3RBIjlRRAQdaYcuaJOcQ7doSjInLpuu2d+3569afo=; b=qZ6r7GW+gWhXMorw3YKY0mcYhdaVeCg5MGghTq8kQULjJUdIFgtW5vr/miOwU9hqBX oqoyJWoLu6QOJtuM6gFpX4pZkg74ouk3oCTaphtvxD6v/5Twmx5ZdCFFKkpkBCHRWzSK KaBxAvsaMViLN0a0MO/iKeJ2Woetmq4a2qx0ramF9xMWBfFMAFwaFDRCNiLR7z/vJchU RSWFwg06KFfhjD4xrWNZ3MDQFZNrerFjsuhCiSLfLWxhH9bXwzihHrxllegfXhwxY0dp z9jno7G7ok7eQhtjP6p2uZjrRVCoeiv3fLO3t48NJPTBYF7lmklMaTc2s07BXJpThhnk xKXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=VS3RBIjlRRAQdaYcuaJOcQ7doSjInLpuu2d+3569afo=; b=Co2X4wfGXtpgpCR6xrArXZnO81iboHzv3lC641QCpztwb3oneA9dDxY2f1MOHTo2rW R7Hh9FkgbwN+wviUAkipn21+6h0nZUtlH+3X7Gvw1B0mU4jVtpzvWDGO/kVnvibg0I/2 MmCEIVx0tbnTGZN8jjtTSc8V95s/Es/a4nZ2TVDYHovVsitTBlqSSREx558kf7ibOPxV elyM4/rG6LKUZA0O4mEwvrm0HuE4IhnwcbD7ib8zq1wIGEWo62goB1oROhTMqRNkrojj Za8y3f0OBGCQU6RH5/oDrS/QHQGn9e2/8EPWkNPgbGFW6alemH0JZZrkPbqZa1p3hVCE JMpA== X-Gm-Message-State: AKS2vOzpm1yEtRN4lPkcgl3zgOHIE2Y7G8aKI5sLiER1bOsaLmvaXTgD 9WdNQt51BQpoew3ndx5/W3BOFCx5iA== X-Received: by 10.223.143.77 with SMTP id p71mr17134911wrb.3.1498583171183; Tue, 27 Jun 2017 10:06:11 -0700 (PDT) MIME-Version: 1.0 Received: by 10.223.160.42 with HTTP; Tue, 27 Jun 2017 10:06:10 -0700 (PDT) In-Reply-To: References: <5ABA962E-A90A-4C25-A5A7-EE5CF66FFDD4@pasteur.fr> <20170627.125426.74697078.sthaug@nethelp.no> From: Matt Joras Date: Tue, 27 Jun 2017 10:06:10 -0700 Message-ID: Subject: Re: Sporadic TCP/RST sent to client To: Youssef GHORBAL Cc: "sthaug@nethelp.no" , "freebsd-net@freebsd.org" , "nparhar@gmail.com" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Jun 2017 17:06:13 -0000 On Tue, Jun 27, 2017 at 5:05 AM, Youssef GHORBAL wrote: > > There is nothing in the 802.3ad that mandates stickiness of flows per NIC= , the only thing explicit is that hash algorithm needs to maintain packet o= rder. In this case, strictly speaking, it's : Packets do leave in "order" a= nd do arrive in "order". I think the important point is that the ordering is not guaranteed in this case, despite whether it's happening or not. As soon as you are using a round-robin lagg on one end you've pretty much lost all guarantees of ordering at the remote end. Unless the switch has some way to know, which as Steinar noted is usually done through a negotiated or statically-configured hash-based lagg, there's no way for it to enforce the ordering you're expecting for proper behaviour. So even if there was some notion of protocol ordering in netisr, the fact that you're using round-robin on one endpoint opens up the possibility for this kind of situation anyway. Further, I would argue that round robin is not a valid 802.3ad/802.1AX algorithm, per how it defines a frame distributor: "This standard does not mandate any particular distribution algorithm(s); however, any distribution algorithm shall ensure that, when frames are received by a Frame Collector as specified in 5.2.3, the algorithm shall not cause: a) Misordering of frames that are part of any given conversation, or b) Duplication of frames. The above requirement to maintain frame ordering is met by ensuring that all frames that compose a given conversation are transmitted on a single link in the order that they are generated by the MAC Client; hence, this requirement does not involve the addition (or modification) of any information to the MAC frame, nor any buffering or processing on the part of the corresponding Frame Collector in order to reorder frames." > Sure, I was just wondering if the FreeBSD network stack was built with th= e fact that each flow needs to arrive on the same NIC and the system was de= signed with this assumption in mind or not. > > I reported it here, thinking that maybe it's a subtle buggy corner case a= nd maybe the community was interesting to know about and maybe fix : > > - If the stack is working as expected and was built with the assumption t= hat each incoming flow needs to stick to a NIC during it's lifetime, maybe = documentation needs to be more explicit regarding this situation. In that c= ase I'll file documentation enhancement bug report. > - If the stack is misbehaving, maybe help the community identify the root= cause and help fixing it > As far as I can tell, as Navdeep noted, there's no unexpected behaviour in your case. "Flows" are a concept that the protocols, in this case TCP, knows about. The devices themselves (Ethernet cards) usually have mechanics to make packet delivery decisions based on flow information (e.g. RSS hashing), but as far as I know that is generally limited within a single port, so it doesn't really help in the general case of a lagg. Matt