From owner-svn-src-all@freebsd.org Wed Nov 18 14:55:50 2020 Return-Path: Delivered-To: svn-src-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4667C469473; Wed, 18 Nov 2020 14:55:50 +0000 (UTC) (envelope-from gallatin@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Cbm8Q1Zcyz3kSm; Wed, 18 Nov 2020 14:55:50 +0000 (UTC) (envelope-from gallatin@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 291C313D2B; Wed, 18 Nov 2020 14:55:50 +0000 (UTC) (envelope-from gallatin@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id 0AIEtoWo022439; Wed, 18 Nov 2020 14:55:50 GMT (envelope-from gallatin@FreeBSD.org) Received: (from gallatin@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id 0AIEtnuV022417; Wed, 18 Nov 2020 14:55:49 GMT (envelope-from gallatin@FreeBSD.org) Message-Id: <202011181455.0AIEtnuV022417@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: gallatin set sender to gallatin@FreeBSD.org using -f From: Andrew Gallatin Date: Wed, 18 Nov 2020 14:55:49 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r367797 - head/sys/net X-SVN-Group: head X-SVN-Commit-Author: gallatin X-SVN-Commit-Paths: head/sys/net X-SVN-Commit-Revision: 367797 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Nov 2020 14:55:50 -0000 Author: gallatin Date: Wed Nov 18 14:55:49 2020 New Revision: 367797 URL: https://svnweb.freebsd.org/changeset/base/367797 Log: LACP: When suppressing distributing, return ENOBUFS When links come and go, lacp goes into a "suppress distributing" mode where it drops traffic for 3 seconds. When in this mode, lagg/lacp historiclally drops traffic with ENETDOWN. That return value causes TCP to close any connection where it gets that value back from the lower parts of the stack. This means that any TCP connection with active traffic during a 3-second windown when an LACP link comes or goes would get closed. TCP treats return values of ENOBUFS as transient errors, and re-schedules transmission later. So rather than returning ENETDOWN, lets return ENOBUFS instead. This allows TCP connections to be preserved. I've tested this by repeatedly bouncing links on a Netlfix CDN server under a moderate (20Gb/s) load and overved ENOBUFS reported back to the TCP stack (as reported by a RACK TCP sysctl). Reviewed by: jhb, jtl, rrs Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D27188 Modified: head/sys/net/ieee8023ad_lacp.c head/sys/net/ieee8023ad_lacp.h head/sys/net/if_lagg.c Modified: head/sys/net/ieee8023ad_lacp.c ============================================================================== --- head/sys/net/ieee8023ad_lacp.c Wed Nov 18 14:54:55 2020 (r367796) +++ head/sys/net/ieee8023ad_lacp.c Wed Nov 18 14:55:49 2020 (r367797) @@ -832,7 +832,8 @@ lacp_stop(struct lagg_softc *sc) } struct lagg_port * -lacp_select_tx_port_by_hash(struct lagg_softc *sc, uint32_t hash, uint8_t numa_domain) +lacp_select_tx_port_by_hash(struct lagg_softc *sc, uint32_t hash, + uint8_t numa_domain, int *err) { struct lacp_softc *lsc = LACP_SOFTC(sc); struct lacp_portmap *pm; @@ -842,12 +843,14 @@ lacp_select_tx_port_by_hash(struct lagg_softc *sc, uin if (__predict_false(lsc->lsc_suppress_distributing)) { LACP_DPRINTF((NULL, "%s: waiting transit\n", __func__)); + *err = ENOBUFS; return (NULL); } pm = &lsc->lsc_pmap[lsc->lsc_activemap]; if (pm->pm_count == 0) { LACP_DPRINTF((NULL, "%s: no active aggregator\n", __func__)); + *err = ENETDOWN; return (NULL); } @@ -879,7 +882,7 @@ lacp_select_tx_port_by_hash(struct lagg_softc *sc, uin } struct lagg_port * -lacp_select_tx_port(struct lagg_softc *sc, struct mbuf *m) +lacp_select_tx_port(struct lagg_softc *sc, struct mbuf *m, int *err) { struct lacp_softc *lsc = LACP_SOFTC(sc); uint32_t hash; @@ -892,7 +895,7 @@ lacp_select_tx_port(struct lagg_softc *sc, struct mbuf hash = m_ether_tcpip_hash(sc->sc_flags, m, lsc->lsc_hashkey); numa_domain = m->m_pkthdr.numa_domain; - return (lacp_select_tx_port_by_hash(sc, hash, numa_domain)); + return (lacp_select_tx_port_by_hash(sc, hash, numa_domain, err)); } /* Modified: head/sys/net/ieee8023ad_lacp.h ============================================================================== --- head/sys/net/ieee8023ad_lacp.h Wed Nov 18 14:54:55 2020 (r367796) +++ head/sys/net/ieee8023ad_lacp.h Wed Nov 18 14:55:49 2020 (r367797) @@ -292,8 +292,10 @@ struct lacp_softc { #define LACP_LOCK_ASSERT(_lsc) mtx_assert(&(_lsc)->lsc_mtx, MA_OWNED) struct mbuf *lacp_input(struct lagg_port *, struct mbuf *); -struct lagg_port *lacp_select_tx_port(struct lagg_softc *, struct mbuf *); -struct lagg_port *lacp_select_tx_port_by_hash(struct lagg_softc *, uint32_t, uint8_t); +struct lagg_port *lacp_select_tx_port(struct lagg_softc *, struct mbuf *, + int *); +struct lagg_port *lacp_select_tx_port_by_hash(struct lagg_softc *, uint32_t, + uint8_t, int *); void lacp_attach(struct lagg_softc *); void lacp_detach(void *); void lacp_init(struct lagg_softc *); Modified: head/sys/net/if_lagg.c ============================================================================== --- head/sys/net/if_lagg.c Wed Nov 18 14:54:55 2020 (r367796) +++ head/sys/net/if_lagg.c Wed Nov 18 14:55:49 2020 (r367797) @@ -1763,6 +1763,7 @@ lookup_snd_tag_port(struct ifnet *ifp, uint32_t flowid struct lagg_port *lp; struct lagg_lb *lb; uint32_t hash, p; + int err; sc = ifp->if_softc; @@ -1783,7 +1784,7 @@ lookup_snd_tag_port(struct ifnet *ifp, uint32_t flowid flowtype == M_HASHTYPE_NONE) return (NULL); hash = flowid >> sc->flowid_shift; - return (lacp_select_tx_port_by_hash(sc, hash, numa_domain)); + return (lacp_select_tx_port_by_hash(sc, hash, numa_domain, &err)); default: return (NULL); } @@ -2580,12 +2581,13 @@ static int lagg_lacp_start(struct lagg_softc *sc, struct mbuf *m) { struct lagg_port *lp; + int err; - lp = lacp_select_tx_port(sc, m); + lp = lacp_select_tx_port(sc, m, &err); if (lp == NULL) { if_inc_counter(sc->sc_ifp, IFCOUNTER_OERRORS, 1); m_freem(m); - return (ENETDOWN); + return (err); } /* Send mbuf */