From nobody Fri Jul 2 02:40:49 2021 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 2B3CE11D5E0F for ; Fri, 2 Jul 2021 02:40:53 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670064.outbound.protection.outlook.com [40.107.67.64]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4GGK836tRcz4S9S for ; Fri, 2 Jul 2021 02:40:51 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=dYIfkX+wRQdqM+qdo+o2HjIjQNTEW8V2J6gw8H1mWtfW7tU6WDkIK2dYqxQEIS2751kmdhPOQY9cVdwbW6nEa0CZLUrLEfwbic9FrlZQBrM2TeQsCegN2LhvLvTluqYUdJQOtxntq7IXc34hXlOEnDsJvwJdWexb4b11Wi9HBf+vpEY3c+4cxs4CVQcmXOCgvHzZY5rgZHFMuHOn4drP7ZLsnAdPMz1TnrP8GM9f6vIAFZJMiIRA3sADtBzZg7Sqwb6ivGRxEz6aKSI5M/NWnGpqeNodbGOdLDNbixiBS1GNfR1znRwAxgNcMkTduuHy0fmUdLkdg8EYkmzEbxQGcQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=umEfovKK+Ips0te99VgXw8eWKRjM8vUw6MYHe44stMI=; b=kREnvr+kiNkN9zfyXAPSjF0i02Mzo0S8UkpgILwYxX6jlLfQp3cWqTEIyPtbHdAhzId0We6AFcjmtL6ZpxmoktcLOCsuIFLG4uKA/reqp3XHJ2TwQe3+GnzFtkc5RIQSO0Hga/1p46gbh54YE7PKAwDT5Tv2dvG4sFh4YAo4SPOzEP2076C8T7uGR/42PHMQ52bfO9Hx4YRCUr3IFo650rKv8GdwTovn9B6gHC6QbZ3JNmi0hYoNkieZQHxpjGPiveqD8NgzEkMYklsDBz1e1Mpwd0YLysl/LDuhRlZxzhL3aQF21dCav2r65K6tkixIGu9sXTxhAu9RE8ahTPqSvw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=umEfovKK+Ips0te99VgXw8eWKRjM8vUw6MYHe44stMI=; b=hTrl3MkmHpCewAPz9WlReyGe7Bb0tKBnfC0rPvyXD1nupzR6BTp2PaItCPDEjKayrUrWwcDpPMqsELkMvWFOXEZPA7s7nt6xfVGq38if2L+RAp/atMiT6uvgLrJyBfNabFJEKqFmLqWXJgvDHqisAVq9U/byHTskG0kV2BFtO33BQIIGvr+B+bn8CiGGqzOpPitnADCBwonGpklvGsBJv5rXiqiXkN9cLSnij9gtB8E8RR16eIgfkmPdElRr98cYMC7/V32RFoLosRB8EteHobC0FS9/xp4f+6mrAv/FLD9MX3h5Lg/gojdkry6nzqz/qgdBSXrg+trjoQs1abVp7w== Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:19::29) by YQBPR0101MB2275.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:c::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4264.23; Fri, 2 Jul 2021 02:40:49 +0000 Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::583:528b:dbac:37bf]) by YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::583:528b:dbac:37bf%4]) with mapi id 15.20.4287.024; Fri, 2 Jul 2021 02:40:49 +0000 From: Rick Macklem To: Peter Eriksson CC: freebsd-net Subject: Re: RFC: NFS trunking (multiple TCP connections for a mount Thread-Topic: RFC: NFS trunking (multiple TCP connections for a mount Thread-Index: AQHXbHjbAJ3BuaRhHUuq5k+hERhS36sqtG0AgAKnXIaAAZ3XpQ== Date: Fri, 2 Jul 2021 02:40:49 +0000 Message-ID: References: ,<362300CE-30DA-4552-A3E4-0F3DFE385B2A@lysator.liu.se>, In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: d0aa4780-a46d-438d-501e-08d93d02cd62 x-ms-traffictypediagnostic: YQBPR0101MB2275: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: HMbHH3L2u/CI37Vke8Mlp/ijrc1xcIX9hajAIapocGqG4V7NjpxcHfEMGlJb9V9y4mFGtCzQK0q6IVnyJkOezN5/X3s9gb0jR91XmDE0sPpPZDY8igWrWN0SyIzs46/pPH6nYU2+LGJsaDpS42QJgbla5J7cvf44zB/9vo3VeIfwj+KnZu3EXZwPUP9VIZhrSYWwTI74wuxxoDTdBuy0DgovKQkeEo9xzNuWq+oAGC8aEzGzjfH/7gMsxTEbu6QjSlHqw/9Zg8/f/o9i9A5MAVnqdi8Rhlh82z1ebF+I4B+1V7Nb/h72Z4iPSiqJ2WVa/ET67T+AfYj4bB6rSc5gAazrVEbdNMHOYppZupm7bjgEF/aS6aos6DPE8C4d6fgjLLveTnAUbIoAswGNQlytkXmiCcLQFHz3N138RIGicFUwVrDFQ3/L+8mwMNPNUI85Q3beutoAeDwjAHQD4OfECl+C2ukOZzWmYMhh2EYiHTbWpvCqpulvC9VFWWvGMvng1non1yVBKEz15RhW8mvmVUnHbyLNA3teYEDk2LB42bPHNRL03dvU+CL+Y1s/nznGubx/fNkm8NkLVDyV82L7u5pZhII/l2KqpgHNLBs1oFhhOsK65avkTGDyRDk2nYbUvb/bXStZyTQEWkOdDHcuI9vNgDFxL5DNHWChJ3f+V+DuNQBZxPu1J8Po9KsEXOfoLeRwK12u3//gag3QtkdPxgTY5zqHmjkhOnyo8kob4ljTF+Lq8oKs54Eq2fIhb9/g7c/vwKPKBsuDqegFK9qXfA== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(346002)(366004)(39860400002)(396003)(136003)(376002)(55016002)(83380400001)(186003)(478600001)(66476007)(66446008)(64756008)(122000001)(966005)(786003)(8676002)(76116006)(316002)(4326008)(38100700002)(66946007)(86362001)(2906002)(91956017)(9686003)(296002)(66556008)(52536014)(8936002)(7696005)(71200400001)(5660300002)(6506007)(6916009)(53546011)(33656002);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?Windows-1252?Q?f341kEbfP72XDrlnF+lL/opWzlKG4j6ygwaGNwgg0K9Bn0s+Fa9iuA5R?= =?Windows-1252?Q?ccuoqrXouQx/1K98SrcpdzOALFRfURq/8iB82MFdBvqX8yDb7W65rUsU?= =?Windows-1252?Q?3rfLQAUCcoWYGbL5wtRm8FRHqxCL+kgd60ysAgpn9tyw4JIWrszzLJDk?= =?Windows-1252?Q?IAgHRFB+2kOn6ahPETsUzw5VYPR823WqoCCBF7ivX+UorZUxpBu+afVY?= =?Windows-1252?Q?1wUZDj9SFxl9iY0zY1zP4X2entP+K0m8ZAdI67j4bIAFgiblGY4Q3mlX?= =?Windows-1252?Q?ZMJL871+g4zi8LThiz4Adh3ngRJIawTkN5E2HXg4sJiKW3Y1hQ04etDr?= =?Windows-1252?Q?jErUdTSx5ZgEoi7WpdNNUTsuzQ3xRnvKu2X6KlgUfqfWiWmLUkMeTXkU?= =?Windows-1252?Q?Rv4vthel0dJezlQ5QOZXdc8bup15UPU76KwrIw1YWujGcvTfuPH1S+vy?= =?Windows-1252?Q?8V91b+Vpl9smbuLvpBkFQ9GT8A8beY5lrh7KKZRQM/kzYCX9Ax0fzkFe?= =?Windows-1252?Q?HSRxQVrX9NzShHAObUih2XnT+WRXhGnKpce5X59nahzdA59fWkqmKLFe?= =?Windows-1252?Q?o0fT9ViqwjPLDhtiji/gVDfTTMqIkJ/3F7VMVcbw5nVcLSsPgClRK82n?= =?Windows-1252?Q?Z+r2BjU67isDp/f6nIjtkExxMR2Otl2Ar4c3DHu10VsrFoP3alfuz4L3?= =?Windows-1252?Q?xjAjs427AtbYkGVHnOvNAZpNUDKCM4KNnDoyIUwj8Af7eLTJR2ExFCHk?= =?Windows-1252?Q?RDdm/hheJ9qVN2wdHp1ZYcfeLhKU/O0MdMdAMi5AB252GBrOxWN5ZP4/?= =?Windows-1252?Q?Fp+GbD1WRrrsktizsnxvJw7kSlu//wbJ2jb1XEQjTBwFkf/C2wXyhcg8?= =?Windows-1252?Q?Dpiy6nnO3oe03IJZk0Dw8HbHuTkHHCOZEeHDLGyBcGEUH/6IZAWXY0ag?= =?Windows-1252?Q?1JUC1POq8efRBhxKrNvWcBboeU95yyeLU+14Eg4JEbnCM0QzJ5abkGiY?= =?Windows-1252?Q?qpbfDcw8ZmZaJGGNhhyipofVP1+pQZeyiyDS3BPZhfAAhrQNm3+JAC8l?= =?Windows-1252?Q?7VR0Rib9rdSFJhR5f7Hakv+MVjvnYQYtlikNAd6gY5piR0mm/a2XhQgg?= =?Windows-1252?Q?h8DyRc12h6Vy68pEhjm5ul9VMv8rpaIJ/8M1MXADPeBZDSHNwOErMCPV?= =?Windows-1252?Q?sMHRU+HBK9ZHxTENbMZB1zS8GVHywQ2ES3vKKjajqVg917Mwc/s3QPvA?= =?Windows-1252?Q?7NeDxNZcc55X8Y5Rdb3whKeS0bzifaXzk9ARZJ7kgMOrTAXt9ADhj45A?= =?Windows-1252?Q?8fVsVEVsmYD7YA3Nqeb4wYi6/TDH/quhqDPh7JQMOa2ccI1JjDRl9IEp?= =?Windows-1252?Q?a0ZHazCefaiDp9IqvV8kYtaWlyBu9qurIKHmZFucwU2R3/3N3XpgB3lD?= =?Windows-1252?Q?kN4XVSMkU2n8C4rtO3TpD1mCVDrs4mKQ2EcdlMyYpbE=3D?= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: d0aa4780-a46d-438d-501e-08d93d02cd62 X-MS-Exchange-CrossTenant-originalarrivaltime: 02 Jul 2021 02:40:49.4207 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: XJYdP/D9mqJ67eO809GrDAbe2dwLC/CAa5UC8AZUkdcP1bbw0Twhx4sSEmWHQtE9t8EnkvjBUl/xO3tKC5ucxQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQBPR0101MB2275 X-Rspamd-Queue-Id: 4GGK836tRcz4S9S X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=uoguelph.ca header.s=selector1 header.b=hTrl3Mkm; arc=pass (microsoft.com:s=arcselector9901:i=1); dmarc=pass (policy=none) header.from=uoguelph.ca; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.67.64 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; RBL_DBL_DONT_QUERY_IPS(0.00)[40.107.67.64:from]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector1]; FREEFALL_USER(0.00)[rmacklem]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; NEURAL_SPAM_SHORT(1.00)[0.999]; SPAMHAUS_ZRD(0.00)[40.107.67.64:from:127.0.2.255]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[uoguelph.ca:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[40.107.67.64:from]; DMARC_POLICY_ALLOW(-0.50)[uoguelph.ca,none]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; MAILMAN_DEST(0.00)[freebsd-net]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.67.64:from] X-ThisMailContainsUnwantedMimeParts: N Rick Macklem wrote:=0A= >In case anyone is interested in testing and/or reviewing the patch,=0A= >it is at https://reviews.freebsd.org/D30970.=0A= >=0A= >Only lightly tested at this point.=0A= >=0A= >The NFS mount option is "nconnect=3D", where 2<=3D N <=3D 16,=0A= >same as Linux. (I haven't done a man page patch yet.)=0A= I have updated the patch so that the original TCP connection is=0A= used for RPCs that consist of small messages (therefore not needing=0A= much network bandwidth) and the RPCs (Read/Readdir/Write) that=0A= use larger messages are sent on the N-1 additional TCP connections=0A= in a round robin fashion.=0A= =0A= The message below was posted a couple of days ago on linux-nfs@vger.kernel.= org.=0A= It might be unfair to put it here, out of context, but I think it at least= =0A= suggests that separating the larger RPC messages from the small ones=0A= (mostly Lookup/Getattr/Access metadata related RPCs) may be useful=0A= under certain circumstances.=0A= > The original issue described was how a high read/write process on the=0A= > client could slow another process trying to do heavy metadata=0A= > operations (like walking the filesystem). Using a different mount to=0A= > the same multi-homed server seems to help a lot (probably because of=0A= > the independent slot table).=0A= --> For this implementation, there is no separate session/slot table.=0A= (Note that each I/O RPC only uses one table slot.)=0A= =0A= I did not make this small vs large RPCs on a separate TCP connection=0A= a separate option, since I believe there are already too many mount options= .=0A= If others feel it should be a separate mount option, please speak up.=0A= =0A= The phabricator patch has been updated. Please test/review/comment.=0A= =0A= Thanks, rick=0A= =0A= Thanks everyone, for your input, rick=0A= =0A= ________________________________________=0A= From: Peter Eriksson =0A= Sent: Tuesday, June 29, 2021 5:11 AM=0A= To: Rick Macklem=0A= Cc: freebsd-net=0A= Subject: Re: RFC: NFS trunking (multiple TCP connections for a mount=0A= =0A= CAUTION: This email originated from outside of the University of Guelph. Do= not click links or open attachments unless you recognize the sender and kn= ow the content is safe. If in doubt, forward suspicious emails to IThelp@uo= guelph.ca=0A= =0A= =0A= > I don't understand how multiple TCP connections to the same=0A= > server IP address will distribute the load across multiple network=0A= > interfaces?=0A= > I thought that lagg would have handled this?=0A= =0A= =0A= A lagg typically keeps all data in a TCP stream on a specific lagg member (= depending on how the lagg is set up, unless you select the =93roundrobin=94= option in freebsd - don=92t do that unless you like out-of-order packets= =85)=0A= =0A= Network equipment with laggs typically hash the IP streams over the lagg me= mbers based on MAC addresses (source&target), IP addresses (source&target) = and port numbers.=0A= =0A= (We have been diagnosing a fun problem locally where we see packet losses/p= erformance drops over our internal backbone network for certain combination= s of odd/even IP addresses/port numbers when things pass certain SPB =93rou= ters=94 (which typically hash the streams over many =93channels=94 between = routers)=85 Fun fun. :-)=0A= =0A= I think the multiple NFS TCP streams could make for some nice performance i= mprovements in certain cases. And it would be a more generalisation of havi= ng multiple streams between two hosts - one-or-many over IPv4 and one-or-ma= ny over IPv6 at the same time. Windows SMB has a similar feature.=0A= =0A= Just avoid the Linux NFS mounting deadlock issue with =93down=94 servers pl= ease :-)=0A= =0A= - Peter=0A= =0A=