From owner-freebsd-fs@freebsd.org Fri Apr 6 00:19:18 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 26AFDF8E469 for ; Fri, 6 Apr 2018 00:19:18 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-eopbgr660086.outbound.protection.outlook.com [40.107.66.86]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT TLS CA 4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id ADA1F78FFD for ; Fri, 6 Apr 2018 00:19:17 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YQBPR0101MB1042.CANPRD01.PROD.OUTLOOK.COM (52.132.66.153) by YQBPR0101MB1394.CANPRD01.PROD.OUTLOOK.COM (52.132.68.155) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.653.12; Fri, 6 Apr 2018 00:19:16 +0000 Received: from YQBPR0101MB1042.CANPRD01.PROD.OUTLOOK.COM ([fe80::185:356:49c5:794c]) by YQBPR0101MB1042.CANPRD01.PROD.OUTLOOK.COM ([fe80::185:356:49c5:794c%13]) with mapi id 15.20.0653.012; Fri, 6 Apr 2018 00:19:16 +0000 From: Rick Macklem To: Ben RUBSON , "freebsd-fs@freebsd.org" Subject: Re: Linux NFS client and FreeBSD server strangeness Thread-Topic: Linux NFS client and FreeBSD server strangeness Thread-Index: AQHTzEKdJ7nozqXV6kKGauYOIuptM6PyXeQAgACA7h8= Date: Fri, 6 Apr 2018 00:19:16 +0000 Message-ID: References: <369fab06-6213-ba87-cc66-c9829e8a76a0@sentex.net>, In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=rmacklem@uoguelph.ca; x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; YQBPR0101MB1394; 7:zsiNatuc3pXH3sdlLgweos+QzNBuMZ8R7hjVtcYkJ2T1vJsx96Ey0MNZNMQUJRucEzOE1e4OkCYPAtAb++lCYO9NHJjTkOA08F52c7bNomKSNJxyMc72pAvr3pJIdTLkepAB0JaJMd2n3XwkQYVQtrefG9+3qvN9ypCcksT2PMdRXGT1qbEu3ltoF6KKNty0WXpLIv4LgqI+KuOzW9dqQ7RB4TzaRQqxBAYyvw5fH6z/BM8yxX2bgZIPiKzRsHVS x-ms-exchange-antispam-srfa-diagnostics: SOS; x-ms-office365-filtering-correlation-id: cbcdf5ff-1dbe-4d45-98fb-08d59b540875 x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(8989080)(5600026)(4604075)(3008032)(4534165)(4627221)(201703031133081)(201702281549075)(8990040)(2017052603328)(7153060)(7193020); SRVR:YQBPR0101MB1394; x-ms-traffictypediagnostic: YQBPR0101MB1394: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(158342451672863); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040522)(2401047)(8121501046)(5005006)(3002001)(10201501046)(3231221)(944501327)(52105095)(93006095)(93001095)(6041310)(20161123564045)(20161123562045)(20161123558120)(20161123560045)(201703131423095)(201702281529075)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(6072148)(201708071742011); SRVR:YQBPR0101MB1394; BCL:0; PCL:0; RULEID:; SRVR:YQBPR0101MB1394; x-forefront-prvs: 0634F37BFF x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(366004)(346002)(376002)(39380400002)(396003)(39860400002)(199004)(189003)(7696005)(110136005)(86362001)(5250100002)(39060400002)(74482002)(2900100001)(97736004)(186003)(76176011)(99286004)(33656002)(446003)(2906002)(26005)(53936002)(6246003)(786003)(5660300001)(229853002)(478600001)(105586002)(2501003)(14454004)(316002)(81156014)(9686003)(68736007)(81166006)(8676002)(8936002)(3280700002)(11346002)(106356001)(6506007)(25786009)(3660700001)(6436002)(305945005)(55016002)(74316002)(486006)(476003)(102836004); DIR:OUT; SFP:1101; SCL:1; SRVR:YQBPR0101MB1394; H:YQBPR0101MB1042.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) x-microsoft-antispam-message-info: V/wO/Iw1/TJFX1koWrQSiYHofEJpIbzGiwAKyYZR7YnyNhbkfLRno1lYsaTfOrZeRwCLZlKPsSJVAekx6pMY9Mbws7lQtL7UQvM9Y7m35HhZKEN9bYikWkjeJix2tb//uWvKGdlCzoFbbpYrL1A2Tp2lLoc4P+izqBHGufEknUc2ecco5onC1EjZ88BzUcSrzW8tLi4CKHOIyRE1RNmu8qHVWyJ1zN14lh+uUEr4UBCCajxdny+/YityVgnihYEDnWlCt6wlM2AuFd1UyG1c7Ag61Lhh6cXqxr/gEApcl3OS6Gkux3p1+qnNPJ1cZjNZkURI5GeEQ/vr3BsI9uLB1n7t9DKjAEv9qnovsZr3SnF9/8Na51IQoCoX+bLYOwPLuqn/Uv1Jt0LvWfBokEeStbBCYbNTPfXrBEtrn7QK3rU= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: cbcdf5ff-1dbe-4d45-98fb-08d59b540875 X-MS-Exchange-CrossTenant-originalarrivaltime: 06 Apr 2018 00:19:16.3338 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQBPR0101MB1394 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Apr 2018 00:19:18 -0000 Ben RUBSON wrote: >On 04 Apr 2018 20:27, Mike Tancsa wrote: >> Note, doing something like >> >> dd if=3D/dev/zero of=3D/backup/test.bin bs=3D4096 count=3D5000000 > >Note that this test may not be really relevant if you have ZFS compression >enabled. > >> I too am using 9000 for the MTU. > >Did you try to use smaller MTU ? >Some network adapters are known to have bugs requesting 9K mbufs for large >MTUs. >Especially Mellanox, not sure about Chelsio though. When the system uses a mix of mbuf cluster sizes (almost always happens whe= n you use jumbo packets), you can fragment the kernel memory pool they are allocated from to the point that jumbo clusters can't be allocated easi= ly. That breaks NFS performance badly. I had some patches that used jumbo mbuf clusters for the NFS read reply any write requests. They worked fine for a while, but would then get hammered by the fragmentat= ion problem. (As such, they never went into head, etc.) (The main advantage of using jumbo clusters was that the mbuf chain for an RPC had fewer mbufs in it and wouldn't get bit by bugs implementing TSO for interfaces that could only handle 32 buffers for a TSO segment. There is code that avoids this problem in tcp_output(), but it only works if the net device driver sets the parameters correctly.) Some have said that 9K jumbo clusters shouldn't exist in FreeBSD because of the fragmentation problem. Others proposed using separate pools for each mbuf cluster size, but nothing has happened as far as I know. rick