From owner-freebsd-fs@freebsd.org Fri Apr 6 00:44:37 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 00BFCF9055E for ; Fri, 6 Apr 2018 00:44:37 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-eopbgr660068.outbound.protection.outlook.com [40.107.66.68]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT TLS CA 4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 84F7A7A38D for ; Fri, 6 Apr 2018 00:44:36 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YQBPR0101MB1042.CANPRD01.PROD.OUTLOOK.COM (52.132.66.153) by YQBPR0101MB1106.CANPRD01.PROD.OUTLOOK.COM (52.132.67.17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.653.12; Fri, 6 Apr 2018 00:44:34 +0000 Received: from YQBPR0101MB1042.CANPRD01.PROD.OUTLOOK.COM ([fe80::185:356:49c5:794c]) by YQBPR0101MB1042.CANPRD01.PROD.OUTLOOK.COM ([fe80::185:356:49c5:794c%13]) with mapi id 15.20.0653.012; Fri, 6 Apr 2018 00:44:34 +0000 From: Rick Macklem To: Bruce Evans , Kaya Saman CC: FreeBSD Filesystems Subject: Re: Linux NFS client and FreeBSD server strangeness Thread-Topic: Linux NFS client and FreeBSD server strangeness Thread-Index: AQHTzEKdJ7nozqXV6kKGauYOIuptM6Pw9uaAgAAF0oCAAAWNgIAABXOAgAAC9oCAAAJwAIAAimAAgAFKdyY= Date: Fri, 6 Apr 2018 00:44:34 +0000 Message-ID: References: <369fab06-6213-ba87-cc66-c9829e8a76a0@sentex.net> <2019ee5a-5b2b-853d-98c5-a365940d93b5@madpilot.net> <4da08f8b-2c28-cf18-77d2-6b498004d435@gmail.com> <2937ffcc-6b47-91af-8745-2117006660db@sentex.net> , <20180405134730.V1123@besplex.bde.org> In-Reply-To: <20180405134730.V1123@besplex.bde.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=rmacklem@uoguelph.ca; x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; YQBPR0101MB1106; 7:ag053OehrJxQ4VxhSRO1RXI6joHmSU3mIMnp8kPcXlOmFEMy914I+SQHqngNgWCt6iS1bcEfj2uh50wDqDE+W4Bw3AnqIKsIpjItXkSWJG3sH3umEqngdKj0ObZdW0Sj6Pq8adcgiREO2nfPCXnPaPVaEaNJRJ1FpXha6GwjVNInYplxf2Sk7IfKQyto430baUvffHWm4cCYImqgw2iAhla30YlxCxigEJAXUGwZ5asEVfTu81KUadit/TAnUCha x-ms-exchange-antispam-srfa-diagnostics: SOS; x-ms-office365-filtering-correlation-id: f803fec3-d1f8-4ba9-ab96-08d59b579181 x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(8989080)(5600026)(4604075)(3008032)(4534165)(4627221)(201703031133081)(201702281549075)(8990040)(2017052603328)(7153060)(7193020); SRVR:YQBPR0101MB1106; x-ms-traffictypediagnostic: YQBPR0101MB1106: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(158342451672863); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040522)(2401047)(8121501046)(5005006)(3002001)(93006095)(93001095)(10201501046)(3231221)(944501327)(52105095)(6041310)(20161123564045)(201703131423095)(201702281529075)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123562045)(20161123560045)(20161123558120)(6072148)(201708071742011); SRVR:YQBPR0101MB1106; BCL:0; PCL:0; RULEID:; SRVR:YQBPR0101MB1106; x-forefront-prvs: 0634F37BFF x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(39850400004)(396003)(346002)(39380400002)(376002)(366004)(189003)(199004)(6436002)(25786009)(9686003)(74316002)(55016002)(53936002)(105586002)(74482002)(86362001)(7696005)(76176011)(5250100002)(93886005)(99286004)(305945005)(106356001)(97736004)(316002)(33656002)(786003)(229853002)(6506007)(102836004)(3660700001)(26005)(3280700002)(14454004)(186003)(486006)(446003)(478600001)(11346002)(476003)(39060400002)(81166006)(68736007)(5660300001)(4326008)(2900100001)(2906002)(8676002)(81156014)(110136005)(8936002)(6246003); DIR:OUT; SFP:1101; SCL:1; SRVR:YQBPR0101MB1106; H:YQBPR0101MB1042.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) x-microsoft-antispam-message-info: obETjdQAtFogoqySd9vIExoxd4bUJQ+MzUcx6vKByE4U3vsHVpC11Q3J/cVRAw8n15P6jFG+8gWkcOqnAYOJidL/nJ/hBrct+49wD4LciBEZMokAqc1xC+k9XFK4FvolWaWiwTkQwuJaycDs4efIxoJB+Ufjy1gk3zMUa4Rl+Pkd6N7TIIk+U3luiR4idlaZWdPZuTbkrabZ6dZulqR4AR0tCGnDgT7kJqzBXecxvh0FlmVQZym79OlOR7a+O6JgO775gHSisEFfxiToZ2mzzPyM7Hbz+3zBycDeB8JpquxsGZ7RM4kfer87jK7/sObRZJxvNpKxO1iGputzv35ACuSIz9jUbfWrCipuvVRj0fHW0h94ElPQ6XPfnnHoOsMdAniEkEJZgEhRE0m5j1lN6PoCWk4b8+zHQOytH26K4Jg= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: f803fec3-d1f8-4ba9-ab96-08d59b579181 X-MS-Exchange-CrossTenant-originalarrivaltime: 06 Apr 2018 00:44:34.7978 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQBPR0101MB1106 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Apr 2018 00:44:37 -0000 Bruce Evans wrote >On Wed, 4 Apr 2018, Kaya Saman wrote: >> If I recall correctly the "sync" option is default. Though it might be >> different depending on the Linux distro in use? >> >> I use this: vers=3D3,defaults,auto,tcp,rsize=3D8192,wsize=3D8192 >> >> though I could get rid of the tcp as that's also a default. The rsize >> and wsize options are for running Jumbo Frames ie. larger MTU then >> 1500; in my case 9000 for 1Gbps links. > >These rsize and wsize options are pessimizations. They override the >default sizes which are usually much larger for tcp. Yes, for TCP, the FreeBSD client uses the largest size supported by the server, up to 128K (because MAXPHYS is set to that and, as such, that is the largest size safely supported by the buffer cache. I chose to make it this large by default for a couple of reasons: 1 - Solaris used 256K by default (and a maximum of 1Mbyte) back when it was Sun and their engineers were pretty good at this stuff. (I believe they argued that fewer RPCs implied lower server load for a given # of bytes. Usually the NFS engineering types have been concern= ed with server load and, therefore, the server's capacity and not the pe= rformance of a single client doing a single file write.) 2 - I don't do ZFS, but some thought that 128K would be a better I/O read/w= rite size for ZFS. Personally, since all I have for testing is 100Mbits/sec networking, I alwa= ys get "wire speed" and don't see any difference for different rsize/wsize ove= r TCP, so long as it is at least 16K. One case where large rsize/wsize plus a larger readahead setting should get better performance is when the network connection is a "long, fat pipe" such as a high bandwidth WAN connection. (Basically, you need to push a lot of bits down the TCP pipe before you wait for an RPC reply, to try and keep the long, fat pipe filled. In theory NFSv4 was meant for the Internet. Does anyone use it on WAN links. Probably yes, but not typically. I have no idea what Linux uses, except that packet traces often show page size (4K) I/O sizes, but not always. For UDP, I think the FreeBSD default is 16K for NFSv3 (UDP is not allowed f= or NFSv4 since congestion control at the transport level is required by the RF= Cs). Congestion control and reliability is why I always use TCP and, again,= for 100Mbit/sec networking, I see wire speed. Both Linux and Solaris use T= CP by default for NFSv3 mounts, which is mainly why it is the default for F= reeBSD too. =20 >The defaults are not documented in the man page, and the current >settings are almost equally impossible to see (e.g., mount -v doesn't >how them). The defaults are not quite impossible to see in the source >code of course, but the source code for them is especially convoluted. For FreeBSD, "nfsstat -m" on the client shows what is actually being used. (I think Linux has a similar option, but I can't remember for sure?) [lots of good stuff snipped] rick=