From owner-freebsd-fs@freebsd.org Thu Apr 5 00:38:44 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BC68BF83A0F for ; Thu, 5 Apr 2018 00:38:43 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670046.outbound.protection.outlook.com [40.107.67.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT TLS CA 4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 512E17C955 for ; Thu, 5 Apr 2018 00:38:42 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YQBPR0101MB1042.CANPRD01.PROD.OUTLOOK.COM (52.132.66.153) by YQBPR0101MB1889.CANPRD01.PROD.OUTLOOK.COM (52.132.71.22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.653.12; Thu, 5 Apr 2018 00:38:41 +0000 Received: from YQBPR0101MB1042.CANPRD01.PROD.OUTLOOK.COM ([fe80::185:356:49c5:794c]) by YQBPR0101MB1042.CANPRD01.PROD.OUTLOOK.COM ([fe80::185:356:49c5:794c%13]) with mapi id 15.20.0653.012; Thu, 5 Apr 2018 00:38:41 +0000 From: Rick Macklem To: Mike Tancsa , "freebsd-fs@freebsd.org" Subject: Re: Linux NFS client and FreeBSD server strangeness Thread-Topic: Linux NFS client and FreeBSD server strangeness Thread-Index: AQHTzEKdJ7nozqXV6kKGauYOIuptM6PxTtMy Date: Thu, 5 Apr 2018 00:38:41 +0000 Message-ID: References: <369fab06-6213-ba87-cc66-c9829e8a76a0@sentex.net> In-Reply-To: <369fab06-6213-ba87-cc66-c9829e8a76a0@sentex.net> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=rmacklem@uoguelph.ca; x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; YQBPR0101MB1889; 7:qN3CsLeg8KVEP7fBcB7fEEdVC9m8tnEbIS5qzuZLWxsjYDUz+HMNtcDPRCngbyxUDgrPWPm4tTJxTzsZFnfTDQaOompIIXEvBSi1q6qYdoCeX8L/QgoTgfyIbLDZztw3Vk3bs54VaTSPdKWpOEaKmoS/q20nTwJ+Pm0IN2IPbZLnlfKlG+aIfxFq5oA/Y5ch9D+Zxf2Yc62dgMZb0D8YUj0hqgN8W502wWeWc4pv2GK30hethLFAGI/N3f+MyqQN x-ms-exchange-antispam-srfa-diagnostics: SOS; x-ms-office365-filtering-correlation-id: b6974e61-0f6b-4fe5-79c7-08d59a8d94a1 x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(8989080)(5600026)(4604075)(3008032)(4534165)(4627221)(201703031133081)(201702281549075)(8990040)(2017052603328)(7153060)(7193020); SRVR:YQBPR0101MB1889; x-ms-traffictypediagnostic: YQBPR0101MB1889: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(158342451672863); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040522)(2401047)(5005006)(8121501046)(10201501046)(93006095)(93001095)(3231221)(944501327)(52105095)(3002001)(6041310)(201703131423095)(201702281529075)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123560045)(20161123562045)(20161123558120)(20161123564045)(6072148)(201708071742011); SRVR:YQBPR0101MB1889; BCL:0; PCL:0; RULEID:; SRVR:YQBPR0101MB1889; x-forefront-prvs: 06339BAE63 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(376002)(39860400002)(366004)(396003)(39380400002)(346002)(189003)(199004)(3660700001)(99286004)(86362001)(106356001)(305945005)(7696005)(26005)(102836004)(97736004)(476003)(6246003)(2900100001)(786003)(74316002)(76176011)(186003)(316002)(8936002)(68736007)(110136005)(2501003)(74482002)(5250100002)(6436002)(9686003)(8676002)(59450400001)(55016002)(229853002)(53936002)(2906002)(3280700002)(105586002)(33656002)(486006)(5660300001)(81166006)(6506007)(25786009)(81156014)(446003)(478600001)(11346002)(14454004); DIR:OUT; SFP:1101; SCL:1; SRVR:YQBPR0101MB1889; H:YQBPR0101MB1042.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) x-microsoft-antispam-message-info: G47CJ7TE0GzkWZMwIkaczJva3nt7eMuziSUvVLm0u/Ue/YIfZoZ1RNatR6uSFk2bE4+WZVX4NGccyGpv48th+bTEukJKdAKwnI0XuQJELuni5mCwOrCMj5FyY2ierxm/9wyj5+rO3ER4lZsNMCgUQRQuJ2jcB/VzvFaVOmHfMzIY4kSKt7gH/uyEA1XyZkWS8iFfdKS3qq5lgL3x9yTk1lyZ5PzkOVBpHJJ1ymfQ3oo6fIhqMSBDrxCdBP7H41Rr9lrJZX3oY+DJpwDJMehEeQpcQ7fuQHDD0hiBcKJK5qtFjs1N4NbZrwW8vINSt/ebcwaWPyMcepRzxE8eh7XN6G7q7j6pmEfzvz1gHHRJdoEaX4i7m2QX93YOxhyGladu+lL6EqC84U96aifg8tkIVPBPnooFRrLDkRszLpGXvkA= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: b6974e61-0f6b-4fe5-79c7-08d59a8d94a1 X-MS-Exchange-CrossTenant-originalarrivaltime: 05 Apr 2018 00:38:41.6376 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQBPR0101MB1889 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Apr 2018 00:38:44 -0000 Mike Tancsa wrote: >Not sure where the tweaking needs to happen, but I am getting strange >behaviour between a Linux nfs client and FreeBSD RELENG_11 NFS server. > >The FreeBSD server starts with > > >nfs_client_enable=3D"YES" >nfs_server_enable=3D"YES" > > >rpcbind_enable=3D"YES" >rpc_lockd_enable=3D"YES" >rpc_statd_enable=3D"YES" Although it probably isn't related to what you are seeing, I avoid the NSM,= NLM since they are fundamentally flawed protocols. You only need them for NFSv3 clien= ts where the clients must see each others byte range locks. If byte range locks only need to be visible to processes within a client, y= ou can get rid of these and use the "nfslockd" mount option, called "nfslock" on Linux= , I think? >nfs_server_flags=3D"-u -t -n 16" 16 nfsd threads is very low. The default (if you don't specify "-n") is 8 p= er core, which is still very low. Extra ones cause very little overhead (a kernel stack fo= r each one), so I usually use "-n 256" if the server is going to be under any amount of loa= d. Another thing you can try is: # sysctl vfs.nfsd.cachetcp=3D0 which disables use of the DRC for TCP mounts. (Many NFS servers never use t= he DRC for TCP mounts. I designed one to try and make NFS over TCP more fault tole= rant, but it does result in quite a bit of overhead for write loads. If this fixe= s the problem, but you want to use it, it can be tuned with something like: vfs.nfsd.tcpcachetimeo=3D300 (five minutes instead of hours) vfs.nfsd.tcphighwater=3D100 (limit of 100 cached entries) --> The smaller you make these, the lower the overheads and the less effect= ive at making NFS over TCP reliable when TCP reconnects occur it becomes. There are several tunables for NFSv4 (but none of these affect NFSv3): vfs.nfsd.sessionhashsize=3D1000 vfs.nfsd.fhhashsize=3D1000 vfs.nfsd.clienthashsize=3D1000 vfs.nfsd.statehashsize=3D100 (A fairly large system dedicated to serving NFS might make the above "1000"= s "10000"s.) >and on the Linux client I have been trying various options to no avail. >The mount works, but on a straight up write to the FreeBSD server, >everything is very bursty. I noticed this (I think) a few months ago >where Linux dumps across an nfs mount seemed to take a lot longer and >were getting very bursty. > >It seems if there are a mixture of reads and writes, everything is >pretty fast. But if a client is just writing to the server, something, >somewhere is blocking. Doing something simple like >ls -l /nfsmount >from the client "wakes" up the server/client so that write stream can >keep going. Otherwise, it will do a big blast of writes and then several >seconds of pausing on the dump. This sounds like a network device driver issue to me. The main difference b= etween a FreeBSD client and a Linux client that I am aware of is that the Linux cl= ient likes to do page size (4K) writes, so it generates lots of them. One example might be interrupt moderation. It's a wonderful thing for some = TCP loads, but can be a terrible thing for NFS loads. Basically anything that adds del= ay to interrupt delivery/processing will increase latency and that kills NFS performance, f= rom what I've seen. Someone else suggested disabling TSO, which is often broken in the net devi= ce drivers. If you have a different type of net interface that uses a different driver,= you might try that and see if it has the same problem. I might look at your packet trace someday, but I haven't yet. Good luck with it, rick [stuff snipped]=