From owner-freebsd-current@freebsd.org Thu Nov 24 12:53:47 2016 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 81AE3C513F3 for ; Thu, 24 Nov 2016 12:53:47 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from na01-by2-obe.outbound.protection.outlook.com (mail-by2on0073.outbound.protection.outlook.com [207.46.100.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2A675CE3; Thu, 24 Nov 2016 12:53:46 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM (10.165.218.133) by YTXPR01MB0192.CANPRD01.PROD.OUTLOOK.COM (10.165.218.136) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.734.8; Thu, 24 Nov 2016 12:53:44 +0000 Received: from YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM ([10.165.218.133]) by YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM ([10.165.218.133]) with mapi id 15.01.0734.014; Thu, 24 Nov 2016 12:53:43 +0000 From: Rick Macklem To: Konstantin Belousov , Alan Somers CC: FreeBSD CURRENT Subject: Re: NFSv4 performance degradation with 12.0-CURRENT client Thread-Topic: NFSv4 performance degradation with 12.0-CURRENT client Thread-Index: AQHSRhIPtzL8Jj0C/0q6PJtKKLQgMaDn2GqAgAA8DvY= Date: Thu, 24 Nov 2016 12:53:43 +0000 Message-ID: References: , <20161124090811.GO54029@kib.kiev.ua> In-Reply-To: <20161124090811.GO54029@kib.kiev.ua> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=rmacklem@uoguelph.ca; x-microsoft-exchange-diagnostics: 1; YTXPR01MB0192; 7:i+lVTn/RVieqIEIgczmSZnXcIVREpcZcRvUruZlgTTOOm7uOxJSvs9nBMYZvGo83KRD9pXMUkoSAVsz2iYqvVt0FBxvAvSxeST3QkNm4OGB9kHa/0poFxLa4Ul/I3MpugpiB+gkytBXCEgdgce5IsdC9b6ocUAYxs/1fj+sXmryPkL8nbQ8LglS0OjYdlYrA71JDnILxsbEFY6Xj5vavo6lzhzxbhI2PsmNoBBifPYG/iSHKpydWBa1HCMmWC99ROe2nclfJ5FdAsoxrfMz5/PNBGkzkblbnhi5GROhZBZCjz70qoD6B/wj8NGNpZV83bmu3P4ZPv745zRytIbow+PrhHqhuEcBGGhGK/S1CryM= x-ms-office365-filtering-correlation-id: a209d3d1-cc75-48c3-59be-08d41468ec33 x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001);SRVR:YTXPR01MB0192; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(158342451672863); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6060326)(6045199)(6040361)(2401047)(5005006)(8121501046)(3002001)(10201501046)(6061324)(6041248)(2016111802025)(20161123555025)(20161123564025)(20161123560025)(20161123562025)(6043046); SRVR:YTXPR01MB0192; BCL:0; PCL:0; RULEID:; SRVR:YTXPR01MB0192; x-forefront-prvs: 0136C1DDA4 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(7916002)(24454002)(199003)(189002)(39380400001)(39410400001)(8676002)(39060400001)(5001770100001)(189998001)(8936002)(81166006)(86362001)(97736004)(38730400001)(74482002)(6506003)(229853002)(3660700001)(305945005)(4326007)(2906002)(74316002)(7846002)(68736007)(5890100001)(9686002)(81156014)(102836003)(92566002)(76176999)(54356999)(50986999)(77096005)(3280700002)(101416001)(2900100001)(33656002)(122556002)(105586002)(106356001)(106116001)(5660300001)(7696004)(2950100002)(39400400001)(21314002); DIR:OUT; SFP:1101; SCL:1; SRVR:YTXPR01MB0192; H:YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-originalarrivaltime: 24 Nov 2016 12:53:43.5533 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTXPR01MB0192 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Nov 2016 12:53:47 -0000 On Wed, Nov 23, 2016 at 10:17:25PM -0700, Alan Somers wrote: > I have a FreeBSD 10.3-RELEASE-p12 server exporting its home > directories over both NFSv3 and NFSv4. I have a TrueOS client (based > on 12.0-CURRENT on the drm-next-4.7 branch, built on 28-October) > mounting the home directories over NFSv4. At first, everything is > fine and performance is good. But if the client does a buildworld > using sources on NFS and locally stored objects, performance slowly > degrades. The degradation is most noticeable with metadata-heavy > operations. For example, "ls -l" in a directory with 153 files takes > less than 0.1 seconds right after booting. But the longer the > buildworld goes on, the slower it gets. Eventually that same "ls -l" > takes 19 seconds. When the home directories are mounted over NFSv3 > instead, I see no degradation. > > top shows negligible CPU consumption on the server, and very high > consumption on the client when using NFSv4 (nearly 100%). The > NFS-using process is spending almost all of its time in system mode, > and dtrace shows that almost all of its time is spent in > ncl_getpages(). > A couple of things you could do when it slow (as well as what Kostik sugges= ted): - nfsstat -c -e on client and nfsstat -e -s on server, to see what RPCs are= being done and how quickly. (nfsstat -s -e will also show you how big the DRC is, al= though a large DRC should show up as increased CPU consumption on the server) - capture packets with tcpdump -s 0 -w test.pcap host - then you can email me test.pcap as an attachment. I can look at it in w= ireshark and see if there seem to protocol and/or TCP issues. (You can look at i= n wireshark yourself, the look for NFS4ERR_xxx, TCP segment retransmits...) If you are using either "intr" or "soft" on the mounts, try without those m= ount options. (The Bugs section of mount_nfs recommends against using them. If an RPC fai= ls due to these options, something called a seqid# can be "out of sync" between clie= nt/server and that causes serious problems.) --> These seqid#s are not used by NFSv4.1, so you could try that by adding "minorversion=3D1" to your mount options. Good luck with it, rick