From owner-freebsd-current@freebsd.org Thu Nov 24 09:08:16 2016 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CCE04C53A7F for ; Thu, 24 Nov 2016 09:08:16 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 491CA9A0; Thu, 24 Nov 2016 09:08:16 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id uAO98BwJ069627 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Thu, 24 Nov 2016 11:08:11 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua uAO98BwJ069627 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id uAO98B6a069626; Thu, 24 Nov 2016 11:08:11 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 24 Nov 2016 11:08:11 +0200 From: Konstantin Belousov To: Alan Somers Cc: FreeBSD CURRENT Subject: Re: NFSv4 performance degradation with 12.0-CURRENT client Message-ID: <20161124090811.GO54029@kib.kiev.ua> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.7.1 (2016-10-04) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Nov 2016 09:08:16 -0000 On Wed, Nov 23, 2016 at 10:17:25PM -0700, Alan Somers wrote: > I have a FreeBSD 10.3-RELEASE-p12 server exporting its home > directories over both NFSv3 and NFSv4. I have a TrueOS client (based > on 12.0-CURRENT on the drm-next-4.7 branch, built on 28-October) > mounting the home directories over NFSv4. At first, everything is > fine and performance is good. But if the client does a buildworld > using sources on NFS and locally stored objects, performance slowly > degrades. The degradation is most noticeable with metadata-heavy > operations. For example, "ls -l" in a directory with 153 files takes > less than 0.1 seconds right after booting. But the longer the > buildworld goes on, the slower it gets. Eventually that same "ls -l" > takes 19 seconds. When the home directories are mounted over NFSv3 > instead, I see no degradation. > > top shows negligible CPU consumption on the server, and very high > consumption on the client when using NFSv4 (nearly 100%). The > NFS-using process is spending almost all of its time in system mode, > and dtrace shows that almost all of its time is spent in > ncl_getpages(). > > I have delegations disabled on the server. On the client, the home > directories are nullfs mounted to two additional locations, and the > buildworld was actually using one of those nullfs mounts, not the NFS > mount directly. > > Any ideas? Try stock FreeBSD first. If reproducable on the stock HEAD, can you point to the lines of ncl_getpages() where the time is spent ? Does reading of the problematic files, as opposed to mmaping it, also cause the behaviour ? E.g. try dd. There is really no time-unbounded loops in the ncl_getpages() itself. I could understand the situation if e.g. time is spent in getpbuf() or ncl_readrpc(), but not in ncl_getpages() directly. Also, as an experiment, you could see if HEAD after r308980 demonstrates any difference.