From owner-freebsd-fs@freebsd.org Sat Dec 31 13:34:01 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B3902C998D4 for ; Sat, 31 Dec 2016 13:34:01 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 357341E43; Sat, 31 Dec 2016 13:34:01 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id uBVDXpWD093460 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Sat, 31 Dec 2016 15:33:52 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua uBVDXpWD093460 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id uBVDXoNd093458; Sat, 31 Dec 2016 15:33:50 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 31 Dec 2016 15:33:50 +0200 From: Konstantin Belousov To: Josh Paetzel Cc: freebsd-fs@freebsd.org, mckusick@freebsd.org Subject: Re: NFS readdirplus on ZFS with > 1 billion files Message-ID: <20161231133350.GU1923@kib.kiev.ua> References: <1483179971.3381747.833629401.5EF242B8@webmail.messagingengine.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1483179971.3381747.833629401.5EF242B8@webmail.messagingengine.com> User-Agent: Mutt/1.7.2 (2016-11-26) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Dec 2016 13:34:01 -0000 On Sat, Dec 31, 2016 at 04:26:11AM -0600, Josh Paetzel wrote: > We've been chasing this bug for a very long time and finally managed to > pin it down. When a ZFS dataset has more than 1 billion files on it and > an NFS client does a readdirplus the file handles for files with high > znode/inode numbers gets truncated due to a 64 -> 32 bit conversion. > > https://reviews.freebsd.org/D9009 > > This isn't a fix so much as a workaround. From a performance standpoint > it's the same as if the client mounts with noreaddirplus; sometimes it's > a win, sometimes it's a lose. CPU usage does go up on the server a bit. > Can you point to the places in ZFS code where the truncation occur ? I have no idea about ZFS code, and my question is mainly is the truncation just occurs due to different types of ino_t and zfs node id, or some code actively does the range reduction. My question is in the context of the long-dragging ino64 work, which might be finished in some visible future. In particular, I am curious if just using the patched kernel fixes your issue. See https://github.com/FreeBSDFoundation/freebsd/tree/ino64 although I do not make any claim about the state of the code yet. Your patch, after a review, might be still useful for stable/10 and 11, since I do not think that ino64 has any bits which could be merged.