From owner-freebsd-fs@freebsd.org Thu Dec 27 10:24:23 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C9AAD13578B0 for ; Thu, 27 Dec 2018 10:24:23 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (smtp.digiware.nl [IPv6:2001:4cb8:90:ffff::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 550186CA28 for ; Thu, 27 Dec 2018 10:24:23 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from router.digiware.nl (localhost.digiware.nl [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id 9BD54B8EC9; Thu, 27 Dec 2018 11:24:19 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.com Received: from smtp.digiware.nl ([127.0.0.1]) by router.digiware.nl (router.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Vwf7iwmbjymr; Thu, 27 Dec 2018 11:24:18 +0100 (CET) Received: from [192.168.101.70] (unknown [192.168.101.70]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.digiware.nl (Postfix) with ESMTPSA id C8EA4B8EC3; Thu, 27 Dec 2018 11:24:18 +0100 (CET) Subject: Re: Suggestion for hardware for ZFS fileserver To: Peter Eriksson , freebsd-fs@freebsd.org References: <4f816be7-79e0-cacb-9502-5fbbe343cfc9@denninger.net> <3160F105-85C1-4CB4-AAD5-D16CF5D6143D@ifm.liu.se> From: Willem Jan Withagen Message-ID: <362ced1c-f39e-cc6d-cf50-aab878c6cd49@digiware.nl> Date: Thu, 27 Dec 2018 11:24:20 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.3.3 MIME-Version: 1.0 In-Reply-To: <3160F105-85C1-4CB4-AAD5-D16CF5D6143D@ifm.liu.se> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Dec 2018 10:24:24 -0000 On 21/12/2018 19:53, Peter Eriksson wrote: Thanx, for the introduction to a serious platform running ZFS. Always interesting to read what other do with their sstuff. :-D But this can be "easily" fixed for the moment. > First thing we noticed was that booting would take forever… Mounting the 20-100k filesystems _and_ enabling them to be shared via NFS is not done efficient at all (for each filesystem it re-reads /etc/zfs/exports (a couple of times) befor appending one line to the end. Repeat 20-100,000 times… Not to mention the big kernel lock for NFS “hold all NFS activity while we flush and reinstalls all sharing information per filesystem” being done by mountd… > > Wish list item #1: A BerkeleyDB-based ’sharetab’ that replaces the horribly slow /etc/zfs/exports text file. > Wish list item #2: A reimplementation of mountd and the kernel interface to allow a “diff” between the contents of the DB-based sharetab above be input into the kernel instead of the brute-force way it’s done now.. > > (I’ve written some code that implements item #1 above and it helps quite a bit. Nothing near production quality yet though. I have looked at item #2 a bit too but not done anything about it.) It would be possible to not user /etc/zfs/exports. The syntax that can be inserted in that file through zfs itself is not quite flexible enough. I remember patching it, and then not getting this accepted, because it was too FreeBSD specific. So I started using /etc/exports. But cram all in /etc/exports, and only load and export files once zfs has loaded all its filesystems. Only speeds up so much, but keeps a few bottlenecks out of the equation. --WjW