From owner-freebsd-fs@FreeBSD.ORG Mon Dec 10 10:35:57 2007 Return-Path: Delivered-To: fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 20A9416A417 for ; Mon, 10 Dec 2007 10:35:57 +0000 (UTC) (envelope-from Artis.Caune@sigmanet.lv) Received: from dzilna.latnet.lv (dzilna.latnet.lv [159.148.19.116]) by mx1.freebsd.org (Postfix) with ESMTP id D272013C44B for ; Mon, 10 Dec 2007 10:35:56 +0000 (UTC) (envelope-from Artis.Caune@sigmanet.lv) Received: from localhost (localhost.localdomain [127.0.0.1]) by dzilna.latnet.lv (Postfix) with ESMTP id DE842D3E8F for ; Mon, 10 Dec 2007 12:16:03 +0200 (EET) X-Virus-Scanned: Debian amavisd-new at dzilna.latnet.lv Received: from dzilna.latnet.lv ([127.0.0.1]) by localhost (dzilna.latnet.lv [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kC9wTxEQmAzq for ; Mon, 10 Dec 2007 12:16:01 +0200 (EET) Received: from [85.254.248.22] (DiDii.sigmanet.lv [85.254.248.22]) by dzilna.latnet.lv (Postfix) with ESMTP id 46265D3EAB for ; Mon, 10 Dec 2007 12:16:01 +0200 (EET) Message-ID: <475D11E0.1050701@sigmanet.lv> Date: Mon, 10 Dec 2007 12:16:00 +0200 From: Artis Caune User-Agent: Thunderbird 1.5.0.10 (X11/20070327) MIME-Version: 1.0 To: fs@FreeBSD.org Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: Subject: +12K ZFS file systems or snapshots X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Dec 2007 10:35:57 -0000 Hello FreeBSD-fs, We want to make ZFS file system for each user: # zfs create tank/user_x # zfs set quota=1G tank/user_x We also want to create snapshots for every user: user_x@month_ago user_x@week_ago user_x@day_ago We have run bonnie tests on amd64, 6G RAM IBM box with 3 SAS drives in raidz1 (7.0-BETA3). Tests failed with "kmem_map too small". Raising kmem_size to 1500M and limiting arc_max to 512M passed bonnie test for 4 days. 'vmstat -m' solaris MemUse is ~600M - 700M We could not create more than 12K file systems with 'zfs create'. ZFS panics with "kmem_map too small" when 'vmstat -m' solaris MemUse reaches 1500M (kmem_size). Solaris recommend 1G for every 10K file systems or snapshots, each fs or snapshot requires 64K of kmem. It looks like kmem_size is u_int and is limited to 4G, but we can boot only with 1500M. With 2G kmem_size console is full with panics. We want +/- 10K users on each box. We can: * create 10K file systems (users), use tank snapshots, not per-user snapshots * create 3K file systems (users) and 3 snapshots per file system +room for zfs cache If kmem_size is limited to ~2G, then there is no need for more than 2G RAM for ZFS file server?