From owner-freebsd-performance@FreeBSD.ORG Thu Dec 20 17:48:46 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4ED2516A417 for ; Thu, 20 Dec 2007 17:48:46 +0000 (UTC) (envelope-from biancalana@gmail.com) Received: from py-out-1112.google.com (py-out-1112.google.com [64.233.166.176]) by mx1.freebsd.org (Postfix) with ESMTP id F2DD013C4CE for ; Thu, 20 Dec 2007 17:48:45 +0000 (UTC) (envelope-from biancalana@gmail.com) Received: by py-out-1112.google.com with SMTP id u77so336477pyb.3 for ; Thu, 20 Dec 2007 09:48:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=tU9WOew7shjfoTmDjjN1u3G5n5K9vUjKoDJ2uARisng=; b=yBLOZZr2gdWg1abp4OoAiilMDzPDANmeYAWm/rB3WoCRQ4P2e+nwREzdbFFpksMWU4zQiluHhkgBir0P7i8GtrPCnNxd34EQohUhDhIQFe2+oZScRGxyX1VHMpy5Gw/fqoiLKruUYIIfd7fT/ITVpAR4RvNdX38s+Pu/AFtBssk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=d4giXjA7hy/QHfRLJJsS1+bEEi8i63Wty6dn4csNicKcSshgB2441ivQkG96LwOVbyPJJ+rxBrImBZGysVoMyPvckG5ial5gQnIbdPuLP+1lIw+uA17wq6NXsLeRgZsnb8vmHq7oyYMf8OuwbvyQy1IaoJmMn8iuHe8MBkMHtA4= Received: by 10.65.253.6 with SMTP id f6mr492189qbs.90.1198172921396; Thu, 20 Dec 2007 09:48:41 -0800 (PST) Received: by 10.64.184.9 with HTTP; Thu, 20 Dec 2007 09:48:41 -0800 (PST) Message-ID: <8e10486b0712200948l2bc41063h54d5579373665fc6@mail.gmail.com> Date: Thu, 20 Dec 2007 15:48:41 -0200 From: "Alexandre Biancalana" To: dima <_pppp@mail.ru> In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <8e10486b0712200910v32ecbfa5h72fb0dc2b82a1009@mail.gmail.com> Cc: freebsd-performance@freebsd.org, =?KOI8-R?Q?Dennis_Kj=E6r_Jensen?= Subject: Re: Re[2]: Bad performance when accessing a lot of small files X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Dec 2007 17:48:46 -0000 On 12/20/07, dima <_pppp@mail.ru> wrote: > >> Increase the dirhash maxmem > >> I have 'sysctl -w vfs.ufs.dirhash_maxmem=268435456' (256Mb)in my > >> rc.local on a (maildir) fileserver here > >> > > This is a zfs filesystem, by the way I raise this limit as suggested. > > > > I let you know > > Can you provide > sysctl vfs.zfs output then? > > ZFS can be tuned by increasing kern.maxvnodes sysctl (don't forget to increase vm.kmem_size and vm.kmem_size_max accordingly). Do it until vfs.numvnodes would stabilize below the kern.maxvnodes value during peak load patterns. $ sysctl -A | grep zfs vfs.zfs.arc_min: 49152000 vfs.zfs.arc_max: 1073741824 vfs.zfs.mdcomp_disable: 0 vfs.zfs.prefetch_disable: 1 vfs.zfs.zio.taskq_threads: 0 vfs.zfs.recover: 0 vfs.zfs.vdev.cache.size: 10485760 vfs.zfs.vdev.cache.max: 16384 vfs.zfs.cache_flush_disable: 0 vfs.zfs.zil_disable: 0 vfs.zfs.debug: 0 kstat.zfs.misc.arcstats.hits: 59096254 kstat.zfs.misc.arcstats.misses: 21362360 kstat.zfs.misc.arcstats.demand_data_hits: 21209779 kstat.zfs.misc.arcstats.demand_data_misses: 17688110 kstat.zfs.misc.arcstats.demand_metadata_hits: 37886475 kstat.zfs.misc.arcstats.demand_metadata_misses: 3674250 kstat.zfs.misc.arcstats.prefetch_data_hits: 0 kstat.zfs.misc.arcstats.prefetch_data_misses: 0 kstat.zfs.misc.arcstats.prefetch_metadata_hits: 0 kstat.zfs.misc.arcstats.prefetch_metadata_misses: 0 kstat.zfs.misc.arcstats.mru_hits: 18311543 kstat.zfs.misc.arcstats.mru_ghost_hits: 16169 kstat.zfs.misc.arcstats.mfu_hits: 40784711 kstat.zfs.misc.arcstats.mfu_ghost_hits: 621909 kstat.zfs.misc.arcstats.deleted: 25885007 kstat.zfs.misc.arcstats.recycle_miss: 9413044 kstat.zfs.misc.arcstats.mutex_miss: 9253 kstat.zfs.misc.arcstats.evict_skip: 4418321 kstat.zfs.misc.arcstats.hash_elements: 232462 kstat.zfs.misc.arcstats.hash_elements_max: 489237 kstat.zfs.misc.arcstats.hash_collisions: 12684452 kstat.zfs.misc.arcstats.hash_chains: 56900 kstat.zfs.misc.arcstats.hash_chain_max: 20 kstat.zfs.misc.arcstats.p: 1023016585 kstat.zfs.misc.arcstats.c: 1040187392 kstat.zfs.misc.arcstats.c_min: 49152000 kstat.zfs.misc.arcstats.c_max: 1073741824 kstat.zfs.misc.arcstats.size: 1040184320 $ cat /etc/sysctl.conf kern.maxvnodes=400000 vfs.ufs.dirhash_maxmem=268435456 $ cat /boot/loader.conf kern.maxdsiz="2G" # Set the max data size to 4GB kern.maxssiz="1G" # Set the max stack size 2GB vfs.zfs.prefetch_disable="1" vfs.zfs.arc_max="1G" vm.kmem_size_max="1500M" vm.kmem_size="1500M" kern.ipc.nmbclusters="32768"