From owner-freebsd-fs@freebsd.org Wed Dec 2 14:00:21 2015 Return-Path: <owner-freebsd-fs@freebsd.org> Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B9B05A3EE72 for <freebsd-fs@mailman.ysv.freebsd.org>; Wed, 2 Dec 2015 14:00:21 +0000 (UTC) (envelope-from kraduk@gmail.com) Received: from mail-wm0-x233.google.com (mail-wm0-x233.google.com [IPv6:2a00:1450:400c:c09::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46FCC13BC for <freebsd-fs@freebsd.org>; Wed, 2 Dec 2015 14:00:21 +0000 (UTC) (envelope-from kraduk@gmail.com) Received: by wmww144 with SMTP id w144so216023294wmw.1 for <freebsd-fs@freebsd.org>; Wed, 02 Dec 2015 06:00:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=Bdth3bVacNKb0WP78Ih/WiTJpcGQY1jzplwiEERk/IQ=; b=ei++WbwFT3htKBEKN2ZDg0MbRrzseNwjfmwF8l7kUWkcnispdxSYv+LM6BRJFb8TWH vYM5b/pXIPv8zsofZbZlinJJQKbcEHIhUIthdD0s8/H1Whxl8EtT2fr5TlAm3iJOUmZx 0iSLPqskqCR4qkIFuZ6X56t0gf4Y0kJ5Q60WEKhua7qvz19lcJu3ZcB+PztRJihkAgPT KBazslylWIosKePeMRIFTYwaoFVzo7EVJfwo8/ehL5SubYWcYSIwB2UbUkXZi5oACjCq tBLEVt/Iwel/NJ2n6PGJccRXYw9TzxfR5gF2ZvlulCeu/Y1UUg8Ro4aEOGEL623aKDtz CqRQ== MIME-Version: 1.0 X-Received: by 10.194.204.202 with SMTP id la10mr5455004wjc.81.1449064819665; Wed, 02 Dec 2015 06:00:19 -0800 (PST) Received: by 10.28.181.213 with HTTP; Wed, 2 Dec 2015 06:00:19 -0800 (PST) In-Reply-To: <CAFHbX1+9JiGeukjePqzy1vTS1-ON1vY27v2iB5daX1XGjpYNDA@mail.gmail.com> References: <20151202133428.35820@smtp.new-ukraine.org> <CAFHbX1+9JiGeukjePqzy1vTS1-ON1vY27v2iB5daX1XGjpYNDA@mail.gmail.com> Date: Wed, 2 Dec 2015 14:00:19 +0000 Message-ID: <CALfReycBOY3i3HRLqNVTChC=fL-=KGJT4eXBhN_=R8uGXQtSJA@mail.gmail.com> Subject: Re: advice needed: zpool of 10 x (raidz2 on (4+2) x 2T HDD) From: krad <kraduk@gmail.com> To: Tom Evans <tevans.uk@googlemail.com> Cc: Zeus Panchenko <zeus@ibs.dn.ua>, FreeBSD Filesystems <freebsd-fs@freebsd.org> Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems <freebsd-fs.freebsd.org> List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-fs>, <mailto:freebsd-fs-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-fs/> List-Post: <mailto:freebsd-fs@freebsd.org> List-Help: <mailto:freebsd-fs-request@freebsd.org?subject=help> List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-fs>, <mailto:freebsd-fs-request@freebsd.org?subject=subscribe> X-List-Received-Date: Wed, 02 Dec 2015 14:00:21 -0000 If this is the case, put more ram in and a good amount of SSD for l2arc. SSD is fairly cheap, so put in a TB of the best you can afford. RAM is always best though if you can afford it and fit it onto the board. It's probably worth going to 10.2 as well, and maybe even stable as there have been a lot of zfs related patches since 10.1. On 2 December 2015 at 13:13, Tom Evans via freebsd-fs < freebsd-fs@freebsd.org> wrote: > On Wed, Dec 2, 2015 at 11:34 AM, Zeus Panchenko <zeus@ibs.dn.ua> wrote: > > -----BEGIN PGP SIGNED MESSAGE----- > > Hash: SHA1 > > > > greetings, > > > > we deployed storage, and as it was filling until now, I see I need > > an advice regarding the configuration and optimization/s ... > > > > the main cause I decided to ask for an advice is this: > > > > once per month (or even more frequently, depends on the load I > > suggest) host hangs and only power reset helps, nothing helpful in log > > files though ... just the fact of restart logged and usual ctld activity > > <snip> > > CPU: Intel(R) Xeon(R) CPU E5-2630L (2 package(s) x 6 core(s) x 2 SMT > threads) > > RAM: 128Gb > > STOR: 3 x LSI SAS2308 PCI-Express Fusion-MPT SAS-2 (jbod) > > 60 x HDD 2T (ATA WDC WD20EFRX-68A 0A80, Fixed Direct Access > SCSI-6 device 600.000MB/s) > > > > OS: FreeBSD 10.1-RELEASE #0 r274401 amd64 > > > > to avoid OS memory shortage sysctl vfs.zfs.arc_max is set to 120275861504 > > > > to clients, storage is provided via iSCSI by ctld (each target is file > backed) > > > > zpool created of 10 x raidz2, each raidz2 consists of 6 geli devices and > > now looks so (yes, deduplication is on): > > > >> zpool list storage > > NAME SIZE ALLOC FREE FRAG EXPANDSZ CAP DEDUP > HEALTH ALTROOT > > storage 109T 33.5T 75.2T - - 30% 1.57x > ONLINE - > > You will need to examine zdb output to correctly determine the size of > your dedupe table. Assuming an average block size of 64kb, your DDT > will be approximately 167GB, ie well outside your RAM (my maths may be > off of course - ((33.5*(2**40)/(64*1024))*320)/(2**30)). > > This article explains in detail: > > http://constantin.glez.de/blog/2011/07/zfs-dedupe-or-not-dedupe > > Cheers > > Tom > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >