From owner-freebsd-fs@freebsd.org Wed Dec 2 13:13:16 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4DE24A3E3A0 for ; Wed, 2 Dec 2015 13:13:16 +0000 (UTC) (envelope-from tevans.uk@googlemail.com) Received: from mail-lf0-x22e.google.com (mail-lf0-x22e.google.com [IPv6:2a00:1450:4010:c07::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EDDBB1ED0 for ; Wed, 2 Dec 2015 13:13:15 +0000 (UTC) (envelope-from tevans.uk@googlemail.com) Received: by lfaz4 with SMTP id z4so49871604lfa.0 for ; Wed, 02 Dec 2015 05:13:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=iMs1HLx3R7R2J1V+XqDFp6rVyU/30o4fD+F+DkcAj+Q=; b=q5cZd6IVmms9/dAX01maomA4e67oqKy32qQ+kkZzi69VblG5+XCXtK0IBKMK6S51hW uKGCyxaqLmkeZaFer8pwUK9o0woTGca76GbSlyHWZS6PizInCQvyyn1ZobbNkt5Zhqcx nDCguvmiVZyINhnf8So587sDrs3BDHt0w3viAw1jlkFOF2vs0J7LBz8TlPT5HmWul95k dc52/e1OEgNQ/Rd8DaSjLyE3rzVjd47vUOwE3GDXQ0BbUtfSCEDdC5x/q33MVVEOcpGR uCQ6h5YdkpvBFQiOKoYlemaD0Y1UJP6Hme52botH7W3Is+EjYhhElYWYcELJTVxtMnRz NqrA== MIME-Version: 1.0 X-Received: by 10.112.173.134 with SMTP id bk6mr2807342lbc.34.1449061993880; Wed, 02 Dec 2015 05:13:13 -0800 (PST) Received: by 10.25.84.134 with HTTP; Wed, 2 Dec 2015 05:13:13 -0800 (PST) In-Reply-To: <20151202133428.35820@smtp.new-ukraine.org> References: <20151202133428.35820@smtp.new-ukraine.org> Date: Wed, 2 Dec 2015 13:13:13 +0000 Message-ID: Subject: Re: advice needed: zpool of 10 x (raidz2 on (4+2) x 2T HDD) From: Tom Evans To: Zeus Panchenko Cc: FreeBSD Filesystems Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Dec 2015 13:13:16 -0000 On Wed, Dec 2, 2015 at 11:34 AM, Zeus Panchenko wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > greetings, > > we deployed storage, and as it was filling until now, I see I need > an advice regarding the configuration and optimization/s ... > > the main cause I decided to ask for an advice is this: > > once per month (or even more frequently, depends on the load I > suggest) host hangs and only power reset helps, nothing helpful in log > files though ... just the fact of restart logged and usual ctld activity > > CPU: Intel(R) Xeon(R) CPU E5-2630L (2 package(s) x 6 core(s) x 2 SMT threads) > RAM: 128Gb > STOR: 3 x LSI SAS2308 PCI-Express Fusion-MPT SAS-2 (jbod) > 60 x HDD 2T (ATA WDC WD20EFRX-68A 0A80, Fixed Direct Access SCSI-6 device 600.000MB/s) > > OS: FreeBSD 10.1-RELEASE #0 r274401 amd64 > > to avoid OS memory shortage sysctl vfs.zfs.arc_max is set to 120275861504 > > to clients, storage is provided via iSCSI by ctld (each target is file backed) > > zpool created of 10 x raidz2, each raidz2 consists of 6 geli devices and > now looks so (yes, deduplication is on): > >> zpool list storage > NAME SIZE ALLOC FREE FRAG EXPANDSZ CAP DEDUP HEALTH ALTROOT > storage 109T 33.5T 75.2T - - 30% 1.57x ONLINE - You will need to examine zdb output to correctly determine the size of your dedupe table. Assuming an average block size of 64kb, your DDT will be approximately 167GB, ie well outside your RAM (my maths may be off of course - ((33.5*(2**40)/(64*1024))*320)/(2**30)). This article explains in detail: http://constantin.glez.de/blog/2011/07/zfs-dedupe-or-not-dedupe Cheers Tom