From owner-freebsd-fs@freebsd.org Fri Jan 29 21:28:30 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 60592A73231 for ; Fri, 29 Jan 2016 21:28:30 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wm0-x232.google.com (mail-wm0-x232.google.com [IPv6:2a00:1450:400c:c09::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id F035C1E83 for ; Fri, 29 Jan 2016 21:28:29 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wm0-x232.google.com with SMTP id l66so71365492wml.0 for ; Fri, 29 Jan 2016 13:28:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-type:content-transfer-encoding; bh=vdZ+NOWx1J97BASTPyFdgV0ZtX3VEp1fv+wW2NzIpyc=; b=bocQzV89pt6Tcfzp7kEUH6UJFonizlpIeWEG27Dis2ac0PqdyIHB9KKLmznfxpi2pT 2/NZQAEqDi9pBPvEXcm4Q9tqHZ0tdOWqyGbIx7LrLwYwor1m2SNv2NAX3fnHtOxuDEwm yG9fm/y9RoyH/35repuBe2SfVm4CmL9SjIc7QwN/zaME1jQcNli/FbADK2w5SNULLpcj KUCGE0XRAv2/CZHATIb3rSNGbZXMcjnoKCAvV39dsGkKRNcKKYh1Ek2JtBKfh2fEI6Ew 2brVw2bPP+VKYayXfPf386wDRlRki5I0OiJP+TmJ6QVJD9iXrvO7544Gs20guG8EPYEt nTeQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-type :content-transfer-encoding; bh=vdZ+NOWx1J97BASTPyFdgV0ZtX3VEp1fv+wW2NzIpyc=; b=N+rw9cg8MTvPgGzTQFsAJ+MrWscQ6eiU+VaTEHtbGck8SYnPRyGdylJlhW+tJLShr+ TMPg5NOjnxwo9muuj3GrzpPrZu30DpRpNOrOagLPTcGjCVFWnoFvTRfkZBvAGfDdccb2 Kl2LLttysE9EF2xoGgOIhB42fB9PJ6L9OQ9iwJ6X1wxYbVpsdUrAHfoRTiOnLYOx+t7V 1cqRnIZnKQf9feJakfafn6+DDpsZa6b6k6+qAWcrOK/LqL86OcaRh9wRd2SAuBy5ZAt/ kwI1xg8FByaS1kaO1KxfWgp516h61EhQ3fYSDvjLOpc8R4BOiLF4FXoAQsgnbdtr1RZi adYQ== X-Gm-Message-State: AG10YOQWBAg8VyaiVLZH5UD3zOO1tZzbgc6xT4xblDXePEml+V/Q29TOi/adQflwbvB6o3cW X-Received: by 10.28.89.69 with SMTP id n66mr11741307wmb.63.1454102908394; Fri, 29 Jan 2016 13:28:28 -0800 (PST) Received: from [10.10.1.58] (liv3d.labs.multiplay.co.uk. [82.69.141.171]) by smtp.gmail.com with ESMTPSA id t76sm9200632wmd.13.2016.01.29.13.28.26 for (version=TLSv1/SSLv3 cipher=OTHER); Fri, 29 Jan 2016 13:28:27 -0800 (PST) Subject: Re: quantifying zpool performance with number of vdevs To: freebsd-fs@freebsd.org References: <56ABAA18.90102@physics.umn.edu> From: Steven Hartland Message-ID: <56ABD98B.3070808@multiplay.co.uk> Date: Fri, 29 Jan 2016 21:28:43 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.5.1 MIME-Version: 1.0 In-Reply-To: <56ABAA18.90102@physics.umn.edu> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Jan 2016 21:28:30 -0000 Always a good read is: http://blog.delphix.com/matt/2014/06/06/zfs-stripe-width/ On 29/01/2016 18:06, Graham Allan wrote: > In many of the storage systems I built to date I was slightly > conservative (?) in wanting to keep any one pool confined to a single > JBOD chassis. In doing this I've generally been using the Supermicro > 45-drive chassis with pools made of 4x (8+2) raidz2, other slots being > kept for spares, ZIL and L2ARC. > > Now I have several servers with 3-4 such chassis, and reliability has > also been such that I'd feel more comfortable about spanning chassis, > if there was worthwhile performance benefit. > > Obviously theory says that iops should scale with number of vdevs but > it would be nice to try and quantify. > > Getting relevant data out of iperf seems problematic on machines with > 128GB+ RAM - it's hard to blow out the ARC. > > It does seem like I get possibly more valid-looking results if I set > "zfs set primarycache=metadata" on my test dataset - it seems like > this should mostly disable the ARC (seems to be borne out by arcstat > output, though there could still be L2ARC effects). > > Wonder if anyone has any thoughts on this, and also on benefits/risks > of moving from 40-drive to 80- or 120-drive pools. > > Graham