From owner-freebsd-fs@freebsd.org Fri Jan 29 18:27:58 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CE9BAA72156 for ; Fri, 29 Jan 2016 18:27:58 +0000 (UTC) (envelope-from allan@physics.umn.edu) Received: from mail.physics.umn.edu (smtp.spa.umn.edu [128.101.220.4]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B0B2A1994 for ; Fri, 29 Jan 2016 18:27:58 +0000 (UTC) (envelope-from allan@physics.umn.edu) Received: from peevish.spa.umn.edu ([128.101.220.230]) by mail.physics.umn.edu with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.77 (FreeBSD)) (envelope-from ) id 1aPDR2-0004cD-Ks for freebsd-fs@freebsd.org; Fri, 29 Jan 2016 12:06:16 -0600 To: freebsd-fs@freebsd.org From: Graham Allan Subject: quantifying zpool performance with number of vdevs Organization: Physics, University of Minnesota Message-ID: <56ABAA18.90102@physics.umn.edu> Date: Fri, 29 Jan 2016 12:06:16 -0600 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Jan 2016 18:27:58 -0000 In many of the storage systems I built to date I was slightly conservative (?) in wanting to keep any one pool confined to a single JBOD chassis. In doing this I've generally been using the Supermicro 45-drive chassis with pools made of 4x (8+2) raidz2, other slots being kept for spares, ZIL and L2ARC. Now I have several servers with 3-4 such chassis, and reliability has also been such that I'd feel more comfortable about spanning chassis, if there was worthwhile performance benefit. Obviously theory says that iops should scale with number of vdevs but it would be nice to try and quantify. Getting relevant data out of iperf seems problematic on machines with 128GB+ RAM - it's hard to blow out the ARC. It does seem like I get possibly more valid-looking results if I set "zfs set primarycache=metadata" on my test dataset - it seems like this should mostly disable the ARC (seems to be borne out by arcstat output, though there could still be L2ARC effects). Wonder if anyone has any thoughts on this, and also on benefits/risks of moving from 40-drive to 80- or 120-drive pools. Graham --