From owner-freebsd-net@FreeBSD.ORG Thu Jul 4 05:03:27 2013 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 6A13D1DC for ; Thu, 4 Jul 2013 05:03:27 +0000 (UTC) (envelope-from peter@wemm.org) Received: from mail-ve0-x230.google.com (mail-ve0-x230.google.com [IPv6:2607:f8b0:400c:c01::230]) by mx1.freebsd.org (Postfix) with ESMTP id 25A231E06 for ; Thu, 4 Jul 2013 05:03:26 +0000 (UTC) Received: by mail-ve0-f176.google.com with SMTP id c13so730463vea.21 for ; Wed, 03 Jul 2013 22:03:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=wemm.org; s=google; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=U8fsYGvQYzX8vrAZed8EHXO8T1CHEQnuMCuEF56hBMs=; b=30+jbIrQoZ47ejNZFMJXylrOcWY1G/BtaQ3+gIeXDwVYy80WJyqk0LTGpJdJp5+eb5 lUZ67M7DFXJQaO9TgP4dn8zxUUp93HjAgp/Dqu7wYZwXb9gZLIdNuGuktltjK4d05h2G 8NjniI8d/U/Po3guZgvLZkkD1SZEZnAwRLjM8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:x-gm-message-state; bh=U8fsYGvQYzX8vrAZed8EHXO8T1CHEQnuMCuEF56hBMs=; b=oDYfj1wbE0yW+2zooqt0X5pD57bGQvy0wH1MmWeMnstpUxXQvPFsjlGvZAhiKqb5Vy UN57ixrbceKggZ4d3Pb1Tu4jUFiDyyN3djAmfXHc4vXmKMnnSWx4RfgrSynWJ4i/ZAbz 7bI7C6JLpUPAF84Ww5z2gwknRvOKDCGRPI5c0N3eXrKOmkV5u9S7ljpNFUoG5CYdLXu6 GWyeNuYebBuOLBGc5cfZfT+85KqAGC3EboEn4kgbVIJpSrW+JUNzqtg1c49gzuwmP/48 SgKgvJ5rNnSH+W2txy2M3AR6UPPBMBgqT2SVa1sFjmvxMOm0LLWmOz2sBc4PH+IfSKEB TLWQ== MIME-Version: 1.0 X-Received: by 10.52.31.170 with SMTP id b10mr1395969vdi.115.1372914206587; Wed, 03 Jul 2013 22:03:26 -0700 (PDT) Received: by 10.221.37.198 with HTTP; Wed, 3 Jul 2013 22:03:26 -0700 (PDT) In-Reply-To: <51D4EECE.4010808@freebsd.org> References: <51D3E5BC.1000604@freebsd.org> <51D42976.9020206@freebsd.org> <51D4D77B.60804@freebsd.org> <51D4EECE.4010808@freebsd.org> Date: Wed, 3 Jul 2013 22:03:26 -0700 Message-ID: Subject: Re: Terrible ix performance From: Peter Wemm To: Lawrence Stewart Content-Type: text/plain; charset=ISO-8859-1 X-Gm-Message-State: ALoCoQkkC8azUuIKqN5Max5YtYc9rOL/6f6EbfXjHfx7UOTTEZ9KAyoW6I/qKYIki7dS81sw3KjK Cc: Outback Dingo , net@freebsd.org, Steven Hartland , Kevin Oberman X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Jul 2013 05:03:27 -0000 On Wed, Jul 3, 2013 at 8:41 PM, Lawrence Stewart wrote: > - I recall some advice that zpool's should not have more than about 8 or > 10 disks in them, and you should instead create multiple zpools if you > have more disks. Perhaps investigate the source of that rumour and if > it's true, try create 2 x 8 disk zpools in Box A and 3 x 8 disk zpools > in box B and see if that changes things at all. http://nex7.blogspot.com/2013/03/readme1st.html Item #1: "1. Virtual Devices Determine IOPS IOPS (I/O per second) are mostly a factor of the number of virtual devices (vdevs) in a zpool. They are not a factor of the raw number of disks in the zpool. This is probably the single most important thing to realize and understand, and is commonly not. ZFS stripes writes across vdevs (not individual disks). A vdev is typically IOPS bound to the speed of the slowest disk within it. So if you have one vdev of 100 disks, your zpool's raw IOPS potential is effectively only a single disk, not 100. " -- end quote I made this mistake myself a number of times before I found out. -- Peter Wemm - peter@wemm.org; peter@FreeBSD.org; peter@yahoo-inc.com; KI6FJV UTF-8: So you can \342\200\231 .. for when a ' just won't do