Date: Thu, 4 Jul 2013 01:40:58 -0400 (EDT) From: Garrett Wollman <wollman@hergotha.csail.mit.edu> To: peter@wemm.org Cc: freebsd-net@freebsd.org Subject: Re: Terrible ix performance Message-ID: <201307040540.r645ewsv044060@hergotha.csail.mit.edu> In-Reply-To: <CAGE5yCpojnENZW%2B6SN9WyeE5RUzpUickE8dB8r0zGrJGBJ2Wqg@mail.gmail.com> References: <CAKYr3zyV74DPLsJRuDoRiYsYdAXs=EoqJ6%2B_k4hJiSnwq5zhUQ@mail.gmail.com> <51D3E5BC.1000604@freebsd.org> <CAKYr3zyWzQsFOrQ-MrGTdTzJzhP1kXNac%2BHu8NXfC_J6YJcOsg@mail.gmail.com> <51D42976.9020206@freebsd.org> <CAKYr3zyFF%2BA-OHsEL7t6rdv6Jc4c2ByvvRhV-Fv%2BPXt9Y-sXwg@mail.gmail.com> <E97FF575ED6A405FB13872854191BF3B@multiplay.co.uk> <CAN6yY1vg=KAAaJhG0p8pO6vRwL%2BypHXUfV2Uth70DYNNy04-Uw@mail.gmail.com> <51D4D77B.60804@freebsd.org> <CAKYr3zyzj=AFcGu62Je3gkZy%2BQP1aDZanYTQp%2BjsMgoWWjrnWA@mail.gmail.com> <51D4EECE.4010808@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
In article <CAGE5yCpojnENZW+6SN9WyeE5RUzpUickE8dB8r0zGrJGBJ2Wqg@mail.gmail.com>, Peter Wemm quotes some advice about ZFS filesystem vdev layout: >"1. Virtual Devices Determine IOPS >IOPS (I/O per second) are mostly a factor of the number of virtual >devices (vdevs) in a zpool. They are not a factor of the raw number of >disks in the zpool. This is probably the single most important thing >to realize and understand, and is commonly not. > >ZFS stripes writes across vdevs (not individual disks). A vdev is >typically IOPS bound to the speed of the slowest disk within it. So if >you have one vdev of 100 disks, your zpool's raw IOPS potential is >effectively only a single disk, not 100. >" -- end quote > >I made this mistake myself a number of times before I found out. Getting a bit off-topic for this mailing-list, but I'll note that the fileservers that I built earlier this year (which also use the Intel 10G NIC) were able to get nearly good-as-local performance over NFS when connected through a single layer-2 switch. This hardware platform can do (local) about 5 GB/s from ARC, or 1.6 GB/s from disk, both reading single-threaded with compression disabled; writes are about 1.2 GB/s; a single-threaded FreeBSD NFS client could do between 750 MB/s and 1 GB/s, or about 80% of full link speed. The disk configuration here is 11 x 8-disk RAID-Z2 with separate ZIL and L2ARC, but reading is no faster on a 44 x 2-disk mirror. Both of these are on 7200-RPM drives on two 16-port LSI eSAS HBAs with gmultipath interposed for labeling and fault-tolerance. (Both layouts are largely determined by the hardware: we have four 24-drive shelves per server, one SSD and one hot spare per shelf, leaving 88 drives, which only factorizes as 44x2, 22x4, or 11x8, and 22x4 RAID-Z2 has the same capacity as 44x2 mirrored but with higher overhead.) A simple four-disk, two-way mirror configuration (i.e., two mirror vdevs of two disks each) on the same machine can do about 120 MB/s in the same tests. -GAWollman
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201307040540.r645ewsv044060>