Date: Tue, 3 Jun 2014 13:48:11 -0700 From: John-Mark Gurney <jmg@funkthat.com> To: Frank Broniewski <brfr@metrico.lu> Cc: freebsd-geom@freebsd.org Subject: Re: Geom stripe bottleneck Message-ID: <20140603204811.GJ31367@funkthat.com> In-Reply-To: <538D9BC3.6040509@metrico.lu> References: <538D9BC3.6040509@metrico.lu>
next in thread | previous in thread | raw e-mail | index | archive | help
Frank Broniewski wrote this message on Tue, Jun 03, 2014 at 11:56 +0200: > I have a stripe (RAID0) geom setup for my database's data. Currently I > am applying some large updates on the data and I think the performance > of my stripe could be better. But I am uncertain and so I thought I'd > request some interpretation help from the community :) > > The stripe consists of two disks (WD Velociraptor with 10.000 rpm): > >diskinfo -v ada2 > ada2 > 512 # sectorsize > 600127266816 # mediasize in bytes (558G) > 1172123568 # mediasize in sectors > 0 # stripesize > 0 # stripeoffset > 1162821 # Cylinders according to firmware. > > 16 # Heads according to firmware. > > 63 # Sectors according to firmware. > > WD-WXH1E61ASNX9 # Disk ident. > > > and /var/log/dmesg.boot > # snip > ada2 at ahcich2 bus 0 scbus2 target 0 lun 0 > ada2: <WDC WD6000HLHX-01JJPV0 04.05G04> ATA-8 SATA 3.x device > ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) > ada2: Command Queueing enabled > ada2: 572325MB (1172123568 512 byte sectors: 16H 63S/T 16383C) > ada2: Previously was known as ad8 > ada3 at ahcich3 bus 0 scbus3 target 0 lun 0 > ada3: <WDC WD6000HLHX-01JJPV0 04.05G04> ATA-8 SATA 3.x device > ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) > ada3: Command Queueing enabled > ada3: 572325MB (1172123568 512 byte sectors: 16H 63S/T 16383C) > ada3: Previously was known as ad10 > #snap > > > And here's some iostat -d -w 10 ada0 ada1 ada2 ada3 example output > #snip > ada0 ada1 ada2 ada3 > KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s > 0.00 0 0.00 0.00 0 0.00 19.33 176 3.32 19.33 176 3.32 > 16.25 0 0.01 16.25 0 0.01 16.87 133 2.20 16.87 133 2.20 > 0.00 0 0.00 0.00 0 0.00 16.77 146 2.40 16.77 147 2.40 > 0.00 0 0.00 0.00 0 0.00 19.46 170 3.24 19.45 170 3.23 > 21.50 0 0.01 21.50 0 0.01 17.00 125 2.08 17.00 125 2.08 > 0.50 0 0.00 0.50 0 0.00 16.88 145 2.38 16.88 145 2.38 > 0.00 0 0.00 0.00 0 0.00 16.96 125 2.07 16.97 125 2.07 > 0.00 0 0.00 0.00 0 0.00 19.82 158 3.06 19.81 158 3.07 > 28.77 1 0.03 28.77 1 0.03 16.83 133 2.19 16.82 133 2.19 > #snap The key here is the tps... Spining drives have a limited number of tps... first you have moving the heads, which on average will be ~4ms, then you have to wait, on average half a rotation, which for a 10k RPM drive is ~3ms, so each seek will take around 7ms, so, as you can see, your best number is 176 TPS, or ~8ms/transaction... so, it looks like your drives are performing as they should... > I think the MB/s output is rather low for such a disk. To gain further > insight I started gstat: > dT: 1.001s w: 1.000s > L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name > 0 27 0 0 0.0 27 2226 4.8 7.0| ada0 > 0 28 1 32 23.9 27 2226 1.3 3.9| ada1 > 2 120 115 1838 6.4 5 96 0.2 74.3| ada2 > 2 121 116 1854 6.3 5 96 0.4 72.9| ada3 > 0 28 1 32 24.0 27 2226 5.0 8.7| mirror/gm > 2 121 116 3708 7.9 5 192 0.4 92.2| stripe/gs > 0 28 1 32 24.0 27 2226 5.0 8.7| mirror/gms1 > 0 12 0 0 0.0 12 1343 9.1 6.9| mirror/gms1a > 0 0 0 0 0.0 0 0 0.0 0.0| mirror/gms1b > 0 0 0 0 0.0 0 0 0.0 0.0| mirror/gms1d > 0 0 0 0 0.0 0 0 0.0 0.0| mirror/gms1e > 0 16 1 32 24.0 15 883 1.7 2.9| mirror/gms1f > > > What bothers me here is that the stripe/gs is 92% busy while the disks > themselves are only at 74/72%. This lead me to my post here and seek > some advice, since I don't know enough about the mechanics and so I > can't really find the problem, if there is any at all. This is because the stripe has to wait for both drives to return data before moving the data up... If you're just running a single threaded benchmark, there isn't multiple IO's in flight, and there for the remaining time is spent in your application before it sends another request down to the stripe... the different between stripe and the drives is the fact each of them is sometimes faster than the other, so again, won't have work to do until another IO is submitted... Try sending more IO at it, like doing 4 or more dd read's such that the between the latency of one IO, there is other IO to server... Also, make sure that you're using NCQ where the OS can submit multiple IO's to the drives at once, this should improve things, but won't change the results you see above as it requires multiple IO's outstanding... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20140603204811.GJ31367>