Date: Mon, 12 Apr 1999 11:21:49 +0930 From: Greg Lehey <grog@lemis.com> To: Andrew Gallatin <gallatin@cs.duke.edu>, freebsd-scsi@FreeBSD.ORG Subject: Re: odd performance 'bug' & other questions Message-ID: <19990412112149.F2142@lemis.com> In-Reply-To: <14097.8430.806061.277769@grasshopper.cs.duke.edu>; from Andrew Gallatin on Sun, Apr 11, 1999 at 07:14:25PM -0400 References: <14097.8430.806061.277769@grasshopper.cs.duke.edu>
index | next in thread | previous in thread | raw e-mail
[-- Attachment #1 --] On Sunday, 11 April 1999 at 19:14:25 -0400, Andrew Gallatin wrote: > > We're setting up a few large servers, each with 6 9GB seagate medalist > pro drives spread across 2 scsi controllers (aic7890 & ncr53c875). > > We've noticed that if we set up the disks using a simple ccd stripe, > after trying various interleaves, the best read bandwidth we can get > is only ~35-40MB/sec (using dd if=/dev/rccd0 of=/dev/null bs=64k), > which is odd because we'd thought we should be getting at least > 55-60MB/sec, as we get about 13.5MB/sec from each drive with the same > test. This kind of test is not very useful, and may be counterproductive. What size stripe did you use? I'm attaching a plot of Vinum performance against stripe size with a volume spanning four very slow disks. I briefly tested ccd and got very similar results. These were done with 'rawio' against the character device with 8 concurrent processes. You'll note that sequential I/O performance peaks at about 80 kB stripes, whereas random transfers (which are presumably closer to what you're doing) improve with increasing stripe size. You'll also notice that the performance ratio is approximately the same as you describe, rather less than 3x the single disk, but this is misleading, since I only had four drives in this test. You'll also notice that the performance figures are terrible; that's because they were done on some ancient drives (look at the throughput of the raw drives without Vinum). > Upon closer examination, we discovered that on some of the drives the > performance wanders all over the place -- if you do a dd if=/dev/rX > of=/dev/null bs=64k on an individual disk on an otherwise idle system > & watch with iostat or systat, you can see the bandwidth jump around > quite a bit. I'm thinking that my performance problems might be due > to the fact that the reads aren't really sequential, rather the disk > arm is moving all over the place to read remapped defective blocks. Have you considered rotational latency? Unless you spindle-sync the drives, you can end up with random delays in moving from one drive to the next. If you spindle-sync them, you may or may not incur a whole-rotation delay :-) I've done some tests here, and they show the same effects for single processes. Greg -- See complete headers for address, home page and phone numbers finger grog@lemis.com for PGP public key [-- Attachment #2 --] GIF87a @@@ @@ @ ` @ ` 0`@@@@ ` ```` ` @`` ` ` ``` @@ @` ```@ ` ` `@ @@` `` @ @ @`` ` @@`p , H*\ȰÇ#JHŋ "&*.26:>BFJNRVZ^bfjnrvz~Ɗ 2 * ( 0 008 8HXhx)9IYiy *:JZjz 8HXhxHP00 0p(x8hX8(h(xH8(xP aa`aa`!@aaa`!@!@!@aa`aaa@a a` @`aaa` a!"@@bb"@@bba@b"@@a@a@"a@bbbbbbbb"b@bbbba@bbbb@bb`@b"a@b"@@"b@ba@"a@bbbbbbbb"b@bbbba@bbb"@@bb"@@bba@b"@@a@a@"a@bbbbD%B( @p10@QQ0@Q11@QQ0@11@0@0@q0@QQQ000 `0 1@QQQQQQQ11@QQQQQ0@QQ0@Q1@Qq0@11@0@0@q0@QQQQQQQ11@QQQQQ0@QQ0@Q11@QQ0 @p0а0P0 @p1Qqq11q1Q0Q10q0Q01q11Q1q01100 10@P`P @P0`pp0 @p1Qqq11q1Q0Q10q0Q01q11Q1q01100 1q11q1Q0Q10q0Q01q11QP a!@ "a "` ""@ a ba """"""""ba """""""""""@ """"""a """ba """a "a "` "b a ba """""""""""@ """"""a """"a ""ba "a " ` a``!@ a!b"b@cccc@cb@c`@b@a@`@`@c@cccccb@cccc@cb@c`@cb@a@#a@#@@c@cccc#c@cccccb@cccc@cb@c`@b@a@`@`@c@cccccb a!baabb`bba"a`bb`a"b`bbaba`bb`abb`bba"a`!@ 1bĈ#F1bĈ#F1bĈ#F !#%')+-//011111111111011111111111011011111111111011111111111011111111111011111111111011011111111a @ ABCDEFGHIJ LLLLLLLLLLLL LLLLLLLLLLL LL @ ABCDEFGHIJKLMNOPQRSTUVẀX̀X̀XXb*h ` ` @pp``pppp`ppp00 @p1Qq2Rr3Ss4Tt5Uuus`ps`p t@rs14us`p@p``p ``@p`p `` ` qp```p` ` ``p@p`` ` ``@p```@``@Pp``@Pp@PPP` ``@```@```@`` ` @p1Q0`qqqqqq1 @p1Qq2Rr3Ss4Tt5Uus `p`p`@p ``p ` ` ` q``p`p `@p` ``p ``p ``@p@ppp`p q`@p ``pp `@p`p@p@p @p1Qqq1 @p1Qq2Rr3Ss4Tt5Uus `p`p`p ``p ` ` `@q@p@P` ` ` ` `P` ` ` @pp``````p`ppp`Pp@P `p ` ` @p1Q0`qqqqqq1 @p1Qq2Rr3Ss4Tt5Uus `p`p`p ``p ` ` ``q `p`p ``p `p`p@pp`p`p`p qpp`p ``p14UVZu4 (80808 0 80 0 0 008008 00(8 80 0 80 00(808 8 80808 800808880(80(8 80 0 80 8HXhxH 8HXhx)9IYiy *:JZjz9@800 808 80 0 0088000800080 0 80 0 800 8080800800080 0 8008`800080 0 80 0 8HXhx)9IYiy *:JZjz*:z8zZʕ+W\rʕ+W\rUP a!bb a!b"c#d$e%f&g'h(i)j*k+lf@,g@llllllllllllllllllllllllf@,g@llllllllllllllllllllllllf@,g@llllllllllllllllXb}D @ @ 8HXhxH@@pxxxXH`Hp8H@pxx(H@HpxxxxxxxxXpxxxxxxxxxx 8HXhx)9IYiy *:JZjzZI0HI0H J IIz ZI0H H@0H @@ H0H @@ @H`H@@0H@ @ @0H H@@ @ @@ H@@@ @@ (H@@ (H (((@ @ @ @@ @@ @@@@ @@@ 8HXhxX 8HXhx)9IYiy *:JZjzjI @@H0H@ H @0H @ @ @H@0H0H @ H@ @0H @0H @@ H H`HpH0HH0H @@ H0H@H@H0HpH 8HXhxX 8H 8HXhx 8HXhx)9IYiy *:JZjzjI @@H0H0H @0H @ @ @H H (@ @ @ @ 0(@ @ @ 8H0@@@@0H088H @ @ @08@0@0 H ( 8HXhxH 8HXhx)9IYiy *:JZjzjI @@H0H0H @0H @ @ @H @PH0H @0H @PH0H H`H0H0H0HH @ @ @pH@H@H* ڪUVZ @P`` @ `p` `P@ @ `P`@@``@``pp `p`@ @p1Q1`qqqqqq1 @p1Qq2Rr3Ss4Tt5Uu@`@ ` `` @ @@```` @ @`1P a!b"c#d$e%f&g'h(i)j*k+'kkgk!kkgkag+W\rʕ+W\rU!@ bŊ @ ABCÂCCÂ @ ABCDEFGHIJKLMNOPQRSTUVÂςȂ WWWWWWWWWWWWWWWWWWWWWĂ @ A@ ABCDE @ ABCDEFGHIJKLMNOPQRSTUÂÂCABCB@ AB @ ABCD @ ABCDEFGHIJKLMNOPQRSTUÂÂBAA @ ABCDE @ ABCDEFGHIJKLMNOPQRSTUÂ WWWWWWWWWWWWWWWWWWW\ @P`` @ `p@@@ ` @ ` `p`P`P@ @ @p1Q0`qqqqqq1 @p1Qq2Rr3Ss4Tt5Uu@`@ ``@ `@ @@`` @ 13Օ+W\rʕ+W "&*.. "&*.26:>BFJNRVZ^bfjnrvz~frfrfrƊ+VXbe 8HXhx 8HXhx)9IYiy *:JZjz kh0h0hi``` ij@h; 8HXhx)9IYiy *:JZjzj0hi0hi`jz j0h h`0h `` h0h `` `h0h h`` h ``0h` ` h` h `` ` 8(h0h ` ``0h`0h```0h`` 8HXhxX 8HXhx)9IYiy *:JZjz ` `@h0h` h `0h ` ` `ph0hPh `` h ` h` `0h ` ` `ph0h `` h0h@h@h0hph 8HXhxX 8H 8HXhx 8HXhx)9IYiy *:JZjz ` `@h0h0h `0h ` ` `ph```0h``` `0h `0h `0h ` ` `ph ` ` `ph@h@h ( 8HXhxH 8HXhx)9IYiy *:JZjz ` `@h0h0h `0h ` ` `ph `0h0h `0h `0h `0h ` ` `ph ` ` `ph@h@hʕ+W\ed (h0h0h ` h` ` ` `08` h h h` `0h ` h` `0h ` ` `08` ` ` `08`0` h ` 8HXhx 8HXhx)9IYiy *:JZjz*h@h`` h0h h` ` `0hph0h h` ` `0h h` ` h`` h0hh ` h`h``@h`0h``ʕ+W\rʕ+W "&*.. "&*.26:>BFJNRVZ^bfjnrvz~frfrfrƊ+VXbe 8HXhx 8HXhx)9IYiy *:JZjz ;ypppy0xxxppz`xxpy 8HXhxH 8HXhx)9IYiy *:JZjzʚx0xpyxx0x z yy ښx0x xpp x0xxp p xp@xpp x xx`xpp0x x p p p ppp p pp pppp pp ppp p 88x pp0xpp0xpp0xp p 8HXhxH 8HXhx)9IYiy *:JZjzʚx0xPx p0xpx xp0x0x0x p pxp0x0x p xp p0x p0x pp x x`xpx0xxp x p0xPx p xppx 8HXhxX 8H 8HXhx 8HXhx)9IYiy *:JZjzʚxppp0xppp p p ppx0x0x@xp0xpx x (p p p p 0(p p p 8x0pppp0x088x0(x ( 0x p p 8HXhxH 8HXhx)9IYiy *:JZjzʚx p0x0x p p ppx0x0x`x x px pPx0x p0x pPx0x x`x0x0x0xxPxPx0x p0x ڪUVZjժUD !#%')+-/13579;=?ACEGIKMOQSUWY !#%')+-/13579;=?ACEGIKMOQSUWY[E\]O\?\]]]]]]]]]]]]]]]]]]]]]]\]O\?\]]]]]]]]]]]]]]]]]]]]]]]]]\ !#%')+-/13579;=?ACEGIKMOQSUWY[]_a3^ bccccccccccccccccccccccc3b bccccccccccccccccŊ+VX2 $,4<DL, 8$,$,4<DLT\dlt| %-5=EMU]emu}M$,$,4<DL$ 8 $,4<DLT\dlt| %-5=EMU]eD@@@DDHD`D@@E0DPD@D Dhh 8hmmmmmmmmmmmmmmmmmmmmmDDDhDXDD E $,D0444$D044444444440444444,0 0 $,4<DLT\dlt| %-5=EMU]eDDD@@DD@D@ @D@ D@@DD@D0D@@D@ @ @DD@@D @@D@@@D@ D@@ DHDD @ @@D@D@@ D @@@ $,4<DLT,X\\\ $,4<DLT\dlt| %-5=EMU]eDD(D @D8DD@DDD @ @PD@DD @D@ @D @D @@DD0D8DDHDD @@DD D DD8D$,4<D,$ $,4<DLT\\ $,4<DLT\dlt| %-5=EMU]eD@@@D@@@ @ @ @8DDD D@D@hDD@ @ @ @ @ @ @ D@@@@DD @ @ @@@ D$,4<DLT$X\\\\\\ $,4<DLT\dlt| %-5=EMU]eD @DD @ @ @8DDD0DD @hD @(DD @D @(DDD0DDDDHD @ @ @8D D Dhh hmmmmmmmmmժUVZjժUV ! " """ """ " " """""" " """ "&*",...... "&*.26:>BFJNRVZ^bfjnrvz~""" " $" " " "" " " " " " """ " " " $" "" " "&*,...... "&*.26:>BFJNRVZ^bfjnrvz~r"~"~~rʕ+W\rʕ+W\ 8HXhx 8HXhx)9IYiy *:JZjz +VX %&
