Date: Sun, 28 Nov 2010 00:29:07 +0300 From: Lev Serebryakov <lev@FreeBSD.org> To: Ivan Voras <ivoras@freebsd.org> Cc: freebsd-geom@freebsd.org Subject: Re: GEOM profiling - how to? Message-ID: <1468561387.20101128002907@serebryakov.spb.ru> In-Reply-To: <icra6r$6q9$1@dough.gmane.org> References: <1299537757.20101127012903@serebryakov.spb.ru> <icra6r$6q9$1@dough.gmane.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Hello, Ivan.
You wrote 27 =D0=BD=D0=BE=D1=8F=D0=B1=D1=80=D1=8F 2010 =D0=B3., 19:04:42:
>> I'm doing some simple benchmarking of geom_raid5 in preparation of
>> putting it into ports. And I notice strange results.
>>
>> It is array of 5 disks, stripsize=3D128k. All disks are SATA2 disks on
>> ICH10R, AHCI driver (8.1-STABLE).
>>
>> Reading from device itself (dd with bs=3D512K) gives speed of one HDD
>> exactly. gstat shows 100% load of RAID geom and 1/5 of this speed
>> (and 18-22% load) on all disk GEOMs.
> This "100% load of RAID geom" is an approximation of disk load, not CPU
I understand this.
> load. I don't know how graid5 module works but if it's like most GEOM=20
> modules, you will probably need to use a very small stripe size,=20
> basically 128 / number_of_disks so that one request can span multiple=20
> drives. In your case, try 32 KiB stripe size or 16 KiB stripe size.
It doesn't work, really. Now I can not reconfigure array (it
contains a lot of valuable data), but I did very extensive
benchmarking before (about 2 years ago), and speed was best with
128Kb stripe. Stripe of 32 or 16Kb gave about half of this speed.
New round of benchamrking was insipired by upgrade to 8.x (and new
AHCI driver) and ampount of spare time which I want to spend on
maintaince and improving RAID5, which is abadoned by author.
>> Reading big file from FS (dd with bs=3D512k, FS block s=
ize 32K,
>> vfs.read_max=3D32) gives about twice speed and every disk GEOM is
>> loaded 38-42%. CPU time is about 8% system, 0.5% interrupt, so CPU
>> is not a bottle neck.
> With big readahead (btw try larger read_max values, like 128) you get
> parallelism on the drive hardware level, not GEOM, this is why it works.
Aagain, going above "32" (which means 32*32 =3D 1024Kb, twice
array-wide stripe size) doesn't work better for one thread and work
much worse in mult-thread loads accroding to my benchmarsk (it was
multi-demensional multi-day run with varying stripe size, FS block
size and pre-read setting).
So, these numbers are local optimum with this RAID
implementation. My interest is, how could I improve RAID
implementation to get better speeds. For this I need to understand
what is bottle neck in it.
>> How could I profile I/O and GEOM?
> There is no specific answer to this question; basically you can use
> gstat to observe performance of every GEOM device individually, and use
> "top" and similar to observe CPU usage. If you turn on GEOM logging,
Yes, I know these tools, but what I see here, that in both cases
"top" GEOM (raid array) loaded 100%, but in one case underlying disks
loaded for 20% (with individual read speeds about 20Mb/s, 80-90Mb/s
combined) and in other numbers are 100% and 40% (40Mb/s individual
speed and 150-160Mb/s combined). It doesn't help a lot :(
> your logs will be swamped by a huge number of messages which you can, in
> theory, create a tool to analyze them with.
Thnaks, I'll try it.
--=20
// Black Lion AKA Lev Serebryakov <lev@FreeBSD.org>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1468561387.20101128002907>