Date: Fri, 18 Jan 2013 18:32:42 -0500 From: Karim Fodil-Lemelin <fodillemlinkarim@gmail.com> To: freebsd-hackers@freebsd.org Cc: gibbs@FreeBSD.org, scottl@FreeBSD.org, mjacob@FreeBSD.org Subject: Re: IBM blade server abysmal disk write performances Message-ID: <50F9DB9A.9050303@gmail.com> In-Reply-To: <50F9CFEB.5060302@feral.com> References: <CAA3ZYrBV9f%2BcHx4jvS0UKTr%2Bp7eNUBA0S2%2Bv9oZAHqwm9VBOWw@mail.gmail.com> <6C0B86E6-195C-4D35-AE40-3D2F9F6D28FB@yahoo.com> <alpine.BSF.2.00.1301182217590.1478@wojtek.tensor.gdynia.pl> <1358544287.32417.251.camel@revolution.hippie.lan> <50F9CFEB.5060302@feral.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 18/01/2013 5:42 PM, Matthew Jacob wrote:
> This is all turning into a bikeshed discussion. As far as I can tell,
> the basic original question was why a *SAS* (not a SATA) drive was not
> performing as well as expected based upon experiences with Linux. I
> still don't know whether reads or writes were being used for dd.
>
> This morning, I ran a fio test with a single threaded read component
> and a multithreaded write component to see if there were differences.
> All I had connected to my MPT system were ATA drives (Seagate 500GBs)
> and I'm remote now and won't be back until Sunday to put one of my
> 'good' SAS drives (140 GB Seagates, i.e., real SAS 15K RPM drives, not
> "fat SATA" bs drives).
>
> The numbers were pretty much the same for both FreeBSD and Linux. In
> fact, FreeBSD was slightly faster. I won't report the exact numbers
> right now, but only mention this as a piece of information that at
> least in my case the differences between the OS platform involved is
> negligible. This would, at least in my case, rule out issues based
> upon different platform access methods and different drivers.
>
> All of this other discussion, about WCE and what not is nice, but for
> all intents and purposes it serves could be moved to *-advocacy.
>
Thanks for the clarifications!
I did mention at some point those were write speeds and reads were just
fine and those were either writes to the filesystem or direct access
(only on SAS again).
Here is what I am planning to do next week when I get the chance:
0) I plan on focusing on the SAS driver tests _only_ since SATA is
working as expected so nothing to report there.
1) Look carefully at how the drives are physically connected. Although
it feels like if the SATA works fine the SAS should also but I'll check
anyway.
2) Boot verbose with "boot -v" and send the dmesg output. mpt driver
might give us a clue.
3) Run gstat -abc in a loop for the test duration. Although I would
think ctlstat(8) might be more interesting here so I'll run it too for
good measure :).
Please note that in all tests write caching was enabled as I think this
is the default with FBSD 9.1 GENERIC but I'll confirm this with
camcontrol(8).
I've also seen quite a lot of 'quirks' for tagged command queuing in the
source code (/sys/cam/scsi/scps_xtp.c) but a particular one got my
attention (thanks to whomever writes good comments in source code :) :
/*
* Slow when tagged queueing is enabled. Write performance
* steadily drops off with more and more concurrent
* transactions. Best sequential write performance with
* tagged queueing turned off and write caching turned on.
*
* PR: kern/10398
* Submitted by: Hideaki Okada <hokada@isl.melco.co.jp>
* Drive: DCAS-34330 w/ "S65A" firmware.
*
* The drive with the problem had the "S65A" firmware
* revision, and has also been reported (by Stephen J.
* Roznowski <sjr@home.net>) for a drive with the "S61A"
* firmware revision.
*
* Although no one has reported problems with the 2 gig
* version of the DCAS drive, the assumption is that it
* has the same problems as the 4 gig version. Therefore
* this quirk entries disables tagged queueing for all
* DCAS drives.
*/
{ T_DIRECT, SIP_MEDIA_FIXED, "IBM", "DCAS*", "*" },
/*quirks*/0, /*mintags*/0, /*maxtags*/0
So I looked at the kern/10398 pr and got some feeling of 'deja vu'
although the original problem was on FreeBSD 3.1 so its most likely not
that but I though I would mention it. The issue described is awfully
familiar. Basically the SAS drive (scsi back then) is slow on writes but
fast on reads with dd. Could be a coincidence or a ghost from the past
who knows...
Cheers,
Karim.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?50F9DB9A.9050303>