Date: Fri, 25 Jan 2013 09:16:36 -0500 From: Karim Fodil-Lemelin <fodillemlinkarim@gmail.com> To: freebsd-hackers@freebsd.org Cc: gibbs@FreeBSD.org, scottl@FreeBSD.org, mjacob@FreeBSD.org Subject: Re: IBM blade server abysmal disk write performances Message-ID: <510293C4.3060304@gmail.com> In-Reply-To: <50F9DB9A.9050303@gmail.com> References: <CAA3ZYrBV9f%2BcHx4jvS0UKTr%2Bp7eNUBA0S2%2Bv9oZAHqwm9VBOWw@mail.gmail.com> <6C0B86E6-195C-4D35-AE40-3D2F9F6D28FB@yahoo.com> <alpine.BSF.2.00.1301182217590.1478@wojtek.tensor.gdynia.pl> <1358544287.32417.251.camel@revolution.hippie.lan> <50F9CFEB.5060302@feral.com> <50F9DB9A.9050303@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi,
Quick follow up on this. As I mentioned in a previous email we have
moved to SATA drives and the SAS drives have been shelved for now. The
current project will be using those so further tests on SAS have been
postponed to an undefined date.
Thanks,
Karim.
PS: I'll keep the SAS tests in my back pocket so I get a head start when
we get around SAS testing again.
On 18/01/2013 6:32 PM, Karim Fodil-Lemelin wrote:
> On 18/01/2013 5:42 PM, Matthew Jacob wrote:
>> This is all turning into a bikeshed discussion. As far as I can tell,
>> the basic original question was why a *SAS* (not a SATA) drive was
>> not performing as well as expected based upon experiences with Linux.
>> I still don't know whether reads or writes were being used for dd.
>>
>> This morning, I ran a fio test with a single threaded read component
>> and a multithreaded write component to see if there were differences.
>> All I had connected to my MPT system were ATA drives (Seagate 500GBs)
>> and I'm remote now and won't be back until Sunday to put one of my
>> 'good' SAS drives (140 GB Seagates, i.e., real SAS 15K RPM drives,
>> not "fat SATA" bs drives).
>>
>> The numbers were pretty much the same for both FreeBSD and Linux. In
>> fact, FreeBSD was slightly faster. I won't report the exact numbers
>> right now, but only mention this as a piece of information that at
>> least in my case the differences between the OS platform involved is
>> negligible. This would, at least in my case, rule out issues based
>> upon different platform access methods and different drivers.
>>
>> All of this other discussion, about WCE and what not is nice, but for
>> all intents and purposes it serves could be moved to *-advocacy.
>>
> Thanks for the clarifications!
>
> I did mention at some point those were write speeds and reads were
> just fine and those were either writes to the filesystem or direct
> access (only on SAS again).
>
> Here is what I am planning to do next week when I get the chance:
>
> 0) I plan on focusing on the SAS driver tests _only_ since SATA is
> working as expected so nothing to report there.
> 1) Look carefully at how the drives are physically connected. Although
> it feels like if the SATA works fine the SAS should also but I'll
> check anyway.
> 2) Boot verbose with "boot -v" and send the dmesg output. mpt driver
> might give us a clue.
> 3) Run gstat -abc in a loop for the test duration. Although I would
> think ctlstat(8) might be more interesting here so I'll run it too for
> good measure :).
>
> Please note that in all tests write caching was enabled as I think
> this is the default with FBSD 9.1 GENERIC but I'll confirm this with
> camcontrol(8).
>
> I've also seen quite a lot of 'quirks' for tagged command queuing in
> the source code (/sys/cam/scsi/scps_xtp.c) but a particular one got my
> attention (thanks to whomever writes good comments in source code :) :
>
> /*
> * Slow when tagged queueing is enabled. Write
> performance
> * steadily drops off with more and more concurrent
> * transactions. Best sequential write performance with
> * tagged queueing turned off and write caching turned
> on.
> *
> * PR: kern/10398
> * Submitted by: Hideaki Okada <hokada@isl.melco.co.jp>
> * Drive: DCAS-34330 w/ "S65A" firmware.
> *
> * The drive with the problem had the "S65A" firmware
> * revision, and has also been reported (by Stephen J.
> * Roznowski <sjr@home.net>) for a drive with the "S61A"
> * firmware revision.
> *
> * Although no one has reported problems with the 2 gig
> * version of the DCAS drive, the assumption is that it
> * has the same problems as the 4 gig version. Therefore
> * this quirk entries disables tagged queueing for all
> * DCAS drives.
> */
> { T_DIRECT, SIP_MEDIA_FIXED, "IBM", "DCAS*", "*" },
> /*quirks*/0, /*mintags*/0, /*maxtags*/0
>
> So I looked at the kern/10398 pr and got some feeling of 'deja vu'
> although the original problem was on FreeBSD 3.1 so its most likely
> not that but I though I would mention it. The issue described is
> awfully familiar. Basically the SAS drive (scsi back then) is slow on
> writes but fast on reads with dd. Could be a coincidence or a ghost
> from the past who knows...
>
> Cheers,
>
> Karim.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?510293C4.3060304>