From owner-freebsd-hackers@FreeBSD.ORG Fri Jan 25 14:16:45 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 69F18FDC; Fri, 25 Jan 2013 14:16:45 +0000 (UTC) (envelope-from fodillemlinkarim@gmail.com) Received: from mail-ie0-x232.google.com (ie-in-x0232.1e100.net [IPv6:2607:f8b0:4001:c03::232]) by mx1.freebsd.org (Postfix) with ESMTP id 14F3ED44; Fri, 25 Jan 2013 14:16:45 +0000 (UTC) Received: by mail-ie0-f178.google.com with SMTP id c12so72686ieb.23 for ; Fri, 25 Jan 2013 06:16:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:message-id:date:from:user-agent:mime-version:to:cc :subject:references:in-reply-to:content-type :content-transfer-encoding; bh=z18AKoVaTYKjNqDNZdeKOpX9kQA19HoqT5K82QlETQk=; b=R1HAvys9bN3cQgOepJpMz0qAtoLIzoWG5oYHgmcz2oupCKquetkzmbby5QE5RSgHoy OdkB5wuDryIwNZfkPiEPbLLozruJebQJTSWVd8Y8SYwTt5X4F+rLmpMDZC/IGFNCaL9t 0G7s6LBgRXW1ldjkXUXH/EgtEHplB4M1aHTRpRKCAUVIlbX+kyj/3/yFwqJwwXp5eWrC bH2HayiwmVz8IqpvrD4OTxcdL6lF6XN8/nVnWJkjkcv9kKd8g8VL7iMGdpJXYeMsu6XR +M51pqw1qcTbZCEk4uRYGh9uw3FwwtSGFhufhq2MNNbPGAELI6YnJgYvRhO04d0ppWa8 9FNA== X-Received: by 10.42.39.80 with SMTP id g16mr3609008ice.26.1359123404682; Fri, 25 Jan 2013 06:16:44 -0800 (PST) Received: from [192.168.1.73] ([208.85.112.101]) by mx.google.com with ESMTPS id fa6sm760658igb.2.2013.01.25.06.16.42 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 25 Jan 2013 06:16:43 -0800 (PST) Message-ID: <510293C4.3060304@gmail.com> Date: Fri, 25 Jan 2013 09:16:36 -0500 From: Karim Fodil-Lemelin User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130107 Thunderbird/17.0.2 MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Subject: Re: IBM blade server abysmal disk write performances References: <6C0B86E6-195C-4D35-AE40-3D2F9F6D28FB@yahoo.com> <1358544287.32417.251.camel@revolution.hippie.lan> <50F9CFEB.5060302@feral.com> <50F9DB9A.9050303@gmail.com> In-Reply-To: <50F9DB9A.9050303@gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Fri, 25 Jan 2013 14:21:45 +0000 Cc: gibbs@FreeBSD.org, scottl@FreeBSD.org, mjacob@FreeBSD.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Jan 2013 14:16:45 -0000 Hi, Quick follow up on this. As I mentioned in a previous email we have moved to SATA drives and the SAS drives have been shelved for now. The current project will be using those so further tests on SAS have been postponed to an undefined date. Thanks, Karim. PS: I'll keep the SAS tests in my back pocket so I get a head start when we get around SAS testing again. On 18/01/2013 6:32 PM, Karim Fodil-Lemelin wrote: > On 18/01/2013 5:42 PM, Matthew Jacob wrote: >> This is all turning into a bikeshed discussion. As far as I can tell, >> the basic original question was why a *SAS* (not a SATA) drive was >> not performing as well as expected based upon experiences with Linux. >> I still don't know whether reads or writes were being used for dd. >> >> This morning, I ran a fio test with a single threaded read component >> and a multithreaded write component to see if there were differences. >> All I had connected to my MPT system were ATA drives (Seagate 500GBs) >> and I'm remote now and won't be back until Sunday to put one of my >> 'good' SAS drives (140 GB Seagates, i.e., real SAS 15K RPM drives, >> not "fat SATA" bs drives). >> >> The numbers were pretty much the same for both FreeBSD and Linux. In >> fact, FreeBSD was slightly faster. I won't report the exact numbers >> right now, but only mention this as a piece of information that at >> least in my case the differences between the OS platform involved is >> negligible. This would, at least in my case, rule out issues based >> upon different platform access methods and different drivers. >> >> All of this other discussion, about WCE and what not is nice, but for >> all intents and purposes it serves could be moved to *-advocacy. >> > Thanks for the clarifications! > > I did mention at some point those were write speeds and reads were > just fine and those were either writes to the filesystem or direct > access (only on SAS again). > > Here is what I am planning to do next week when I get the chance: > > 0) I plan on focusing on the SAS driver tests _only_ since SATA is > working as expected so nothing to report there. > 1) Look carefully at how the drives are physically connected. Although > it feels like if the SATA works fine the SAS should also but I'll > check anyway. > 2) Boot verbose with "boot -v" and send the dmesg output. mpt driver > might give us a clue. > 3) Run gstat -abc in a loop for the test duration. Although I would > think ctlstat(8) might be more interesting here so I'll run it too for > good measure :). > > Please note that in all tests write caching was enabled as I think > this is the default with FBSD 9.1 GENERIC but I'll confirm this with > camcontrol(8). > > I've also seen quite a lot of 'quirks' for tagged command queuing in > the source code (/sys/cam/scsi/scps_xtp.c) but a particular one got my > attention (thanks to whomever writes good comments in source code :) : > > /* > * Slow when tagged queueing is enabled. Write > performance > * steadily drops off with more and more concurrent > * transactions. Best sequential write performance with > * tagged queueing turned off and write caching turned > on. > * > * PR: kern/10398 > * Submitted by: Hideaki Okada > * Drive: DCAS-34330 w/ "S65A" firmware. > * > * The drive with the problem had the "S65A" firmware > * revision, and has also been reported (by Stephen J. > * Roznowski ) for a drive with the "S61A" > * firmware revision. > * > * Although no one has reported problems with the 2 gig > * version of the DCAS drive, the assumption is that it > * has the same problems as the 4 gig version. Therefore > * this quirk entries disables tagged queueing for all > * DCAS drives. > */ > { T_DIRECT, SIP_MEDIA_FIXED, "IBM", "DCAS*", "*" }, > /*quirks*/0, /*mintags*/0, /*maxtags*/0 > > So I looked at the kern/10398 pr and got some feeling of 'deja vu' > although the original problem was on FreeBSD 3.1 so its most likely > not that but I though I would mention it. The issue described is > awfully familiar. Basically the SAS drive (scsi back then) is slow on > writes but fast on reads with dd. Could be a coincidence or a ghost > from the past who knows... > > Cheers, > > Karim.