From owner-freebsd-hackers@FreeBSD.ORG Fri Jan 18 23:32:53 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 9A341934; Fri, 18 Jan 2013 23:32:53 +0000 (UTC) (envelope-from fodillemlinkarim@gmail.com) Received: from mail-ie0-f171.google.com (mail-ie0-f171.google.com [209.85.223.171]) by mx1.freebsd.org (Postfix) with ESMTP id 4ECC2E08; Fri, 18 Jan 2013 23:32:53 +0000 (UTC) Received: by mail-ie0-f171.google.com with SMTP id 9so3183084iec.2 for ; Fri, 18 Jan 2013 15:32:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:message-id:date:from:user-agent:mime-version:to:cc :subject:references:in-reply-to:content-type :content-transfer-encoding; bh=/2iKFgdXoaUaS36OnHTe7YFbD2qoisU1USH1YdS9uWg=; b=W8RmK6g/mydXTrzaRpcNYNSzJX1DGmm48f05JdS7tqJnBVKQKhChif/2aZVb0LWVta 7zMjRuBJcR584CoI47ZjpOaDSzLLwpIyqM7/NnRTCzH5qfhmqzh/3vhSeVSiTHUL7zmt KvFJFenZ2figmX6E7y76qhkvf1vVrHwOL5Rt/ubOeFzxVsrn2oNtTaLmQcOxs0re4+10 PM3RmGaZecFoiwzXNr26XjfEbicfyT5617lWPl+wv1GLEwjCPr6BJtuZssORD6BSGyrV yeCTF+ygd+xGwP+elYy9i5RykiQjgt+TRuChhW2Z2pYdKO/r0JMFGEvXNq42qKEG80Nl ONXQ== X-Received: by 10.50.190.199 with SMTP id gs7mr3496466igc.89.1358551972683; Fri, 18 Jan 2013 15:32:52 -0800 (PST) Received: from [10.0.0.131] ([24.225.136.71]) by mx.google.com with ESMTPS id eo7sm3192101igc.12.2013.01.18.15.32.50 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 18 Jan 2013 15:32:51 -0800 (PST) Message-ID: <50F9DB9A.9050303@gmail.com> Date: Fri, 18 Jan 2013 18:32:42 -0500 From: Karim Fodil-Lemelin User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130107 Thunderbird/17.0.2 MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Subject: Re: IBM blade server abysmal disk write performances References: <6C0B86E6-195C-4D35-AE40-3D2F9F6D28FB@yahoo.com> <1358544287.32417.251.camel@revolution.hippie.lan> <50F9CFEB.5060302@feral.com> In-Reply-To: <50F9CFEB.5060302@feral.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Sat, 19 Jan 2013 02:33:18 +0000 Cc: gibbs@FreeBSD.org, scottl@FreeBSD.org, mjacob@FreeBSD.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Jan 2013 23:32:53 -0000 On 18/01/2013 5:42 PM, Matthew Jacob wrote: > This is all turning into a bikeshed discussion. As far as I can tell, > the basic original question was why a *SAS* (not a SATA) drive was not > performing as well as expected based upon experiences with Linux. I > still don't know whether reads or writes were being used for dd. > > This morning, I ran a fio test with a single threaded read component > and a multithreaded write component to see if there were differences. > All I had connected to my MPT system were ATA drives (Seagate 500GBs) > and I'm remote now and won't be back until Sunday to put one of my > 'good' SAS drives (140 GB Seagates, i.e., real SAS 15K RPM drives, not > "fat SATA" bs drives). > > The numbers were pretty much the same for both FreeBSD and Linux. In > fact, FreeBSD was slightly faster. I won't report the exact numbers > right now, but only mention this as a piece of information that at > least in my case the differences between the OS platform involved is > negligible. This would, at least in my case, rule out issues based > upon different platform access methods and different drivers. > > All of this other discussion, about WCE and what not is nice, but for > all intents and purposes it serves could be moved to *-advocacy. > Thanks for the clarifications! I did mention at some point those were write speeds and reads were just fine and those were either writes to the filesystem or direct access (only on SAS again). Here is what I am planning to do next week when I get the chance: 0) I plan on focusing on the SAS driver tests _only_ since SATA is working as expected so nothing to report there. 1) Look carefully at how the drives are physically connected. Although it feels like if the SATA works fine the SAS should also but I'll check anyway. 2) Boot verbose with "boot -v" and send the dmesg output. mpt driver might give us a clue. 3) Run gstat -abc in a loop for the test duration. Although I would think ctlstat(8) might be more interesting here so I'll run it too for good measure :). Please note that in all tests write caching was enabled as I think this is the default with FBSD 9.1 GENERIC but I'll confirm this with camcontrol(8). I've also seen quite a lot of 'quirks' for tagged command queuing in the source code (/sys/cam/scsi/scps_xtp.c) but a particular one got my attention (thanks to whomever writes good comments in source code :) : /* * Slow when tagged queueing is enabled. Write performance * steadily drops off with more and more concurrent * transactions. Best sequential write performance with * tagged queueing turned off and write caching turned on. * * PR: kern/10398 * Submitted by: Hideaki Okada * Drive: DCAS-34330 w/ "S65A" firmware. * * The drive with the problem had the "S65A" firmware * revision, and has also been reported (by Stephen J. * Roznowski ) for a drive with the "S61A" * firmware revision. * * Although no one has reported problems with the 2 gig * version of the DCAS drive, the assumption is that it * has the same problems as the 4 gig version. Therefore * this quirk entries disables tagged queueing for all * DCAS drives. */ { T_DIRECT, SIP_MEDIA_FIXED, "IBM", "DCAS*", "*" }, /*quirks*/0, /*mintags*/0, /*maxtags*/0 So I looked at the kern/10398 pr and got some feeling of 'deja vu' although the original problem was on FreeBSD 3.1 so its most likely not that but I though I would mention it. The issue described is awfully familiar. Basically the SAS drive (scsi back then) is slow on writes but fast on reads with dd. Could be a coincidence or a ghost from the past who knows... Cheers, Karim.