From owner-freebsd-hackers@FreeBSD.ORG Thu Jan 17 22:12:26 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 97269B47 for ; Thu, 17 Jan 2013 22:12:26 +0000 (UTC) (envelope-from fodillemlinkarim@gmail.com) Received: from mail-ia0-x22c.google.com (ia-in-x022c.1e100.net [IPv6:2607:f8b0:4001:c02::22c]) by mx1.freebsd.org (Postfix) with ESMTP id 6A07488F for ; Thu, 17 Jan 2013 22:12:26 +0000 (UTC) Received: by mail-ia0-f172.google.com with SMTP id u8so846762iag.3 for ; Thu, 17 Jan 2013 14:12:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:message-id:date:from:user-agent:mime-version:to:subject :references:in-reply-to:content-type; bh=LS+j0ANYOad5rgxk0qca4G9OqYZn+1Kqdrb7VPfKhVc=; b=DPoSJoHdEemicAwgoxgmoQN4QQyuEBZlDQeXIQkpO7ViPzQgthUa8nFLtRUkqCO47d M66369x3lgBfC/JGpmNB6ZD4GnJxFc+NWlCxj07kvQO+UTrLhnpK6yUcqBZ/m3Wblp1H YFge610JM5PcFB4oRZdaL0EMgwWSYyBdEjeW14WCxkGc19asmn2W0tu7aGYhi8vrJ6It 2Ob5Y059TUxq6H3jO3RQnvGI/fjQwcIAGo64O4TR4eKrpShQiSoLRESaUcafrRuq5IZe AgvRoDXaTyCac5owA2rsS6T0QYqYbfPX6cgZUZoGxjUjx3V/Lr5LyzTF24HKFrBRwbtm FP4A== X-Received: by 10.50.208.41 with SMTP id mb9mr324254igc.42.1358460745721; Thu, 17 Jan 2013 14:12:25 -0800 (PST) Received: from [192.168.1.73] ([208.85.112.101]) by mx.google.com with ESMTPS id px5sm2399099igc.0.2013.01.17.14.12.23 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 17 Jan 2013 14:12:24 -0800 (PST) Message-ID: <50F87741.4030201@gmail.com> Date: Thu, 17 Jan 2013 17:12:17 -0500 From: Karim Fodil-Lemelin User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130107 Thunderbird/17.0.2 MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Subject: Re: IBM blade server abysmal disk write performances References: In-Reply-To: X-Mailman-Approved-At: Thu, 17 Jan 2013 23:05:40 +0000 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jan 2013 22:12:26 -0000 On 16/01/2013 2:48 AM, Dieter BSD wrote: > Karim writes: >> It is quite obvious that something is awfully slow on SAS drives, >> whatever it is and regardless of OS comparison. We swapped the SAS >> drives for SATA and we're seeing much higher speeds. Basically on par >> with what we were expecting (roughly 300 to 400 times faster then what >> we see with SAS...). > Major clue there! According to wikipedia: "Most SAS drives provide > tagged command queuing, while most newer SATA drives provide native > command queuing" [1] > > Note that the driver says "Command Queueing enabled" without > specifying which. If the driver is trying to use SATA's NCQ but > the drive only speaks SCSI's TCQ, that could explain it. Or if > the TCQ isn't working for some other reason. > > See if there are any error messages in dmesg or /var/log. > If not, perhaps the driver has extra debugging you could turn on. > > Get TCQ working and make sure your partitions are aligned on > 4 KiB boundaries (in case the drive actually has 4 KiB sectors), > and you should get the expected performance. > > [1] http://en.wikipedia.org/wiki/Serial_attached_SCSI > Thanks for the wiki article reference it is very interesting and confirms our current setup. I'm mostly thinking about this line: "SAS controllers may connect to SATA devices, either directly connected using native SATA protocol or through SAS expanders using SATA Tunneled Protocol (STP)." The systems is currently put in place using SATA instead of SAS although its using the same interface and backplane connectors and the drives (SATA) show as da0 in BSD _but_ with the SATA drive we get *much* better performances. I am thinking that something fancy in that SAS drive is not being handled correctly by the FreeBSD driver. I am planning to revisit the SAS drive issue at a later point (sometimes next week). Here is some trimmed and hopefully relevant information (from dmesg): SAS drive: mpt0: port 0x1000-0x10ff mem 0x99910000-0x99913fff,0x99900000-0x9990ffff irq 28 at device 0.0 on pci11 mpt0: MPI Version=1.5.20.0 mpt0: Capabilities: ( RAID-0 RAID-1E RAID-1 ) mpt0: 0 Active Volumes (2 Max) mpt0: 0 Hidden Drive Members (14 Max) ... da0 at mpt0 bus 0 scbus0 target 1 lun 0 da0: Fixed Direct Access SCSI-6 device da0: 300.000MB/s transfers da0: Command Queueing enabled da0: 286102MB (585937500 512 byte sectors: 255H 63S/T 36472C) ... GEOM: da0: the primary GPT table is corrupt or invalid. GEOM: da0: using the secondary instead -- recovery strongly advised. SATA drive: mpt0: port 0x1000-0x10ff mem 0x9b910000-0x9b913fff,0x9b900000-0x9b90ffff irq 28 at device 0.0 on pci11 mpt0: MPI Version=1.5.20.0 mpt0: Capabilities: ( RAID-0 RAID-1E RAID-1 ) mpt0: 0 Active Volumes (2 Max) mpt0: 0 Hidden Drive Members (14 Max) ... da0 at mpt0 bus 0 scbus0 target 2 lun 0 da0: Fixed Direct Access SCSI-5 device da0: 300.000MB/s transfers da0: Command Queueing enabled da0: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C) ... GEOM: da0s1: geometry does not match label (16h,63s != 255h,63s). Please let me know if there is anything you would like me to run on the BSD 9.1 system to help diagnose this issue? Thank you, Karim.