Date: Thu, 17 Jan 2013 17:12:17 -0500 From: Karim Fodil-Lemelin <fodillemlinkarim@gmail.com> To: freebsd-hackers@freebsd.org Subject: Re: IBM blade server abysmal disk write performances Message-ID: <50F87741.4030201@gmail.com> In-Reply-To: <CAA3ZYrB%2B4LU=YRQsvYB19sE9ywjejyP7Gc4j8fpTGx2LaZm4kw@mail.gmail.com> References: <CAA3ZYrB%2B4LU=YRQsvYB19sE9ywjejyP7Gc4j8fpTGx2LaZm4kw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 16/01/2013 2:48 AM, Dieter BSD wrote: > Karim writes: >> It is quite obvious that something is awfully slow on SAS drives, >> whatever it is and regardless of OS comparison. We swapped the SAS >> drives for SATA and we're seeing much higher speeds. Basically on par >> with what we were expecting (roughly 300 to 400 times faster then what >> we see with SAS...). > Major clue there! According to wikipedia: "Most SAS drives provide > tagged command queuing, while most newer SATA drives provide native > command queuing" [1] > > Note that the driver says "Command Queueing enabled" without > specifying which. If the driver is trying to use SATA's NCQ but > the drive only speaks SCSI's TCQ, that could explain it. Or if > the TCQ isn't working for some other reason. > > See if there are any error messages in dmesg or /var/log. > If not, perhaps the driver has extra debugging you could turn on. > > Get TCQ working and make sure your partitions are aligned on > 4 KiB boundaries (in case the drive actually has 4 KiB sectors), > and you should get the expected performance. > > [1] http://en.wikipedia.org/wiki/Serial_attached_SCSI > Thanks for the wiki article reference it is very interesting and confirms our current setup. I'm mostly thinking about this line: "SAS controllers may connect to SATA devices, either directly connected using native SATA protocol or through SAS expanders using SATA Tunneled Protocol (STP)." The systems is currently put in place using SATA instead of SAS although its using the same interface and backplane connectors and the drives (SATA) show as da0 in BSD _but_ with the SATA drive we get *much* better performances. I am thinking that something fancy in that SAS drive is not being handled correctly by the FreeBSD driver. I am planning to revisit the SAS drive issue at a later point (sometimes next week). Here is some trimmed and hopefully relevant information (from dmesg): SAS drive: mpt0: <LSILogic SAS/SATA Adapter> port 0x1000-0x10ff mem 0x99910000-0x99913fff,0x99900000-0x9990ffff irq 28 at device 0.0 on pci11 mpt0: MPI Version=1.5.20.0 mpt0: Capabilities: ( RAID-0 RAID-1E RAID-1 ) mpt0: 0 Active Volumes (2 Max) mpt0: 0 Hidden Drive Members (14 Max) ... da0 at mpt0 bus 0 scbus0 target 1 lun 0 da0: <IBM-ESXS HUC106030CSS60 D3A6> Fixed Direct Access SCSI-6 device da0: 300.000MB/s transfers da0: Command Queueing enabled da0: 286102MB (585937500 512 byte sectors: 255H 63S/T 36472C) ... GEOM: da0: the primary GPT table is corrupt or invalid. GEOM: da0: using the secondary instead -- recovery strongly advised. SATA drive: mpt0: <LSILogic SAS/SATA Adapter> port 0x1000-0x10ff mem 0x9b910000-0x9b913fff,0x9b900000-0x9b90ffff irq 28 at device 0.0 on pci11 mpt0: MPI Version=1.5.20.0 mpt0: Capabilities: ( RAID-0 RAID-1E RAID-1 ) mpt0: 0 Active Volumes (2 Max) mpt0: 0 Hidden Drive Members (14 Max) ... da0 at mpt0 bus 0 scbus0 target 2 lun 0 da0: <ATA ST91000640NS SN03> Fixed Direct Access SCSI-5 device da0: 300.000MB/s transfers da0: Command Queueing enabled da0: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C) ... GEOM: da0s1: geometry does not match label (16h,63s != 255h,63s). Please let me know if there is anything you would like me to run on the BSD 9.1 system to help diagnose this issue? Thank you, Karim.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?50F87741.4030201>