From owner-freebsd-scsi@FreeBSD.ORG Mon Oct 6 16:13:09 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 56A74DE8; Mon, 6 Oct 2014 16:13:09 +0000 (UTC) Received: from mail-wi0-x236.google.com (mail-wi0-x236.google.com [IPv6:2a00:1450:400c:c05::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A7923224; Mon, 6 Oct 2014 16:13:08 +0000 (UTC) Received: by mail-wi0-f182.google.com with SMTP id n3so5127990wiv.15 for ; Mon, 06 Oct 2014 09:13:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type:content-transfer-encoding; bh=OZavf+lodR1q+NhBOqBVSkWxib9r8cfeVXvwji2ujHE=; b=NgSefnYQFWUEZNqI/XgBV3bAdcOFAxFXYOLuQoukftjMkzPA/izc8o+qC0BN+lvlpO n/FMJ9z6hYu1EDqSCETxypN2RGTQx+l2eUE/F5mIuyGmEqU8qvAFcG9Zi3O00l09xjz8 Vm9cKIcluVrowKfkYxyY+DfLkHLk3fM+Eom8PsLiXITT9ynROl+27negHUFr+P45hNeQ sWJQrdWtEvk7sInxxGsnahKlCmNMDaIs1oC34AGIBYF79kimA6MS9ngci9DktYEWWBHa gXFDacRdhcva9834+a9brDQbSOIzKwjC2UEaY2KVSDImLhrBL/KUV3ckh97gYhaUUcTo bnog== MIME-Version: 1.0 X-Received: by 10.180.206.230 with SMTP id lr6mr20886913wic.82.1412611986420; Mon, 06 Oct 2014 09:13:06 -0700 (PDT) Sender: asomers@gmail.com Received: by 10.194.220.227 with HTTP; Mon, 6 Oct 2014 09:13:06 -0700 (PDT) In-Reply-To: References: Date: Mon, 6 Oct 2014 10:13:06 -0600 X-Google-Sender-Auth: zZc4jGKST3Ps0nqv0nNB2PA9SGc Message-ID: Subject: Re: mpr vs mps performance From: Alan Somers To: Borja Marcos , slm@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: FreeBSD-scsi X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Oct 2014 16:13:09 -0000 On Mon, Sep 22, 2014 at 9:39 AM, Borja Marcos wrote: > > Hello, > > I have been playing with the new SAS3 cards supported by the mpr driver, = and I=C2=B4ve found out that they are, in the same hardware > configuration, considerably slower writing data. Moreover, running two si= multaneous "bonnie" benchmarks (I am using SSDs, and one "bonnie" sometime= s hits 100% CPU usage, unable to really saturate the I/O) I see the writing= activity somewhat stalling, with disk bandwidth going from 600 MB/s to aro= und 50 for 20 seconds or so. > > I'd like to know if this matches anyone else's experiences. Also, I can t= ry and make some tests if needed. But for now it seems we will stick to the= SAS2 HBAs. > > > > > The Bonnie results are: > > With mpr driver, SAS3: > > (each bonnie instance, so multiply the results by 2 to get the actual ban= dwidth achieved) > > Seq output: (writing) > Block: 292155 KB/s > Rewrite: 139713 KB/s > Seq input: > Block: 862861 KB/s > > With mps driver: SAS2, again, total is 2x the following figures. > > Seq output: (writing) > Block: 587950 KB/s > Rewrite: 208239 KB/s > Seq. input: (reading) > Block: 842169 KB/s > > > The storage is a ZFS pool with a 9-disk raidz2 vdev, made of Samsung 840= EVO 1 TB SSDs. The pool has been created with an ashift of 12 (zpool appli= ed it thanks to the 4 KB block quirk for these SSDs) > > at scbus0 target 9 lun 0 (pass0,da0) > at scbus0 target 10 lun 0 (pass1,da1) > at scbus0 target 11 lun 0 (pass2,da2) > at scbus0 target 12 lun 0 (pass3,da3) > at scbus0 target 13 lun 0 (pass4,da4) > at scbus0 target 14 lun 0 (pass5,da5) > at scbus0 target 17 lun 0 (pass7,da6) > at scbus0 target 18 lun 0 (pass8,da7) > at scbus0 target 27 lun 0 (pass14,da12= ) > > > > > The mpr card details follow: > Sep 17 09:49:39 elibm kernel: mpr0: port 0x3f00-0x3fff mem = 0x912f0000-0x912fffff irq 32 at device 0.0 on pci17 > Sep 17 09:49:39 elibm kernel: mpr0: IOCFacts : > Sep 17 09:49:39 elibm kernel: MsgVersion: 0x205 > Sep 17 09:49:39 elibm kernel: HeaderVersion: 0x1d00 > Sep 17 09:49:39 elibm kernel: IOCNumber: 0 > Sep 17 09:49:39 elibm kernel: IOCExceptions: 0x0 > Sep 17 09:49:39 elibm kernel: MaxChainDepth: 128 > Sep 17 09:49:39 elibm kernel: NumberOfPorts: 1 > Sep 17 09:49:39 elibm kernel: RequestCredit: 11264 > Sep 17 09:49:39 elibm kernel: ProductID: 0x2221 > Sep 17 09:49:39 elibm kernel: IOCRequestFrameSize: 32 > Sep 17 09:49:39 elibm kernel: MaxInitiators: 1 > Sep 17 09:49:39 elibm kernel: MaxTargets: 1024 > Sep 17 09:49:39 elibm kernel: MaxSasExpanders: 14 > Sep 17 09:49:39 elibm kernel: MaxEnclosures: 15 > Sep 17 09:49:39 elibm kernel: HighPriorityCredit: 60 > Sep 17 09:49:39 elibm kernel: MaxReplyDescriptorPostQueueDepth: 65504 > Sep 17 09:49:39 elibm kernel: ReplyFrameSize: 32 > Sep 17 09:49:39 elibm kernel: MaxVolumes: 0 > Sep 17 09:49:39 elibm kernel: MaxDevHandle: 1047 > Sep 17 09:49:39 elibm kernel: MaxPersistentEntries: 128 > Sep 17 09:49:39 elibm kernel: mpr0: Firmware: 01.00.03.00, Driver: 05.255= .05.00-fbsd > Sep 17 09:49:39 elibm kernel: mpr0: IOCCapabilities: 3a85c > > > And the mps card is a classic: > Sep 22 17:18:24 elibm kernel: mps0: port 0x3f00-0x3fff mem = 0x90ebc000-0x90ebffff,0x912c0000-0x912fffff irq 32 at device 0.0 on pci17 > Sep 22 17:18:24 elibm kernel: mps0: Firmware: 18.00.00.00, Driver: 19.00.= 00.00-fbsd > Sep 22 17:18:24 elibm kernel: mps0: IOCCapabilities: 1285c > > > The connected devices follow. Both use the same hardware (except for the = cables and HBA of course), but currently there's no way to check this with = the SAS3 card, as sas3ircu nor sas3flash detect it on FreeBSD. > > > > # sas2ircu 0 display > LSI Corporation SAS2 IR Configuration Utility. > Version 18.00.00.00 (2013.11.18) > Copyright (c) 2009-2013 LSI Corporation. All rights reserved. > > Read configuration has been initiated for controller 0 > ------------------------------------------------------------------------ > Controller information > ------------------------------------------------------------------------ > Controller type : SAS2008 > BIOS version : 7.35.00.00 > Firmware version : 18.00.00.00 > Channel description : 1 Serial Attached SCSI > Initiator ID : 0 > Maximum physical devices : 255 > Concurrent commands supported : 3432 > Slot : 3 > Segment : 0 > Bus : 17 > Device : 0 > Function : 0 > RAID Support : No > ------------------------------------------------------------------------ > IR Volume information > ------------------------------------------------------------------------ > ------------------------------------------------------------------------ > Physical device information > ------------------------------------------------------------------------ > Initiator at ID #0 > > Device is a Hard disk > Enclosure # : 2 > Slot # : 16 > SAS Address : 5000c50-0-05b5-ce25 > State : Ready (RDY) > Size (in MB)/(in sectors) : 140014/286749479 > Manufacturer : SEAGATE > Model Number : ST9146803SS > Firmware Revision : FS03 > Serial No : 3SD02W5L > GUID : N/A > Protocol : SAS > Drive Type : SAS_HDD > > Device is a Hard disk > Enclosure # : 2 > Slot # : 17 > SAS Address : 5005076-0-3e8e-81a2 > State : Ready (RDY) > Size (in MB)/(in sectors) : 953869/1953525167 > Manufacturer : ATA > Model Number : Samsung SSD 840 > Firmware Revision : BB0Q > Serial No : S1D9NEADA08549F > GUID : N/A > Protocol : SATA > Drive Type : SATA_SSD > > Device is a Hard disk > Enclosure # : 2 > Slot # : 18 > SAS Address : 5005076-0-3e8e-81a3 > State : Ready (RDY) > Size (in MB)/(in sectors) : 953869/1953525167 > Manufacturer : ATA > Model Number : Samsung SSD 840 > Firmware Revision : BB0Q > Serial No : S1D9NEADA08548T > GUID : N/A > Protocol : SATA > Drive Type : SATA_SSD > > Device is a Hard disk > Enclosure # : 2 > Slot # : 19 > SAS Address : 5005076-0-3e8e-81a4 > State : Ready (RDY) > Size (in MB)/(in sectors) : 953869/1953525167 > Manufacturer : ATA > Model Number : Samsung SSD 840 > Firmware Revision : BB0Q > Serial No : S1D9NEADA08568E > GUID : N/A > Protocol : SATA > Drive Type : SATA_SSD > > Device is a Hard disk > Enclosure # : 2 > Slot # : 20 > SAS Address : 5005076-0-3e8e-81a5 > State : Ready (RDY) > Size (in MB)/(in sectors) : 953869/1953525167 > Manufacturer : ATA > Model Number : Samsung SSD 840 > Firmware Revision : BB0Q > Serial No : S1D9NEADA08547X > GUID : N/A > Protocol : SATA > Drive Type : SATA_SSD > > Device is a Hard disk > Enclosure # : 2 > Slot # : 21 > SAS Address : 5005076-0-3e8e-81a6 > State : Ready (RDY) > Size (in MB)/(in sectors) : 953869/1953525167 > Manufacturer : ATA > Model Number : Samsung SSD 840 > Firmware Revision : BB0Q > Serial No : S1D9NEADA08518Y > GUID : N/A > Protocol : SATA > Drive Type : SATA_SSD > > Device is a Hard disk > Enclosure # : 2 > Slot # : 22 > SAS Address : 5005076-0-3e8e-81a7 > State : Ready (RDY) > Size (in MB)/(in sectors) : 953869/1953525167 > Manufacturer : ATA > Model Number : Samsung SSD 840 > Firmware Revision : BB0Q > Serial No : S1D9NEADA08556K > GUID : N/A > Protocol : SATA > Drive Type : SATA_SSD > > Device is a Enclosure services device > Enclosure # : 2 > Slot # : 255 > SAS Address : 5005076-0-3e8e-81b9 > State : Standby (SBY) > Manufacturer : IBM-ESXS > Model Number : SAS EXP BP > Firmware Revision : 61A6 > Serial No : 00000006 > GUID : N/A > Protocol : SAS > Device Type : Enclosure services device > > Device is a Hard disk > Enclosure # : 3 > Slot # : 0 > SAS Address : 5005076-0-3e8e-86e9 > State : Ready (RDY) > Size (in MB)/(in sectors) : 953869/1953525167 > Manufacturer : ATA > Model Number : Samsung SSD 840 > Firmware Revision : BB0Q > Serial No : S1D9NEADA08550R > GUID : N/A > Protocol : SATA > Drive Type : SATA_SSD > > Device is a Hard disk > Enclosure # : 3 > Slot # : 1 > SAS Address : 5005076-0-3e8e-86ea > State : Ready (RDY) > Size (in MB)/(in sectors) : 953869/1953525167 > Manufacturer : ATA > Model Number : Samsung SSD 840 > Firmware Revision : BB0Q > Serial No : S1D9NEADA08911Y > GUID : N/A > Protocol : SATA > Drive Type : SATA_SSD > > Device is a Hard disk > Enclosure # : 3 > Slot # : 2 > SAS Address : 5005076-0-3e8e-86eb > State : Ready (RDY) > Size (in MB)/(in sectors) : 953869/1953525167 > Manufacturer : ATA > Model Number : Samsung SSD 840 > Firmware Revision : BB0Q > Serial No : S1D9NEADA08811L > GUID : N/A > Protocol : SATA > Drive Type : SATA_SSD > > Device is a Hard disk > Enclosure # : 3 > Slot # : 13 > SAS Address : 5000c50-0-05b5-e531 > State : Ready (RDY) > Size (in MB)/(in sectors) : 140014/286749479 > Manufacturer : SEAGATE > Model Number : ST9146803SS > Firmware Revision : FS03 > Serial No : 3SD02STR > GUID : N/A > Protocol : SAS > Drive Type : SAS_HDD > > Device is a Hard disk > Enclosure # : 3 > Slot # : 14 > SAS Address : 5000c50-0-05b5-d489 > State : Ready (RDY) > Size (in MB)/(in sectors) : 140014/286749479 > Manufacturer : SEAGATE > Model Number : ST9146803SS > Firmware Revision : FS03 > Serial No : 3SD02TV1 > GUID : N/A > Protocol : SAS > Drive Type : SAS_HDD > > Device is a Hard disk > Enclosure # : 3 > Slot # : 15 > SAS Address : 5000c50-0-05b5-f0ad > State : Ready (RDY) > Size (in MB)/(in sectors) : 140014/286749479 > Manufacturer : SEAGATE > Model Number : ST9146803SS > Firmware Revision : FS03 > Serial No : 3SD03F4C > GUID : N/A > Protocol : SAS > Drive Type : SAS_HDD > > Device is a Enclosure services device > Enclosure # : 3 > Slot # : 255 > SAS Address : 5005076-0-3e8e-86f9 > State : Standby (SBY) > Manufacturer : IBM-ESXS > Model Number : SAS EXP BP > Firmware Revision : 61A6 > Serial No : 00000006 > GUID : N/A > Protocol : SAS > Device Type : Enclosure services device > ------------------------------------------------------------------------ > Enclosure information > ------------------------------------------------------------------------ > Enclosure# : 1 > Logical ID : 500605b0:07ba2100 > Numslots : 8 > StartSlot : 0 > Enclosure# : 2 > Logical ID : 50050760:3e8e81a0 > Numslots : 25 > StartSlot : 0 > Enclosure# : 3 > Logical ID : 50050760:3e8e86e0 > Numslots : 25 > StartSlot : 0 > ------------------------------------------------------------------------ > SAS2IRCU: Command DISPLAY Completed Successfully. > SAS2IRCU: Utility Completed Successfully. > > > > > _______________________________________________ > freebsd-scsi@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-scsi > To unsubscribe, send any mail to "freebsd-scsi-unsubscribe@freebsd.org" Adding Steve McConnell, who may not be on the freebsd-scsi list. From owner-freebsd-scsi@FreeBSD.ORG Tue Oct 7 00:53:28 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EAFDC62F for ; Tue, 7 Oct 2014 00:53:27 +0000 (UTC) Received: from exprod7og129.obsmtp.com (exprod7og129.obsmtp.com [64.18.2.122]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 893A961C for ; Tue, 7 Oct 2014 00:53:27 +0000 (UTC) Received: from mail-pd0-f180.google.com ([209.85.192.180]) (using TLSv1) by exprod7ob129.postini.com ([64.18.6.12]) with SMTP ID DSNKVDM5hm3F9mhHvnoK5Rur6keAZ7rJi8Nl@postini.com; Mon, 06 Oct 2014 17:53:27 PDT Received: by mail-pd0-f180.google.com with SMTP id fp1so4168816pdb.11 for ; Mon, 06 Oct 2014 17:53:26 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:references:in-reply-to:mime-version :thread-index:date:message-id:subject:to:cc:content-type :content-transfer-encoding; bh=ZLoVmtbR77WQ6BpqvOLDaHQ0Tx8H/YVsKaC8zIoNWQU=; b=JWcHQVemp1kNYr4xqKFQVtyA+fkrTBn88A6RKlh1aF/R8fEljTDbj60WaKctTSn/kX JAYKUOU5S+A2FPKOnyGJqlCJKdnh4DeLr+9/F3OEcddTybylnuU+0djABNk7oFHTbIFT OPmbjfo6btsrlYmZq9Kyyt1W7x4zbjytPPBbc5itptIxGd1j+/tL4EM69Rva6hOl29ZZ BVsfIIRmsFtOKQ+s/wswjTGNHDrX+T46Y5pAvtK0OPKo1+lMkxA0BKYirb8kDo/zvYoy q2w9WLqqWqEOH9Nb/sABRH7aokVED+XSVka/WNMVEvixedT+37IZFk59X3c1mvTqYils bfPA== X-Received: by 10.70.138.9 with SMTP id qm9mr158491pdb.131.1412636125152; Mon, 06 Oct 2014 15:55:25 -0700 (PDT) X-Gm-Message-State: ALoCoQmwi2kEMIvhIHceeJrvFO3xqfNLC/OAuXD2Ymnpsp6/XSbuk5dVSr6w2r5vr2Gc9/hHgElu17vaaYayrwmbJkSFqfWd7tmMIlWrDw0AUqIXiWJhV+yGn/UoF/Chan06fGwf0+zA3t8nh3E3StG2LuUbR7TvSw== X-Received: by 10.70.138.9 with SMTP id qm9mr158482pdb.131.1412636125045; Mon, 06 Oct 2014 15:55:25 -0700 (PDT) From: Stephen Mcconnell References: In-Reply-To: MIME-Version: 1.0 X-Mailer: Microsoft Outlook 14.0 Thread-Index: AQHAvPDqV/NT6Mjmvn39KZFmH6q6QwHpuJfJnDKoYmA= Date: Mon, 6 Oct 2014 16:55:02 -0600 Message-ID: <26cd5d2251523a832e25f393f2379227@mail.gmail.com> Subject: RE: mpr vs mps performance To: Alan Somers , Borja Marcos Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: FreeBSD-scsi X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Oct 2014 00:53:28 -0000 > -----Original Message----- > From: asomers@gmail.com [mailto:asomers@gmail.com] On Behalf Of Alan > Somers > Sent: Monday, October 06, 2014 10:13 AM > To: Borja Marcos; slm@freebsd.org > Cc: FreeBSD-scsi > Subject: Re: mpr vs mps performance > > On Mon, Sep 22, 2014 at 9:39 AM, Borja Marcos wrote: > > > > Hello, > > > > I have been playing with the new SAS3 cards supported by the mpr > > driver, and I=C2=B4ve found out that they are, in the same hardware > > configuration, > considerably slower writing data. Moreover, running two simultaneous > "bonnie" > benchmarks (I am using SSDs, and one "bonnie" sometimes hits 100% CPU > usage, unable to really saturate the I/O) I see the writing activity > somewhat > stalling, with disk bandwidth going from 600 MB/s to around 50 for 20 > seconds > or so. > > > > I'd like to know if this matches anyone else's experiences. Also, I can > > try and > make some tests if needed. But for now it seems we will stick to the SAS2 > HBAs. > > > > > > > > > > The Bonnie results are: > > > > With mpr driver, SAS3: > > > > (each bonnie instance, so multiply the results by 2 to get the actual > > bandwidth achieved) > > > > Seq output: (writing) > > Block: 292155 KB/s > > Rewrite: 139713 KB/s > > Seq input: > > Block: 862861 KB/s > > > > With mps driver: SAS2, again, total is 2x the following figures. > > > > Seq output: (writing) > > Block: 587950 KB/s > > Rewrite: 208239 KB/s > > Seq. input: (reading) > > Block: 842169 KB/s > > > > > > The storage is a ZFS pool with a 9-disk raidz2 vdev, made of Samsung > > 840 EVO 1 TB SSDs. The pool has been created with an ashift of 12 > > (zpool applied it thanks to the 4 KB block quirk for these SSDs) > > > > at scbus0 target 9 lun 0 (pass0,da0) > > at scbus0 target 10 lun 0 (pass1,da1= ) > > at scbus0 target 11 lun 0 (pass2,da2= ) > > at scbus0 target 12 lun 0 (pass3,da3= ) > > at scbus0 target 13 lun 0 (pass4,da4= ) > > at scbus0 target 14 lun 0 (pass5,da5= ) > > at scbus0 target 17 lun 0 (pass7,da6= ) > > at scbus0 target 18 lun 0 (pass8,da7= ) > > at scbus0 target 27 lun 0 > > (pass14,da12) > > > > > > > > > > The mpr card details follow: > > Sep 17 09:49:39 elibm kernel: mpr0: port 0x3f00-0x3fff me= m > 0x912f0000-0x912fffff irq 32 at device 0.0 on pci17 > > Sep 17 09:49:39 elibm kernel: mpr0: IOCFacts : > > Sep 17 09:49:39 elibm kernel: MsgVersion: 0x205 Sep 17 09:49:39 elibm > > kernel: HeaderVersion: 0x1d00 Sep 17 09:49:39 elibm kernel: IOCNumber: > > 0 Sep 17 09:49:39 elibm kernel: IOCExceptions: 0x0 Sep 17 09:49:39 > > elibm kernel: MaxChainDepth: 128 Sep 17 09:49:39 elibm kernel: > > NumberOfPorts: 1 Sep 17 09:49:39 elibm kernel: RequestCredit: 11264 > > Sep 17 09:49:39 elibm kernel: ProductID: 0x2221 Sep 17 09:49:39 elibm > > kernel: IOCRequestFrameSize: 32 Sep 17 09:49:39 elibm kernel: > > MaxInitiators: 1 Sep 17 09:49:39 elibm kernel: MaxTargets: 1024 Sep 17 > > 09:49:39 elibm kernel: MaxSasExpanders: 14 Sep 17 09:49:39 elibm > > kernel: MaxEnclosures: 15 Sep 17 09:49:39 elibm kernel: > > HighPriorityCredit: 60 Sep 17 09:49:39 elibm kernel: > > MaxReplyDescriptorPostQueueDepth: 65504 Sep 17 09:49:39 elibm kernel: > > ReplyFrameSize: 32 Sep 17 09:49:39 elibm kernel: MaxVolumes: 0 Sep 17 > > 09:49:39 elibm kernel: MaxDevHandle: 1047 Sep 17 09:49:39 elibm > > kernel: MaxPersistentEntries: 128 Sep 17 09:49:39 elibm kernel: mpr0: > > Firmware: 01.00.03.00, Driver: 05.255.05.00-fbsd Sep 17 09:49:39 elibm > > kernel: mpr0: IOCCapabilities: > > 3a85c > ndex,HostDisc> > > > > > > And the mps card is a classic: > > Sep 22 17:18:24 elibm kernel: mps0: port 0x3f00-0x3fff > > mem 0x90ebc000-0x90ebffff,0x912c0000-0x912fffff irq 32 at device 0.0 > > on pci17 Sep 22 17:18:24 elibm kernel: mps0: Firmware: 18.00.00.00, > > Driver: 19.00.00.00-fbsd Sep 22 17:18:24 elibm kernel: mps0: > > IOCCapabilities: > > 1285c > isc> > > > > > > The connected devices follow. Both use the same hardware (except for th= e > cables and HBA of course), but currently there's no way to check this wit= h > the > SAS3 card, as sas3ircu nor sas3flash detect it on FreeBSD. > > > > > > > > # sas2ircu 0 display > > LSI Corporation SAS2 IR Configuration Utility. > > Version 18.00.00.00 (2013.11.18) > > Copyright (c) 2009-2013 LSI Corporation. All rights reserved. > > > > Read configuration has been initiated for controller 0 > > ---------------------------------------------------------------------- > > -- > > Controller information > > -----------------------------------------------------------------------= - > > Controller type : SAS2008 > > BIOS version : 7.35.00.00 > > Firmware version : 18.00.00.00 > > Channel description : 1 Serial Attached SCSI > > Initiator ID : 0 > > Maximum physical devices : 255 > > Concurrent commands supported : 3432 > > Slot : 3 > > Segment : 0 > > Bus : 17 > > Device : 0 > > Function : 0 > > RAID Support : No > > ---------------------------------------------------------------------- > > -- > > IR Volume information > > ---------------------------------------------------------------------- > > -- > > ---------------------------------------------------------------------- > > -- > > Physical device information > > ---------------------------------------------------------------------- > > -- > > Initiator at ID #0 > > > > Device is a Hard disk > > Enclosure # : 2 > > Slot # : 16 > > SAS Address : 5000c50-0-05b5-ce25 > > State : Ready (RDY) > > Size (in MB)/(in sectors) : 140014/286749479 > > Manufacturer : SEAGATE > > Model Number : ST9146803SS > > Firmware Revision : FS03 > > Serial No : 3SD02W5L > > GUID : N/A > > Protocol : SAS > > Drive Type : SAS_HDD > > > > Device is a Hard disk > > Enclosure # : 2 > > Slot # : 17 > > SAS Address : 5005076-0-3e8e-81a2 > > State : Ready (RDY) > > Size (in MB)/(in sectors) : 953869/1953525167 > > Manufacturer : ATA > > Model Number : Samsung SSD 840 > > Firmware Revision : BB0Q > > Serial No : S1D9NEADA08549F > > GUID : N/A > > Protocol : SATA > > Drive Type : SATA_SSD > > > > Device is a Hard disk > > Enclosure # : 2 > > Slot # : 18 > > SAS Address : 5005076-0-3e8e-81a3 > > State : Ready (RDY) > > Size (in MB)/(in sectors) : 953869/1953525167 > > Manufacturer : ATA > > Model Number : Samsung SSD 840 > > Firmware Revision : BB0Q > > Serial No : S1D9NEADA08548T > > GUID : N/A > > Protocol : SATA > > Drive Type : SATA_SSD > > > > Device is a Hard disk > > Enclosure # : 2 > > Slot # : 19 > > SAS Address : 5005076-0-3e8e-81a4 > > State : Ready (RDY) > > Size (in MB)/(in sectors) : 953869/1953525167 > > Manufacturer : ATA > > Model Number : Samsung SSD 840 > > Firmware Revision : BB0Q > > Serial No : S1D9NEADA08568E > > GUID : N/A > > Protocol : SATA > > Drive Type : SATA_SSD > > > > Device is a Hard disk > > Enclosure # : 2 > > Slot # : 20 > > SAS Address : 5005076-0-3e8e-81a5 > > State : Ready (RDY) > > Size (in MB)/(in sectors) : 953869/1953525167 > > Manufacturer : ATA > > Model Number : Samsung SSD 840 > > Firmware Revision : BB0Q > > Serial No : S1D9NEADA08547X > > GUID : N/A > > Protocol : SATA > > Drive Type : SATA_SSD > > > > Device is a Hard disk > > Enclosure # : 2 > > Slot # : 21 > > SAS Address : 5005076-0-3e8e-81a6 > > State : Ready (RDY) > > Size (in MB)/(in sectors) : 953869/1953525167 > > Manufacturer : ATA > > Model Number : Samsung SSD 840 > > Firmware Revision : BB0Q > > Serial No : S1D9NEADA08518Y > > GUID : N/A > > Protocol : SATA > > Drive Type : SATA_SSD > > > > Device is a Hard disk > > Enclosure # : 2 > > Slot # : 22 > > SAS Address : 5005076-0-3e8e-81a7 > > State : Ready (RDY) > > Size (in MB)/(in sectors) : 953869/1953525167 > > Manufacturer : ATA > > Model Number : Samsung SSD 840 > > Firmware Revision : BB0Q > > Serial No : S1D9NEADA08556K > > GUID : N/A > > Protocol : SATA > > Drive Type : SATA_SSD > > > > Device is a Enclosure services device > > Enclosure # : 2 > > Slot # : 255 > > SAS Address : 5005076-0-3e8e-81b9 > > State : Standby (SBY) > > Manufacturer : IBM-ESXS > > Model Number : SAS EXP BP > > Firmware Revision : 61A6 > > Serial No : 00000006 > > GUID : N/A > > Protocol : SAS > > Device Type : Enclosure services device > > > > Device is a Hard disk > > Enclosure # : 3 > > Slot # : 0 > > SAS Address : 5005076-0-3e8e-86e9 > > State : Ready (RDY) > > Size (in MB)/(in sectors) : 953869/1953525167 > > Manufacturer : ATA > > Model Number : Samsung SSD 840 > > Firmware Revision : BB0Q > > Serial No : S1D9NEADA08550R > > GUID : N/A > > Protocol : SATA > > Drive Type : SATA_SSD > > > > Device is a Hard disk > > Enclosure # : 3 > > Slot # : 1 > > SAS Address : 5005076-0-3e8e-86ea > > State : Ready (RDY) > > Size (in MB)/(in sectors) : 953869/1953525167 > > Manufacturer : ATA > > Model Number : Samsung SSD 840 > > Firmware Revision : BB0Q > > Serial No : S1D9NEADA08911Y > > GUID : N/A > > Protocol : SATA > > Drive Type : SATA_SSD > > > > Device is a Hard disk > > Enclosure # : 3 > > Slot # : 2 > > SAS Address : 5005076-0-3e8e-86eb > > State : Ready (RDY) > > Size (in MB)/(in sectors) : 953869/1953525167 > > Manufacturer : ATA > > Model Number : Samsung SSD 840 > > Firmware Revision : BB0Q > > Serial No : S1D9NEADA08811L > > GUID : N/A > > Protocol : SATA > > Drive Type : SATA_SSD > > > > Device is a Hard disk > > Enclosure # : 3 > > Slot # : 13 > > SAS Address : 5000c50-0-05b5-e531 > > State : Ready (RDY) > > Size (in MB)/(in sectors) : 140014/286749479 > > Manufacturer : SEAGATE > > Model Number : ST9146803SS > > Firmware Revision : FS03 > > Serial No : 3SD02STR > > GUID : N/A > > Protocol : SAS > > Drive Type : SAS_HDD > > > > Device is a Hard disk > > Enclosure # : 3 > > Slot # : 14 > > SAS Address : 5000c50-0-05b5-d489 > > State : Ready (RDY) > > Size (in MB)/(in sectors) : 140014/286749479 > > Manufacturer : SEAGATE > > Model Number : ST9146803SS > > Firmware Revision : FS03 > > Serial No : 3SD02TV1 > > GUID : N/A > > Protocol : SAS > > Drive Type : SAS_HDD > > > > Device is a Hard disk > > Enclosure # : 3 > > Slot # : 15 > > SAS Address : 5000c50-0-05b5-f0ad > > State : Ready (RDY) > > Size (in MB)/(in sectors) : 140014/286749479 > > Manufacturer : SEAGATE > > Model Number : ST9146803SS > > Firmware Revision : FS03 > > Serial No : 3SD03F4C > > GUID : N/A > > Protocol : SAS > > Drive Type : SAS_HDD > > > > Device is a Enclosure services device > > Enclosure # : 3 > > Slot # : 255 > > SAS Address : 5005076-0-3e8e-86f9 > > State : Standby (SBY) > > Manufacturer : IBM-ESXS > > Model Number : SAS EXP BP > > Firmware Revision : 61A6 > > Serial No : 00000006 > > GUID : N/A > > Protocol : SAS > > Device Type : Enclosure services device > > ---------------------------------------------------------------------- > > -- > > Enclosure information > > -----------------------------------------------------------------------= - > > Enclosure# : 1 > > Logical ID : 500605b0:07ba2100 > > Numslots : 8 > > StartSlot : 0 > > Enclosure# : 2 > > Logical ID : 50050760:3e8e81a0 > > Numslots : 25 > > StartSlot : 0 > > Enclosure# : 3 > > Logical ID : 50050760:3e8e86e0 > > Numslots : 25 > > StartSlot : 0 > > ---------------------------------------------------------------------- > > -- > > SAS2IRCU: Command DISPLAY Completed Successfully. > > SAS2IRCU: Utility Completed Successfully. > > > > > > > > > > _______________________________________________ > > freebsd-scsi@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-scsi > > To unsubscribe, send any mail to "freebsd-scsi-unsubscribe@freebsd.org" > > Adding Steve McConnell, who may not be on the freebsd-scsi list. Right, I'm not on the freebsd-scsi list. I suspect that the reason for thi= s is that most of the Gen3 performance gains come from MSI-x support, but tha= t is not in the mpr driver yet, and MSI-x is not supported in the 2008 card. But, I have a question out to another person and I'm waiting for more info. Steve From owner-freebsd-scsi@FreeBSD.ORG Tue Oct 7 15:33:36 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 94C4BF35 for ; Tue, 7 Oct 2014 15:33:36 +0000 (UTC) Received: from exprod7og126.obsmtp.com (exprod7og126.obsmtp.com [64.18.2.206]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2731D150 for ; Tue, 7 Oct 2014 15:33:35 +0000 (UTC) Received: from mail-pa0-f53.google.com ([209.85.220.53]) (using TLSv1) by exprod7ob126.postini.com ([64.18.6.12]) with SMTP ID DSNKVDQHzkEJF+kW12HzgDKI9bIW0KGGDXSP@postini.com; Tue, 07 Oct 2014 08:33:36 PDT Received: by mail-pa0-f53.google.com with SMTP id kq14so7390111pab.12 for ; Tue, 07 Oct 2014 08:33:34 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:references:in-reply-to:mime-version :thread-index:date:message-id:subject:to:cc:content-type :content-transfer-encoding; bh=rmvOVp56HH9sei0s4UNRrBI5eSkprm63YBD4FIg7CgM=; b=RE6nFHnRzd5jKnO3sg+luwUUlvmNOmknmDhuJkg8XWMGlbMHcruUV5v8mSyXtlFWum MVC+N+sDyg6jGsKbVFwoJR4FdnCQrgk49h37Hzq831Qxb4bZJ/uwk5665hd9wf8oWHSO /8DTAandgql/rBT0CsUohgEHhSoixJK57j/ZhEwPHDyhnBN9XDY2sGLk3DfM+cAFQM0T 8mVwMCPVYnpHs1YMLfmvEF2usqc3sbuGcHJQ/gFFAXiPw6KupaWT5DOT+3x6NXY2TMTF U58QJ4z2aBAqtTYKKvMh+QCfAt7iF5Xyni8Tppf0I7aJQlsDoeg3plKOyI41XoqXqFNo yn6w== X-Received: by 10.70.36.79 with SMTP id o15mr4760279pdj.6.1412695529975; Tue, 07 Oct 2014 08:25:29 -0700 (PDT) X-Gm-Message-State: ALoCoQllbp53xjXObbEDA6bZQkj6wwCaQUHqDfIPz/h+uhLWI618QLD0pIpj57MwAEoIoRg0KH5fSwzWrDdoO4yWAOMXkxkkZS/B3mO04s0Rw8rADNLjp1VKJBfUbQ181CAw9RdtnHroh8OtjrDzE6q2A7jdM3MyOQ== X-Received: by 10.70.36.79 with SMTP id o15mr4760262pdj.6.1412695529804; Tue, 07 Oct 2014 08:25:29 -0700 (PDT) From: Stephen Mcconnell References: 26cd5d2251523a832e25f393f2379227@mail.gmail.com In-Reply-To: 26cd5d2251523a832e25f393f2379227@mail.gmail.com MIME-Version: 1.0 X-Mailer: Microsoft Outlook 14.0 Thread-Index: AQHAvPDqV/NT6Mjmvn39KZFmH6q6QwHpuJfJnDKoYmCAARS+kA== Date: Tue, 7 Oct 2014 09:25:29 -0600 Message-ID: Subject: RE: mpr vs mps performance To: Alan Somers , Borja Marcos Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: FreeBSD-scsi X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Oct 2014 15:33:36 -0000 > -----Original Message----- > From: Stephen Mcconnell [mailto:stephen.mcconnell@avagotech.com] > Sent: Monday, October 06, 2014 4:55 PM > To: 'Alan Somers'; 'Borja Marcos' > Cc: 'FreeBSD-scsi' > Subject: RE: mpr vs mps performance > > > > > -----Original Message----- > > From: asomers@gmail.com [mailto:asomers@gmail.com] On Behalf Of Alan > > Somers > > Sent: Monday, October 06, 2014 10:13 AM > > To: Borja Marcos; slm@freebsd.org > > Cc: FreeBSD-scsi > > Subject: Re: mpr vs mps performance > > > > On Mon, Sep 22, 2014 at 9:39 AM, Borja Marcos wrote= : > > > > > > Hello, > > > > > > I have been playing with the new SAS3 cards supported by the mpr > > > driver, and I=C2=B4ve found out that they are, in the same hardware > > > configuration, > > considerably slower writing data. Moreover, running two simultaneous > "bonnie" > > benchmarks (I am using SSDs, and one "bonnie" sometimes hits 100% CPU > > usage, unable to really saturate the I/O) I see the writing activity > > somewhat stalling, with disk bandwidth going from 600 MB/s to around > > 50 for 20 seconds or so. > > > > > > I'd like to know if this matches anyone else's experiences. Also, I > > > can try and > > make some tests if needed. But for now it seems we will stick to the > > SAS2 > HBAs. > > > > > > > > > > > > > > > The Bonnie results are: > > > > > > With mpr driver, SAS3: > > > > > > (each bonnie instance, so multiply the results by 2 to get the > > > actual bandwidth achieved) > > > > > > Seq output: (writing) > > > Block: 292155 KB/s > > > Rewrite: 139713 KB/s > > > Seq input: > > > Block: 862861 KB/s > > > > > > With mps driver: SAS2, again, total is 2x the following figures. > > > > > > Seq output: (writing) > > > Block: 587950 KB/s > > > Rewrite: 208239 KB/s > > > Seq. input: (reading) > > > Block: 842169 KB/s > > > > > > > > > The storage is a ZFS pool with a 9-disk raidz2 vdev, made of > > > Samsung > > > 840 EVO 1 TB SSDs. The pool has been created with an ashift of 12 > > > (zpool applied it thanks to the 4 KB block quirk for these SSDs) > > > > > > at scbus0 target 9 lun 0 > > > (pass0,da0) > > > at scbus0 target 10 lun 0 > > > (pass1,da1) > > > at scbus0 target 11 lun 0 > > > (pass2,da2) > > > at scbus0 target 12 lun 0 > > > (pass3,da3) > > > at scbus0 target 13 lun 0 > > > (pass4,da4) > > > at scbus0 target 14 lun 0 > > > (pass5,da5) > > > at scbus0 target 17 lun 0 > > > (pass7,da6) > > > at scbus0 target 18 lun 0 > > > (pass8,da7) > > > at scbus0 target 27 lun 0 > > > (pass14,da12) > > > > > > > > > > > > > > > The mpr card details follow: > > > Sep 17 09:49:39 elibm kernel: mpr0: port 0x3f00-0x3fff > > > mem > > 0x912f0000-0x912fffff irq 32 at device 0.0 on pci17 > > > Sep 17 09:49:39 elibm kernel: mpr0: IOCFacts : > > > Sep 17 09:49:39 elibm kernel: MsgVersion: 0x205 Sep 17 09:49:39 > > > elibm > > > kernel: HeaderVersion: 0x1d00 Sep 17 09:49:39 elibm kernel: IOCNumber= : > > > 0 Sep 17 09:49:39 elibm kernel: IOCExceptions: 0x0 Sep 17 09:49:39 > > > elibm kernel: MaxChainDepth: 128 Sep 17 09:49:39 elibm kernel: > > > NumberOfPorts: 1 Sep 17 09:49:39 elibm kernel: RequestCredit: 11264 > > > Sep 17 09:49:39 elibm kernel: ProductID: 0x2221 Sep 17 09:49:39 > > > elibm > > > kernel: IOCRequestFrameSize: 32 Sep 17 09:49:39 elibm kernel: > > > MaxInitiators: 1 Sep 17 09:49:39 elibm kernel: MaxTargets: 1024 Sep > > > 17 > > > 09:49:39 elibm kernel: MaxSasExpanders: 14 Sep 17 09:49:39 elibm > > > kernel: MaxEnclosures: 15 Sep 17 09:49:39 elibm kernel: > > > HighPriorityCredit: 60 Sep 17 09:49:39 elibm kernel: > > > MaxReplyDescriptorPostQueueDepth: 65504 Sep 17 09:49:39 elibm kernel: > > > ReplyFrameSize: 32 Sep 17 09:49:39 elibm kernel: MaxVolumes: 0 Sep > > > 17 > > > 09:49:39 elibm kernel: MaxDevHandle: 1047 Sep 17 09:49:39 elibm > > > kernel: MaxPersistentEntries: 128 Sep 17 09:49:39 elibm kernel: mpr0: > > > Firmware: 01.00.03.00, Driver: 05.255.05.00-fbsd Sep 17 09:49:39 > > > elibm > > > kernel: mpr0: IOCCapabilities: > > > 3a85c > > XI > > > ndex,HostDisc> > > > > > > > > > And the mps card is a classic: > > > Sep 22 17:18:24 elibm kernel: mps0: port 0x3f00-0x3fff > > > mem 0x90ebc000-0x90ebffff,0x912c0000-0x912fffff irq 32 at device 0.0 > > > on pci17 Sep 22 17:18:24 elibm kernel: mps0: Firmware: 18.00.00.00, > > > Driver: 19.00.00.00-fbsd Sep 22 17:18:24 elibm kernel: mps0: > > > IOCCapabilities: > > > 1285c > > tD > > > isc> > > > > > > > > > The connected devices follow. Both use the same hardware (except for > > > the > > cables and HBA of course), but currently there's no way to check this > > with the > > SAS3 card, as sas3ircu nor sas3flash detect it on FreeBSD. > > > > > > > > > > > > # sas2ircu 0 display > > > LSI Corporation SAS2 IR Configuration Utility. > > > Version 18.00.00.00 (2013.11.18) > > > Copyright (c) 2009-2013 LSI Corporation. All rights reserved. > > > > > > Read configuration has been initiated for controller 0 > > > -------------------------------------------------------------------- > > > -- > > > -- > > > Controller information > > > ---------------------------------------------------------------------= --- > > > Controller type : SAS2008 > > > BIOS version : 7.35.00.00 > > > Firmware version : 18.00.00.00 > > > Channel description : 1 Serial Attached SCSI > > > Initiator ID : 0 > > > Maximum physical devices : 255 > > > Concurrent commands supported : 3432 > > > Slot : 3 > > > Segment : 0 > > > Bus : 17 > > > Device : 0 > > > Function : 0 > > > RAID Support : No > > > -------------------------------------------------------------------- > > > -- > > > -- > > > IR Volume information > > > -------------------------------------------------------------------- > > > -- > > > -- > > > -------------------------------------------------------------------- > > > -- > > > -- > > > Physical device information > > > -------------------------------------------------------------------- > > > -- > > > -- > > > Initiator at ID #0 > > > > > > Device is a Hard disk > > > Enclosure # : 2 > > > Slot # : 16 > > > SAS Address : 5000c50-0-05b5-ce25 > > > State : Ready (RDY) > > > Size (in MB)/(in sectors) : 140014/286749479 > > > Manufacturer : SEAGATE > > > Model Number : ST9146803SS > > > Firmware Revision : FS03 > > > Serial No : 3SD02W5L > > > GUID : N/A > > > Protocol : SAS > > > Drive Type : SAS_HDD > > > > > > Device is a Hard disk > > > Enclosure # : 2 > > > Slot # : 17 > > > SAS Address : 5005076-0-3e8e-81a2 > > > State : Ready (RDY) > > > Size (in MB)/(in sectors) : 953869/1953525167 > > > Manufacturer : ATA > > > Model Number : Samsung SSD 840 > > > Firmware Revision : BB0Q > > > Serial No : S1D9NEADA08549F > > > GUID : N/A > > > Protocol : SATA > > > Drive Type : SATA_SSD > > > > > > Device is a Hard disk > > > Enclosure # : 2 > > > Slot # : 18 > > > SAS Address : 5005076-0-3e8e-81a3 > > > State : Ready (RDY) > > > Size (in MB)/(in sectors) : 953869/1953525167 > > > Manufacturer : ATA > > > Model Number : Samsung SSD 840 > > > Firmware Revision : BB0Q > > > Serial No : S1D9NEADA08548T > > > GUID : N/A > > > Protocol : SATA > > > Drive Type : SATA_SSD > > > > > > Device is a Hard disk > > > Enclosure # : 2 > > > Slot # : 19 > > > SAS Address : 5005076-0-3e8e-81a4 > > > State : Ready (RDY) > > > Size (in MB)/(in sectors) : 953869/1953525167 > > > Manufacturer : ATA > > > Model Number : Samsung SSD 840 > > > Firmware Revision : BB0Q > > > Serial No : S1D9NEADA08568E > > > GUID : N/A > > > Protocol : SATA > > > Drive Type : SATA_SSD > > > > > > Device is a Hard disk > > > Enclosure # : 2 > > > Slot # : 20 > > > SAS Address : 5005076-0-3e8e-81a5 > > > State : Ready (RDY) > > > Size (in MB)/(in sectors) : 953869/1953525167 > > > Manufacturer : ATA > > > Model Number : Samsung SSD 840 > > > Firmware Revision : BB0Q > > > Serial No : S1D9NEADA08547X > > > GUID : N/A > > > Protocol : SATA > > > Drive Type : SATA_SSD > > > > > > Device is a Hard disk > > > Enclosure # : 2 > > > Slot # : 21 > > > SAS Address : 5005076-0-3e8e-81a6 > > > State : Ready (RDY) > > > Size (in MB)/(in sectors) : 953869/1953525167 > > > Manufacturer : ATA > > > Model Number : Samsung SSD 840 > > > Firmware Revision : BB0Q > > > Serial No : S1D9NEADA08518Y > > > GUID : N/A > > > Protocol : SATA > > > Drive Type : SATA_SSD > > > > > > Device is a Hard disk > > > Enclosure # : 2 > > > Slot # : 22 > > > SAS Address : 5005076-0-3e8e-81a7 > > > State : Ready (RDY) > > > Size (in MB)/(in sectors) : 953869/1953525167 > > > Manufacturer : ATA > > > Model Number : Samsung SSD 840 > > > Firmware Revision : BB0Q > > > Serial No : S1D9NEADA08556K > > > GUID : N/A > > > Protocol : SATA > > > Drive Type : SATA_SSD > > > > > > Device is a Enclosure services device > > > Enclosure # : 2 > > > Slot # : 255 > > > SAS Address : 5005076-0-3e8e-81b9 > > > State : Standby (SBY) > > > Manufacturer : IBM-ESXS > > > Model Number : SAS EXP BP > > > Firmware Revision : 61A6 > > > Serial No : 00000006 > > > GUID : N/A > > > Protocol : SAS > > > Device Type : Enclosure services device > > > > > > Device is a Hard disk > > > Enclosure # : 3 > > > Slot # : 0 > > > SAS Address : 5005076-0-3e8e-86e9 > > > State : Ready (RDY) > > > Size (in MB)/(in sectors) : 953869/1953525167 > > > Manufacturer : ATA > > > Model Number : Samsung SSD 840 > > > Firmware Revision : BB0Q > > > Serial No : S1D9NEADA08550R > > > GUID : N/A > > > Protocol : SATA > > > Drive Type : SATA_SSD > > > > > > Device is a Hard disk > > > Enclosure # : 3 > > > Slot # : 1 > > > SAS Address : 5005076-0-3e8e-86ea > > > State : Ready (RDY) > > > Size (in MB)/(in sectors) : 953869/1953525167 > > > Manufacturer : ATA > > > Model Number : Samsung SSD 840 > > > Firmware Revision : BB0Q > > > Serial No : S1D9NEADA08911Y > > > GUID : N/A > > > Protocol : SATA > > > Drive Type : SATA_SSD > > > > > > Device is a Hard disk > > > Enclosure # : 3 > > > Slot # : 2 > > > SAS Address : 5005076-0-3e8e-86eb > > > State : Ready (RDY) > > > Size (in MB)/(in sectors) : 953869/1953525167 > > > Manufacturer : ATA > > > Model Number : Samsung SSD 840 > > > Firmware Revision : BB0Q > > > Serial No : S1D9NEADA08811L > > > GUID : N/A > > > Protocol : SATA > > > Drive Type : SATA_SSD > > > > > > Device is a Hard disk > > > Enclosure # : 3 > > > Slot # : 13 > > > SAS Address : 5000c50-0-05b5-e531 > > > State : Ready (RDY) > > > Size (in MB)/(in sectors) : 140014/286749479 > > > Manufacturer : SEAGATE > > > Model Number : ST9146803SS > > > Firmware Revision : FS03 > > > Serial No : 3SD02STR > > > GUID : N/A > > > Protocol : SAS > > > Drive Type : SAS_HDD > > > > > > Device is a Hard disk > > > Enclosure # : 3 > > > Slot # : 14 > > > SAS Address : 5000c50-0-05b5-d489 > > > State : Ready (RDY) > > > Size (in MB)/(in sectors) : 140014/286749479 > > > Manufacturer : SEAGATE > > > Model Number : ST9146803SS > > > Firmware Revision : FS03 > > > Serial No : 3SD02TV1 > > > GUID : N/A > > > Protocol : SAS > > > Drive Type : SAS_HDD > > > > > > Device is a Hard disk > > > Enclosure # : 3 > > > Slot # : 15 > > > SAS Address : 5000c50-0-05b5-f0ad > > > State : Ready (RDY) > > > Size (in MB)/(in sectors) : 140014/286749479 > > > Manufacturer : SEAGATE > > > Model Number : ST9146803SS > > > Firmware Revision : FS03 > > > Serial No : 3SD03F4C > > > GUID : N/A > > > Protocol : SAS > > > Drive Type : SAS_HDD > > > > > > Device is a Enclosure services device > > > Enclosure # : 3 > > > Slot # : 255 > > > SAS Address : 5005076-0-3e8e-86f9 > > > State : Standby (SBY) > > > Manufacturer : IBM-ESXS > > > Model Number : SAS EXP BP > > > Firmware Revision : 61A6 > > > Serial No : 00000006 > > > GUID : N/A > > > Protocol : SAS > > > Device Type : Enclosure services device > > > -------------------------------------------------------------------- > > > -- > > > -- > > > Enclosure information > > > ---------------------------------------------------------------------= --- > > > Enclosure# : 1 > > > Logical ID : 500605b0:07ba2100 > > > Numslots : 8 > > > StartSlot : 0 > > > Enclosure# : 2 > > > Logical ID : 50050760:3e8e81a0 > > > Numslots : 25 > > > StartSlot : 0 > > > Enclosure# : 3 > > > Logical ID : 50050760:3e8e86e0 > > > Numslots : 25 > > > StartSlot : 0 > > > -------------------------------------------------------------------- > > > -- > > > -- > > > SAS2IRCU: Command DISPLAY Completed Successfully. > > > SAS2IRCU: Utility Completed Successfully. > > > > > > > > > > > > > > > _______________________________________________ > > > freebsd-scsi@freebsd.org mailing list > > > http://lists.freebsd.org/mailman/listinfo/freebsd-scsi > > > To unsubscribe, send any mail to > > > "freebsd-scsi-unsubscribe@freebsd.org" > > > > Adding Steve McConnell, who may not be on the freebsd-scsi list. > > Right, I'm not on the freebsd-scsi list. I suspect that the reason for > this is that > most of the Gen3 performance gains come from MSI-x support, but that is > not in > the mpr driver yet, and MSI-x is not supported in the 2008 card. But, I > have a > question out to another person and I'm waiting for more info. > > Steve Borja, can you upgrade your card to the latest FW. Your version is really old (1.00.03.00). We should be up to 7.00.00.00 now. There could be a lot of fixes that would affect performance. Steve From owner-freebsd-scsi@FreeBSD.ORG Tue Oct 7 16:11:54 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 956E128A; Tue, 7 Oct 2014 16:11:54 +0000 (UTC) Received: from mail-wi0-x22a.google.com (mail-wi0-x22a.google.com [IPv6:2a00:1450:400c:c05::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E804585C; Tue, 7 Oct 2014 16:11:53 +0000 (UTC) Received: by mail-wi0-f170.google.com with SMTP id hi2so10209444wib.3 for ; Tue, 07 Oct 2014 09:11:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type:content-transfer-encoding; bh=ViDEj39oTBF9iyjap9nlu8k5MFI8lFCTXNsi4tlEq30=; b=vdp9TszxwVZw4eh7kCSoQGusQshZjymP3v9A4ig37yZiv2ZaMqrHS1pDouMwdeRor+ sfVS1GtflnDRKt0o4kN9zmbd4tgGhkwShggHCnV6/msy3AErjnMkYHg8YRo0n/zGSeku T3+a7IXST/FaAN/dqq0MhLpdxKYKLINUH6DT1rf+CY5nxXzK1Y+OmHxRhnsxKSBo3BG1 vctXK44J54sOBZpLirpXPKqCzfOCiMrgqcZnp2ep5HILKWzZbJbGDPnqkeNMqPqTqU8P rrT9QQc3rpWyjE/NpHXBA8EOiFzLKkPMhNogcNBfW0Eta341PJ6NqXUNT/q7DwPxkc9w 6QxQ== MIME-Version: 1.0 X-Received: by 10.181.27.132 with SMTP id jg4mr28988334wid.82.1412698312080; Tue, 07 Oct 2014 09:11:52 -0700 (PDT) Sender: asomers@gmail.com Received: by 10.194.220.227 with HTTP; Tue, 7 Oct 2014 09:11:52 -0700 (PDT) In-Reply-To: References: Date: Tue, 7 Oct 2014 10:11:52 -0600 X-Google-Sender-Auth: klHiADk3rFgxjYBA135XtgpzfoQ Message-ID: Subject: Re: mpr vs mps performance From: Alan Somers To: Stephen Mcconnell Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: FreeBSD-scsi X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Oct 2014 16:11:54 -0000 On Tue, Oct 7, 2014 at 9:25 AM, Stephen Mcconnell wrote: >> -----Original Message----- >> From: Stephen Mcconnell [mailto:stephen.mcconnell@avagotech.com] >> Sent: Monday, October 06, 2014 4:55 PM >> To: 'Alan Somers'; 'Borja Marcos' >> Cc: 'FreeBSD-scsi' >> Subject: RE: mpr vs mps performance >> >> >> >> > -----Original Message----- >> > From: asomers@gmail.com [mailto:asomers@gmail.com] On Behalf Of Alan >> > Somers >> > Sent: Monday, October 06, 2014 10:13 AM >> > To: Borja Marcos; slm@freebsd.org >> > Cc: FreeBSD-scsi >> > Subject: Re: mpr vs mps performance >> > >> > On Mon, Sep 22, 2014 at 9:39 AM, Borja Marcos wrot= e: >> > > >> > > Hello, >> > > >> > > I have been playing with the new SAS3 cards supported by the mpr >> > > driver, and I=C2=B4ve found out that they are, in the same hardware >> > > configuration, >> > considerably slower writing data. Moreover, running two simultaneous >> "bonnie" >> > benchmarks (I am using SSDs, and one "bonnie" sometimes hits 100% CPU >> > usage, unable to really saturate the I/O) I see the writing activity >> > somewhat stalling, with disk bandwidth going from 600 MB/s to around >> > 50 for 20 seconds or so. >> > > >> > > I'd like to know if this matches anyone else's experiences. Also, I >> > > can try and >> > make some tests if needed. But for now it seems we will stick to the >> > SAS2 >> HBAs. >> > > >> > > >> > > >> > > >> > > The Bonnie results are: >> > > >> > > With mpr driver, SAS3: >> > > >> > > (each bonnie instance, so multiply the results by 2 to get the >> > > actual bandwidth achieved) >> > > >> > > Seq output: (writing) >> > > Block: 292155 KB/s >> > > Rewrite: 139713 KB/s >> > > Seq input: >> > > Block: 862861 KB/s >> > > >> > > With mps driver: SAS2, again, total is 2x the following figures. >> > > >> > > Seq output: (writing) >> > > Block: 587950 KB/s >> > > Rewrite: 208239 KB/s >> > > Seq. input: (reading) >> > > Block: 842169 KB/s >> > > >> > > >> > > The storage is a ZFS pool with a 9-disk raidz2 vdev, made of >> > > Samsung >> > > 840 EVO 1 TB SSDs. The pool has been created with an ashift of 12 >> > > (zpool applied it thanks to the 4 KB block quirk for these SSDs) >> > > >> > > at scbus0 target 9 lun 0 >> > > (pass0,da0) >> > > at scbus0 target 10 lun 0 >> > > (pass1,da1) >> > > at scbus0 target 11 lun 0 >> > > (pass2,da2) >> > > at scbus0 target 12 lun 0 >> > > (pass3,da3) >> > > at scbus0 target 13 lun 0 >> > > (pass4,da4) >> > > at scbus0 target 14 lun 0 >> > > (pass5,da5) >> > > at scbus0 target 17 lun 0 >> > > (pass7,da6) >> > > at scbus0 target 18 lun 0 >> > > (pass8,da7) >> > > at scbus0 target 27 lun 0 >> > > (pass14,da12) >> > > >> > > >> > > >> > > >> > > The mpr card details follow: >> > > Sep 17 09:49:39 elibm kernel: mpr0: port 0x3f00-0x3fff >> > > mem >> > 0x912f0000-0x912fffff irq 32 at device 0.0 on pci17 >> > > Sep 17 09:49:39 elibm kernel: mpr0: IOCFacts : >> > > Sep 17 09:49:39 elibm kernel: MsgVersion: 0x205 Sep 17 09:49:39 >> > > elibm >> > > kernel: HeaderVersion: 0x1d00 Sep 17 09:49:39 elibm kernel: IOCNumbe= r: >> > > 0 Sep 17 09:49:39 elibm kernel: IOCExceptions: 0x0 Sep 17 09:49:39 >> > > elibm kernel: MaxChainDepth: 128 Sep 17 09:49:39 elibm kernel: >> > > NumberOfPorts: 1 Sep 17 09:49:39 elibm kernel: RequestCredit: 11264 >> > > Sep 17 09:49:39 elibm kernel: ProductID: 0x2221 Sep 17 09:49:39 >> > > elibm >> > > kernel: IOCRequestFrameSize: 32 Sep 17 09:49:39 elibm kernel: >> > > MaxInitiators: 1 Sep 17 09:49:39 elibm kernel: MaxTargets: 1024 Sep >> > > 17 >> > > 09:49:39 elibm kernel: MaxSasExpanders: 14 Sep 17 09:49:39 elibm >> > > kernel: MaxEnclosures: 15 Sep 17 09:49:39 elibm kernel: >> > > HighPriorityCredit: 60 Sep 17 09:49:39 elibm kernel: >> > > MaxReplyDescriptorPostQueueDepth: 65504 Sep 17 09:49:39 elibm kernel= : >> > > ReplyFrameSize: 32 Sep 17 09:49:39 elibm kernel: MaxVolumes: 0 Sep >> > > 17 >> > > 09:49:39 elibm kernel: MaxDevHandle: 1047 Sep 17 09:49:39 elibm >> > > kernel: MaxPersistentEntries: 128 Sep 17 09:49:39 elibm kernel: mpr0= : >> > > Firmware: 01.00.03.00, Driver: 05.255.05.00-fbsd Sep 17 09:49:39 >> > > elibm >> > > kernel: mpr0: IOCCapabilities: >> > > 3a85c> > > XI >> > > ndex,HostDisc> >> > > >> > > >> > > And the mps card is a classic: >> > > Sep 22 17:18:24 elibm kernel: mps0: port 0x3f00-0x3fff >> > > mem 0x90ebc000-0x90ebffff,0x912c0000-0x912fffff irq 32 at device 0.0 >> > > on pci17 Sep 22 17:18:24 elibm kernel: mps0: Firmware: 18.00.00.00, >> > > Driver: 19.00.00.00-fbsd Sep 22 17:18:24 elibm kernel: mps0: >> > > IOCCapabilities: >> > > 1285c> > > tD >> > > isc> >> > > >> > > >> > > The connected devices follow. Both use the same hardware (except for >> > > the >> > cables and HBA of course), but currently there's no way to check this >> > with the >> > SAS3 card, as sas3ircu nor sas3flash detect it on FreeBSD. >> > > >> > > >> > > >> > > # sas2ircu 0 display >> > > LSI Corporation SAS2 IR Configuration Utility. >> > > Version 18.00.00.00 (2013.11.18) >> > > Copyright (c) 2009-2013 LSI Corporation. All rights reserved. >> > > >> > > Read configuration has been initiated for controller 0 >> > > -------------------------------------------------------------------- >> > > -- >> > > -- >> > > Controller information >> > > --------------------------------------------------------------------= ---- >> > > Controller type : SAS2008 >> > > BIOS version : 7.35.00.00 >> > > Firmware version : 18.00.00.00 >> > > Channel description : 1 Serial Attached SCSI >> > > Initiator ID : 0 >> > > Maximum physical devices : 255 >> > > Concurrent commands supported : 3432 >> > > Slot : 3 >> > > Segment : 0 >> > > Bus : 17 >> > > Device : 0 >> > > Function : 0 >> > > RAID Support : No >> > > -------------------------------------------------------------------- >> > > -- >> > > -- >> > > IR Volume information >> > > -------------------------------------------------------------------- >> > > -- >> > > -- >> > > -------------------------------------------------------------------- >> > > -- >> > > -- >> > > Physical device information >> > > -------------------------------------------------------------------- >> > > -- >> > > -- >> > > Initiator at ID #0 >> > > >> > > Device is a Hard disk >> > > Enclosure # : 2 >> > > Slot # : 16 >> > > SAS Address : 5000c50-0-05b5-ce25 >> > > State : Ready (RDY) >> > > Size (in MB)/(in sectors) : 140014/286749479 >> > > Manufacturer : SEAGATE >> > > Model Number : ST9146803SS >> > > Firmware Revision : FS03 >> > > Serial No : 3SD02W5L >> > > GUID : N/A >> > > Protocol : SAS >> > > Drive Type : SAS_HDD >> > > >> > > Device is a Hard disk >> > > Enclosure # : 2 >> > > Slot # : 17 >> > > SAS Address : 5005076-0-3e8e-81a2 >> > > State : Ready (RDY) >> > > Size (in MB)/(in sectors) : 953869/1953525167 >> > > Manufacturer : ATA >> > > Model Number : Samsung SSD 840 >> > > Firmware Revision : BB0Q >> > > Serial No : S1D9NEADA08549F >> > > GUID : N/A >> > > Protocol : SATA >> > > Drive Type : SATA_SSD >> > > >> > > Device is a Hard disk >> > > Enclosure # : 2 >> > > Slot # : 18 >> > > SAS Address : 5005076-0-3e8e-81a3 >> > > State : Ready (RDY) >> > > Size (in MB)/(in sectors) : 953869/1953525167 >> > > Manufacturer : ATA >> > > Model Number : Samsung SSD 840 >> > > Firmware Revision : BB0Q >> > > Serial No : S1D9NEADA08548T >> > > GUID : N/A >> > > Protocol : SATA >> > > Drive Type : SATA_SSD >> > > >> > > Device is a Hard disk >> > > Enclosure # : 2 >> > > Slot # : 19 >> > > SAS Address : 5005076-0-3e8e-81a4 >> > > State : Ready (RDY) >> > > Size (in MB)/(in sectors) : 953869/1953525167 >> > > Manufacturer : ATA >> > > Model Number : Samsung SSD 840 >> > > Firmware Revision : BB0Q >> > > Serial No : S1D9NEADA08568E >> > > GUID : N/A >> > > Protocol : SATA >> > > Drive Type : SATA_SSD >> > > >> > > Device is a Hard disk >> > > Enclosure # : 2 >> > > Slot # : 20 >> > > SAS Address : 5005076-0-3e8e-81a5 >> > > State : Ready (RDY) >> > > Size (in MB)/(in sectors) : 953869/1953525167 >> > > Manufacturer : ATA >> > > Model Number : Samsung SSD 840 >> > > Firmware Revision : BB0Q >> > > Serial No : S1D9NEADA08547X >> > > GUID : N/A >> > > Protocol : SATA >> > > Drive Type : SATA_SSD >> > > >> > > Device is a Hard disk >> > > Enclosure # : 2 >> > > Slot # : 21 >> > > SAS Address : 5005076-0-3e8e-81a6 >> > > State : Ready (RDY) >> > > Size (in MB)/(in sectors) : 953869/1953525167 >> > > Manufacturer : ATA >> > > Model Number : Samsung SSD 840 >> > > Firmware Revision : BB0Q >> > > Serial No : S1D9NEADA08518Y >> > > GUID : N/A >> > > Protocol : SATA >> > > Drive Type : SATA_SSD >> > > >> > > Device is a Hard disk >> > > Enclosure # : 2 >> > > Slot # : 22 >> > > SAS Address : 5005076-0-3e8e-81a7 >> > > State : Ready (RDY) >> > > Size (in MB)/(in sectors) : 953869/1953525167 >> > > Manufacturer : ATA >> > > Model Number : Samsung SSD 840 >> > > Firmware Revision : BB0Q >> > > Serial No : S1D9NEADA08556K >> > > GUID : N/A >> > > Protocol : SATA >> > > Drive Type : SATA_SSD >> > > >> > > Device is a Enclosure services device >> > > Enclosure # : 2 >> > > Slot # : 255 >> > > SAS Address : 5005076-0-3e8e-81b9 >> > > State : Standby (SBY) >> > > Manufacturer : IBM-ESXS >> > > Model Number : SAS EXP BP >> > > Firmware Revision : 61A6 >> > > Serial No : 00000006 >> > > GUID : N/A >> > > Protocol : SAS >> > > Device Type : Enclosure services devic= e >> > > >> > > Device is a Hard disk >> > > Enclosure # : 3 >> > > Slot # : 0 >> > > SAS Address : 5005076-0-3e8e-86e9 >> > > State : Ready (RDY) >> > > Size (in MB)/(in sectors) : 953869/1953525167 >> > > Manufacturer : ATA >> > > Model Number : Samsung SSD 840 >> > > Firmware Revision : BB0Q >> > > Serial No : S1D9NEADA08550R >> > > GUID : N/A >> > > Protocol : SATA >> > > Drive Type : SATA_SSD >> > > >> > > Device is a Hard disk >> > > Enclosure # : 3 >> > > Slot # : 1 >> > > SAS Address : 5005076-0-3e8e-86ea >> > > State : Ready (RDY) >> > > Size (in MB)/(in sectors) : 953869/1953525167 >> > > Manufacturer : ATA >> > > Model Number : Samsung SSD 840 >> > > Firmware Revision : BB0Q >> > > Serial No : S1D9NEADA08911Y >> > > GUID : N/A >> > > Protocol : SATA >> > > Drive Type : SATA_SSD >> > > >> > > Device is a Hard disk >> > > Enclosure # : 3 >> > > Slot # : 2 >> > > SAS Address : 5005076-0-3e8e-86eb >> > > State : Ready (RDY) >> > > Size (in MB)/(in sectors) : 953869/1953525167 >> > > Manufacturer : ATA >> > > Model Number : Samsung SSD 840 >> > > Firmware Revision : BB0Q >> > > Serial No : S1D9NEADA08811L >> > > GUID : N/A >> > > Protocol : SATA >> > > Drive Type : SATA_SSD >> > > >> > > Device is a Hard disk >> > > Enclosure # : 3 >> > > Slot # : 13 >> > > SAS Address : 5000c50-0-05b5-e531 >> > > State : Ready (RDY) >> > > Size (in MB)/(in sectors) : 140014/286749479 >> > > Manufacturer : SEAGATE >> > > Model Number : ST9146803SS >> > > Firmware Revision : FS03 >> > > Serial No : 3SD02STR >> > > GUID : N/A >> > > Protocol : SAS >> > > Drive Type : SAS_HDD >> > > >> > > Device is a Hard disk >> > > Enclosure # : 3 >> > > Slot # : 14 >> > > SAS Address : 5000c50-0-05b5-d489 >> > > State : Ready (RDY) >> > > Size (in MB)/(in sectors) : 140014/286749479 >> > > Manufacturer : SEAGATE >> > > Model Number : ST9146803SS >> > > Firmware Revision : FS03 >> > > Serial No : 3SD02TV1 >> > > GUID : N/A >> > > Protocol : SAS >> > > Drive Type : SAS_HDD >> > > >> > > Device is a Hard disk >> > > Enclosure # : 3 >> > > Slot # : 15 >> > > SAS Address : 5000c50-0-05b5-f0ad >> > > State : Ready (RDY) >> > > Size (in MB)/(in sectors) : 140014/286749479 >> > > Manufacturer : SEAGATE >> > > Model Number : ST9146803SS >> > > Firmware Revision : FS03 >> > > Serial No : 3SD03F4C >> > > GUID : N/A >> > > Protocol : SAS >> > > Drive Type : SAS_HDD >> > > >> > > Device is a Enclosure services device >> > > Enclosure # : 3 >> > > Slot # : 255 >> > > SAS Address : 5005076-0-3e8e-86f9 >> > > State : Standby (SBY) >> > > Manufacturer : IBM-ESXS >> > > Model Number : SAS EXP BP >> > > Firmware Revision : 61A6 >> > > Serial No : 00000006 >> > > GUID : N/A >> > > Protocol : SAS >> > > Device Type : Enclosure services devic= e >> > > -------------------------------------------------------------------- >> > > -- >> > > -- >> > > Enclosure information >> > > --------------------------------------------------------------------= ---- >> > > Enclosure# : 1 >> > > Logical ID : 500605b0:07ba2100 >> > > Numslots : 8 >> > > StartSlot : 0 >> > > Enclosure# : 2 >> > > Logical ID : 50050760:3e8e81a0 >> > > Numslots : 25 >> > > StartSlot : 0 >> > > Enclosure# : 3 >> > > Logical ID : 50050760:3e8e86e0 >> > > Numslots : 25 >> > > StartSlot : 0 >> > > -------------------------------------------------------------------- >> > > -- >> > > -- >> > > SAS2IRCU: Command DISPLAY Completed Successfully. >> > > SAS2IRCU: Utility Completed Successfully. >> > > >> > > >> > > >> > > >> > > _______________________________________________ >> > > freebsd-scsi@freebsd.org mailing list >> > > http://lists.freebsd.org/mailman/listinfo/freebsd-scsi >> > > To unsubscribe, send any mail to >> > > "freebsd-scsi-unsubscribe@freebsd.org" >> > >> > Adding Steve McConnell, who may not be on the freebsd-scsi list. >> >> Right, I'm not on the freebsd-scsi list. I suspect that the reason for >> this is that >> most of the Gen3 performance gains come from MSI-x support, but that is >> not in >> the mpr driver yet, and MSI-x is not supported in the 2008 card. But, I >> have a >> question out to another person and I'm waiting for more info. >> >> Steve > > Borja, can you upgrade your card to the latest FW. Your version is reall= y > old (1.00.03.00). We should be up to 7.00.00.00 now. There could be a l= ot > of fixes that would affect performance. To update the firmware, you must download LSI's Installer_P6_for_FreeBSD from the link below. Then the tricky part: you must hexedit the executable and replace all "mps3_" strings with "mpr", suitably padded with NULLs. It sucks, I know; complain to LSI/Avago. Then you must download the 9300_8e_Package_P6_IT_Firmware_BIOS_for_MSDOS_Windows (or whichever is appropriate for your hardware) to get the actual firmware files. Then you can use the hacked FreeBSD installer binary to download the firmware from the Windows firmware package. http://www.lsi.com/support/pages/download-results.aspx?component=3DStorage+= Component&productfamily=3DHost+Bus+Adapters&productcode=3DP00766&assettype= =3D0&productname=3DLSI+SAS+9300-8e -Alan From owner-freebsd-scsi@FreeBSD.ORG Tue Oct 7 16:27:04 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E9F7CA71; Tue, 7 Oct 2014 16:27:04 +0000 (UTC) Received: from cu01176a.smtpx.saremail.com (cu01176a.smtpx.saremail.com [195.16.150.151]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A2A07A33; Tue, 7 Oct 2014 16:27:04 +0000 (UTC) Received: from [172.16.2.2] (izaro.sarenet.es [192.148.167.11]) by proxypop03.sare.net (Postfix) with ESMTPSA id D8A059DD449; Tue, 7 Oct 2014 18:19:15 +0200 (CEST) Subject: Re: mpr vs mps performance Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=us-ascii From: Borja Marcos In-Reply-To: Date: Tue, 7 Oct 2014 18:19:12 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: References: To: Alan Somers X-Mailer: Apple Mail (2.1283) Cc: Stephen Mcconnell , FreeBSD-scsi X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Oct 2014 16:27:05 -0000 On Oct 7, 2014, at 6:11 PM, Alan Somers wrote: >> Borja, can you upgrade your card to the latest FW. Your version is = really >> old (1.00.03.00). We should be up to 7.00.00.00 now. There could be = a lot >> of fixes that would affect performance. >=20 > To update the firmware, you must download LSI's > Installer_P6_for_FreeBSD from the link below. Then the tricky part: > you must hexedit the executable and replace all "mps3_" strings with > "mpr", suitably padded with NULLs. It sucks, I know; complain to > LSI/Avago. Then you must download the > 9300_8e_Package_P6_IT_Firmware_BIOS_for_MSDOS_Windows (or whichever is > appropriate for your hardware) to get the actual firmware files. Then > you can use the hacked FreeBSD installer binary to download the > firmware from the Windows firmware package. >=20 > = http://www.lsi.com/support/pages/download-results.aspx?component=3DStorage= +Component&productfamily=3DHost+Bus+Adapters&productcode=3DP00766&assettyp= e=3D0&productname=3DLSI+SAS+9300-8e Thanks, I'll try. Probably the SAS2 cards will go into production but = I'm happy to help with some tests. The performance problem was not sustained, but writing performance = stalled for several seconds now and then. It did not happen with the = SAS2 HBA (which is a Dell branded as H200 flashed to IT firmware). Borja. From owner-freebsd-scsi@FreeBSD.ORG Tue Oct 7 17:01:34 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 05E43BD2 for ; Tue, 7 Oct 2014 17:01:34 +0000 (UTC) Received: from smtp.mullet.se (smtp.mullet.se [94.247.168.122]) by mx1.freebsd.org (Postfix) with ESMTP id B4D68E41 for ; Tue, 7 Oct 2014 17:01:33 +0000 (UTC) Received: from Martin-Nilssons-MacBook-Pro.local (ua-83-227-181-30.cust.bredbandsbolaget.se [83.227.181.30]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by smtp.mullet.se (Postfix) with ESMTPSA id 834776270B35 for ; Tue, 7 Oct 2014 18:55:33 +0200 (CEST) Message-ID: <54341B05.3090408@gneto.com> Date: Tue, 07 Oct 2014 18:55:33 +0200 From: Martin Nilsson User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:31.0) Gecko/20100101 Thunderbird/31.1.1 MIME-Version: 1.0 To: freebsd-scsi@freebsd.org Subject: Re: mpr vs mps performance References: In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Oct 2014 17:01:34 -0000 On 2014-10-07 18:11, Alan Somers wrote: > On Tue, Oct 7, 2014 at 9:25 AM, Stephen Mcconnell > wrote: >> Borja, can you upgrade your card to the latest FW. Your version is really >> old (1.00.03.00). We should be up to 7.00.00.00 now. There could be a lot >> of fixes that would affect performance. > To update the firmware, you must download LSI's > Installer_P6_for_FreeBSD from the link below. Then the tricky part: > you must hexedit the executable and replace all "mps3_" strings with > "mpr", suitably padded with NULLs. It sucks, I know; complain to > LSI/Avago. Then you must download the > 9300_8e_Package_P6_IT_Firmware_BIOS_for_MSDOS_Windows (or whichever is > appropriate for your hardware) to get the actual firmware files. Then > you can use the hacked FreeBSD installer binary to download the > firmware from the Windows firmware package. > > http://www.lsi.com/support/pages/download-results.aspx?component=Storage+Component&productfamily=Host+Bus+Adapters&productcode=P00766&assettype=0&productname=LSI+SAS+9300-8e Or you can put the firmware and the efi version of the installer on FAT formatted USB stick and boot into an EFI shell after POST. Then change to the usb stick with fs0: From there it works like a normal command interface. Much simpler... From owner-freebsd-scsi@FreeBSD.ORG Tue Oct 7 19:08:22 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 38562722; Tue, 7 Oct 2014 19:08:22 +0000 (UTC) Received: from smtp.infotech.no (smtp.infotech.no [82.134.31.41]) by mx1.freebsd.org (Postfix) with ESMTP id E509EFB8; Tue, 7 Oct 2014 19:08:21 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp.infotech.no (Postfix) with ESMTP id 07E67204190; Tue, 7 Oct 2014 21:08:19 +0200 (CEST) X-Virus-Scanned: by amavisd-new-2.6.6 (20110518) (Debian) at infotech.no Received: from smtp.infotech.no ([127.0.0.1]) by localhost (smtp.infotech.no [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id tvTBVIhVkqqj; Tue, 7 Oct 2014 21:08:18 +0200 (CEST) Received: from [10.7.0.6] (unknown [10.7.0.6]) by smtp.infotech.no (Postfix) with ESMTPA id 0D52F204198; Tue, 7 Oct 2014 21:08:17 +0200 (CEST) Message-ID: <54343A0F.6080905@interlog.com> Date: Tue, 07 Oct 2014 15:07:59 -0400 From: Douglas Gilbert Reply-To: dgilbert@interlog.com User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: Borja Marcos , Alan Somers Subject: Re: mpr vs mps performance References: In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Cc: Stephen Mcconnell , FreeBSD-scsi X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Oct 2014 19:08:22 -0000 On 14-10-07 12:19 PM, Borja Marcos wrote: > > On Oct 7, 2014, at 6:11 PM, Alan Somers wrote: > >>> Borja, can you upgrade your card to the latest FW. Your version is really >>> old (1.00.03.00). We should be up to 7.00.00.00 now. There could be a lot >>> of fixes that would affect performance. >> >> To update the firmware, you must download LSI's >> Installer_P6_for_FreeBSD from the link below. Then the tricky part: >> you must hexedit the executable and replace all "mps3_" strings with >> "mpr", suitably padded with NULLs. It sucks, I know; complain to >> LSI/Avago. Then you must download the >> 9300_8e_Package_P6_IT_Firmware_BIOS_for_MSDOS_Windows (or whichever is >> appropriate for your hardware) to get the actual firmware files. Then >> you can use the hacked FreeBSD installer binary to download the >> firmware from the Windows firmware package. >> >> http://www.lsi.com/support/pages/download-results.aspx?component=Storage+Component&productfamily=Host+Bus+Adapters&productcode=P00766&assettype=0&productname=LSI+SAS+9300-8e > > Thanks, I'll try. Probably the SAS2 cards will go into production but I'm happy to help with some tests. > > The performance problem was not sustained, but writing performance stalled for several seconds now and then. It did not happen with the SAS2 HBA (which is a Dell branded as H200 flashed to IT firmware). I'm not seeing a big difference between FreeBSD and Linux using a LSI SAS-3 HBA with a 12 Gbps SAS SSD. With the same 9300-4i HBA (P6 firmware (where is P7 that Stephen mentioned ??)) and the same disk (SEAGATE ST200FM0073 0005), using Linux as the client: Via iSCSI against FreeNAS 9.3-M4 (nightly build, 3 days old): # ddpt --odx if=/dev/sg5 bs=512 of=/dev/sg5 seek=4m count=4m 4194304+0 records in 4194304+0 records out time to transfer data: 11.275797 secs at 190.45 MB/sec On Linux lk 3.17.0 against the same disk, directly connected: # ddpt if=/dev/sg2 bs=512 of=/dev/sg2 seek=4m count=4m 4194304+0 records in 4194304+0 records out time to transfer data: 10.807345 secs at 198.71 MB/sec The SSD is probably not optimized for an "up-copy" like that. With raw READs it yields better than 700 MB/sec, using both ports it yields better than 900 MB/sec. "--odx" is a Microsoft term for a token based remote copy. It was proposed to T10 as "XCOPYv2: Extended Copy Plus & Lite" and the result is found in SBC-3. Doug Gilbert From owner-freebsd-scsi@FreeBSD.ORG Tue Oct 7 19:23:41 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EB958C3F for ; Tue, 7 Oct 2014 19:23:41 +0000 (UTC) Received: from exprod7og111.obsmtp.com (exprod7og111.obsmtp.com [64.18.2.175]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7B657251 for ; Tue, 7 Oct 2014 19:23:41 +0000 (UTC) Received: from mail-pd0-f169.google.com ([209.85.192.169]) (using TLSv1) by exprod7ob111.postini.com ([64.18.6.12]) with SMTP ID DSNKVDQ9vNf9+cYlTI67378/bGR48M91uOsR@postini.com; Tue, 07 Oct 2014 12:23:41 PDT Received: by mail-pd0-f169.google.com with SMTP id w10so5550334pde.28 for ; Tue, 07 Oct 2014 12:23:39 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:references:in-reply-to:mime-version :thread-index:date:message-id:subject:to:cc:content-type; bh=A4Uzz3P67d+Usv3ObXYIxpxpSADZm+dXYGRQ3ujYZGQ=; b=nLqZDwCrU6aA6LmjSO2tJ1zDEvCmi0XJhokKS2KH+E6RXd/mho1uJ7FJxwEP74z5iR IsTKTokNLSyeYNKV1VHEfOWrsxUY6SbVTWXufgizpZTQNwVQ1W9kDNZORYxlgZabCq8R z1CEx9RgRF+sdW2bseShEkDEmCFq5YRiKkPEQ7JFTPqPPDy2sVe3dc+Fmf05PTTpvSlU 2DK8AoVUIb+FRbtMFKj9Qd2OFMryy2sIRG9o3IcjGGSRMj/RWECmSWk3Tacos5pZMswZ NytSzbQV6RS7LUJow1PF4/C6+dxC3JdvAEMUx8JXXfY8u/JgkJcI+KK27+L8OEb2jQ61 +6Hg== X-Received: by 10.70.95.229 with SMTP id dn5mr5857432pdb.4.1412709507727; Tue, 07 Oct 2014 12:18:27 -0700 (PDT) X-Gm-Message-State: ALoCoQls8YR5NYnsha/hKiHQtOfAQNtUr7iAu0pDxMsLoFEcEJQ7SnfZD/uDVghKYkk4uVMwoMiiYdRBg0qxFaee3OU1l4LZL+FmeYzjQx5AAeeK6oY9kk5zVM/yVkGQRoyX4FEXexyG8kKWJQ+NDJ/rxTVXimPF9A== X-Received: by 10.70.95.229 with SMTP id dn5mr5857419pdb.4.1412709507561; Tue, 07 Oct 2014 12:18:27 -0700 (PDT) From: Stephen Mcconnell References: <54343A0F.6080905@interlog.com> In-Reply-To: <54343A0F.6080905@interlog.com> MIME-Version: 1.0 X-Mailer: Microsoft Outlook 14.0 Thread-Index: AQHAvPDqV/NT6Mjmvn39KZFmH6q6QwHpuJfJAoXUFXgBjNvBvwHFdtxGAVPo6gCb+p2JAA== Date: Tue, 7 Oct 2014 13:18:26 -0600 Message-ID: <2c53e472677ca9fc35b8e6e20ab1d994@mail.gmail.com> Subject: RE: mpr vs mps performance To: dgilbert@interlog.com, Borja Marcos , Alan Somers Content-Type: text/plain; charset=UTF-8 Cc: FreeBSD-scsi X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Oct 2014 19:23:42 -0000 > -----Original Message----- > From: Douglas Gilbert [mailto:dgilbert@interlog.com] > Sent: Tuesday, October 07, 2014 1:08 PM > To: Borja Marcos; Alan Somers > Cc: Stephen Mcconnell; FreeBSD-scsi > Subject: Re: mpr vs mps performance > > On 14-10-07 12:19 PM, Borja Marcos wrote: > > > > On Oct 7, 2014, at 6:11 PM, Alan Somers wrote: > > > >>> Borja, can you upgrade your card to the latest FW. Your version is > >>> really old (1.00.03.00). We should be up to 7.00.00.00 now. There > >>> could be a lot of fixes that would affect performance. > >> > >> To update the firmware, you must download LSI's > >> Installer_P6_for_FreeBSD from the link below. Then the tricky part: > >> you must hexedit the executable and replace all "mps3_" strings with > >> "mpr", suitably padded with NULLs. It sucks, I know; complain to > >> LSI/Avago. Then you must download the > >> 9300_8e_Package_P6_IT_Firmware_BIOS_for_MSDOS_Windows (or > whichever > >> is appropriate for your hardware) to get the actual firmware files. > >> Then you can use the hacked FreeBSD installer binary to download the > >> firmware from the Windows firmware package. > >> > >> http://www.lsi.com/support/pages/download-results.aspx?component=Stor > >> > age+Component&productfamily=Host+Bus+Adapters&productcode=P00766&as > se > >> ttype=0&productname=LSI+SAS+9300-8e > > > > Thanks, I'll try. Probably the SAS2 cards will go into production but I'm happy > to help with some tests. > > > > The performance problem was not sustained, but writing performance stalled > for several seconds now and then. It did not happen with the SAS2 HBA (which is > a Dell branded as H200 flashed to IT firmware). > > I'm not seeing a big difference between FreeBSD and Linux using a LSI SAS-3 HBA > with a 12 Gbps SAS SSD. > > With the same 9300-4i HBA (P6 firmware (where is P7 that Stephen mentioned > ??)) and the same disk (SEAGATE ST200FM0073 0005), using Linux as the client: It's Phase 6, but the version is 7.00.00.00 > > Via iSCSI against FreeNAS 9.3-M4 (nightly build, 3 days old): > # ddpt --odx if=/dev/sg5 bs=512 of=/dev/sg5 seek=4m count=4m > 4194304+0 records in > 4194304+0 records out > time to transfer data: 11.275797 secs at 190.45 MB/sec > > On Linux lk 3.17.0 against the same disk, directly connected: > # ddpt if=/dev/sg2 bs=512 of=/dev/sg2 seek=4m count=4m > 4194304+0 records in > 4194304+0 records out > time to transfer data: 10.807345 secs at 198.71 MB/sec > > > The SSD is probably not optimized for an "up-copy" like that. With raw READs it > yields better than 700 MB/sec, using both ports it yields better than 900 MB/sec. > > "--odx" is a Microsoft term for a token based remote copy. It was proposed to > T10 as "XCOPYv2: Extended Copy Plus & Lite" and the result is found in SBC-3. > > Doug Gilbert > > > From owner-freebsd-scsi@FreeBSD.ORG Tue Oct 7 22:43:36 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D9590328 for ; Tue, 7 Oct 2014 22:43:36 +0000 (UTC) Received: from exprod7og115.obsmtp.com (exprod7og115.obsmtp.com [64.18.2.217]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 84964BE6 for ; Tue, 7 Oct 2014 22:43:36 +0000 (UTC) Received: from mail-pa0-f44.google.com ([209.85.220.44]) (using TLSv1) by exprod7ob115.postini.com ([64.18.6.12]) with SMTP ID DSNKVDRskdC7ZQ7HGlg/TQBWPDwGaOM78XdI@postini.com; Tue, 07 Oct 2014 15:43:36 PDT Received: by mail-pa0-f44.google.com with SMTP id et14so7927972pad.31 for ; Tue, 07 Oct 2014 15:43:29 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:references:in-reply-to:mime-version :thread-index:date:message-id:subject:to:cc:content-type; bh=cuiV9/2ZvCk3bjaQflQLoipjJ5akzoOFmY6JptGjTow=; b=be25BZqw4DQeIN/A7Xl2R1WgRznTNoG137q6ZdfMP47TcdasDjNT6kcJcfiJIA+rSn bdezCeQXQSgW/8oXrfpoZ15YDtUed1XNKrVEfK8vi3Wc7BDM9foirAu0PESWvGTvjOh/ P93Ls7NzwAHxhyX1lkeBJZb/yCseXI8JM4GZKr0WkheukMgLKqev/uiYSVKFQ5lAd4Ri uSb2/Im7ExYNGABqQT2yLL+2QYHBNVKFb9KPi5+inHkSr8X3q9AaSjhUHLMkBdADGAcU akeH6VlJr207KQppTgloyl3ujUuTMJiveO1y0X0N4YjNL3u2UmqvHBhRFLEPSOLCPMTN Y1kQ== X-Gm-Message-State: ALoCoQnNNXOYIG9GvNXs2aCz9dkG+y9ALl34abOs5S1MaVkehHauGzfH32qGc26Lzm/etc4FrCJPCGKVQe3/Ov2f5YtQq6aFSndu3FOoCv022zWzez5SuV0GYTlNspA3Ymoye2OgCMfVMafJ8ZUkRpInmxtgVhlYtA== X-Received: by 10.66.220.194 with SMTP id py2mr6295193pac.81.1412721809423; Tue, 07 Oct 2014 15:43:29 -0700 (PDT) X-Received: by 10.66.220.194 with SMTP id py2mr6295181pac.81.1412721809251; Tue, 07 Oct 2014 15:43:29 -0700 (PDT) From: Stephen Mcconnell References: <54343A0F.6080905@interlog.com> In-Reply-To: <54343A0F.6080905@interlog.com> MIME-Version: 1.0 X-Mailer: Microsoft Outlook 14.0 Thread-Index: AQHAvPDqV/NT6Mjmvn39KZFmH6q6QwHpuJfJAoXUFXgBjNvBvwHFdtxGAVPo6gCb+tZ6YA== Date: Tue, 7 Oct 2014 16:43:30 -0600 Message-ID: Subject: RE: mpr vs mps performance To: dgilbert@interlog.com, Borja Marcos , Alan Somers Content-Type: text/plain; charset=UTF-8 Cc: FreeBSD-scsi X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Oct 2014 22:43:36 -0000 > -----Original Message----- > From: Douglas Gilbert [mailto:dgilbert@interlog.com] > Sent: Tuesday, October 07, 2014 1:08 PM > To: Borja Marcos; Alan Somers > Cc: Stephen Mcconnell; FreeBSD-scsi > Subject: Re: mpr vs mps performance > > On 14-10-07 12:19 PM, Borja Marcos wrote: > > > > On Oct 7, 2014, at 6:11 PM, Alan Somers wrote: > > > >>> Borja, can you upgrade your card to the latest FW. Your version is > >>> really old (1.00.03.00). We should be up to 7.00.00.00 now. There > >>> could be a lot of fixes that would affect performance. > >> > >> To update the firmware, you must download LSI's > >> Installer_P6_for_FreeBSD from the link below. Then the tricky part: > >> you must hexedit the executable and replace all "mps3_" strings with > >> "mpr", suitably padded with NULLs. It sucks, I know; complain to > >> LSI/Avago. Then you must download the > >> 9300_8e_Package_P6_IT_Firmware_BIOS_for_MSDOS_Windows (or > whichever > >> is appropriate for your hardware) to get the actual firmware files. > >> Then you can use the hacked FreeBSD installer binary to download the > >> firmware from the Windows firmware package. > >> > >> http://www.lsi.com/support/pages/download-results.aspx?component=Stor > >> > age+Component&productfamily=Host+Bus+Adapters&productcode=P00766&as > se > >> ttype=0&productname=LSI+SAS+9300-8e > > > > Thanks, I'll try. Probably the SAS2 cards will go into production but I'm happy > to help with some tests. > > > > The performance problem was not sustained, but writing performance stalled > for several seconds now and then. It did not happen with the SAS2 HBA (which is > a Dell branded as H200 flashed to IT firmware). > > I'm not seeing a big difference between FreeBSD and Linux using a LSI SAS-3 HBA > with a 12 Gbps SAS SSD. > > With the same 9300-4i HBA (P6 firmware (where is P7 that Stephen mentioned > ??)) and the same disk (SEAGATE ST200FM0073 0005), using Linux as the client: > > Via iSCSI against FreeNAS 9.3-M4 (nightly build, 3 days old): > # ddpt --odx if=/dev/sg5 bs=512 of=/dev/sg5 seek=4m count=4m > 4194304+0 records in > 4194304+0 records out > time to transfer data: 11.275797 secs at 190.45 MB/sec > > On Linux lk 3.17.0 against the same disk, directly connected: > # ddpt if=/dev/sg2 bs=512 of=/dev/sg2 seek=4m count=4m > 4194304+0 records in > 4194304+0 records out > time to transfer data: 10.807345 secs at 198.71 MB/sec > > > The SSD is probably not optimized for an "up-copy" like that. With raw READs it > yields better than 700 MB/sec, using both ports it yields better than 900 MB/sec. > > "--odx" is a Microsoft term for a token based remote copy. It was proposed to > T10 as "XCOPYv2: Extended Copy Plus & Lite" and the result is found in SBC-3. > > Doug Gilbert Yesterday I mentioned that most of the performance gains on the Gen3 card come from MSI-x support. In fact, MSI-x Firmware support does add to performance, but I forgot about the actual hardware I/O acceleration in the Gen3 cards (Fast Path I/O), which increases performance significantly. Just wanted to add that. Also, I did a little testing and I got about the same numbers as Doug with the 3008 and a 12Gb SAS SSD (a little better than 700 MB/sec). With the 2008, I saw raw reads of around 425 MB/sec. I don't do a lot of benchmark testing, but that's what I saw in my little test. Steve > > >