Date: Mon, 6 Oct 2014 10:13:06 -0600 From: Alan Somers <asomers@freebsd.org> To: Borja Marcos <borjam@sarenet.es>, slm@freebsd.org Cc: FreeBSD-scsi <freebsd-scsi@freebsd.org> Subject: Re: mpr vs mps performance Message-ID: <CAOtMX2jVR-8wkHcTyudOVwMNmxOvDkWPNsTBuyu%2BWZQ6WtDvmg@mail.gmail.com> In-Reply-To: <A8A5F62D-ED7B-420F-B5B5-88842A8F8827@sarenet.es> References: <A8A5F62D-ED7B-420F-B5B5-88842A8F8827@sarenet.es>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Sep 22, 2014 at 9:39 AM, Borja Marcos <borjam@sarenet.es> wrote: > > Hello, > > I have been playing with the new SAS3 cards supported by the mpr driver, = and I=C2=B4ve found out that they are, in the same hardware > configuration, considerably slower writing data. Moreover, running two si= multaneous "bonnie" benchmarks (I am using SSDs, and one "bonnie" sometime= s hits 100% CPU usage, unable to really saturate the I/O) I see the writing= activity somewhat stalling, with disk bandwidth going from 600 MB/s to aro= und 50 for 20 seconds or so. > > I'd like to know if this matches anyone else's experiences. Also, I can t= ry and make some tests if needed. But for now it seems we will stick to the= SAS2 HBAs. > > > > > The Bonnie results are: > > With mpr driver, SAS3: > > (each bonnie instance, so multiply the results by 2 to get the actual ban= dwidth achieved) > > Seq output: (writing) > Block: 292155 KB/s > Rewrite: 139713 KB/s > Seq input: > Block: 862861 KB/s > > With mps driver: SAS2, again, total is 2x the following figures. > > Seq output: (writing) > Block: 587950 KB/s > Rewrite: 208239 KB/s > Seq. input: (reading) > Block: 842169 KB/s > > > The storage is a ZFS pool with a 9-disk raidz2 vdev, made of Samsung 840= EVO 1 TB SSDs. The pool has been created with an ashift of 12 (zpool appli= ed it thanks to the 4 KB block quirk for these SSDs) > > <ATA Samsung SSD 840 BB0Q> at scbus0 target 9 lun 0 (pass0,da0) > <ATA Samsung SSD 840 BB0Q> at scbus0 target 10 lun 0 (pass1,da1) > <ATA Samsung SSD 840 BB0Q> at scbus0 target 11 lun 0 (pass2,da2) > <ATA Samsung SSD 840 BB0Q> at scbus0 target 12 lun 0 (pass3,da3) > <ATA Samsung SSD 840 BB0Q> at scbus0 target 13 lun 0 (pass4,da4) > <ATA Samsung SSD 840 BB0Q> at scbus0 target 14 lun 0 (pass5,da5) > <ATA Samsung SSD 840 BB0Q> at scbus0 target 17 lun 0 (pass7,da6) > <ATA Samsung SSD 840 BB0Q> at scbus0 target 18 lun 0 (pass8,da7) > <ATA Samsung SSD 840 BB0Q> at scbus0 target 27 lun 0 (pass14,da12= ) > > > > > The mpr card details follow: > Sep 17 09:49:39 elibm kernel: mpr0: <LSI SAS3008> port 0x3f00-0x3fff mem = 0x912f0000-0x912fffff irq 32 at device 0.0 on pci17 > Sep 17 09:49:39 elibm kernel: mpr0: IOCFacts : > Sep 17 09:49:39 elibm kernel: MsgVersion: 0x205 > Sep 17 09:49:39 elibm kernel: HeaderVersion: 0x1d00 > Sep 17 09:49:39 elibm kernel: IOCNumber: 0 > Sep 17 09:49:39 elibm kernel: IOCExceptions: 0x0 > Sep 17 09:49:39 elibm kernel: MaxChainDepth: 128 > Sep 17 09:49:39 elibm kernel: NumberOfPorts: 1 > Sep 17 09:49:39 elibm kernel: RequestCredit: 11264 > Sep 17 09:49:39 elibm kernel: ProductID: 0x2221 > Sep 17 09:49:39 elibm kernel: IOCRequestFrameSize: 32 > Sep 17 09:49:39 elibm kernel: MaxInitiators: 1 > Sep 17 09:49:39 elibm kernel: MaxTargets: 1024 > Sep 17 09:49:39 elibm kernel: MaxSasExpanders: 14 > Sep 17 09:49:39 elibm kernel: MaxEnclosures: 15 > Sep 17 09:49:39 elibm kernel: HighPriorityCredit: 60 > Sep 17 09:49:39 elibm kernel: MaxReplyDescriptorPostQueueDepth: 65504 > Sep 17 09:49:39 elibm kernel: ReplyFrameSize: 32 > Sep 17 09:49:39 elibm kernel: MaxVolumes: 0 > Sep 17 09:49:39 elibm kernel: MaxDevHandle: 1047 > Sep 17 09:49:39 elibm kernel: MaxPersistentEntries: 128 > Sep 17 09:49:39 elibm kernel: mpr0: Firmware: 01.00.03.00, Driver: 05.255= .05.00-fbsd > Sep 17 09:49:39 elibm kernel: mpr0: IOCCapabilities: 3a85c<ScsiTaskFull,D= iagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc> > > > And the mps card is a classic: > Sep 22 17:18:24 elibm kernel: mps0: <LSI SAS2008> port 0x3f00-0x3fff mem = 0x90ebc000-0x90ebffff,0x912c0000-0x912fffff irq 32 at device 0.0 on pci17 > Sep 22 17:18:24 elibm kernel: mps0: Firmware: 18.00.00.00, Driver: 19.00.= 00.00-fbsd > Sep 22 17:18:24 elibm kernel: mps0: IOCCapabilities: 1285c<ScsiTaskFull,D= iagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc> > > > The connected devices follow. Both use the same hardware (except for the = cables and HBA of course), but currently there's no way to check this with = the SAS3 card, as sas3ircu nor sas3flash detect it on FreeBSD. > > > > # sas2ircu 0 display > LSI Corporation SAS2 IR Configuration Utility. > Version 18.00.00.00 (2013.11.18) > Copyright (c) 2009-2013 LSI Corporation. All rights reserved. > > Read configuration has been initiated for controller 0 > ------------------------------------------------------------------------ > Controller information > ------------------------------------------------------------------------ > Controller type : SAS2008 > BIOS version : 7.35.00.00 > Firmware version : 18.00.00.00 > Channel description : 1 Serial Attached SCSI > Initiator ID : 0 > Maximum physical devices : 255 > Concurrent commands supported : 3432 > Slot : 3 > Segment : 0 > Bus : 17 > Device : 0 > Function : 0 > RAID Support : No > ------------------------------------------------------------------------ > IR Volume information > ------------------------------------------------------------------------ > ------------------------------------------------------------------------ > Physical device information > ------------------------------------------------------------------------ > Initiator at ID #0 > > Device is a Hard disk > Enclosure # : 2 > Slot # : 16 > SAS Address : 5000c50-0-05b5-ce25 > State : Ready (RDY) > Size (in MB)/(in sectors) : 140014/286749479 > Manufacturer : SEAGATE > Model Number : ST9146803SS > Firmware Revision : FS03 > Serial No : 3SD02W5L > GUID : N/A > Protocol : SAS > Drive Type : SAS_HDD > > Device is a Hard disk > Enclosure # : 2 > Slot # : 17 > SAS Address : 5005076-0-3e8e-81a2 > State : Ready (RDY) > Size (in MB)/(in sectors) : 953869/1953525167 > Manufacturer : ATA > Model Number : Samsung SSD 840 > Firmware Revision : BB0Q > Serial No : S1D9NEADA08549F > GUID : N/A > Protocol : SATA > Drive Type : SATA_SSD > > Device is a Hard disk > Enclosure # : 2 > Slot # : 18 > SAS Address : 5005076-0-3e8e-81a3 > State : Ready (RDY) > Size (in MB)/(in sectors) : 953869/1953525167 > Manufacturer : ATA > Model Number : Samsung SSD 840 > Firmware Revision : BB0Q > Serial No : S1D9NEADA08548T > GUID : N/A > Protocol : SATA > Drive Type : SATA_SSD > > Device is a Hard disk > Enclosure # : 2 > Slot # : 19 > SAS Address : 5005076-0-3e8e-81a4 > State : Ready (RDY) > Size (in MB)/(in sectors) : 953869/1953525167 > Manufacturer : ATA > Model Number : Samsung SSD 840 > Firmware Revision : BB0Q > Serial No : S1D9NEADA08568E > GUID : N/A > Protocol : SATA > Drive Type : SATA_SSD > > Device is a Hard disk > Enclosure # : 2 > Slot # : 20 > SAS Address : 5005076-0-3e8e-81a5 > State : Ready (RDY) > Size (in MB)/(in sectors) : 953869/1953525167 > Manufacturer : ATA > Model Number : Samsung SSD 840 > Firmware Revision : BB0Q > Serial No : S1D9NEADA08547X > GUID : N/A > Protocol : SATA > Drive Type : SATA_SSD > > Device is a Hard disk > Enclosure # : 2 > Slot # : 21 > SAS Address : 5005076-0-3e8e-81a6 > State : Ready (RDY) > Size (in MB)/(in sectors) : 953869/1953525167 > Manufacturer : ATA > Model Number : Samsung SSD 840 > Firmware Revision : BB0Q > Serial No : S1D9NEADA08518Y > GUID : N/A > Protocol : SATA > Drive Type : SATA_SSD > > Device is a Hard disk > Enclosure # : 2 > Slot # : 22 > SAS Address : 5005076-0-3e8e-81a7 > State : Ready (RDY) > Size (in MB)/(in sectors) : 953869/1953525167 > Manufacturer : ATA > Model Number : Samsung SSD 840 > Firmware Revision : BB0Q > Serial No : S1D9NEADA08556K > GUID : N/A > Protocol : SATA > Drive Type : SATA_SSD > > Device is a Enclosure services device > Enclosure # : 2 > Slot # : 255 > SAS Address : 5005076-0-3e8e-81b9 > State : Standby (SBY) > Manufacturer : IBM-ESXS > Model Number : SAS EXP BP > Firmware Revision : 61A6 > Serial No : 00000006 > GUID : N/A > Protocol : SAS > Device Type : Enclosure services device > > Device is a Hard disk > Enclosure # : 3 > Slot # : 0 > SAS Address : 5005076-0-3e8e-86e9 > State : Ready (RDY) > Size (in MB)/(in sectors) : 953869/1953525167 > Manufacturer : ATA > Model Number : Samsung SSD 840 > Firmware Revision : BB0Q > Serial No : S1D9NEADA08550R > GUID : N/A > Protocol : SATA > Drive Type : SATA_SSD > > Device is a Hard disk > Enclosure # : 3 > Slot # : 1 > SAS Address : 5005076-0-3e8e-86ea > State : Ready (RDY) > Size (in MB)/(in sectors) : 953869/1953525167 > Manufacturer : ATA > Model Number : Samsung SSD 840 > Firmware Revision : BB0Q > Serial No : S1D9NEADA08911Y > GUID : N/A > Protocol : SATA > Drive Type : SATA_SSD > > Device is a Hard disk > Enclosure # : 3 > Slot # : 2 > SAS Address : 5005076-0-3e8e-86eb > State : Ready (RDY) > Size (in MB)/(in sectors) : 953869/1953525167 > Manufacturer : ATA > Model Number : Samsung SSD 840 > Firmware Revision : BB0Q > Serial No : S1D9NEADA08811L > GUID : N/A > Protocol : SATA > Drive Type : SATA_SSD > > Device is a Hard disk > Enclosure # : 3 > Slot # : 13 > SAS Address : 5000c50-0-05b5-e531 > State : Ready (RDY) > Size (in MB)/(in sectors) : 140014/286749479 > Manufacturer : SEAGATE > Model Number : ST9146803SS > Firmware Revision : FS03 > Serial No : 3SD02STR > GUID : N/A > Protocol : SAS > Drive Type : SAS_HDD > > Device is a Hard disk > Enclosure # : 3 > Slot # : 14 > SAS Address : 5000c50-0-05b5-d489 > State : Ready (RDY) > Size (in MB)/(in sectors) : 140014/286749479 > Manufacturer : SEAGATE > Model Number : ST9146803SS > Firmware Revision : FS03 > Serial No : 3SD02TV1 > GUID : N/A > Protocol : SAS > Drive Type : SAS_HDD > > Device is a Hard disk > Enclosure # : 3 > Slot # : 15 > SAS Address : 5000c50-0-05b5-f0ad > State : Ready (RDY) > Size (in MB)/(in sectors) : 140014/286749479 > Manufacturer : SEAGATE > Model Number : ST9146803SS > Firmware Revision : FS03 > Serial No : 3SD03F4C > GUID : N/A > Protocol : SAS > Drive Type : SAS_HDD > > Device is a Enclosure services device > Enclosure # : 3 > Slot # : 255 > SAS Address : 5005076-0-3e8e-86f9 > State : Standby (SBY) > Manufacturer : IBM-ESXS > Model Number : SAS EXP BP > Firmware Revision : 61A6 > Serial No : 00000006 > GUID : N/A > Protocol : SAS > Device Type : Enclosure services device > ------------------------------------------------------------------------ > Enclosure information > ------------------------------------------------------------------------ > Enclosure# : 1 > Logical ID : 500605b0:07ba2100 > Numslots : 8 > StartSlot : 0 > Enclosure# : 2 > Logical ID : 50050760:3e8e81a0 > Numslots : 25 > StartSlot : 0 > Enclosure# : 3 > Logical ID : 50050760:3e8e86e0 > Numslots : 25 > StartSlot : 0 > ------------------------------------------------------------------------ > SAS2IRCU: Command DISPLAY Completed Successfully. > SAS2IRCU: Utility Completed Successfully. > > > > > _______________________________________________ > freebsd-scsi@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-scsi > To unsubscribe, send any mail to "freebsd-scsi-unsubscribe@freebsd.org" Adding Steve McConnell, who may not be on the freebsd-scsi list.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOtMX2jVR-8wkHcTyudOVwMNmxOvDkWPNsTBuyu%2BWZQ6WtDvmg>