Date: Mon, 22 Sep 2014 17:39:58 +0200 From: Borja Marcos <borjam@sarenet.es> To: FreeBSD-scsi <freebsd-scsi@freebsd.org> Subject: mpr vs mps performance Message-ID: <A8A5F62D-ED7B-420F-B5B5-88842A8F8827@sarenet.es>
next in thread | raw e-mail | index | archive | help
Hello, I have been playing with the new SAS3 cards supported by the mpr driver, and I´ve found out that they are, in the same hardware configuration, considerably slower writing data. Moreover, running two simultaneous "bonnie" benchmarks (I am using SSDs, and one "bonnie" sometimes hits 100% CPU usage, unable to really saturate the I/O) I see the writing activity somewhat stalling, with disk bandwidth going from 600 MB/s to around 50 for 20 seconds or so. I'd like to know if this matches anyone else's experiences. Also, I can try and make some tests if needed. But for now it seems we will stick to the SAS2 HBAs. The Bonnie results are: With mpr driver, SAS3: (each bonnie instance, so multiply the results by 2 to get the actual bandwidth achieved) Seq output: (writing) Block: 292155 KB/s Rewrite: 139713 KB/s Seq input: Block: 862861 KB/s With mps driver: SAS2, again, total is 2x the following figures. Seq output: (writing) Block: 587950 KB/s Rewrite: 208239 KB/s Seq. input: (reading) Block: 842169 KB/s The storage is a ZFS pool with a 9-disk raidz2 vdev, made of Samsung 840 EVO 1 TB SSDs. The pool has been created with an ashift of 12 (zpool applied it thanks to the 4 KB block quirk for these SSDs) <ATA Samsung SSD 840 BB0Q> at scbus0 target 9 lun 0 (pass0,da0) <ATA Samsung SSD 840 BB0Q> at scbus0 target 10 lun 0 (pass1,da1) <ATA Samsung SSD 840 BB0Q> at scbus0 target 11 lun 0 (pass2,da2) <ATA Samsung SSD 840 BB0Q> at scbus0 target 12 lun 0 (pass3,da3) <ATA Samsung SSD 840 BB0Q> at scbus0 target 13 lun 0 (pass4,da4) <ATA Samsung SSD 840 BB0Q> at scbus0 target 14 lun 0 (pass5,da5) <ATA Samsung SSD 840 BB0Q> at scbus0 target 17 lun 0 (pass7,da6) <ATA Samsung SSD 840 BB0Q> at scbus0 target 18 lun 0 (pass8,da7) <ATA Samsung SSD 840 BB0Q> at scbus0 target 27 lun 0 (pass14,da12) The mpr card details follow: Sep 17 09:49:39 elibm kernel: mpr0: <LSI SAS3008> port 0x3f00-0x3fff mem 0x912f0000-0x912fffff irq 32 at device 0.0 on pci17 Sep 17 09:49:39 elibm kernel: mpr0: IOCFacts : Sep 17 09:49:39 elibm kernel: MsgVersion: 0x205 Sep 17 09:49:39 elibm kernel: HeaderVersion: 0x1d00 Sep 17 09:49:39 elibm kernel: IOCNumber: 0 Sep 17 09:49:39 elibm kernel: IOCExceptions: 0x0 Sep 17 09:49:39 elibm kernel: MaxChainDepth: 128 Sep 17 09:49:39 elibm kernel: NumberOfPorts: 1 Sep 17 09:49:39 elibm kernel: RequestCredit: 11264 Sep 17 09:49:39 elibm kernel: ProductID: 0x2221 Sep 17 09:49:39 elibm kernel: IOCRequestFrameSize: 32 Sep 17 09:49:39 elibm kernel: MaxInitiators: 1 Sep 17 09:49:39 elibm kernel: MaxTargets: 1024 Sep 17 09:49:39 elibm kernel: MaxSasExpanders: 14 Sep 17 09:49:39 elibm kernel: MaxEnclosures: 15 Sep 17 09:49:39 elibm kernel: HighPriorityCredit: 60 Sep 17 09:49:39 elibm kernel: MaxReplyDescriptorPostQueueDepth: 65504 Sep 17 09:49:39 elibm kernel: ReplyFrameSize: 32 Sep 17 09:49:39 elibm kernel: MaxVolumes: 0 Sep 17 09:49:39 elibm kernel: MaxDevHandle: 1047 Sep 17 09:49:39 elibm kernel: MaxPersistentEntries: 128 Sep 17 09:49:39 elibm kernel: mpr0: Firmware: 01.00.03.00, Driver: 05.255.05.00-fbsd Sep 17 09:49:39 elibm kernel: mpr0: IOCCapabilities: 3a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc> And the mps card is a classic: Sep 22 17:18:24 elibm kernel: mps0: <LSI SAS2008> port 0x3f00-0x3fff mem 0x90ebc000-0x90ebffff,0x912c0000-0x912fffff irq 32 at device 0.0 on pci17 Sep 22 17:18:24 elibm kernel: mps0: Firmware: 18.00.00.00, Driver: 19.00.00.00-fbsd Sep 22 17:18:24 elibm kernel: mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc> The connected devices follow. Both use the same hardware (except for the cables and HBA of course), but currently there's no way to check this with the SAS3 card, as sas3ircu nor sas3flash detect it on FreeBSD. # sas2ircu 0 display LSI Corporation SAS2 IR Configuration Utility. Version 18.00.00.00 (2013.11.18) Copyright (c) 2009-2013 LSI Corporation. All rights reserved. Read configuration has been initiated for controller 0 ------------------------------------------------------------------------ Controller information ------------------------------------------------------------------------ Controller type : SAS2008 BIOS version : 7.35.00.00 Firmware version : 18.00.00.00 Channel description : 1 Serial Attached SCSI Initiator ID : 0 Maximum physical devices : 255 Concurrent commands supported : 3432 Slot : 3 Segment : 0 Bus : 17 Device : 0 Function : 0 RAID Support : No ------------------------------------------------------------------------ IR Volume information ------------------------------------------------------------------------ ------------------------------------------------------------------------ Physical device information ------------------------------------------------------------------------ Initiator at ID #0 Device is a Hard disk Enclosure # : 2 Slot # : 16 SAS Address : 5000c50-0-05b5-ce25 State : Ready (RDY) Size (in MB)/(in sectors) : 140014/286749479 Manufacturer : SEAGATE Model Number : ST9146803SS Firmware Revision : FS03 Serial No : 3SD02W5L GUID : N/A Protocol : SAS Drive Type : SAS_HDD Device is a Hard disk Enclosure # : 2 Slot # : 17 SAS Address : 5005076-0-3e8e-81a2 State : Ready (RDY) Size (in MB)/(in sectors) : 953869/1953525167 Manufacturer : ATA Model Number : Samsung SSD 840 Firmware Revision : BB0Q Serial No : S1D9NEADA08549F GUID : N/A Protocol : SATA Drive Type : SATA_SSD Device is a Hard disk Enclosure # : 2 Slot # : 18 SAS Address : 5005076-0-3e8e-81a3 State : Ready (RDY) Size (in MB)/(in sectors) : 953869/1953525167 Manufacturer : ATA Model Number : Samsung SSD 840 Firmware Revision : BB0Q Serial No : S1D9NEADA08548T GUID : N/A Protocol : SATA Drive Type : SATA_SSD Device is a Hard disk Enclosure # : 2 Slot # : 19 SAS Address : 5005076-0-3e8e-81a4 State : Ready (RDY) Size (in MB)/(in sectors) : 953869/1953525167 Manufacturer : ATA Model Number : Samsung SSD 840 Firmware Revision : BB0Q Serial No : S1D9NEADA08568E GUID : N/A Protocol : SATA Drive Type : SATA_SSD Device is a Hard disk Enclosure # : 2 Slot # : 20 SAS Address : 5005076-0-3e8e-81a5 State : Ready (RDY) Size (in MB)/(in sectors) : 953869/1953525167 Manufacturer : ATA Model Number : Samsung SSD 840 Firmware Revision : BB0Q Serial No : S1D9NEADA08547X GUID : N/A Protocol : SATA Drive Type : SATA_SSD Device is a Hard disk Enclosure # : 2 Slot # : 21 SAS Address : 5005076-0-3e8e-81a6 State : Ready (RDY) Size (in MB)/(in sectors) : 953869/1953525167 Manufacturer : ATA Model Number : Samsung SSD 840 Firmware Revision : BB0Q Serial No : S1D9NEADA08518Y GUID : N/A Protocol : SATA Drive Type : SATA_SSD Device is a Hard disk Enclosure # : 2 Slot # : 22 SAS Address : 5005076-0-3e8e-81a7 State : Ready (RDY) Size (in MB)/(in sectors) : 953869/1953525167 Manufacturer : ATA Model Number : Samsung SSD 840 Firmware Revision : BB0Q Serial No : S1D9NEADA08556K GUID : N/A Protocol : SATA Drive Type : SATA_SSD Device is a Enclosure services device Enclosure # : 2 Slot # : 255 SAS Address : 5005076-0-3e8e-81b9 State : Standby (SBY) Manufacturer : IBM-ESXS Model Number : SAS EXP BP Firmware Revision : 61A6 Serial No : 00000006 GUID : N/A Protocol : SAS Device Type : Enclosure services device Device is a Hard disk Enclosure # : 3 Slot # : 0 SAS Address : 5005076-0-3e8e-86e9 State : Ready (RDY) Size (in MB)/(in sectors) : 953869/1953525167 Manufacturer : ATA Model Number : Samsung SSD 840 Firmware Revision : BB0Q Serial No : S1D9NEADA08550R GUID : N/A Protocol : SATA Drive Type : SATA_SSD Device is a Hard disk Enclosure # : 3 Slot # : 1 SAS Address : 5005076-0-3e8e-86ea State : Ready (RDY) Size (in MB)/(in sectors) : 953869/1953525167 Manufacturer : ATA Model Number : Samsung SSD 840 Firmware Revision : BB0Q Serial No : S1D9NEADA08911Y GUID : N/A Protocol : SATA Drive Type : SATA_SSD Device is a Hard disk Enclosure # : 3 Slot # : 2 SAS Address : 5005076-0-3e8e-86eb State : Ready (RDY) Size (in MB)/(in sectors) : 953869/1953525167 Manufacturer : ATA Model Number : Samsung SSD 840 Firmware Revision : BB0Q Serial No : S1D9NEADA08811L GUID : N/A Protocol : SATA Drive Type : SATA_SSD Device is a Hard disk Enclosure # : 3 Slot # : 13 SAS Address : 5000c50-0-05b5-e531 State : Ready (RDY) Size (in MB)/(in sectors) : 140014/286749479 Manufacturer : SEAGATE Model Number : ST9146803SS Firmware Revision : FS03 Serial No : 3SD02STR GUID : N/A Protocol : SAS Drive Type : SAS_HDD Device is a Hard disk Enclosure # : 3 Slot # : 14 SAS Address : 5000c50-0-05b5-d489 State : Ready (RDY) Size (in MB)/(in sectors) : 140014/286749479 Manufacturer : SEAGATE Model Number : ST9146803SS Firmware Revision : FS03 Serial No : 3SD02TV1 GUID : N/A Protocol : SAS Drive Type : SAS_HDD Device is a Hard disk Enclosure # : 3 Slot # : 15 SAS Address : 5000c50-0-05b5-f0ad State : Ready (RDY) Size (in MB)/(in sectors) : 140014/286749479 Manufacturer : SEAGATE Model Number : ST9146803SS Firmware Revision : FS03 Serial No : 3SD03F4C GUID : N/A Protocol : SAS Drive Type : SAS_HDD Device is a Enclosure services device Enclosure # : 3 Slot # : 255 SAS Address : 5005076-0-3e8e-86f9 State : Standby (SBY) Manufacturer : IBM-ESXS Model Number : SAS EXP BP Firmware Revision : 61A6 Serial No : 00000006 GUID : N/A Protocol : SAS Device Type : Enclosure services device ------------------------------------------------------------------------ Enclosure information ------------------------------------------------------------------------ Enclosure# : 1 Logical ID : 500605b0:07ba2100 Numslots : 8 StartSlot : 0 Enclosure# : 2 Logical ID : 50050760:3e8e81a0 Numslots : 25 StartSlot : 0 Enclosure# : 3 Logical ID : 50050760:3e8e86e0 Numslots : 25 StartSlot : 0 ------------------------------------------------------------------------ SAS2IRCU: Command DISPLAY Completed Successfully. SAS2IRCU: Utility Completed Successfully.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?A8A5F62D-ED7B-420F-B5B5-88842A8F8827>