Skip site navigation (1)Skip section navigation (2)
Date:      Mon, 6 Oct 2014 10:13:06 -0600
From:      Alan Somers <asomers@freebsd.org>
To:        Borja Marcos <borjam@sarenet.es>, slm@freebsd.org
Cc:        FreeBSD-scsi <freebsd-scsi@freebsd.org>
Subject:   Re: mpr vs mps performance
Message-ID:  <CAOtMX2jVR-8wkHcTyudOVwMNmxOvDkWPNsTBuyu%2BWZQ6WtDvmg@mail.gmail.com>
In-Reply-To: <A8A5F62D-ED7B-420F-B5B5-88842A8F8827@sarenet.es>
References:  <A8A5F62D-ED7B-420F-B5B5-88842A8F8827@sarenet.es>

next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Sep 22, 2014 at 9:39 AM, Borja Marcos <borjam@sarenet.es> wrote:
>
> Hello,
>
> I have been playing with the new SAS3 cards supported by the mpr driver, =
and I=C2=B4ve found out that they are, in the same hardware
> configuration, considerably slower writing data. Moreover, running two si=
multaneous "bonnie"  benchmarks (I am using SSDs, and one "bonnie" sometime=
s hits 100% CPU usage, unable to really saturate the I/O) I see the writing=
 activity somewhat stalling, with disk bandwidth going from 600 MB/s to aro=
und 50 for 20 seconds or so.
>
> I'd like to know if this matches anyone else's experiences. Also, I can t=
ry and make some tests if needed. But for now it seems we will stick to the=
 SAS2 HBAs.
>
>
>
>
> The Bonnie results are:
>
> With mpr driver, SAS3:
>
> (each bonnie instance, so multiply the results by 2 to get the actual ban=
dwidth achieved)
>
> Seq output: (writing)
>         Block:  292155 KB/s
>         Rewrite: 139713 KB/s
> Seq input:
>         Block:    862861 KB/s
>
> With mps driver: SAS2, again, total is 2x the following figures.
>
> Seq output: (writing)
>         Block:  587950  KB/s
>         Rewrite: 208239 KB/s
> Seq. input: (reading)
>         Block: 842169 KB/s
>
>
> The storage is a ZFS pool with a 9-disk raidz2 vdev, made  of Samsung 840=
 EVO 1 TB SSDs. The pool has been created with an ashift of 12 (zpool appli=
ed it thanks to the 4 KB block quirk for these SSDs)
>
> <ATA Samsung SSD 840 BB0Q>         at scbus0 target 9 lun 0 (pass0,da0)
> <ATA Samsung SSD 840 BB0Q>         at scbus0 target 10 lun 0 (pass1,da1)
> <ATA Samsung SSD 840 BB0Q>         at scbus0 target 11 lun 0 (pass2,da2)
> <ATA Samsung SSD 840 BB0Q>         at scbus0 target 12 lun 0 (pass3,da3)
> <ATA Samsung SSD 840 BB0Q>         at scbus0 target 13 lun 0 (pass4,da4)
> <ATA Samsung SSD 840 BB0Q>         at scbus0 target 14 lun 0 (pass5,da5)
> <ATA Samsung SSD 840 BB0Q>         at scbus0 target 17 lun 0 (pass7,da6)
> <ATA Samsung SSD 840 BB0Q>         at scbus0 target 18 lun 0 (pass8,da7)
> <ATA Samsung SSD 840 BB0Q>         at scbus0 target 27 lun 0 (pass14,da12=
)
>
>
>
>
> The mpr card details follow:
> Sep 17 09:49:39 elibm kernel: mpr0: <LSI SAS3008> port 0x3f00-0x3fff mem =
0x912f0000-0x912fffff irq 32 at device 0.0 on pci17
> Sep 17 09:49:39 elibm kernel: mpr0: IOCFacts    :
> Sep 17 09:49:39 elibm kernel: MsgVersion: 0x205
> Sep 17 09:49:39 elibm kernel: HeaderVersion: 0x1d00
> Sep 17 09:49:39 elibm kernel: IOCNumber: 0
> Sep 17 09:49:39 elibm kernel: IOCExceptions: 0x0
> Sep 17 09:49:39 elibm kernel: MaxChainDepth: 128
> Sep 17 09:49:39 elibm kernel: NumberOfPorts: 1
> Sep 17 09:49:39 elibm kernel: RequestCredit: 11264
> Sep 17 09:49:39 elibm kernel: ProductID: 0x2221
> Sep 17 09:49:39 elibm kernel: IOCRequestFrameSize: 32
> Sep 17 09:49:39 elibm kernel: MaxInitiators: 1
> Sep 17 09:49:39 elibm kernel: MaxTargets: 1024
> Sep 17 09:49:39 elibm kernel: MaxSasExpanders: 14
> Sep 17 09:49:39 elibm kernel: MaxEnclosures: 15
> Sep 17 09:49:39 elibm kernel: HighPriorityCredit: 60
> Sep 17 09:49:39 elibm kernel: MaxReplyDescriptorPostQueueDepth: 65504
> Sep 17 09:49:39 elibm kernel: ReplyFrameSize: 32
> Sep 17 09:49:39 elibm kernel: MaxVolumes: 0
> Sep 17 09:49:39 elibm kernel: MaxDevHandle: 1047
> Sep 17 09:49:39 elibm kernel: MaxPersistentEntries: 128
> Sep 17 09:49:39 elibm kernel: mpr0: Firmware: 01.00.03.00, Driver: 05.255=
.05.00-fbsd
> Sep 17 09:49:39 elibm kernel: mpr0: IOCCapabilities: 3a85c<ScsiTaskFull,D=
iagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc>
>
>
> And the  mps card is a classic:
> Sep 22 17:18:24 elibm kernel: mps0: <LSI SAS2008> port 0x3f00-0x3fff mem =
0x90ebc000-0x90ebffff,0x912c0000-0x912fffff irq 32 at device 0.0 on pci17
> Sep 22 17:18:24 elibm kernel: mps0: Firmware: 18.00.00.00, Driver: 19.00.=
00.00-fbsd
> Sep 22 17:18:24 elibm kernel: mps0: IOCCapabilities: 1285c<ScsiTaskFull,D=
iagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
>
>
> The connected devices follow. Both use the same hardware (except for the =
cables and HBA of course), but currently there's no way to check this with =
the SAS3 card, as sas3ircu nor sas3flash detect it on FreeBSD.
>
>
>
> # sas2ircu 0 display
> LSI Corporation SAS2 IR Configuration Utility.
> Version 18.00.00.00 (2013.11.18)
> Copyright (c) 2009-2013 LSI Corporation. All rights reserved.
>
> Read configuration has been initiated for controller 0
> ------------------------------------------------------------------------
> Controller information
> ------------------------------------------------------------------------
>   Controller type                         : SAS2008
>   BIOS version                            : 7.35.00.00
>   Firmware version                        : 18.00.00.00
>   Channel description                     : 1 Serial Attached SCSI
>   Initiator ID                            : 0
>   Maximum physical devices                : 255
>   Concurrent commands supported           : 3432
>   Slot                                    : 3
>   Segment                                 : 0
>   Bus                                     : 17
>   Device                                  : 0
>   Function                                : 0
>   RAID Support                            : No
> ------------------------------------------------------------------------
> IR Volume information
> ------------------------------------------------------------------------
> ------------------------------------------------------------------------
> Physical device information
> ------------------------------------------------------------------------
> Initiator at ID #0
>
> Device is a Hard disk
>   Enclosure #                             : 2
>   Slot #                                  : 16
>   SAS Address                             : 5000c50-0-05b5-ce25
>   State                                   : Ready (RDY)
>   Size (in MB)/(in sectors)               : 140014/286749479
>   Manufacturer                            : SEAGATE
>   Model Number                            : ST9146803SS
>   Firmware Revision                       : FS03
>   Serial No                               : 3SD02W5L
>   GUID                                    : N/A
>   Protocol                                : SAS
>   Drive Type                              : SAS_HDD
>
> Device is a Hard disk
>   Enclosure #                             : 2
>   Slot #                                  : 17
>   SAS Address                             : 5005076-0-3e8e-81a2
>   State                                   : Ready (RDY)
>   Size (in MB)/(in sectors)               : 953869/1953525167
>   Manufacturer                            : ATA
>   Model Number                            : Samsung SSD 840
>   Firmware Revision                       : BB0Q
>   Serial No                               : S1D9NEADA08549F
>   GUID                                    : N/A
>   Protocol                                : SATA
>   Drive Type                              : SATA_SSD
>
> Device is a Hard disk
>   Enclosure #                             : 2
>   Slot #                                  : 18
>   SAS Address                             : 5005076-0-3e8e-81a3
>   State                                   : Ready (RDY)
>   Size (in MB)/(in sectors)               : 953869/1953525167
>   Manufacturer                            : ATA
>   Model Number                            : Samsung SSD 840
>   Firmware Revision                       : BB0Q
>   Serial No                               : S1D9NEADA08548T
>   GUID                                    : N/A
>   Protocol                                : SATA
>   Drive Type                              : SATA_SSD
>
> Device is a Hard disk
>   Enclosure #                             : 2
>   Slot #                                  : 19
>   SAS Address                             : 5005076-0-3e8e-81a4
>   State                                   : Ready (RDY)
>   Size (in MB)/(in sectors)               : 953869/1953525167
>   Manufacturer                            : ATA
>   Model Number                            : Samsung SSD 840
>   Firmware Revision                       : BB0Q
>   Serial No                               : S1D9NEADA08568E
>   GUID                                    : N/A
>   Protocol                                : SATA
>   Drive Type                              : SATA_SSD
>
> Device is a Hard disk
>   Enclosure #                             : 2
>   Slot #                                  : 20
>   SAS Address                             : 5005076-0-3e8e-81a5
>   State                                   : Ready (RDY)
>   Size (in MB)/(in sectors)               : 953869/1953525167
>   Manufacturer                            : ATA
>   Model Number                            : Samsung SSD 840
>   Firmware Revision                       : BB0Q
>   Serial No                               : S1D9NEADA08547X
>   GUID                                    : N/A
>   Protocol                                : SATA
>   Drive Type                              : SATA_SSD
>
> Device is a Hard disk
>   Enclosure #                             : 2
>   Slot #                                  : 21
>   SAS Address                             : 5005076-0-3e8e-81a6
>   State                                   : Ready (RDY)
>   Size (in MB)/(in sectors)               : 953869/1953525167
>   Manufacturer                            : ATA
>   Model Number                            : Samsung SSD 840
>   Firmware Revision                       : BB0Q
>   Serial No                               : S1D9NEADA08518Y
>   GUID                                    : N/A
>   Protocol                                : SATA
>   Drive Type                              : SATA_SSD
>
> Device is a Hard disk
>   Enclosure #                             : 2
>   Slot #                                  : 22
>   SAS Address                             : 5005076-0-3e8e-81a7
>   State                                   : Ready (RDY)
>   Size (in MB)/(in sectors)               : 953869/1953525167
>   Manufacturer                            : ATA
>   Model Number                            : Samsung SSD 840
>   Firmware Revision                       : BB0Q
>   Serial No                               : S1D9NEADA08556K
>   GUID                                    : N/A
>   Protocol                                : SATA
>   Drive Type                              : SATA_SSD
>
> Device is a Enclosure services device
>   Enclosure #                             : 2
>   Slot #                                  : 255
>   SAS Address                             : 5005076-0-3e8e-81b9
>   State                                   : Standby (SBY)
>   Manufacturer                            : IBM-ESXS
>   Model Number                            : SAS EXP BP
>   Firmware Revision                       : 61A6
>   Serial No                               : 00000006
>   GUID                                    : N/A
>   Protocol                                : SAS
>   Device Type                             : Enclosure services device
>
> Device is a Hard disk
>   Enclosure #                             : 3
>   Slot #                                  : 0
>   SAS Address                             : 5005076-0-3e8e-86e9
>   State                                   : Ready (RDY)
>   Size (in MB)/(in sectors)               : 953869/1953525167
>   Manufacturer                            : ATA
>   Model Number                            : Samsung SSD 840
>   Firmware Revision                       : BB0Q
>   Serial No                               : S1D9NEADA08550R
>   GUID                                    : N/A
>   Protocol                                : SATA
>   Drive Type                              : SATA_SSD
>
> Device is a Hard disk
>   Enclosure #                             : 3
>   Slot #                                  : 1
>   SAS Address                             : 5005076-0-3e8e-86ea
>   State                                   : Ready (RDY)
>   Size (in MB)/(in sectors)               : 953869/1953525167
>   Manufacturer                            : ATA
>   Model Number                            : Samsung SSD 840
>   Firmware Revision                       : BB0Q
>   Serial No                               : S1D9NEADA08911Y
>   GUID                                    : N/A
>   Protocol                                : SATA
>   Drive Type                              : SATA_SSD
>
> Device is a Hard disk
>   Enclosure #                             : 3
>   Slot #                                  : 2
>   SAS Address                             : 5005076-0-3e8e-86eb
>   State                                   : Ready (RDY)
>   Size (in MB)/(in sectors)               : 953869/1953525167
>   Manufacturer                            : ATA
>   Model Number                            : Samsung SSD 840
>   Firmware Revision                       : BB0Q
>   Serial No                               : S1D9NEADA08811L
>   GUID                                    : N/A
>   Protocol                                : SATA
>   Drive Type                              : SATA_SSD
>
> Device is a Hard disk
>   Enclosure #                             : 3
>   Slot #                                  : 13
>   SAS Address                             : 5000c50-0-05b5-e531
>   State                                   : Ready (RDY)
>   Size (in MB)/(in sectors)               : 140014/286749479
>   Manufacturer                            : SEAGATE
>   Model Number                            : ST9146803SS
>   Firmware Revision                       : FS03
>   Serial No                               : 3SD02STR
>   GUID                                    : N/A
>   Protocol                                : SAS
>   Drive Type                              : SAS_HDD
>
> Device is a Hard disk
>   Enclosure #                             : 3
>   Slot #                                  : 14
>   SAS Address                             : 5000c50-0-05b5-d489
>   State                                   : Ready (RDY)
>   Size (in MB)/(in sectors)               : 140014/286749479
>   Manufacturer                            : SEAGATE
>   Model Number                            : ST9146803SS
>   Firmware Revision                       : FS03
>   Serial No                               : 3SD02TV1
>   GUID                                    : N/A
>   Protocol                                : SAS
>   Drive Type                              : SAS_HDD
>
> Device is a Hard disk
>   Enclosure #                             : 3
>   Slot #                                  : 15
>   SAS Address                             : 5000c50-0-05b5-f0ad
>   State                                   : Ready (RDY)
>   Size (in MB)/(in sectors)               : 140014/286749479
>   Manufacturer                            : SEAGATE
>   Model Number                            : ST9146803SS
>   Firmware Revision                       : FS03
>   Serial No                               : 3SD03F4C
>   GUID                                    : N/A
>   Protocol                                : SAS
>   Drive Type                              : SAS_HDD
>
> Device is a Enclosure services device
>   Enclosure #                             : 3
>   Slot #                                  : 255
>   SAS Address                             : 5005076-0-3e8e-86f9
>   State                                   : Standby (SBY)
>   Manufacturer                            : IBM-ESXS
>   Model Number                            : SAS EXP BP
>   Firmware Revision                       : 61A6
>   Serial No                               : 00000006
>   GUID                                    : N/A
>   Protocol                                : SAS
>   Device Type                             : Enclosure services device
> ------------------------------------------------------------------------
> Enclosure information
> ------------------------------------------------------------------------
>   Enclosure#                              : 1
>   Logical ID                              : 500605b0:07ba2100
>   Numslots                                : 8
>   StartSlot                               : 0
>   Enclosure#                              : 2
>   Logical ID                              : 50050760:3e8e81a0
>   Numslots                                : 25
>   StartSlot                               : 0
>   Enclosure#                              : 3
>   Logical ID                              : 50050760:3e8e86e0
>   Numslots                                : 25
>   StartSlot                               : 0
> ------------------------------------------------------------------------
> SAS2IRCU: Command DISPLAY Completed Successfully.
> SAS2IRCU: Utility Completed Successfully.
>
>
>
>
> _______________________________________________
> freebsd-scsi@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-scsi
> To unsubscribe, send any mail to "freebsd-scsi-unsubscribe@freebsd.org"

Adding Steve McConnell, who may not be on the freebsd-scsi list.



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOtMX2jVR-8wkHcTyudOVwMNmxOvDkWPNsTBuyu%2BWZQ6WtDvmg>