From owner-freebsd-scsi@FreeBSD.ORG Tue Oct 7 15:33:36 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 94C4BF35 for ; Tue, 7 Oct 2014 15:33:36 +0000 (UTC) Received: from exprod7og126.obsmtp.com (exprod7og126.obsmtp.com [64.18.2.206]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2731D150 for ; Tue, 7 Oct 2014 15:33:35 +0000 (UTC) Received: from mail-pa0-f53.google.com ([209.85.220.53]) (using TLSv1) by exprod7ob126.postini.com ([64.18.6.12]) with SMTP ID DSNKVDQHzkEJF+kW12HzgDKI9bIW0KGGDXSP@postini.com; Tue, 07 Oct 2014 08:33:36 PDT Received: by mail-pa0-f53.google.com with SMTP id kq14so7390111pab.12 for ; Tue, 07 Oct 2014 08:33:34 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:references:in-reply-to:mime-version :thread-index:date:message-id:subject:to:cc:content-type :content-transfer-encoding; bh=rmvOVp56HH9sei0s4UNRrBI5eSkprm63YBD4FIg7CgM=; b=RE6nFHnRzd5jKnO3sg+luwUUlvmNOmknmDhuJkg8XWMGlbMHcruUV5v8mSyXtlFWum MVC+N+sDyg6jGsKbVFwoJR4FdnCQrgk49h37Hzq831Qxb4bZJ/uwk5665hd9wf8oWHSO /8DTAandgql/rBT0CsUohgEHhSoixJK57j/ZhEwPHDyhnBN9XDY2sGLk3DfM+cAFQM0T 8mVwMCPVYnpHs1YMLfmvEF2usqc3sbuGcHJQ/gFFAXiPw6KupaWT5DOT+3x6NXY2TMTF U58QJ4z2aBAqtTYKKvMh+QCfAt7iF5Xyni8Tppf0I7aJQlsDoeg3plKOyI41XoqXqFNo yn6w== X-Received: by 10.70.36.79 with SMTP id o15mr4760279pdj.6.1412695529975; Tue, 07 Oct 2014 08:25:29 -0700 (PDT) X-Gm-Message-State: ALoCoQllbp53xjXObbEDA6bZQkj6wwCaQUHqDfIPz/h+uhLWI618QLD0pIpj57MwAEoIoRg0KH5fSwzWrDdoO4yWAOMXkxkkZS/B3mO04s0Rw8rADNLjp1VKJBfUbQ181CAw9RdtnHroh8OtjrDzE6q2A7jdM3MyOQ== X-Received: by 10.70.36.79 with SMTP id o15mr4760262pdj.6.1412695529804; Tue, 07 Oct 2014 08:25:29 -0700 (PDT) From: Stephen Mcconnell References: 26cd5d2251523a832e25f393f2379227@mail.gmail.com In-Reply-To: 26cd5d2251523a832e25f393f2379227@mail.gmail.com MIME-Version: 1.0 X-Mailer: Microsoft Outlook 14.0 Thread-Index: AQHAvPDqV/NT6Mjmvn39KZFmH6q6QwHpuJfJnDKoYmCAARS+kA== Date: Tue, 7 Oct 2014 09:25:29 -0600 Message-ID: Subject: RE: mpr vs mps performance To: Alan Somers , Borja Marcos Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: FreeBSD-scsi X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Oct 2014 15:33:36 -0000 > -----Original Message----- > From: Stephen Mcconnell [mailto:stephen.mcconnell@avagotech.com] > Sent: Monday, October 06, 2014 4:55 PM > To: 'Alan Somers'; 'Borja Marcos' > Cc: 'FreeBSD-scsi' > Subject: RE: mpr vs mps performance > > > > > -----Original Message----- > > From: asomers@gmail.com [mailto:asomers@gmail.com] On Behalf Of Alan > > Somers > > Sent: Monday, October 06, 2014 10:13 AM > > To: Borja Marcos; slm@freebsd.org > > Cc: FreeBSD-scsi > > Subject: Re: mpr vs mps performance > > > > On Mon, Sep 22, 2014 at 9:39 AM, Borja Marcos wrote= : > > > > > > Hello, > > > > > > I have been playing with the new SAS3 cards supported by the mpr > > > driver, and I=C2=B4ve found out that they are, in the same hardware > > > configuration, > > considerably slower writing data. Moreover, running two simultaneous > "bonnie" > > benchmarks (I am using SSDs, and one "bonnie" sometimes hits 100% CPU > > usage, unable to really saturate the I/O) I see the writing activity > > somewhat stalling, with disk bandwidth going from 600 MB/s to around > > 50 for 20 seconds or so. > > > > > > I'd like to know if this matches anyone else's experiences. Also, I > > > can try and > > make some tests if needed. But for now it seems we will stick to the > > SAS2 > HBAs. > > > > > > > > > > > > > > > The Bonnie results are: > > > > > > With mpr driver, SAS3: > > > > > > (each bonnie instance, so multiply the results by 2 to get the > > > actual bandwidth achieved) > > > > > > Seq output: (writing) > > > Block: 292155 KB/s > > > Rewrite: 139713 KB/s > > > Seq input: > > > Block: 862861 KB/s > > > > > > With mps driver: SAS2, again, total is 2x the following figures. > > > > > > Seq output: (writing) > > > Block: 587950 KB/s > > > Rewrite: 208239 KB/s > > > Seq. input: (reading) > > > Block: 842169 KB/s > > > > > > > > > The storage is a ZFS pool with a 9-disk raidz2 vdev, made of > > > Samsung > > > 840 EVO 1 TB SSDs. The pool has been created with an ashift of 12 > > > (zpool applied it thanks to the 4 KB block quirk for these SSDs) > > > > > > at scbus0 target 9 lun 0 > > > (pass0,da0) > > > at scbus0 target 10 lun 0 > > > (pass1,da1) > > > at scbus0 target 11 lun 0 > > > (pass2,da2) > > > at scbus0 target 12 lun 0 > > > (pass3,da3) > > > at scbus0 target 13 lun 0 > > > (pass4,da4) > > > at scbus0 target 14 lun 0 > > > (pass5,da5) > > > at scbus0 target 17 lun 0 > > > (pass7,da6) > > > at scbus0 target 18 lun 0 > > > (pass8,da7) > > > at scbus0 target 27 lun 0 > > > (pass14,da12) > > > > > > > > > > > > > > > The mpr card details follow: > > > Sep 17 09:49:39 elibm kernel: mpr0: port 0x3f00-0x3fff > > > mem > > 0x912f0000-0x912fffff irq 32 at device 0.0 on pci17 > > > Sep 17 09:49:39 elibm kernel: mpr0: IOCFacts : > > > Sep 17 09:49:39 elibm kernel: MsgVersion: 0x205 Sep 17 09:49:39 > > > elibm > > > kernel: HeaderVersion: 0x1d00 Sep 17 09:49:39 elibm kernel: IOCNumber= : > > > 0 Sep 17 09:49:39 elibm kernel: IOCExceptions: 0x0 Sep 17 09:49:39 > > > elibm kernel: MaxChainDepth: 128 Sep 17 09:49:39 elibm kernel: > > > NumberOfPorts: 1 Sep 17 09:49:39 elibm kernel: RequestCredit: 11264 > > > Sep 17 09:49:39 elibm kernel: ProductID: 0x2221 Sep 17 09:49:39 > > > elibm > > > kernel: IOCRequestFrameSize: 32 Sep 17 09:49:39 elibm kernel: > > > MaxInitiators: 1 Sep 17 09:49:39 elibm kernel: MaxTargets: 1024 Sep > > > 17 > > > 09:49:39 elibm kernel: MaxSasExpanders: 14 Sep 17 09:49:39 elibm > > > kernel: MaxEnclosures: 15 Sep 17 09:49:39 elibm kernel: > > > HighPriorityCredit: 60 Sep 17 09:49:39 elibm kernel: > > > MaxReplyDescriptorPostQueueDepth: 65504 Sep 17 09:49:39 elibm kernel: > > > ReplyFrameSize: 32 Sep 17 09:49:39 elibm kernel: MaxVolumes: 0 Sep > > > 17 > > > 09:49:39 elibm kernel: MaxDevHandle: 1047 Sep 17 09:49:39 elibm > > > kernel: MaxPersistentEntries: 128 Sep 17 09:49:39 elibm kernel: mpr0: > > > Firmware: 01.00.03.00, Driver: 05.255.05.00-fbsd Sep 17 09:49:39 > > > elibm > > > kernel: mpr0: IOCCapabilities: > > > 3a85c > > XI > > > ndex,HostDisc> > > > > > > > > > And the mps card is a classic: > > > Sep 22 17:18:24 elibm kernel: mps0: port 0x3f00-0x3fff > > > mem 0x90ebc000-0x90ebffff,0x912c0000-0x912fffff irq 32 at device 0.0 > > > on pci17 Sep 22 17:18:24 elibm kernel: mps0: Firmware: 18.00.00.00, > > > Driver: 19.00.00.00-fbsd Sep 22 17:18:24 elibm kernel: mps0: > > > IOCCapabilities: > > > 1285c > > tD > > > isc> > > > > > > > > > The connected devices follow. Both use the same hardware (except for > > > the > > cables and HBA of course), but currently there's no way to check this > > with the > > SAS3 card, as sas3ircu nor sas3flash detect it on FreeBSD. > > > > > > > > > > > > # sas2ircu 0 display > > > LSI Corporation SAS2 IR Configuration Utility. > > > Version 18.00.00.00 (2013.11.18) > > > Copyright (c) 2009-2013 LSI Corporation. All rights reserved. > > > > > > Read configuration has been initiated for controller 0 > > > -------------------------------------------------------------------- > > > -- > > > -- > > > Controller information > > > ---------------------------------------------------------------------= --- > > > Controller type : SAS2008 > > > BIOS version : 7.35.00.00 > > > Firmware version : 18.00.00.00 > > > Channel description : 1 Serial Attached SCSI > > > Initiator ID : 0 > > > Maximum physical devices : 255 > > > Concurrent commands supported : 3432 > > > Slot : 3 > > > Segment : 0 > > > Bus : 17 > > > Device : 0 > > > Function : 0 > > > RAID Support : No > > > -------------------------------------------------------------------- > > > -- > > > -- > > > IR Volume information > > > -------------------------------------------------------------------- > > > -- > > > -- > > > -------------------------------------------------------------------- > > > -- > > > -- > > > Physical device information > > > -------------------------------------------------------------------- > > > -- > > > -- > > > Initiator at ID #0 > > > > > > Device is a Hard disk > > > Enclosure # : 2 > > > Slot # : 16 > > > SAS Address : 5000c50-0-05b5-ce25 > > > State : Ready (RDY) > > > Size (in MB)/(in sectors) : 140014/286749479 > > > Manufacturer : SEAGATE > > > Model Number : ST9146803SS > > > Firmware Revision : FS03 > > > Serial No : 3SD02W5L > > > GUID : N/A > > > Protocol : SAS > > > Drive Type : SAS_HDD > > > > > > Device is a Hard disk > > > Enclosure # : 2 > > > Slot # : 17 > > > SAS Address : 5005076-0-3e8e-81a2 > > > State : Ready (RDY) > > > Size (in MB)/(in sectors) : 953869/1953525167 > > > Manufacturer : ATA > > > Model Number : Samsung SSD 840 > > > Firmware Revision : BB0Q > > > Serial No : S1D9NEADA08549F > > > GUID : N/A > > > Protocol : SATA > > > Drive Type : SATA_SSD > > > > > > Device is a Hard disk > > > Enclosure # : 2 > > > Slot # : 18 > > > SAS Address : 5005076-0-3e8e-81a3 > > > State : Ready (RDY) > > > Size (in MB)/(in sectors) : 953869/1953525167 > > > Manufacturer : ATA > > > Model Number : Samsung SSD 840 > > > Firmware Revision : BB0Q > > > Serial No : S1D9NEADA08548T > > > GUID : N/A > > > Protocol : SATA > > > Drive Type : SATA_SSD > > > > > > Device is a Hard disk > > > Enclosure # : 2 > > > Slot # : 19 > > > SAS Address : 5005076-0-3e8e-81a4 > > > State : Ready (RDY) > > > Size (in MB)/(in sectors) : 953869/1953525167 > > > Manufacturer : ATA > > > Model Number : Samsung SSD 840 > > > Firmware Revision : BB0Q > > > Serial No : S1D9NEADA08568E > > > GUID : N/A > > > Protocol : SATA > > > Drive Type : SATA_SSD > > > > > > Device is a Hard disk > > > Enclosure # : 2 > > > Slot # : 20 > > > SAS Address : 5005076-0-3e8e-81a5 > > > State : Ready (RDY) > > > Size (in MB)/(in sectors) : 953869/1953525167 > > > Manufacturer : ATA > > > Model Number : Samsung SSD 840 > > > Firmware Revision : BB0Q > > > Serial No : S1D9NEADA08547X > > > GUID : N/A > > > Protocol : SATA > > > Drive Type : SATA_SSD > > > > > > Device is a Hard disk > > > Enclosure # : 2 > > > Slot # : 21 > > > SAS Address : 5005076-0-3e8e-81a6 > > > State : Ready (RDY) > > > Size (in MB)/(in sectors) : 953869/1953525167 > > > Manufacturer : ATA > > > Model Number : Samsung SSD 840 > > > Firmware Revision : BB0Q > > > Serial No : S1D9NEADA08518Y > > > GUID : N/A > > > Protocol : SATA > > > Drive Type : SATA_SSD > > > > > > Device is a Hard disk > > > Enclosure # : 2 > > > Slot # : 22 > > > SAS Address : 5005076-0-3e8e-81a7 > > > State : Ready (RDY) > > > Size (in MB)/(in sectors) : 953869/1953525167 > > > Manufacturer : ATA > > > Model Number : Samsung SSD 840 > > > Firmware Revision : BB0Q > > > Serial No : S1D9NEADA08556K > > > GUID : N/A > > > Protocol : SATA > > > Drive Type : SATA_SSD > > > > > > Device is a Enclosure services device > > > Enclosure # : 2 > > > Slot # : 255 > > > SAS Address : 5005076-0-3e8e-81b9 > > > State : Standby (SBY) > > > Manufacturer : IBM-ESXS > > > Model Number : SAS EXP BP > > > Firmware Revision : 61A6 > > > Serial No : 00000006 > > > GUID : N/A > > > Protocol : SAS > > > Device Type : Enclosure services device > > > > > > Device is a Hard disk > > > Enclosure # : 3 > > > Slot # : 0 > > > SAS Address : 5005076-0-3e8e-86e9 > > > State : Ready (RDY) > > > Size (in MB)/(in sectors) : 953869/1953525167 > > > Manufacturer : ATA > > > Model Number : Samsung SSD 840 > > > Firmware Revision : BB0Q > > > Serial No : S1D9NEADA08550R > > > GUID : N/A > > > Protocol : SATA > > > Drive Type : SATA_SSD > > > > > > Device is a Hard disk > > > Enclosure # : 3 > > > Slot # : 1 > > > SAS Address : 5005076-0-3e8e-86ea > > > State : Ready (RDY) > > > Size (in MB)/(in sectors) : 953869/1953525167 > > > Manufacturer : ATA > > > Model Number : Samsung SSD 840 > > > Firmware Revision : BB0Q > > > Serial No : S1D9NEADA08911Y > > > GUID : N/A > > > Protocol : SATA > > > Drive Type : SATA_SSD > > > > > > Device is a Hard disk > > > Enclosure # : 3 > > > Slot # : 2 > > > SAS Address : 5005076-0-3e8e-86eb > > > State : Ready (RDY) > > > Size (in MB)/(in sectors) : 953869/1953525167 > > > Manufacturer : ATA > > > Model Number : Samsung SSD 840 > > > Firmware Revision : BB0Q > > > Serial No : S1D9NEADA08811L > > > GUID : N/A > > > Protocol : SATA > > > Drive Type : SATA_SSD > > > > > > Device is a Hard disk > > > Enclosure # : 3 > > > Slot # : 13 > > > SAS Address : 5000c50-0-05b5-e531 > > > State : Ready (RDY) > > > Size (in MB)/(in sectors) : 140014/286749479 > > > Manufacturer : SEAGATE > > > Model Number : ST9146803SS > > > Firmware Revision : FS03 > > > Serial No : 3SD02STR > > > GUID : N/A > > > Protocol : SAS > > > Drive Type : SAS_HDD > > > > > > Device is a Hard disk > > > Enclosure # : 3 > > > Slot # : 14 > > > SAS Address : 5000c50-0-05b5-d489 > > > State : Ready (RDY) > > > Size (in MB)/(in sectors) : 140014/286749479 > > > Manufacturer : SEAGATE > > > Model Number : ST9146803SS > > > Firmware Revision : FS03 > > > Serial No : 3SD02TV1 > > > GUID : N/A > > > Protocol : SAS > > > Drive Type : SAS_HDD > > > > > > Device is a Hard disk > > > Enclosure # : 3 > > > Slot # : 15 > > > SAS Address : 5000c50-0-05b5-f0ad > > > State : Ready (RDY) > > > Size (in MB)/(in sectors) : 140014/286749479 > > > Manufacturer : SEAGATE > > > Model Number : ST9146803SS > > > Firmware Revision : FS03 > > > Serial No : 3SD03F4C > > > GUID : N/A > > > Protocol : SAS > > > Drive Type : SAS_HDD > > > > > > Device is a Enclosure services device > > > Enclosure # : 3 > > > Slot # : 255 > > > SAS Address : 5005076-0-3e8e-86f9 > > > State : Standby (SBY) > > > Manufacturer : IBM-ESXS > > > Model Number : SAS EXP BP > > > Firmware Revision : 61A6 > > > Serial No : 00000006 > > > GUID : N/A > > > Protocol : SAS > > > Device Type : Enclosure services device > > > -------------------------------------------------------------------- > > > -- > > > -- > > > Enclosure information > > > ---------------------------------------------------------------------= --- > > > Enclosure# : 1 > > > Logical ID : 500605b0:07ba2100 > > > Numslots : 8 > > > StartSlot : 0 > > > Enclosure# : 2 > > > Logical ID : 50050760:3e8e81a0 > > > Numslots : 25 > > > StartSlot : 0 > > > Enclosure# : 3 > > > Logical ID : 50050760:3e8e86e0 > > > Numslots : 25 > > > StartSlot : 0 > > > -------------------------------------------------------------------- > > > -- > > > -- > > > SAS2IRCU: Command DISPLAY Completed Successfully. > > > SAS2IRCU: Utility Completed Successfully. > > > > > > > > > > > > > > > _______________________________________________ > > > freebsd-scsi@freebsd.org mailing list > > > http://lists.freebsd.org/mailman/listinfo/freebsd-scsi > > > To unsubscribe, send any mail to > > > "freebsd-scsi-unsubscribe@freebsd.org" > > > > Adding Steve McConnell, who may not be on the freebsd-scsi list. > > Right, I'm not on the freebsd-scsi list. I suspect that the reason for > this is that > most of the Gen3 performance gains come from MSI-x support, but that is > not in > the mpr driver yet, and MSI-x is not supported in the 2008 card. But, I > have a > question out to another person and I'm waiting for more info. > > Steve Borja, can you upgrade your card to the latest FW. Your version is really old (1.00.03.00). We should be up to 7.00.00.00 now. There could be a lot of fixes that would affect performance. Steve