Skip site navigation (1)Skip section navigation (2)
Date:      Thu, 20 Mar 2014 16:55:34 -0700
From:      Doug Ambrisko <ambrisko@cisco.com>
To:        "Desai, Kashyap" <Kashyap.Desai@lsi.com>
Cc:        "scottl@netflix.com" <scottl@netflix.com>, "Radford, Adam" <Adam.Radford@lsi.com>, "Kenneth D. Merry" <ken@kdm.org>, "sean_bruno@yahoo.com" <sean_bruno@yahoo.com>, "Mankani, Krishnaraddi" <Krishnaraddi.Mankani@lsi.com>, "dwhite@ixsystems.com" <dwhite@ixsystems.com>, "Maloy, Joe" <Joe.Maloy@lsi.com>, "jpaetzel@freebsd.org" <jpaetzel@freebsd.org>, "freebsd-scsi@freebsd.org" <freebsd-scsi@freebsd.org>, "McConnell, Stephen" <Stephen.McConnell@lsi.com>
Subject:   Re: LSI - MR-Fusion controller driver <mrsas> patch and man page
Message-ID:  <20140320235534.GA92797@cisco.com>
In-Reply-To: <20140318143738.GA65955@cisco.com>
References:  <08ba2a262fba45f687cdd3225f325110@BN1PR07MB247.namprd07.prod.outlook.com> <20140103211449.GA69721@cisco.com> <8c423414ecc2421fbace3eb9f386be91@BN1PR07MB247.namprd07.prod.outlook.com> <20140106182935.GC93278@cisco.com> <e59396595152456dbcde63d48f70aa8f@BN1PR07MB247.namprd07.prod.outlook.com> <20140107181139.GC2080@cisco.com> <20140124185356.GA28724@ambrisko.com> <20140124190047.GA34975@ambrisko.com> <9c3fd2b15e9b4c2cb967519a3b7f98ad@BN1PR07MB247.namprd07.prod.outlook.com> <20140318143738.GA65955@cisco.com>

next in thread | previous in thread | raw e-mail | index | archive | help
I kicked the tires.  I did some tests with our home grown management
tool that uses the standard LSI ioctl format and that has worked okay.
I got a panic before when recreating a RAID.  So things have improved.

Doing some performance tests, I seem to have tracked down how it
happens and why I didn't see a long time ago.  Performance is okay
with mfi versus mrasas on ThunderBolt cards (ie. 9266-8i) but when
I used an Invader card (9361-8i) read performance is really bad.
I'm using 4, 300G, 2.5" SAS disk (Toshiba AL13SEB300) in a RAID 10
configuration.  Write performance is fine, read performance is
really bad.  With mfi performance is fine for read and write.
The mrsas write performance is ~302974Kbytes and ~7738Kbytes for
read.  That is really bad.  It can be seen with just a
	dd if=/dev/da0 of=/dev/zero bs=1m
With mfi I see ~364128Kbytes writes and ~214817Kbytes reads.  So
something is strange.  The firmware on card is:
  Name: LSI MegaRAID 9361-8i
  Serial Number: SV33505325
  Firmware BIOS 6.13.00.1_4.14.05.00_0x06010600
  Firmware CTLR 5.01-0005
  Firmware APP  4.210.50-3015
  Firmware NVDT 3.1310.00-0062
  Firmware BTBL 3.00.00.00-0009
which was the lastest when I looks on the LSI web site a month ago or so.

Any ideas of how to debug this?

I recently started testing with the Invader card so that is why I started
seeing performance issues.  With ThunderBolt performance is okay and
about the same.

Thanks,

Doug A.

On Tue, Mar 18, 2014 at 07:37:38AM -0700, Doug Ambrisko wrote:
| Sounds good.  I'll try it out and see how it works.
| 
| I won't be able to make today's meeting. 
| 
| Thanks,
| 
| Doug A.
| 
| On Tue, Mar 18, 2014 at 09:23:23AM +0000, Desai, Kashyap wrote:
| | Hi,
| | 
| | Please find the latest <mrsas> patch for upstream commit.
| | 
| | - This patch include all latest and greatest feature supported by 
| |   <mrsas> driver released at LSI external site.
| | - mfi vs mrsas interopt is resolved using latest <mfi> commit which uses 
| |   tunable " hw.mfi.mrsas_enable".
| | 
| | <mrsas> man page is still under re-work. I will be posting <mrsas> man 
| | page as a follow up action item.
| | 
| | Thanks, Kashyap
| | 
| | > -----Original Message-----
| | > From: Doug Ambrisko [mailto:ambrisko@ambrisko.com]
| | > Sent: Saturday, January 25, 2014 12:31 AM
| | > To: Doug Ambrisko
| | > Cc: Desai, Kashyap; freebsd-scsi@freebsd.org; scottl@netflix.com;
| | > sean_bruno@yahoo.com; dwhite@ixsystems.com; jpaetzel@freebsd.org;
| | > Maloy, Joe; Mankani, Krishnaraddi; McConnell, Stephen; Saxena, Sumit;
| | > Radford, Adam; Kenneth D. Merry
| | > Subject: Re: LSI - MR-Fusion controller driver <mrsas> patch and man page
| | > 
| | > On Fri, Jan 24, 2014 at 10:53:56AM -0800, Doug Ambrisko wrote:
| | > | On Tue, Jan 07, 2014 at 10:11:39AM -0800, Doug Ambrisko wrote:
| | > | 	[snip]
| | > | | Yes, we can probably make the minimal change to mfi to allow mrsas
| | > | | to optionally take over.  That can probably be done the quickest.
| | > |
| | > | Here is the patch I propose to mfi(4) to allow mrsas(4) to optionally
| | > | take newer cards.
| | > 
| | > I noticed that this patch is partially incomplete since I didn't have
| | > FLAGS_MRSAS added to all of the TBOLT ID's.  I'll fix that in the commit.
| | > 
| | > | Index: mfi_pci.c
| | > |
| | > ==========================================================
| | > =========
| | > | --- mfi_pci.c	(revision 260231)
| | > | +++ mfi_pci.c	(working copy)
| | > | @@ -112,6 +112,11 @@
| | > |  SYSCTL_INT(_hw_mfi, OID_AUTO, msi, CTLFLAG_RDTUN, &mfi_msi, 0,
| | > |      "Enable use of MSI interrupts");
| | > |
| | > | +static int	mfi_mrsas_enable = 0;
| | > | +TUNABLE_INT("hw.mfi.mrsas_enable", &mfi_msi); SYSCTL_INT(_hw_mfi,
| | > | +OID_AUTO, mrsas_enable, CTLFLAG_RDTUN, &mfi_mrsas_enable,
| | > | +     0, "Allow mrasas to take newer cards");
| | > | +
| | > |  struct mfi_ident {
| | > |  	uint16_t	vendor;
| | > |  	uint16_t	device;
| | > | @@ -127,11 +132,11 @@
| | > |  	{0x1000, 0x005b, 0x1028, 0x1f34, MFI_FLAGS_SKINNY|
| | > MFI_FLAGS_TBOLT, "Dell PERC H710P Mini (monolithics)"},
| | > |  	{0x1000, 0x005b, 0x1028, 0x1f35, MFI_FLAGS_SKINNY|
| | > MFI_FLAGS_TBOLT, "Dell PERC H710 Adapter"},
| | > |  	{0x1000, 0x005b, 0x1028, 0x1f37, MFI_FLAGS_SKINNY|
| | > MFI_FLAGS_TBOLT, "Dell PERC H710 Mini (blades)"},
| | > | -	{0x1000, 0x005b, 0x1028, 0x1f38, MFI_FLAGS_SKINNY|
| | > MFI_FLAGS_TBOLT, "Dell PERC H710 Mini (monolithics)"},
| | > | -	{0x1000, 0x005b, 0x8086, 0x9265, MFI_FLAGS_SKINNY|
| | > MFI_FLAGS_TBOLT, "Intel (R) RAID Controller RS25DB080"},
| | > | -	{0x1000, 0x005b, 0x8086, 0x9285, MFI_FLAGS_SKINNY|
| | > MFI_FLAGS_TBOLT, "Intel (R) RAID Controller RS25NB008"},
| | > | -	{0x1000, 0x005b, 0xffff, 0xffff, MFI_FLAGS_SKINNY|
| | > MFI_FLAGS_TBOLT, "ThunderBolt"},
| | > | -	{0x1000, 0x005d, 0xffff, 0xffff, MFI_FLAGS_SKINNY|
| | > MFI_FLAGS_TBOLT, "Invader"},
| | > | +	{0x1000, 0x005b, 0x1028, 0x1f38, MFI_FLAGS_SKINNY|
| | > MFI_FLAGS_TBOLT| MFI_FLAGS_MRSAS, "Dell PERC H710 Mini
| | > (monolithics)"},
| | > | +	{0x1000, 0x005b, 0x8086, 0x9265, MFI_FLAGS_SKINNY|
| | > MFI_FLAGS_TBOLT| MFI_FLAGS_MRSAS, "Intel (R) RAID Controller
| | > RS25DB080"},
| | > | +	{0x1000, 0x005b, 0x8086, 0x9285, MFI_FLAGS_SKINNY|
| | > MFI_FLAGS_TBOLT| MFI_FLAGS_MRSAS, "Intel (R) RAID Controller
| | > RS25NB008"},
| | > | +	{0x1000, 0x005b, 0xffff, 0xffff, MFI_FLAGS_SKINNY|
| | > MFI_FLAGS_TBOLT| MFI_FLAGS_MRSAS, "ThunderBolt"},
| | > | +	{0x1000, 0x005d, 0xffff, 0xffff, MFI_FLAGS_SKINNY|
| | > MFI_FLAGS_TBOLT|
| | > | +MFI_FLAGS_MRSAS, "Invader"},
| | > |  	{0x1000, 0x0060, 0x1028, 0xffff, MFI_FLAGS_1078,  "Dell PERC 6"},
| | > |  	{0x1000, 0x0060, 0xffff, 0xffff, MFI_FLAGS_1078,  "LSI MegaSAS
| | > 1078"},
| | > |  	{0x1000, 0x0071, 0xffff, 0xffff, MFI_FLAGS_SKINNY, "Drake Skinny"},
| | > | @@ -178,7 +183,13 @@
| | > |
| | > |  	if ((id = mfi_find_ident(dev)) != NULL) {
| | > |  		device_set_desc(dev, id->desc);
| | > | -		return (BUS_PROBE_DEFAULT);
| | > | +
| | > | +		/* give priority to mrsas if tunable set */
| | > | +		TUNABLE_INT_FETCH("hw.mfi.mrsas_enable",
| | > &mfi_mrsas_enable);
| | > | +		if ((id->flags & MFI_FLAGS_MRSAS) && mfi_mrsas_enable)
| | > | +			return (BUS_PROBE_LOW_PRIORITY);
| | > | +		else
| | > | +			return (BUS_PROBE_DEFAULT);
| | > |  	}
| | > |  	return (ENXIO);
| | > |  }
| | > | Index: mfivar.h
| | > |
| | > ==========================================================
| | > =========
| | > | --- mfivar.h	(revision 260231)
| | > | +++ mfivar.h	(working copy)
| | > | @@ -199,6 +199,7 @@
| | > |  #define MFI_FLAGS_GEN2		(1<<6)
| | > |  #define MFI_FLAGS_SKINNY	(1<<7)
| | > |  #define MFI_FLAGS_TBOLT		(1<<8)
| | > | +#define MFI_FLAGS_MRSAS		(1<<9)
| | > |  	// Start: LSIP200113393
| | > |  	bus_dma_tag_t			verbuf_h_dmat;
| | > |  	bus_dmamap_t			verbuf_h_dmamap;
| | > |
| | > | This creates a hw.mfi.mrsas_enable tunable to control it.  The method
| | > | via hints wasn't the best since for one the unit index was being
| | > | abused a non-unit specfic option.  It was also a little strange to
| | > | have mrsas hint be in mfi(4).
| | > |
| | > | Then we need a minor change to mrsas.c
| | > |
| | > |
| | > | --- ../mrsas.orig/mrsas.c	2014-01-03 11:30:28.000000000 -0800
| | > | +++ ./mrsas.c	2014-01-24 10:43:20.000000000 -0800
| | > | @@ -328,25 +328,11 @@ static struct mrsas_ident * mrsas_find_i  static
| | > | int mrsas_probe(device_t dev)  {
| | > |      struct mrsas_ident *id;
| | > | -	unsigned int force = 0, ivar;
| | > |
| | > |      if ((id = mrsas_find_ident(dev)) != NULL) {
| | > | -		if (id->device == 0x005b || id->device == 0x005d) {
| | > | -            resource_int_value("mrsas", 0, "fusion_force", &ivar);
| | > | -
| | > | -			if (ivar == 0 || ivar == 1)
| | > | -		            force = ivar;
| | > | -
| | > | -            device_set_desc(dev, id->desc);
| | > | -            if (force)
| | > | -            	return (BUS_PROBE_DEFAULT);
| | > | -                //return (BUS_PROBE_SPECIFIC);  //give priority to MFI driver
| | > | -            else
| | > | -                return (BUS_PROBE_LOW_PRIORITY);
| | > | -        }
| | > | -        else
| | > | -            device_set_desc(dev, id->desc);
| | > | -            return (BUS_PROBE_DEFAULT);
| | > | +	device_set_desc(dev, id->desc);
| | > | +	/* between BUS_PROBE_DEFAULT and BUS_PROBE_LOW_PRIORITY
| | > */
| | > | +	return (-30);
| | > |      }
| | > |      return (ENXIO);
| | > |
| | > | So that its probe part way between mfi(4) results and then it doesn't
| | > | have to change.
| | > |
| | > | If no one has concerns then I'll check in the mfi(4) change.
| | > 
| | > Thanks,
| | > 
| | > Doug A.
| | 
| 
| 



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20140320235534.GA92797>