Date: Tue, 7 Jan 2014 05:37:17 +0000 From: "Desai, Kashyap" <Kashyap.Desai@lsi.com> To: Doug Ambrisko <ambrisko@cisco.com> Cc: "scottl@netflix.com" <scottl@netflix.com>, "Radford, Adam" <Adam.Radford@lsi.com>, "Kenneth D. Merry" <ken@kdm.org>, "sean_bruno@yahoo.com" <sean_bruno@yahoo.com>, "Mankani, Krishnaraddi" <Krishnaraddi.Mankani@lsi.com>, "dwhite@ixsystems.com" <dwhite@ixsystems.com>, "Maloy, Joe" <Joe.Maloy@lsi.com>, "jpaetzel@freebsd.org" <jpaetzel@freebsd.org>, "freebsd-scsi@freebsd.org" <freebsd-scsi@freebsd.org>, "McConnell, Stephen" <Stephen.McConnell@lsi.com> Subject: RE: LSI - MR-Fusion controller driver <mrsas> patch and man page Message-ID: <e59396595152456dbcde63d48f70aa8f@BN1PR07MB247.namprd07.prod.outlook.com> In-Reply-To: <20140106182935.GC93278@cisco.com> References: <08ba2a262fba45f687cdd3225f325110@BN1PR07MB247.namprd07.prod.outlook.com> <20140103211449.GA69721@cisco.com> <8c423414ecc2421fbace3eb9f386be91@BN1PR07MB247.namprd07.prod.outlook.com> <20140106182935.GC93278@cisco.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Doug: I have replied to your mail at very high level and not going for inline rep= ly to your respond to capture key issue and possible resolution.=20 Hi All, There are two biggest challenge for <mrsas> drivers. 1. <mrsas> conflict with <mfi> for Thunderbolt, Invader and Fury controller= s.=20 As we discussed, we have resolve this giving priority to <mfi> controller a= nd only when customer wants use <mrsas> they need proper hint support. Currently, there are customer who is using <mrsas> posted from LSI external= site and able to configure using storcli and driver is working well for th= eir need. Definitely, we are doing lots of enhancement and defect fixes in those area= s, but note that before we work on actual issue, lots of customer complain = that <mrsas> is not loaded due to <mfi> and <mrsas> PnP ID conflict. Please= understand that, this is bit critical for customers who want to use <mrsas= >. To this issue, we need svn commits in <mfi> driver (from submitted patch s= ee only <mfi> related changes) to understand device.hints at least, so that= customer can use <mrsas> driver from lsi external site. Anything related to <mfi>/<mrsas> interopt can be considered a different de= sign goal and requirement. =20 E.a "We should be fine not have alias of /dev/daX to /dev/mfidX for initia= l drop. Probably we should document this constraint for initial commit of = mrsas" E.a "Please consider that customer is not using <mfiutils> to configure wh= en <mrsas> driver is used."=20 Till now customer is not having any issue using <strocli64> with <mrsas>. = We may go for good amount of interopt development to support configuration = utils as <mfiutils>, but that may need time. 2. <mrsas> is only available via LSI external site. We need <mrsas> upstre= am, so that customer should not wait for LSI external site downloads for mo= re generic use case. This can be done in different phases, as LSI is doing for <mps> driver. Th= ere are good comment posted by Doug and we are working on those. We do not wanted to change code without posting for internal Q/A testing. W= e should take advantage of LSI internal Q/A testing to fix most of the defe= ct internally. To resolve above issue, we should find middle way to post the <mrsas> check= -in in mainline. I mean, keep few know limitation as documented and do not= wait for gathering of all feature in single drop.=20 E.a If we start changing code to meet style(9) coding standard, we may add= some manual mistakes in code. We can take each action item as Enhancement= request (Including style(9) change as well) for LSI and we will be doing a= ll those step by step.( As we are doing for <mps> driver). I am not aware of project method to do svn check, since I worked with Ken D= Merry earlier to do actual commit in FreeBSD head repo. I thought it is go= ing to be same way for <mrsas>. Sorry, but doing lots of discussion in single mail thread is prone to diver= t actual agenda and we may hang forever without any conclusion. So, let's open defect in <mrsas> as soon as it get upstream, LSI will work = for all the defect against <mrsas>.=20 At least immediate check in <mfi> diver to understand device.hints, will al= low customer to use <mrsas> from LSI external site. Thanks, Kashyap > -----Original Message----- > From: Doug Ambrisko [mailto:ambrisko@cisco.com] > Sent: Tuesday, January 07, 2014 12:00 AM > To: Desai, Kashyap > Cc: freebsd-scsi@freebsd.org; scottl@netflix.com; sean_bruno@yahoo.com; > dwhite@ixsystems.com; jpaetzel@freebsd.org; Maloy, Joe; Mankani, > Krishnaraddi; McConnell, Stephen; Saxena, Sumit; Radford, Adam; Kenneth > D. Merry > Subject: Re: LSI - MR-Fusion controller driver <mrsas> patch and man page >=20 > On Mon, Jan 06, 2014 at 07:43:27AM +0000, Desai, Kashyap wrote: > | Doug: > | > | I have replied inline for your comment. For most of the comment you > | posted below can be work out and we can prioritize completion of those > | based on nature of issue. > | I will work on READ/WRITE comparison w.r.t <mfi> and <mrsas> as a > | priority item. >=20 > IO performance is the highest priority. >=20 > | Please consider that it will be really helpful for further > | development, if we have common code check-in in Upstream FreeBSD. We > | will actively work on each comment posted for <mrsas> and address > | those before we have <mrsas> in-box for any FreeBSD release. >=20 > What we have done in the past is to create a project branch to do this an= d it > has worked well. It would be nice if you could check in directly to a th= is > project branch so we could use svn to resolve differences. >=20 > I poked around at adding the ioctl compatibility support and noticed that > mrsas seems to lack a general method to send a dcmd to the controller. > What Scott did in mfi was to create a generic function to send dcmd's wit= h a > simple data buffer. That data buffer was then put into the sg list in th= e > generic function. This made it easier to translate the various ioctl's s= ince we > could convert the sg lists passed down from the user land into a simple > malloced data base and then covert back out. > In the mfituil ioctl method it just passes down the data buffer and then > let's the driver convert it to a sg list. To make ioctl emulation > easier in these cases: > 32 bit FreeBSD LSI ioctl on 64 bit > 32 bit Linux LSI ioctl on 32 bit > 32 bit Linux LSI ioctl on 64 bit > 32 bit mfiutil ioctl on 32 bit > 32 bit mfiutil ioctl on 64 bit >=20 > Of high priority is style, please make it style(9) compliant. There are = a trailing > spaces, spaces instead of tabs, I think lack of tab aligned indent. I no= ticed > functions being externally defined in .c files instead of .h. There shou= ld be .h > files that can be included in user land applications for ioctls and there= can be > .h files just for the kernel. > These files will need to be installed as appropriate. I found this with = the ioctl > function. >=20 > | > -----Original Message----- > | > From: Doug Ambrisko [mailto:ambrisko@cisco.com] > | > Sent: Saturday, January 04, 2014 2:45 AM > | > To: Desai, Kashyap > | > Cc: freebsd-scsi@freebsd.org; scottl@netflix.com; > | > sean_bruno@yahoo.com; dwhite@ixsystems.com; > jpaetzel@freebsd.org; > | > Maloy, Joe; Mankani, Krishnaraddi; McConnell, Stephen; Saxena, > | > Sumit; Radford, Adam; Kenneth D. Merry > | > Subject: Re: LSI - MR-Fusion controller driver <mrsas> patch and man > | > page > | > > | > On Fri, Dec 06, 2013 at 02:37:21PM +0000, Desai, Kashyap wrote: > | > | Hi, > | > | > | > | Please consider attached patch for FreeBSD upstream check-in. > | > | > | > | Please find attached patch for <mrsas> driver for LSI's MegaRaid > | > Controllers. This driver supports Thunderbolt onwards Device IDs of > | > MR controllers. > | > | Currently it supports 0x005B and 0x005D Device IDs. > | > | > | > | NOTE : This driver will not eliminate or by pass any functionality > | > | of <mfi> > | > driver which already support above to Device IDs to keep existing > | > user experience unchanged. > | > | > | > | <mfi> Driver will be always given priority over <mrsas> driver and > | > | only if customer/user wants to use/migrate from <mfi> to <mrsas>, > | > | it will hook up into kernel via device.hint rules. (Attached is > | > | mrsas man page for more info.) > | > | > | > | LSI will continue to update <mrsas> driver in future in timely base= s. > | > | We have another set of patch in pipeline to be submitted for > | > | <mrsas>, but need first go ahead for attached patch and later we > | > | will continue to keep <mrsas> up-to-date (In sync with LSI > | > | released driver which is available at lsi's external site) > | > | > | > | Apply patch with "patch -p0 < patchname.patch" from head directory. > | > | > | > | -- Few notes for user-- > | > | LSI recommends using fusion_force bit In hint settings at start of > | > | the day, if > | > they want to use <mrsas>. ( <mfi> will be a default choice for > | > MR-Fusion HBA), if will be changed only with fusion_force hint > | > settings. (See mrsas man > | > page) Changing any default behavior is well tested for most of the > condition. > | > | Switching from <mfi> to <mrsas> for MR-Fusion options is designed > | > | to > | > allow user as one time choice, though multiple time switch from > | > <mfi> to <mrsas> is possible, it is not recommended. So, user needs > | > to decide from start of the day, which driver they want to use for MR= - > Fusion card. > | > | > | > | -- Implementation details -- > | > | To support this feature, we have modify <mfi> code to change > | > | default > | > return type from probe. Currently <mfi> driver return > | > "BUS_PROBE_DEFAULT". <mfi> driver has been be changed to return > | > "BUS_PROBE_LOW_PRIORITY" if fusion_force hint from device.hints is > set. > | > | Please notice, above mentioned implementation in <mfi> driver is > | > | only > | > applicable in case of MR-Fusion controller detection. For any other > | > controllers, supported by <mfi> driver, the behavior of probe return > | > will remain same as before. > | > | > | > | > | > | -- High level feature list of <mrsas> -- 1. Supports Fast Path > | > | feature of LSI controllers. > | > | 2. Supports 4K sector Drives. > | > | 3. CAM layer based interface. All VDs will be attached to CAM > | > | layer (Expected storage will be visible in "camcontroll") 4. > | > | Complete support of Online Controller Reset. (OCR) 5. OCR on > | > | Fimrware fault and IO > | > timeout case. > | > | 6. Work well with <storcli> management application which is > | > | generic > | > application provided by LSI for all other Operating system. > | > | 7. Supporst DIF enabled VDs (Same support as provided in Linux and > | > | other OSes in FreeBSD) 8. Fast Path Load balance support. > | > | > | > | - In summary, this driver is in part with Linux based MR drivers > | > | and all other features will be available to <mrsas> as planned > | > | activity from LSI > | > | > | > | This code is well tested by LSI Q/A team on 32 bit and 64 bit > | > | FreeBSD > | > Released OSes. > | > > | > Sorry it is has taken so long to start playing with this. I found > | > one critical issue with read performance. I added a printf to the > | > driver to make sure the fast path is not being used and it claims it = isn't. > | > Doing either a dd or using the > | > dt(http://www.scsifaq.org/RMiller_Tools/dt.html) > | > I see good write performance with both mrsas and mfi but on read > | > mrsas is backing up and appears to be single threaded. With mrsas > | > doing a dd of=3D/dev/zero if=3D/dev/da0 bs=3D1m I'm seeing 45685 kBps= and > | > with mfi seeing > | > 597072 kBps via gstat. Using dt via: > | > dt of=3D/dev/da0 bs=3D64k limit=3D1g aios=3D32 I see writes at 53897= 7 > | > Kbytes/sec and reads at bouncing around between > | > 20000 - 8000 Kbytes/sec, mfi reports 244744 Kbytes/sec. The > | > interesting piece is seeing the L(q) via gstat being around 30 when > | > doing mrsas reads, writes are 1 or less. With mfi both read and writ= es are > 1 or less. > | > > | Thanks for for providing feedback. This is very important observation. > | We will work on this and get back to you. Meanwhile, if we have high > | level acceptance(considering few defect fixes and optimization as a > | follow up activity) of current <mrsas> code, can have <mrsas> commit > | in current FreeBSD driver. This will definitely help us to manage > | source code from single destination. Currently there are multiple > | flavors/version of <mrsas> driver code is floated due to Upstream > | commit is pending. We can work for each observation/optimization > | proposed from upstream community on priority bases. >=20 > Yes a project branch can help with this. >=20 > | > This is with a GENERIC kernel on -current/amd64. I updated my > | > source to > | > 260231 and see the same. Witness, etc. are all on. I don't recall > | > seeing this when I tested mrsas a long time ago. > | > | Thanks for additional info. > | > | > > | > The card I'm using is: > | > LSI MegaRAID SAS 9266-8i > | > Firmware BIOS 5.38.00_4.12.05.00_0x05180000 > | > Firmware BCON 6.1-49-e_49-Rel > | > Firmware PCLI 05.05-03:#%00011 > | > Firmware APP 3.220.75-2196 > | > Firmware NVDT 2.1209.03-0117 > | > Firmware BTBL 2.05.00.00-0010 > | > Firmware BOOT 07.26.13.219 > | > > | > Some other issues I ran into when I kldunload mrsas I got a: > | > sysctl_unregister_oid: failed to unregister sysctl and then when I > | > | I do not recall if we have seen this issue, but will try this on our se= tup. >=20 > It think this and the panic might have been an artifact of my inition tes= ting. I > followed the instruction of kldload mrsas from /boot/loader.conf and saw > this. However, when I started working the emulation I modified my > GERNERIC config to not have mfi in it so I could switch between mfi and > mrsas via kldload/kldunload to test ioctl emulation. When I did this I n= oticed > that the patch included mrsas in the GENERIC configuration. > That might have caused this issue having both the module and static drive= r in > the same system. So this is probably a bad bug report. > Doing kldload and kldunload without mrsas and mfi static in the kernel I = didn't > run into this again. >=20 > | > kldload mrsas I got a panic: > | > mrsas0: <LSI Thunderbolt SAS Controller> port 0x7000-0x70ff mem > | > 0xdf060000-0xdf0 63fff,0xdf000000-0xdf03ffff irq 32 at device 0.0 on > | > pci4 > ######################################################### > | > <mrsas> Driver has detected MR-Fusion Controller.If you wish to > | > switch to <mfi> driver for MR-Fusion,read manual of mrsas. Once you > | > chose to use <mrsas> Driverf or MR-Fusion Controllers, you should > | > not switch to <mfi>.LSI recommend to use <m > | > rsas> driver for MR-Fusion.Use <mrsas> if you are not sure about > | > rsas> choice.Quick he > | > lp to use <mfi> for MR-Fustion Card: > | > Remove hint.mrsas.0.fusion_force=3D1 from /boot/device.hints. > | > > ######################################################### > | > mrsas0: Waiting for FW to come to ready state > | > > | > <panic message from malloc> > | > > | > db> tr > | > Tracing pid 534 tid 100245 td 0xfffff80015035490 > | > malloc() at malloc+0x1c8/frame 0xfffffe03fbab3200 > | > cpuctl_devs() at cpuctl_devs+0x61df/frame 0xfffffe03fbab3290 > | > cpuctl_devs() at cpuctl_devs+0x6189/frame 0xfffffe03fbab32f0 > | > cpuctl_devs() at cpuctl_devs+0x990a/frame 0xfffffe03fbab34b0 > | > device_attach() at device_attach+0x3a2/frame 0xfffffe03fbab3510 > | > pci_driver_added() at pci_driver_added+0xfa/frame 0xfffffe03fbab3550 > | > devclass_driver_added() at devclass_driver_added+0x7d/frame > | > 0xfffffe03fbab3590 > | > devclass_add_driver() at devclass_add_driver+0x127/frame > | > 0xfffffe03fbab35d0 > | > module_register_init() at module_register_init+0xb0/frame > | > 0xfffffe03fbab3600 > | > linker_load_module() at linker_load_module+0xbda/frame > | > 0xfffffe03fbab3930 > | > kern_kldload() at kern_kldload+0xad/frame 0xfffffe03fbab3970 > | > sys_kldload() at sys_kldload+0x5b/frame 0xfffffe03fbab39a0 > | > amd64_syscall() at amd64_syscall+0x265/frame 0xfffffe03fbab3ab0 > | > Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe03fbab3ab0 > | > > | > I played with storcli/storcli64 and MegaCli/MegaCli64 from LSI's web = site. > | > I noticed that the 32 bit versions are built for FreeBSD 8.1 and the > | > 64 bit FreeBSD 7.4. It might not be a critical problem but some > | > companies might still be on earlier releases. > | > > | > I need to look at adding the ioctl translation support for mfiutil > | > and Linux to mrsas. This means we should have a 32 bit to 64 bit > | > ioctl translator in the driver since our Linux emulation is 32 bit. > | We can pick this work as next action item. >=20 > I started working on this and mentioned some things that would make this > easier. >=20 > | > It's also handy to run 32 bit tools on the 64 bit kernel. > | > > | > It would be good to add alias support for /dev/mfid* such as was > | > done via ATA CAM sys/cam/ata/ata_da.c. Search for > | > ada_legacy_aliases and legacy_id. The helped the migration from > /dev/ad* to /dev/ada*. > | > > | Let me check this. You can help me to understand what is expected here > | and we can provide solution for this. > | Again, we can mark this as next to-do for mrsas and will provide patch > | as an when we have code ready. >=20 > The idea is that when /dev/da<x> node is created and alias of /dev/mfi<y>= is > created point to /dev/da<x>. This way existing fstab that reference > /dev/mfid0 for example will continue to work. It just makes it easier fo= r > migration. >=20 > A trickier part will be the feature to allow both LD and PD of an LD to b= e > available in a system. It is a feature that people use. With mfi it is = easy to see > RAID controlled devices via /dev/mfid<x> and mfisyspd<x>. > The physical device access come up as /dev/da<x>. So people should know > be very carefull when accessing /dev/da<x> since it could upset the RAID > controller, such as flashing firmware and when the disk goes offline it c= an kick > out the disk from the RAID. >=20 > I haven't looked at how mrsas deals with deleting and recreating RAID on = the > fly. mfi can do it with the proper sysctl/tunables set. >=20 > | > I'm not sure hint.mrsas.<x>.fusion_force=3D"1" is a good toggle to > | > enable/disable mrsas. I can see why you did that but it might be > | > more obvious to make hint.mfi.<x>.disabled=3D"1" to work since that > | > follows more of what people are used to. > | > | Either way is fine, since hint.mfi.<X>.disabled may miss lead that we > | are going to disable <mfi> completely. > | We may have <mfi> driver working for Falcon and other old MFI based > | controllers. >=20 > Since it is a hint.mfi.<x>.disabled, the x refers to card x, ie. > hint.mfi.1.disabled would disable the 2nd card in the system what would b= e > the ThunderBolt based can and leave the others to be mfi. We should get > more input from FreeBSD folks. fusion_force seems pretty non-obvious to > me. I'd suggest force_mrsas to be more obvious and it should probably be= a > generic tunabled and device instance specific. >=20 > Adding printing events to the kernel messages seems to be missing. > This needs to be controlled by sysctl/tunable but I might not have seen t= his > yet. >=20 > I'd say the highest priority is on performance and style. >=20 > Getting a review from Scott would be good but I'm not sure he has had tim= e > to do that. It might be good to look at mfi to see how things are laid o= ut. >=20 > FYI, my biggest problem working on FreeBSD is time. My day job prioritie= s > end up eating into time to work on "less critical" issues. Unfortunately= a > bunch of FreeBSD things fall into that catagory. I need to really resume= on > my mount path changes. I know what I need to do, I just need to carve ou= t > some time to work on it. >=20 > Thanks, >=20 > Doug A.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?e59396595152456dbcde63d48f70aa8f>