Date: Mon, 6 Jan 2014 07:43:27 +0000 From: "Desai, Kashyap" <Kashyap.Desai@lsi.com> To: Doug Ambrisko <ambrisko@cisco.com> Cc: "scottl@netflix.com" <scottl@netflix.com>, "Radford, Adam" <Adam.Radford@lsi.com>, "Kenneth D. Merry" <ken@kdm.org>, "sean_bruno@yahoo.com" <sean_bruno@yahoo.com>, "Mankani, Krishnaraddi" <Krishnaraddi.Mankani@lsi.com>, "dwhite@ixsystems.com" <dwhite@ixsystems.com>, "Maloy, Joe" <Joe.Maloy@lsi.com>, "jpaetzel@freebsd.org" <jpaetzel@freebsd.org>, "freebsd-scsi@freebsd.org" <freebsd-scsi@freebsd.org>, "McConnell, Stephen" <Stephen.McConnell@lsi.com> Subject: RE: LSI - MR-Fusion controller driver <mrsas> patch and man page Message-ID: <8c423414ecc2421fbace3eb9f386be91@BN1PR07MB247.namprd07.prod.outlook.com> In-Reply-To: <20140103211449.GA69721@cisco.com> References: <08ba2a262fba45f687cdd3225f325110@BN1PR07MB247.namprd07.prod.outlook.com> <20140103211449.GA69721@cisco.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Doug: I have replied inline for your comment. For most of the comment you posted = below can be work out and we can prioritize completion of those based on na= ture of issue. I will work on READ/WRITE comparison w.r.t <mfi> and <mrsas> as a priority = item.=20 Please consider that it will be really helpful for further development, if = we have common code check-in in Upstream FreeBSD. We will actively work on = each comment posted for <mrsas> and address those before we have <mrsas> in= -box for any FreeBSD release. Thanks, Kashyap > -----Original Message----- > From: Doug Ambrisko [mailto:ambrisko@cisco.com] > Sent: Saturday, January 04, 2014 2:45 AM > To: Desai, Kashyap > Cc: freebsd-scsi@freebsd.org; scottl@netflix.com; sean_bruno@yahoo.com; > dwhite@ixsystems.com; jpaetzel@freebsd.org; Maloy, Joe; Mankani, > Krishnaraddi; McConnell, Stephen; Saxena, Sumit; Radford, Adam; Kenneth > D. Merry > Subject: Re: LSI - MR-Fusion controller driver <mrsas> patch and man page >=20 > On Fri, Dec 06, 2013 at 02:37:21PM +0000, Desai, Kashyap wrote: > | Hi, > | > | Please consider attached patch for FreeBSD upstream check-in. > | > | Please find attached patch for <mrsas> driver for LSI's MegaRaid > Controllers. This driver supports Thunderbolt onwards Device IDs of MR > controllers. > | Currently it supports 0x005B and 0x005D Device IDs. > | > | NOTE : This driver will not eliminate or by pass any functionality of <= mfi> > driver which already support above to Device IDs to keep existing user > experience unchanged. > | > | <mfi> Driver will be always given priority over <mrsas> driver and > | only if customer/user wants to use/migrate from <mfi> to <mrsas>, it > | will hook up into kernel via device.hint rules. (Attached is mrsas man > | page for more info.) > | > | LSI will continue to update <mrsas> driver in future in timely bases. > | We have another set of patch in pipeline to be submitted for <mrsas>, > | but need first go ahead for attached patch and later we will continue > | to keep <mrsas> up-to-date (In sync with LSI released driver which is > | available at lsi's external site) > | > | Apply patch with "patch -p0 < patchname.patch" from head directory. > | > | -- Few notes for user-- > | LSI recommends using fusion_force bit In hint settings at start of the = day, if > they want to use <mrsas>. ( <mfi> will be a default choice for MR-Fusion > HBA), if will be changed only with fusion_force hint settings. (See mrsas= man > page) Changing any default behavior is well tested for most of the condit= ion. > | Switching from <mfi> to <mrsas> for MR-Fusion options is designed to > allow user as one time choice, though multiple time switch from <mfi> to > <mrsas> is possible, it is not recommended. So, user needs to decide from > start of the day, which driver they want to use for MR-Fusion card. > | > | -- Implementation details -- > | To support this feature, we have modify <mfi> code to change default > return type from probe. Currently <mfi> driver return > "BUS_PROBE_DEFAULT". <mfi> driver has been be changed to return > "BUS_PROBE_LOW_PRIORITY" if fusion_force hint from device.hints is set. > | Please notice, above mentioned implementation in <mfi> driver is only > applicable in case of MR-Fusion controller detection. For any other > controllers, supported by <mfi> driver, the behavior of probe return will > remain same as before. > | > | > | -- High level feature list of <mrsas> -- 1. Supports Fast Path feature > | of LSI controllers. > | 2. Supports 4K sector Drives. > | 3. CAM layer based interface. All VDs will be attached to CAM layer > | (Expected storage will be visible in "camcontroll") 4. Complete > | support of Online Controller Reset. (OCR) 5. OCR on Fimrware fault and= IO > timeout case. > | 6. Work well with <storcli> management application which is generic > application provided by LSI for all other Operating system. > | 7. Supporst DIF enabled VDs (Same support as provided in Linux and > | other OSes in FreeBSD) 8. Fast Path Load balance support. > | > | - In summary, this driver is in part with Linux based MR drivers and > | all other features will be available to <mrsas> as planned activity > | from LSI > | > | This code is well tested by LSI Q/A team on 32 bit and 64 bit FreeBSD > Released OSes. >=20 > Sorry it is has taken so long to start playing with this. I found one cr= itical issue > with read performance. I added a printf to the driver to make sure the f= ast > path is not being used and it claims it isn't. > Doing either a dd or using the > dt(http://www.scsifaq.org/RMiller_Tools/dt.html) > I see good write performance with both mrsas and mfi but on read mrsas is > backing up and appears to be single threaded. With mrsas doing a dd > of=3D/dev/zero if=3D/dev/da0 bs=3D1m I'm seeing 45685 kBps and with mfi s= eeing > 597072 kBps via gstat. Using dt via: > dt of=3D/dev/da0 bs=3D64k limit=3D1g aios=3D32 > I see writes at 538977 Kbytes/sec and reads at bouncing around between > 20000 - 8000 Kbytes/sec, mfi reports 244744 Kbytes/sec. The interesting > piece is seeing the L(q) via gstat being around 30 when doing mrsas reads= , > writes are 1 or less. With mfi both read and writes are 1 or less. >=20 Thanks for for providing feedback. This is very important observation. We w= ill work on this and get back to you. Meanwhile, if we have high level ac= ceptance(considering few defect fixes and optimization as a follow up activ= ity) of current <mrsas> code, can have <mrsas> commit in current FreeBSD dr= iver. This will definitely help us to manage source code from single destin= ation. Currently there are multiple flavors/version of <mrsas> driver code = is floated due to Upstream commit is pending. We can work for each observat= ion/optimization proposed from upstream community on priority bases. > This is with a GENERIC kernel on -current/amd64. I updated my source to > 260231 and see the same. Witness, etc. are all on. I don't recall seein= g this > when I tested mrsas a long time ago. Thanks for additional info. >=20 > The card I'm using is: > LSI MegaRAID SAS 9266-8i > Firmware BIOS 5.38.00_4.12.05.00_0x05180000 > Firmware BCON 6.1-49-e_49-Rel > Firmware PCLI 05.05-03:#%00011 > Firmware APP 3.220.75-2196 > Firmware NVDT 2.1209.03-0117 > Firmware BTBL 2.05.00.00-0010 > Firmware BOOT 07.26.13.219 >=20 > Some other issues I ran into when I kldunload mrsas I got a: > sysctl_unregister_oid: failed to unregister sysctl and then when I I do not recall if we have seen this issue, but will try this on our setup. > kldload mrsas I got a panic: > mrsas0: <LSI Thunderbolt SAS Controller> port 0x7000-0x70ff mem > 0xdf060000-0xdf0 63fff,0xdf000000-0xdf03ffff irq 32 at device 0.0 on pci4 > ######################################################### > <mrsas> Driver has detected MR-Fusion Controller.If you wish to switch to > <mfi> driver for MR-Fusion,read manual of mrsas. Once you chose to use > <mrsas> Driverf or MR-Fusion Controllers, you should not switch to <mfi>.= LSI > recommend to use <m > rsas> driver for MR-Fusion.Use <mrsas> if you are not sure about > rsas> choice.Quick he > lp to use <mfi> for MR-Fustion Card: > Remove hint.mrsas.0.fusion_force=3D1 from /boot/device.hints. > ######################################################### > mrsas0: Waiting for FW to come to ready state >=20 > <panic message from malloc> >=20 > db> tr > Tracing pid 534 tid 100245 td 0xfffff80015035490 > malloc() at malloc+0x1c8/frame 0xfffffe03fbab3200 > cpuctl_devs() at cpuctl_devs+0x61df/frame 0xfffffe03fbab3290 > cpuctl_devs() at cpuctl_devs+0x6189/frame 0xfffffe03fbab32f0 > cpuctl_devs() at cpuctl_devs+0x990a/frame 0xfffffe03fbab34b0 > device_attach() at device_attach+0x3a2/frame 0xfffffe03fbab3510 > pci_driver_added() at pci_driver_added+0xfa/frame 0xfffffe03fbab3550 > devclass_driver_added() at devclass_driver_added+0x7d/frame > 0xfffffe03fbab3590 > devclass_add_driver() at devclass_add_driver+0x127/frame > 0xfffffe03fbab35d0 > module_register_init() at module_register_init+0xb0/frame > 0xfffffe03fbab3600 > linker_load_module() at linker_load_module+0xbda/frame > 0xfffffe03fbab3930 > kern_kldload() at kern_kldload+0xad/frame 0xfffffe03fbab3970 > sys_kldload() at sys_kldload+0x5b/frame 0xfffffe03fbab39a0 > amd64_syscall() at amd64_syscall+0x265/frame 0xfffffe03fbab3ab0 > Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe03fbab3ab0 >=20 > I played with storcli/storcli64 and MegaCli/MegaCli64 from LSI's web site= . > I noticed that the 32 bit versions are built for FreeBSD 8.1 and the 64 b= it > FreeBSD 7.4. It might not be a critical problem but some companies might= still > be on earlier releases. >=20 > I need to look at adding the ioctl translation support for mfiutil and Li= nux to > mrsas. This means we should have a 32 bit to 64 bit ioctl translator in = the > driver since our Linux emulation is 32 bit. We can pick this work as next action item. > It's also handy to run 32 bit tools on the 64 bit kernel. >=20 > It would be good to add alias support for /dev/mfid* such as was done via > ATA CAM sys/cam/ata/ata_da.c. Search for ada_legacy_aliases and > legacy_id. The helped the migration from /dev/ad* to /dev/ada*. >=20 Let me check this. You can help me to understand what is expected here and = we can provide solution for this. Again, we can mark this as next to-do for mrsas and will provide patch as a= n when we have code ready. > I'm not sure hint.mrsas.<x>.fusion_force=3D"1" is a good toggle to > enable/disable mrsas. I can see why you did that but it might be more > obvious to make hint.mfi.<x>.disabled=3D"1" to work since that follows mo= re > of what people are used to. Either way is fine, since hint.mfi.<X>.disabled may miss lead that we are g= oing to disable <mfi> completely. We may have <mfi> driver working for Falcon and other old MFI based control= lers. >=20 > Thanks, >=20 > Doug A.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?8c423414ecc2421fbace3eb9f386be91>