Date: Fri, 3 Jan 2014 13:14:49 -0800 From: Doug Ambrisko <ambrisko@cisco.com> To: "Desai, Kashyap" <Kashyap.Desai@lsi.com> Cc: "scottl@netflix.com" <scottl@netflix.com>, "Radford, Adam" <Adam.Radford@lsi.com>, "Kenneth D. Merry" <ken@kdm.org>, "sean_bruno@yahoo.com" <sean_bruno@yahoo.com>, "Mankani, Krishnaraddi" <Krishnaraddi.Mankani@lsi.com>, "dwhite@ixsystems.com" <dwhite@ixsystems.com>, "Maloy, Joe" <Joe.Maloy@lsi.com>, "jpaetzel@freebsd.org" <jpaetzel@freebsd.org>, "freebsd-scsi@freebsd.org" <freebsd-scsi@freebsd.org>, "McConnell, Stephen" <Stephen.McConnell@lsi.com> Subject: Re: LSI - MR-Fusion controller driver <mrsas> patch and man page Message-ID: <20140103211449.GA69721@cisco.com> In-Reply-To: <08ba2a262fba45f687cdd3225f325110@BN1PR07MB247.namprd07.prod.outlook.com> References: <08ba2a262fba45f687cdd3225f325110@BN1PR07MB247.namprd07.prod.outlook.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, Dec 06, 2013 at 02:37:21PM +0000, Desai, Kashyap wrote: | Hi, | | Please consider attached patch for FreeBSD upstream check-in. | | Please find attached patch for <mrsas> driver for LSI's MegaRaid Controllers. This driver supports Thunderbolt onwards Device IDs of MR controllers. | Currently it supports 0x005B and 0x005D Device IDs. | | NOTE : This driver will not eliminate or by pass any functionality of <mfi> driver which already support above to Device IDs to keep existing user experience unchanged. | | <mfi> Driver will be always given priority over <mrsas> driver and only if customer/user wants to use/migrate from <mfi> to <mrsas>, it will hook up into kernel via device.hint rules. (Attached is mrsas man page for more info.) | | LSI will continue to update <mrsas> driver in future in timely bases. We have another set of patch in pipeline to be submitted for <mrsas>, but need first go ahead for attached patch and later we will continue to keep <mrsas> up-to-date (In sync with LSI released driver which is available at lsi's external site) | | Apply patch with "patch -p0 < patchname.patch" from head directory. | | -- Few notes for user-- | LSI recommends using fusion_force bit In hint settings at start of the day, if they want to use <mrsas>. ( <mfi> will be a default choice for MR-Fusion HBA), if will be changed only with fusion_force hint settings. (See mrsas man page) Changing any default behavior is well tested for most of the condition. | Switching from <mfi> to <mrsas> for MR-Fusion options is designed to allow user as one time choice, though multiple time switch from <mfi> to <mrsas> is possible, it is not recommended. So, user needs to decide from start of the day, which driver they want to use for MR-Fusion card. | | -- Implementation details -- | To support this feature, we have modify <mfi> code to change default return type from probe. Currently <mfi> driver return "BUS_PROBE_DEFAULT". <mfi> driver has been be changed to return "BUS_PROBE_LOW_PRIORITY" if fusion_force hint from device.hints is set. | Please notice, above mentioned implementation in <mfi> driver is only applicable in case of MR-Fusion controller detection. For any other controllers, supported by <mfi> driver, the behavior of probe return will remain same as before. | | | -- High level feature list of <mrsas> -- | 1. Supports Fast Path feature of LSI controllers. | 2. Supports 4K sector Drives. | 3. CAM layer based interface. All VDs will be attached to CAM layer (Expected storage will be visible in "camcontroll") | 4. Complete support of Online Controller Reset. (OCR) | 5. OCR on Fimrware fault and IO timeout case. | 6. Work well with <storcli> management application which is generic application provided by LSI for all other Operating system. | 7. Supporst DIF enabled VDs (Same support as provided in Linux and other OSes in FreeBSD) | 8. Fast Path Load balance support. | | - In summary, this driver is in part with Linux based MR drivers and all other features will be available to <mrsas> as planned activity from LSI | | This code is well tested by LSI Q/A team on 32 bit and 64 bit FreeBSD Released OSes. Sorry it is has taken so long to start playing with this. I found one critical issue with read performance. I added a printf to the driver to make sure the fast path is not being used and it claims it isn't. Doing either a dd or using the dt(http://www.scsifaq.org/RMiller_Tools/dt.html) I see good write performance with both mrsas and mfi but on read mrsas is backing up and appears to be single threaded. With mrsas doing a dd of=/dev/zero if=/dev/da0 bs=1m I'm seeing 45685 kBps and with mfi seeing 597072 kBps via gstat. Using dt via: dt of=/dev/da0 bs=64k limit=1g aios=32 I see writes at 538977 Kbytes/sec and reads at bouncing around between 20000 - 8000 Kbytes/sec, mfi reports 244744 Kbytes/sec. The interesting piece is seeing the L(q) via gstat being around 30 when doing mrsas reads, writes are 1 or less. With mfi both read and writes are 1 or less. This is with a GENERIC kernel on -current/amd64. I updated my source to 260231 and see the same. Witness, etc. are all on. I don't recall seeing this when I tested mrsas a long time ago. The card I'm using is: LSI MegaRAID SAS 9266-8i Firmware BIOS 5.38.00_4.12.05.00_0x05180000 Firmware BCON 6.1-49-e_49-Rel Firmware PCLI 05.05-03:#%00011 Firmware APP 3.220.75-2196 Firmware NVDT 2.1209.03-0117 Firmware BTBL 2.05.00.00-0010 Firmware BOOT 07.26.13.219 Some other issues I ran into when I kldunload mrsas I got a: sysctl_unregister_oid: failed to unregister sysctl and then when I kldload mrsas I got a panic: mrsas0: <LSI Thunderbolt SAS Controller> port 0x7000-0x70ff mem 0xdf060000-0xdf0 63fff,0xdf000000-0xdf03ffff irq 32 at device 0.0 on pci4 ######################################################### <mrsas> Driver has detected MR-Fusion Controller.If you wish to switch to <mfi> driver for MR-Fusion,read manual of mrsas. Once you chose to use <mrsas> Driverf or MR-Fusion Controllers, you should not switch to <mfi>.LSI recommend to use <m rsas> driver for MR-Fusion.Use <mrsas> if you are not sure about choice.Quick he lp to use <mfi> for MR-Fustion Card: Remove hint.mrsas.0.fusion_force=1 from /boot/device.hints. ######################################################### mrsas0: Waiting for FW to come to ready state <panic message from malloc> db> tr Tracing pid 534 tid 100245 td 0xfffff80015035490 malloc() at malloc+0x1c8/frame 0xfffffe03fbab3200 cpuctl_devs() at cpuctl_devs+0x61df/frame 0xfffffe03fbab3290 cpuctl_devs() at cpuctl_devs+0x6189/frame 0xfffffe03fbab32f0 cpuctl_devs() at cpuctl_devs+0x990a/frame 0xfffffe03fbab34b0 device_attach() at device_attach+0x3a2/frame 0xfffffe03fbab3510 pci_driver_added() at pci_driver_added+0xfa/frame 0xfffffe03fbab3550 devclass_driver_added() at devclass_driver_added+0x7d/frame 0xfffffe03fbab3590 devclass_add_driver() at devclass_add_driver+0x127/frame 0xfffffe03fbab35d0 module_register_init() at module_register_init+0xb0/frame 0xfffffe03fbab3600 linker_load_module() at linker_load_module+0xbda/frame 0xfffffe03fbab3930 kern_kldload() at kern_kldload+0xad/frame 0xfffffe03fbab3970 sys_kldload() at sys_kldload+0x5b/frame 0xfffffe03fbab39a0 amd64_syscall() at amd64_syscall+0x265/frame 0xfffffe03fbab3ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe03fbab3ab0 I played with storcli/storcli64 and MegaCli/MegaCli64 from LSI's web site. I noticed that the 32 bit versions are built for FreeBSD 8.1 and the 64 bit FreeBSD 7.4. It might not be a critical problem but some companies might still be on earlier releases. I need to look at adding the ioctl translation support for mfiutil and Linux to mrsas. This means we should have a 32 bit to 64 bit ioctl translator in the driver since our Linux emulation is 32 bit. It's also handy to run 32 bit tools on the 64 bit kernel. It would be good to add alias support for /dev/mfid* such as was done via ATA CAM sys/cam/ata/ata_da.c. Search for ada_legacy_aliases and legacy_id. The helped the migration from /dev/ad* to /dev/ada*. I'm not sure hint.mrsas.<x>.fusion_force="1" is a good toggle to enable/disable mrsas. I can see why you did that but it might be more obvious to make hint.mfi.<x>.disabled="1" to work since that follows more of what people are used to. Thanks, Doug A.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20140103211449.GA69721>