From owner-freebsd-scsi@FreeBSD.ORG Mon Jan 6 08:17:49 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 03DDE67F; Mon, 6 Jan 2014 08:17:49 +0000 (UTC) Received: from na01-bl2-obe.outbound.protection.outlook.com (mail-bl2lp0209.outbound.protection.outlook.com [207.46.163.209]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 6037B1178; Mon, 6 Jan 2014 08:17:48 +0000 (UTC) Received: from BN1PR07MB247.namprd07.prod.outlook.com (10.141.64.146) by BN1PR07MB438.namprd07.prod.outlook.com (10.141.63.13) with Microsoft SMTP Server (TLS) id 15.0.842.7; Mon, 6 Jan 2014 07:43:30 +0000 Received: from BN1PR07MB247.namprd07.prod.outlook.com ([169.254.7.143]) by BN1PR07MB247.namprd07.prod.outlook.com ([169.254.7.143]) with mapi id 15.00.0842.003; Mon, 6 Jan 2014 07:43:29 +0000 From: "Desai, Kashyap" To: Doug Ambrisko Subject: RE: LSI - MR-Fusion controller driver patch and man page Thread-Topic: LSI - MR-Fusion controller driver patch and man page Thread-Index: Ac7ykKZzGPABTpK8R3+D5UAv3TD3WQWOC9SAAHVVybA= Date: Mon, 6 Jan 2014 07:43:27 +0000 Message-ID: <8c423414ecc2421fbace3eb9f386be91@BN1PR07MB247.namprd07.prod.outlook.com> References: <08ba2a262fba45f687cdd3225f325110@BN1PR07MB247.namprd07.prod.outlook.com> <20140103211449.GA69721@cisco.com> In-Reply-To: <20140103211449.GA69721@cisco.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.19.239.250] x-forefront-prvs: 0083A7F08A x-forefront-antispam-report: SFV:NSPM; SFS:(10019001)(164054003)(199002)(189002)(24454002)(377454003)(13464003)(51704005)(74876001)(76796001)(56776001)(33646001)(54316002)(74706001)(47736001)(74316001)(4396001)(74662001)(15202345003)(53806001)(87266001)(74502001)(46102001)(90146001)(59766001)(65816001)(81686001)(66066001)(81542001)(49866001)(51856001)(77096001)(83322001)(80022001)(81342001)(77982001)(56816005)(83072002)(47446002)(80976001)(2656002)(63696002)(50986001)(19580395003)(54356001)(76786001)(76576001)(85852003)(69226001)(76482001)(81816001)(31966008)(74366001)(87936001)(19580405001)(47976001)(15975445006)(79102001)(85306002)(24736002); DIR:OUT; SFP:1102; SCL:1; SRVR:BN1PR07MB438; H:BN1PR07MB247.namprd07.prod.outlook.com; CLIP:192.19.239.250; FPR:; RD:InfoNoRecords; A:1; MX:1; LANG:en; Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: lsi.com X-Mailman-Approved-At: Mon, 06 Jan 2014 12:45:32 +0000 Cc: "scottl@netflix.com" , "Radford, Adam" , "Kenneth D. Merry" , "sean_bruno@yahoo.com" , "Mankani, Krishnaraddi" , "dwhite@ixsystems.com" , "Maloy, Joe" , "jpaetzel@freebsd.org" , "freebsd-scsi@freebsd.org" , "McConnell, Stephen" X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jan 2014 08:17:49 -0000 Doug: I have replied inline for your comment. For most of the comment you posted = below can be work out and we can prioritize completion of those based on na= ture of issue. I will work on READ/WRITE comparison w.r.t and as a priority = item.=20 Please consider that it will be really helpful for further development, if = we have common code check-in in Upstream FreeBSD. We will actively work on = each comment posted for and address those before we have in= -box for any FreeBSD release. Thanks, Kashyap > -----Original Message----- > From: Doug Ambrisko [mailto:ambrisko@cisco.com] > Sent: Saturday, January 04, 2014 2:45 AM > To: Desai, Kashyap > Cc: freebsd-scsi@freebsd.org; scottl@netflix.com; sean_bruno@yahoo.com; > dwhite@ixsystems.com; jpaetzel@freebsd.org; Maloy, Joe; Mankani, > Krishnaraddi; McConnell, Stephen; Saxena, Sumit; Radford, Adam; Kenneth > D. Merry > Subject: Re: LSI - MR-Fusion controller driver patch and man page >=20 > On Fri, Dec 06, 2013 at 02:37:21PM +0000, Desai, Kashyap wrote: > | Hi, > | > | Please consider attached patch for FreeBSD upstream check-in. > | > | Please find attached patch for driver for LSI's MegaRaid > Controllers. This driver supports Thunderbolt onwards Device IDs of MR > controllers. > | Currently it supports 0x005B and 0x005D Device IDs. > | > | NOTE : This driver will not eliminate or by pass any functionality of <= mfi> > driver which already support above to Device IDs to keep existing user > experience unchanged. > | > | Driver will be always given priority over driver and > | only if customer/user wants to use/migrate from to , it > | will hook up into kernel via device.hint rules. (Attached is mrsas man > | page for more info.) > | > | LSI will continue to update driver in future in timely bases. > | We have another set of patch in pipeline to be submitted for , > | but need first go ahead for attached patch and later we will continue > | to keep up-to-date (In sync with LSI released driver which is > | available at lsi's external site) > | > | Apply patch with "patch -p0 < patchname.patch" from head directory. > | > | -- Few notes for user-- > | LSI recommends using fusion_force bit In hint settings at start of the = day, if > they want to use . ( will be a default choice for MR-Fusion > HBA), if will be changed only with fusion_force hint settings. (See mrsas= man > page) Changing any default behavior is well tested for most of the condit= ion. > | Switching from to for MR-Fusion options is designed to > allow user as one time choice, though multiple time switch from to > is possible, it is not recommended. So, user needs to decide from > start of the day, which driver they want to use for MR-Fusion card. > | > | -- Implementation details -- > | To support this feature, we have modify code to change default > return type from probe. Currently driver return > "BUS_PROBE_DEFAULT". driver has been be changed to return > "BUS_PROBE_LOW_PRIORITY" if fusion_force hint from device.hints is set. > | Please notice, above mentioned implementation in driver is only > applicable in case of MR-Fusion controller detection. For any other > controllers, supported by driver, the behavior of probe return will > remain same as before. > | > | > | -- High level feature list of -- 1. Supports Fast Path feature > | of LSI controllers. > | 2. Supports 4K sector Drives. > | 3. CAM layer based interface. All VDs will be attached to CAM layer > | (Expected storage will be visible in "camcontroll") 4. Complete > | support of Online Controller Reset. (OCR) 5. OCR on Fimrware fault and= IO > timeout case. > | 6. Work well with management application which is generic > application provided by LSI for all other Operating system. > | 7. Supporst DIF enabled VDs (Same support as provided in Linux and > | other OSes in FreeBSD) 8. Fast Path Load balance support. > | > | - In summary, this driver is in part with Linux based MR drivers and > | all other features will be available to as planned activity > | from LSI > | > | This code is well tested by LSI Q/A team on 32 bit and 64 bit FreeBSD > Released OSes. >=20 > Sorry it is has taken so long to start playing with this. I found one cr= itical issue > with read performance. I added a printf to the driver to make sure the f= ast > path is not being used and it claims it isn't. > Doing either a dd or using the > dt(http://www.scsifaq.org/RMiller_Tools/dt.html) > I see good write performance with both mrsas and mfi but on read mrsas is > backing up and appears to be single threaded. With mrsas doing a dd > of=3D/dev/zero if=3D/dev/da0 bs=3D1m I'm seeing 45685 kBps and with mfi s= eeing > 597072 kBps via gstat. Using dt via: > dt of=3D/dev/da0 bs=3D64k limit=3D1g aios=3D32 > I see writes at 538977 Kbytes/sec and reads at bouncing around between > 20000 - 8000 Kbytes/sec, mfi reports 244744 Kbytes/sec. The interesting > piece is seeing the L(q) via gstat being around 30 when doing mrsas reads= , > writes are 1 or less. With mfi both read and writes are 1 or less. >=20 Thanks for for providing feedback. This is very important observation. We w= ill work on this and get back to you. Meanwhile, if we have high level ac= ceptance(considering few defect fixes and optimization as a follow up activ= ity) of current code, can have commit in current FreeBSD dr= iver. This will definitely help us to manage source code from single destin= ation. Currently there are multiple flavors/version of driver code = is floated due to Upstream commit is pending. We can work for each observat= ion/optimization proposed from upstream community on priority bases. > This is with a GENERIC kernel on -current/amd64. I updated my source to > 260231 and see the same. Witness, etc. are all on. I don't recall seein= g this > when I tested mrsas a long time ago. Thanks for additional info. >=20 > The card I'm using is: > LSI MegaRAID SAS 9266-8i > Firmware BIOS 5.38.00_4.12.05.00_0x05180000 > Firmware BCON 6.1-49-e_49-Rel > Firmware PCLI 05.05-03:#%00011 > Firmware APP 3.220.75-2196 > Firmware NVDT 2.1209.03-0117 > Firmware BTBL 2.05.00.00-0010 > Firmware BOOT 07.26.13.219 >=20 > Some other issues I ran into when I kldunload mrsas I got a: > sysctl_unregister_oid: failed to unregister sysctl and then when I I do not recall if we have seen this issue, but will try this on our setup. > kldload mrsas I got a panic: > mrsas0: port 0x7000-0x70ff mem > 0xdf060000-0xdf0 63fff,0xdf000000-0xdf03ffff irq 32 at device 0.0 on pci4 > ######################################################### > Driver has detected MR-Fusion Controller.If you wish to switch to > driver for MR-Fusion,read manual of mrsas. Once you chose to use > Driverf or MR-Fusion Controllers, you should not switch to .= LSI > recommend to use rsas> driver for MR-Fusion.Use if you are not sure about > rsas> choice.Quick he > lp to use for MR-Fustion Card: > Remove hint.mrsas.0.fusion_force=3D1 from /boot/device.hints. > ######################################################### > mrsas0: Waiting for FW to come to ready state >=20 > >=20 > db> tr > Tracing pid 534 tid 100245 td 0xfffff80015035490 > malloc() at malloc+0x1c8/frame 0xfffffe03fbab3200 > cpuctl_devs() at cpuctl_devs+0x61df/frame 0xfffffe03fbab3290 > cpuctl_devs() at cpuctl_devs+0x6189/frame 0xfffffe03fbab32f0 > cpuctl_devs() at cpuctl_devs+0x990a/frame 0xfffffe03fbab34b0 > device_attach() at device_attach+0x3a2/frame 0xfffffe03fbab3510 > pci_driver_added() at pci_driver_added+0xfa/frame 0xfffffe03fbab3550 > devclass_driver_added() at devclass_driver_added+0x7d/frame > 0xfffffe03fbab3590 > devclass_add_driver() at devclass_add_driver+0x127/frame > 0xfffffe03fbab35d0 > module_register_init() at module_register_init+0xb0/frame > 0xfffffe03fbab3600 > linker_load_module() at linker_load_module+0xbda/frame > 0xfffffe03fbab3930 > kern_kldload() at kern_kldload+0xad/frame 0xfffffe03fbab3970 > sys_kldload() at sys_kldload+0x5b/frame 0xfffffe03fbab39a0 > amd64_syscall() at amd64_syscall+0x265/frame 0xfffffe03fbab3ab0 > Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe03fbab3ab0 >=20 > I played with storcli/storcli64 and MegaCli/MegaCli64 from LSI's web site= . > I noticed that the 32 bit versions are built for FreeBSD 8.1 and the 64 b= it > FreeBSD 7.4. It might not be a critical problem but some companies might= still > be on earlier releases. >=20 > I need to look at adding the ioctl translation support for mfiutil and Li= nux to > mrsas. This means we should have a 32 bit to 64 bit ioctl translator in = the > driver since our Linux emulation is 32 bit. We can pick this work as next action item. > It's also handy to run 32 bit tools on the 64 bit kernel. >=20 > It would be good to add alias support for /dev/mfid* such as was done via > ATA CAM sys/cam/ata/ata_da.c. Search for ada_legacy_aliases and > legacy_id. The helped the migration from /dev/ad* to /dev/ada*. >=20 Let me check this. You can help me to understand what is expected here and = we can provide solution for this. Again, we can mark this as next to-do for mrsas and will provide patch as a= n when we have code ready. > I'm not sure hint.mrsas..fusion_force=3D"1" is a good toggle to > enable/disable mrsas. I can see why you did that but it might be more > obvious to make hint.mfi..disabled=3D"1" to work since that follows mo= re > of what people are used to. Either way is fine, since hint.mfi..disabled may miss lead that we are g= oing to disable completely. We may have driver working for Falcon and other old MFI based control= lers. >=20 > Thanks, >=20 > Doug A.