From owner-freebsd-scsi@FreeBSD.ORG Mon Jan 6 00:09:16 2014 Return-Path: Delivered-To: freebsd-scsi@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 50343D88; Mon, 6 Jan 2014 00:09:16 +0000 (UTC) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 23D89116E; Mon, 6 Jan 2014 00:09:16 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id s0609FpG075972; Mon, 6 Jan 2014 00:09:15 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id s0609Fi0075971; Mon, 6 Jan 2014 00:09:15 GMT (envelope-from linimon) Date: Mon, 6 Jan 2014 00:09:15 GMT Message-Id: <201401060009.s0609Fi0075971@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-scsi@FreeBSD.org From: linimon@FreeBSD.org Subject: Re: kern/185498: [iscsi] Bug in native iscsi-portal with many ip addresses X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jan 2014 00:09:16 -0000 Old Synopsis: Bug in native iscsi-portal with many ip addresses New Synopsis: [iscsi] Bug in native iscsi-portal with many ip addresses Responsible-Changed-From-To: freebsd-bugs->freebsd-scsi Responsible-Changed-By: linimon Responsible-Changed-When: Mon Jan 6 00:08:45 UTC 2014 Responsible-Changed-Why: reclassify. http://www.freebsd.org/cgi/query-pr.cgi?pr=185498 From owner-freebsd-scsi@FreeBSD.ORG Mon Jan 6 11:06:54 2014 Return-Path: Delivered-To: freebsd-scsi@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 79B6672C for ; Mon, 6 Jan 2014 11:06:54 +0000 (UTC) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 646551095 for ; Mon, 6 Jan 2014 11:06:54 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id s06B6sfW045374 for ; Mon, 6 Jan 2014 11:06:54 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id s06B6rXw045372 for freebsd-scsi@FreeBSD.org; Mon, 6 Jan 2014 11:06:53 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 6 Jan 2014 11:06:53 GMT Message-Id: <201401061106.s06B6rXw045372@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-scsi@FreeBSD.org Subject: Current problem reports assigned to freebsd-scsi@FreeBSD.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jan 2014 11:06:54 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/185498 scsi [iscsi] Bug in native iscsi-portal with many ip addres o kern/184059 scsi [mps] mps SCSI driver causes FreeBSD to hang during bo o kern/179932 scsi [ciss] ciss i/o stall problem with HP Bl Gen8 (and HP o kern/178795 scsi [mps] MSI for mps driver doesn't work under vmware o kern/165982 scsi [mpt] mpt instability, drive resets, and losses on Fre o kern/165740 scsi [cam] SCSI code must drain callbacks before free f kern/162256 scsi [mpt] QUEUE FULL EVENT and 'mpt_cam_event: 0x0' o docs/151336 scsi Missing documentation of scsi_ and ata_ functions in c o kern/148083 scsi [aac] Strange device reporting o kern/144648 scsi [aac] Strange values of speed and bus width in dmesg o kern/142351 scsi [mpt] LSILogic driver performance problems o kern/134488 scsi [mpt] MPT SCSI driver probes max. 8 LUNs per device o kern/130621 scsi [mpt] tranfer rate is inscrutable slow when use lsi213 f kern/129602 scsi [ahd] ahd(4) gets confused and wedges SCSI bus f kern/123674 scsi [ahc] ahc driver dumping o sparc/121676 scsi [iscsi] iscontrol do not connect iscsi-target on sparc 16 problems total. From owner-freebsd-scsi@FreeBSD.ORG Mon Jan 6 08:17:49 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 03DDE67F; Mon, 6 Jan 2014 08:17:49 +0000 (UTC) Received: from na01-bl2-obe.outbound.protection.outlook.com (mail-bl2lp0209.outbound.protection.outlook.com [207.46.163.209]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 6037B1178; Mon, 6 Jan 2014 08:17:48 +0000 (UTC) Received: from BN1PR07MB247.namprd07.prod.outlook.com (10.141.64.146) by BN1PR07MB438.namprd07.prod.outlook.com (10.141.63.13) with Microsoft SMTP Server (TLS) id 15.0.842.7; Mon, 6 Jan 2014 07:43:30 +0000 Received: from BN1PR07MB247.namprd07.prod.outlook.com ([169.254.7.143]) by BN1PR07MB247.namprd07.prod.outlook.com ([169.254.7.143]) with mapi id 15.00.0842.003; Mon, 6 Jan 2014 07:43:29 +0000 From: "Desai, Kashyap" To: Doug Ambrisko Subject: RE: LSI - MR-Fusion controller driver patch and man page Thread-Topic: LSI - MR-Fusion controller driver patch and man page Thread-Index: Ac7ykKZzGPABTpK8R3+D5UAv3TD3WQWOC9SAAHVVybA= Date: Mon, 6 Jan 2014 07:43:27 +0000 Message-ID: <8c423414ecc2421fbace3eb9f386be91@BN1PR07MB247.namprd07.prod.outlook.com> References: <08ba2a262fba45f687cdd3225f325110@BN1PR07MB247.namprd07.prod.outlook.com> <20140103211449.GA69721@cisco.com> In-Reply-To: <20140103211449.GA69721@cisco.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.19.239.250] x-forefront-prvs: 0083A7F08A x-forefront-antispam-report: SFV:NSPM; SFS:(10019001)(164054003)(199002)(189002)(24454002)(377454003)(13464003)(51704005)(74876001)(76796001)(56776001)(33646001)(54316002)(74706001)(47736001)(74316001)(4396001)(74662001)(15202345003)(53806001)(87266001)(74502001)(46102001)(90146001)(59766001)(65816001)(81686001)(66066001)(81542001)(49866001)(51856001)(77096001)(83322001)(80022001)(81342001)(77982001)(56816005)(83072002)(47446002)(80976001)(2656002)(63696002)(50986001)(19580395003)(54356001)(76786001)(76576001)(85852003)(69226001)(76482001)(81816001)(31966008)(74366001)(87936001)(19580405001)(47976001)(15975445006)(79102001)(85306002)(24736002); DIR:OUT; SFP:1102; SCL:1; SRVR:BN1PR07MB438; H:BN1PR07MB247.namprd07.prod.outlook.com; CLIP:192.19.239.250; FPR:; RD:InfoNoRecords; A:1; MX:1; LANG:en; Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: lsi.com X-Mailman-Approved-At: Mon, 06 Jan 2014 12:45:32 +0000 Cc: "scottl@netflix.com" , "Radford, Adam" , "Kenneth D. Merry" , "sean_bruno@yahoo.com" , "Mankani, Krishnaraddi" , "dwhite@ixsystems.com" , "Maloy, Joe" , "jpaetzel@freebsd.org" , "freebsd-scsi@freebsd.org" , "McConnell, Stephen" X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jan 2014 08:17:49 -0000 Doug: I have replied inline for your comment. For most of the comment you posted = below can be work out and we can prioritize completion of those based on na= ture of issue. I will work on READ/WRITE comparison w.r.t and as a priority = item.=20 Please consider that it will be really helpful for further development, if = we have common code check-in in Upstream FreeBSD. We will actively work on = each comment posted for and address those before we have in= -box for any FreeBSD release. Thanks, Kashyap > -----Original Message----- > From: Doug Ambrisko [mailto:ambrisko@cisco.com] > Sent: Saturday, January 04, 2014 2:45 AM > To: Desai, Kashyap > Cc: freebsd-scsi@freebsd.org; scottl@netflix.com; sean_bruno@yahoo.com; > dwhite@ixsystems.com; jpaetzel@freebsd.org; Maloy, Joe; Mankani, > Krishnaraddi; McConnell, Stephen; Saxena, Sumit; Radford, Adam; Kenneth > D. Merry > Subject: Re: LSI - MR-Fusion controller driver patch and man page >=20 > On Fri, Dec 06, 2013 at 02:37:21PM +0000, Desai, Kashyap wrote: > | Hi, > | > | Please consider attached patch for FreeBSD upstream check-in. > | > | Please find attached patch for driver for LSI's MegaRaid > Controllers. This driver supports Thunderbolt onwards Device IDs of MR > controllers. > | Currently it supports 0x005B and 0x005D Device IDs. > | > | NOTE : This driver will not eliminate or by pass any functionality of <= mfi> > driver which already support above to Device IDs to keep existing user > experience unchanged. > | > | Driver will be always given priority over driver and > | only if customer/user wants to use/migrate from to , it > | will hook up into kernel via device.hint rules. (Attached is mrsas man > | page for more info.) > | > | LSI will continue to update driver in future in timely bases. > | We have another set of patch in pipeline to be submitted for , > | but need first go ahead for attached patch and later we will continue > | to keep up-to-date (In sync with LSI released driver which is > | available at lsi's external site) > | > | Apply patch with "patch -p0 < patchname.patch" from head directory. > | > | -- Few notes for user-- > | LSI recommends using fusion_force bit In hint settings at start of the = day, if > they want to use . ( will be a default choice for MR-Fusion > HBA), if will be changed only with fusion_force hint settings. (See mrsas= man > page) Changing any default behavior is well tested for most of the condit= ion. > | Switching from to for MR-Fusion options is designed to > allow user as one time choice, though multiple time switch from to > is possible, it is not recommended. So, user needs to decide from > start of the day, which driver they want to use for MR-Fusion card. > | > | -- Implementation details -- > | To support this feature, we have modify code to change default > return type from probe. Currently driver return > "BUS_PROBE_DEFAULT". driver has been be changed to return > "BUS_PROBE_LOW_PRIORITY" if fusion_force hint from device.hints is set. > | Please notice, above mentioned implementation in driver is only > applicable in case of MR-Fusion controller detection. For any other > controllers, supported by driver, the behavior of probe return will > remain same as before. > | > | > | -- High level feature list of -- 1. Supports Fast Path feature > | of LSI controllers. > | 2. Supports 4K sector Drives. > | 3. CAM layer based interface. All VDs will be attached to CAM layer > | (Expected storage will be visible in "camcontroll") 4. Complete > | support of Online Controller Reset. (OCR) 5. OCR on Fimrware fault and= IO > timeout case. > | 6. Work well with management application which is generic > application provided by LSI for all other Operating system. > | 7. Supporst DIF enabled VDs (Same support as provided in Linux and > | other OSes in FreeBSD) 8. Fast Path Load balance support. > | > | - In summary, this driver is in part with Linux based MR drivers and > | all other features will be available to as planned activity > | from LSI > | > | This code is well tested by LSI Q/A team on 32 bit and 64 bit FreeBSD > Released OSes. >=20 > Sorry it is has taken so long to start playing with this. I found one cr= itical issue > with read performance. I added a printf to the driver to make sure the f= ast > path is not being used and it claims it isn't. > Doing either a dd or using the > dt(http://www.scsifaq.org/RMiller_Tools/dt.html) > I see good write performance with both mrsas and mfi but on read mrsas is > backing up and appears to be single threaded. With mrsas doing a dd > of=3D/dev/zero if=3D/dev/da0 bs=3D1m I'm seeing 45685 kBps and with mfi s= eeing > 597072 kBps via gstat. Using dt via: > dt of=3D/dev/da0 bs=3D64k limit=3D1g aios=3D32 > I see writes at 538977 Kbytes/sec and reads at bouncing around between > 20000 - 8000 Kbytes/sec, mfi reports 244744 Kbytes/sec. The interesting > piece is seeing the L(q) via gstat being around 30 when doing mrsas reads= , > writes are 1 or less. With mfi both read and writes are 1 or less. >=20 Thanks for for providing feedback. This is very important observation. We w= ill work on this and get back to you. Meanwhile, if we have high level ac= ceptance(considering few defect fixes and optimization as a follow up activ= ity) of current code, can have commit in current FreeBSD dr= iver. This will definitely help us to manage source code from single destin= ation. Currently there are multiple flavors/version of driver code = is floated due to Upstream commit is pending. We can work for each observat= ion/optimization proposed from upstream community on priority bases. > This is with a GENERIC kernel on -current/amd64. I updated my source to > 260231 and see the same. Witness, etc. are all on. I don't recall seein= g this > when I tested mrsas a long time ago. Thanks for additional info. >=20 > The card I'm using is: > LSI MegaRAID SAS 9266-8i > Firmware BIOS 5.38.00_4.12.05.00_0x05180000 > Firmware BCON 6.1-49-e_49-Rel > Firmware PCLI 05.05-03:#%00011 > Firmware APP 3.220.75-2196 > Firmware NVDT 2.1209.03-0117 > Firmware BTBL 2.05.00.00-0010 > Firmware BOOT 07.26.13.219 >=20 > Some other issues I ran into when I kldunload mrsas I got a: > sysctl_unregister_oid: failed to unregister sysctl and then when I I do not recall if we have seen this issue, but will try this on our setup. > kldload mrsas I got a panic: > mrsas0: port 0x7000-0x70ff mem > 0xdf060000-0xdf0 63fff,0xdf000000-0xdf03ffff irq 32 at device 0.0 on pci4 > ######################################################### > Driver has detected MR-Fusion Controller.If you wish to switch to > driver for MR-Fusion,read manual of mrsas. Once you chose to use > Driverf or MR-Fusion Controllers, you should not switch to .= LSI > recommend to use rsas> driver for MR-Fusion.Use if you are not sure about > rsas> choice.Quick he > lp to use for MR-Fustion Card: > Remove hint.mrsas.0.fusion_force=3D1 from /boot/device.hints. > ######################################################### > mrsas0: Waiting for FW to come to ready state >=20 > >=20 > db> tr > Tracing pid 534 tid 100245 td 0xfffff80015035490 > malloc() at malloc+0x1c8/frame 0xfffffe03fbab3200 > cpuctl_devs() at cpuctl_devs+0x61df/frame 0xfffffe03fbab3290 > cpuctl_devs() at cpuctl_devs+0x6189/frame 0xfffffe03fbab32f0 > cpuctl_devs() at cpuctl_devs+0x990a/frame 0xfffffe03fbab34b0 > device_attach() at device_attach+0x3a2/frame 0xfffffe03fbab3510 > pci_driver_added() at pci_driver_added+0xfa/frame 0xfffffe03fbab3550 > devclass_driver_added() at devclass_driver_added+0x7d/frame > 0xfffffe03fbab3590 > devclass_add_driver() at devclass_add_driver+0x127/frame > 0xfffffe03fbab35d0 > module_register_init() at module_register_init+0xb0/frame > 0xfffffe03fbab3600 > linker_load_module() at linker_load_module+0xbda/frame > 0xfffffe03fbab3930 > kern_kldload() at kern_kldload+0xad/frame 0xfffffe03fbab3970 > sys_kldload() at sys_kldload+0x5b/frame 0xfffffe03fbab39a0 > amd64_syscall() at amd64_syscall+0x265/frame 0xfffffe03fbab3ab0 > Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe03fbab3ab0 >=20 > I played with storcli/storcli64 and MegaCli/MegaCli64 from LSI's web site= . > I noticed that the 32 bit versions are built for FreeBSD 8.1 and the 64 b= it > FreeBSD 7.4. It might not be a critical problem but some companies might= still > be on earlier releases. >=20 > I need to look at adding the ioctl translation support for mfiutil and Li= nux to > mrsas. This means we should have a 32 bit to 64 bit ioctl translator in = the > driver since our Linux emulation is 32 bit. We can pick this work as next action item. > It's also handy to run 32 bit tools on the 64 bit kernel. >=20 > It would be good to add alias support for /dev/mfid* such as was done via > ATA CAM sys/cam/ata/ata_da.c. Search for ada_legacy_aliases and > legacy_id. The helped the migration from /dev/ad* to /dev/ada*. >=20 Let me check this. You can help me to understand what is expected here and = we can provide solution for this. Again, we can mark this as next to-do for mrsas and will provide patch as a= n when we have code ready. > I'm not sure hint.mrsas..fusion_force=3D"1" is a good toggle to > enable/disable mrsas. I can see why you did that but it might be more > obvious to make hint.mfi..disabled=3D"1" to work since that follows mo= re > of what people are used to. Either way is fine, since hint.mfi..disabled may miss lead that we are g= oing to disable completely. We may have driver working for Falcon and other old MFI based control= lers. >=20 > Thanks, >=20 > Doug A. From owner-freebsd-scsi@FreeBSD.ORG Mon Jan 6 18:33:24 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EC0C34AE; Mon, 6 Jan 2014 18:33:23 +0000 (UTC) Received: from mtv-iport-2.cisco.com (mtv-iport-2.cisco.com [173.36.130.13]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id B80E51DCF; Mon, 6 Jan 2014 18:33:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=14864; q=dns/txt; s=iport; t=1389033203; x=1390242803; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=8rqcPJLySllLqB6nSi+j4LtOUZtv/3siqybD2UfBNPU=; b=jzi5Xy+Xjr6NGtwBYLHeVTui2kpE2DsSrge+x4L2HsXajktZT3lTdqqc r2IMrWNEQqJexRKMle1w+HE6TE2t13xrr/14oUrYoGYTgbt00o42c/NnZ Pstx5ajatALC7hGPjQiJbWaxOX1asbG9pwKke8l3T+ceExMVR+kXjq6dW Y=; X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ah8FAB/2ylKrRDoI/2dsb2JhbABYgws4qAeSF4EMFnSCJQEBAQQOLCsUDAQLEQQBAQoeBw8FMQEJDhMJCweHVQMQDsMiF4x8gScVDkkHBoMegRMEiUOENog4gWUBjFqFO4NOgUgIFw X-IronPort-AV: E=Sophos;i="4.95,614,1384300800"; d="scan'208";a="102244297" Received: from mtv-core-3.cisco.com ([171.68.58.8]) by mtv-iport-2.cisco.com with ESMTP; 06 Jan 2014 18:32:15 +0000 Received: from dambriskodt.cisco.com (dambriskodt.cisco.com [171.71.230.115]) by mtv-core-3.cisco.com (8.14.5/8.14.5) with ESMTP id s06IW3Ma028053 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 6 Jan 2014 18:32:10 GMT Received: from dambriskodt.cisco.com (localhost [127.0.0.1]) by dambriskodt.cisco.com (8.14.7/8.14.3) with ESMTP id s06ITa8b093701; Mon, 6 Jan 2014 10:29:36 -0800 (PST) (envelope-from ambrisko@dambriskodt.cisco.com) Received: (from ambrisko@localhost) by dambriskodt.cisco.com (8.14.7/8.14.3/Submit) id s06ITZ4W093700; Mon, 6 Jan 2014 10:29:35 -0800 (PST) (envelope-from ambrisko) Date: Mon, 6 Jan 2014 10:29:35 -0800 From: Doug Ambrisko To: "Desai, Kashyap" Subject: Re: LSI - MR-Fusion controller driver patch and man page Message-ID: <20140106182935.GC93278@cisco.com> References: <08ba2a262fba45f687cdd3225f325110@BN1PR07MB247.namprd07.prod.outlook.com> <20140103211449.GA69721@cisco.com> <8c423414ecc2421fbace3eb9f386be91@BN1PR07MB247.namprd07.prod.outlook.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <8c423414ecc2421fbace3eb9f386be91@BN1PR07MB247.namprd07.prod.outlook.com> User-Agent: Mutt/1.4.2.3i X-Mailman-Approved-At: Mon, 06 Jan 2014 18:39:40 +0000 Cc: "scottl@netflix.com" , "Radford, Adam" , "Kenneth D. Merry" , "sean_bruno@yahoo.com" , "Mankani, Krishnaraddi" , "dwhite@ixsystems.com" , "Maloy, Joe" , "jpaetzel@freebsd.org" , "freebsd-scsi@freebsd.org" , "McConnell, Stephen" X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jan 2014 18:33:24 -0000 On Mon, Jan 06, 2014 at 07:43:27AM +0000, Desai, Kashyap wrote: | Doug: | | I have replied inline for your comment. For most of the comment you | posted below can be work out and we can prioritize completion of those | based on nature of issue. | I will work on READ/WRITE comparison w.r.t and as a | priority item. IO performance is the highest priority. | Please consider that it will be really helpful for further development, | if we have common code check-in in Upstream FreeBSD. We will actively | work on each comment posted for and address those before we | have in-box for any FreeBSD release. What we have done in the past is to create a project branch to do this and it has worked well. It would be nice if you could check in directly to a this project branch so we could use svn to resolve differences. I poked around at adding the ioctl compatibility support and noticed that mrsas seems to lack a general method to send a dcmd to the controller. What Scott did in mfi was to create a generic function to send dcmd's with a simple data buffer. That data buffer was then put into the sg list in the generic function. This made it easier to translate the various ioctl's since we could convert the sg lists passed down from the user land into a simple malloced data base and then covert back out. In the mfituil ioctl method it just passes down the data buffer and then let's the driver convert it to a sg list. To make ioctl emulation easier in these cases: 32 bit FreeBSD LSI ioctl on 64 bit 32 bit Linux LSI ioctl on 32 bit 32 bit Linux LSI ioctl on 64 bit 32 bit mfiutil ioctl on 32 bit 32 bit mfiutil ioctl on 64 bit Of high priority is style, please make it style(9) compliant. There are a trailing spaces, spaces instead of tabs, I think lack of tab aligned indent. I noticed functions being externally defined in .c files instead of .h. There should be .h files that can be included in user land applications for ioctls and there can be .h files just for the kernel. These files will need to be installed as appropriate. I found this with the ioctl function. | > -----Original Message----- | > From: Doug Ambrisko [mailto:ambrisko@cisco.com] | > Sent: Saturday, January 04, 2014 2:45 AM | > To: Desai, Kashyap | > Cc: freebsd-scsi@freebsd.org; scottl@netflix.com; sean_bruno@yahoo.com; | > dwhite@ixsystems.com; jpaetzel@freebsd.org; Maloy, Joe; Mankani, | > Krishnaraddi; McConnell, Stephen; Saxena, Sumit; Radford, Adam; Kenneth | > D. Merry | > Subject: Re: LSI - MR-Fusion controller driver patch and man page | > | > On Fri, Dec 06, 2013 at 02:37:21PM +0000, Desai, Kashyap wrote: | > | Hi, | > | | > | Please consider attached patch for FreeBSD upstream check-in. | > | | > | Please find attached patch for driver for LSI's MegaRaid | > Controllers. This driver supports Thunderbolt onwards Device IDs of MR | > controllers. | > | Currently it supports 0x005B and 0x005D Device IDs. | > | | > | NOTE : This driver will not eliminate or by pass any functionality of | > driver which already support above to Device IDs to keep existing user | > experience unchanged. | > | | > | Driver will be always given priority over driver and | > | only if customer/user wants to use/migrate from to , it | > | will hook up into kernel via device.hint rules. (Attached is mrsas man | > | page for more info.) | > | | > | LSI will continue to update driver in future in timely bases. | > | We have another set of patch in pipeline to be submitted for , | > | but need first go ahead for attached patch and later we will continue | > | to keep up-to-date (In sync with LSI released driver which is | > | available at lsi's external site) | > | | > | Apply patch with "patch -p0 < patchname.patch" from head directory. | > | | > | -- Few notes for user-- | > | LSI recommends using fusion_force bit In hint settings at start of the day, if | > they want to use . ( will be a default choice for MR-Fusion | > HBA), if will be changed only with fusion_force hint settings. (See mrsas man | > page) Changing any default behavior is well tested for most of the condition. | > | Switching from to for MR-Fusion options is designed to | > allow user as one time choice, though multiple time switch from to | > is possible, it is not recommended. So, user needs to decide from | > start of the day, which driver they want to use for MR-Fusion card. | > | | > | -- Implementation details -- | > | To support this feature, we have modify code to change default | > return type from probe. Currently driver return | > "BUS_PROBE_DEFAULT". driver has been be changed to return | > "BUS_PROBE_LOW_PRIORITY" if fusion_force hint from device.hints is set. | > | Please notice, above mentioned implementation in driver is only | > applicable in case of MR-Fusion controller detection. For any other | > controllers, supported by driver, the behavior of probe return will | > remain same as before. | > | | > | | > | -- High level feature list of -- 1. Supports Fast Path feature | > | of LSI controllers. | > | 2. Supports 4K sector Drives. | > | 3. CAM layer based interface. All VDs will be attached to CAM layer | > | (Expected storage will be visible in "camcontroll") 4. Complete | > | support of Online Controller Reset. (OCR) 5. OCR on Fimrware fault and IO | > timeout case. | > | 6. Work well with management application which is generic | > application provided by LSI for all other Operating system. | > | 7. Supporst DIF enabled VDs (Same support as provided in Linux and | > | other OSes in FreeBSD) 8. Fast Path Load balance support. | > | | > | - In summary, this driver is in part with Linux based MR drivers and | > | all other features will be available to as planned activity | > | from LSI | > | | > | This code is well tested by LSI Q/A team on 32 bit and 64 bit FreeBSD | > Released OSes. | > | > Sorry it is has taken so long to start playing with this. I found one critical issue | > with read performance. I added a printf to the driver to make sure the fast | > path is not being used and it claims it isn't. | > Doing either a dd or using the | > dt(http://www.scsifaq.org/RMiller_Tools/dt.html) | > I see good write performance with both mrsas and mfi but on read mrsas is | > backing up and appears to be single threaded. With mrsas doing a dd | > of=/dev/zero if=/dev/da0 bs=1m I'm seeing 45685 kBps and with mfi seeing | > 597072 kBps via gstat. Using dt via: | > dt of=/dev/da0 bs=64k limit=1g aios=32 | > I see writes at 538977 Kbytes/sec and reads at bouncing around between | > 20000 - 8000 Kbytes/sec, mfi reports 244744 Kbytes/sec. The interesting | > piece is seeing the L(q) via gstat being around 30 when doing mrsas reads, | > writes are 1 or less. With mfi both read and writes are 1 or less. | > | Thanks for for providing feedback. This is very important observation. | We will work on this and get back to you. Meanwhile, if we have high | level acceptance(considering few defect fixes and optimization as a | follow up activity) of current code, can have commit in | current FreeBSD driver. This will definitely help us to manage source | code from single destination. Currently there are multiple flavors/version | of driver code is floated due to Upstream commit is pending. We | can work for each observation/optimization proposed from upstream | community on priority bases. Yes a project branch can help with this. | > This is with a GENERIC kernel on -current/amd64. I updated my source to | > 260231 and see the same. Witness, etc. are all on. I don't recall | > seeing this | > when I tested mrsas a long time ago. | | Thanks for additional info. | | > | > The card I'm using is: | > LSI MegaRAID SAS 9266-8i | > Firmware BIOS 5.38.00_4.12.05.00_0x05180000 | > Firmware BCON 6.1-49-e_49-Rel | > Firmware PCLI 05.05-03:#%00011 | > Firmware APP 3.220.75-2196 | > Firmware NVDT 2.1209.03-0117 | > Firmware BTBL 2.05.00.00-0010 | > Firmware BOOT 07.26.13.219 | > | > Some other issues I ran into when I kldunload mrsas I got a: | > sysctl_unregister_oid: failed to unregister sysctl and then when I | | I do not recall if we have seen this issue, but will try this on our setup. It think this and the panic might have been an artifact of my inition testing. I followed the instruction of kldload mrsas from /boot/loader.conf and saw this. However, when I started working the emulation I modified my GERNERIC config to not have mfi in it so I could switch between mfi and mrsas via kldload/kldunload to test ioctl emulation. When I did this I noticed that the patch included mrsas in the GENERIC configuration. That might have caused this issue having both the module and static driver in the same system. So this is probably a bad bug report. Doing kldload and kldunload without mrsas and mfi static in the kernel I didn't run into this again. | > kldload mrsas I got a panic: | > mrsas0: port 0x7000-0x70ff mem | > 0xdf060000-0xdf0 63fff,0xdf000000-0xdf03ffff irq 32 at device 0.0 on pci4 | > ######################################################### | > Driver has detected MR-Fusion Controller.If you wish to switch to | > driver for MR-Fusion,read manual of mrsas. Once you chose to use | > Driverf or MR-Fusion Controllers, you should not switch to .LSI | > recommend to use rsas> driver for MR-Fusion.Use if you are not sure about | > rsas> choice.Quick he | > lp to use for MR-Fustion Card: | > Remove hint.mrsas.0.fusion_force=1 from /boot/device.hints. | > ######################################################### | > mrsas0: Waiting for FW to come to ready state | > | > | > | > db> tr | > Tracing pid 534 tid 100245 td 0xfffff80015035490 | > malloc() at malloc+0x1c8/frame 0xfffffe03fbab3200 | > cpuctl_devs() at cpuctl_devs+0x61df/frame 0xfffffe03fbab3290 | > cpuctl_devs() at cpuctl_devs+0x6189/frame 0xfffffe03fbab32f0 | > cpuctl_devs() at cpuctl_devs+0x990a/frame 0xfffffe03fbab34b0 | > device_attach() at device_attach+0x3a2/frame 0xfffffe03fbab3510 | > pci_driver_added() at pci_driver_added+0xfa/frame 0xfffffe03fbab3550 | > devclass_driver_added() at devclass_driver_added+0x7d/frame | > 0xfffffe03fbab3590 | > devclass_add_driver() at devclass_add_driver+0x127/frame | > 0xfffffe03fbab35d0 | > module_register_init() at module_register_init+0xb0/frame | > 0xfffffe03fbab3600 | > linker_load_module() at linker_load_module+0xbda/frame | > 0xfffffe03fbab3930 | > kern_kldload() at kern_kldload+0xad/frame 0xfffffe03fbab3970 | > sys_kldload() at sys_kldload+0x5b/frame 0xfffffe03fbab39a0 | > amd64_syscall() at amd64_syscall+0x265/frame 0xfffffe03fbab3ab0 | > Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe03fbab3ab0 | > | > I played with storcli/storcli64 and MegaCli/MegaCli64 from LSI's web site. | > I noticed that the 32 bit versions are built for FreeBSD 8.1 and the 64 bit | > FreeBSD 7.4. It might not be a critical problem but some companies might still | > be on earlier releases. | > | > I need to look at adding the ioctl translation support for mfiutil and | > Linux to mrsas. This means we should have a 32 bit to 64 bit ioctl | > translator in the | > driver since our Linux emulation is 32 bit. | We can pick this work as next action item. I started working on this and mentioned some things that would make this easier. | > It's also handy to run 32 bit tools on the 64 bit kernel. | > | > It would be good to add alias support for /dev/mfid* such as was done via | > ATA CAM sys/cam/ata/ata_da.c. Search for ada_legacy_aliases and | > legacy_id. The helped the migration from /dev/ad* to /dev/ada*. | > | Let me check this. You can help me to understand what is expected here | and we can provide solution for this. | Again, we can mark this as next to-do for mrsas and will provide patch | as an when we have code ready. The idea is that when /dev/da node is created and alias of /dev/mfi is created point to /dev/da. This way existing fstab that reference /dev/mfid0 for example will continue to work. It just makes it easier for migration. A trickier part will be the feature to allow both LD and PD of an LD to be available in a system. It is a feature that people use. With mfi it is easy to see RAID controlled devices via /dev/mfid and mfisyspd. The physical device access come up as /dev/da. So people should know be very carefull when accessing /dev/da since it could upset the RAID controller, such as flashing firmware and when the disk goes offline it can kick out the disk from the RAID. I haven't looked at how mrsas deals with deleting and recreating RAID on the fly. mfi can do it with the proper sysctl/tunables set. | > I'm not sure hint.mrsas..fusion_force="1" is a good toggle to | > enable/disable mrsas. I can see why you did that but it might be more | > obvious to make hint.mfi..disabled="1" to work since that follows more | > of what people are used to. | | Either way is fine, since hint.mfi..disabled may miss lead that we are | going to disable completely. | We may have driver working for Falcon and other old MFI based | controllers. Since it is a hint.mfi..disabled, the x refers to card x, ie. hint.mfi.1.disabled would disable the 2nd card in the system what would be the ThunderBolt based can and leave the others to be mfi. We should get more input from FreeBSD folks. fusion_force seems pretty non-obvious to me. I'd suggest force_mrsas to be more obvious and it should probably be a generic tunabled and device instance specific. Adding printing events to the kernel messages seems to be missing. This needs to be controlled by sysctl/tunable but I might not have seen this yet. I'd say the highest priority is on performance and style. Getting a review from Scott would be good but I'm not sure he has had time to do that. It might be good to look at mfi to see how things are laid out. FYI, my biggest problem working on FreeBSD is time. My day job priorities end up eating into time to work on "less critical" issues. Unfortunately a bunch of FreeBSD things fall into that catagory. I need to really resume on my mount path changes. I know what I need to do, I just need to carve out some time to work on it. Thanks, Doug A. From owner-freebsd-scsi@FreeBSD.ORG Mon Jan 6 21:56:29 2014 Return-Path: Delivered-To: freebsd-scsi@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3718D70; Mon, 6 Jan 2014 21:56:29 +0000 (UTC) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 0874F1E3E; Mon, 6 Jan 2014 21:56:29 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id s06LuS2d097000; Mon, 6 Jan 2014 21:56:28 GMT (envelope-from trasz@freefall.freebsd.org) Received: (from trasz@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id s06LuSil096999; Mon, 6 Jan 2014 21:56:28 GMT (envelope-from trasz) Date: Mon, 6 Jan 2014 21:56:28 GMT Message-Id: <201401062156.s06LuSil096999@freefall.freebsd.org> To: trasz@FreeBSD.org, freebsd-scsi@FreeBSD.org, trasz@FreeBSD.org From: trasz@FreeBSD.org Subject: Re: kern/185498: [iscsi] Bug in native iscsi-portal with many ip addresses X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jan 2014 21:56:29 -0000 Synopsis: [iscsi] Bug in native iscsi-portal with many ip addresses Responsible-Changed-From-To: freebsd-scsi->trasz Responsible-Changed-By: trasz Responsible-Changed-When: Mon Jan 6 21:56:28 UTC 2014 Responsible-Changed-Why: I'll take it. http://www.freebsd.org/cgi/query-pr.cgi?pr=185498 From owner-freebsd-scsi@FreeBSD.ORG Mon Jan 6 22:01:26 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7AF9D100 for ; Mon, 6 Jan 2014 22:01:26 +0000 (UTC) Received: from mail-qe0-x22e.google.com (mail-qe0-x22e.google.com [IPv6:2607:f8b0:400d:c02::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 3924F1EB0 for ; Mon, 6 Jan 2014 22:01:26 +0000 (UTC) Received: by mail-qe0-f46.google.com with SMTP id a11so18451268qen.5 for ; Mon, 06 Jan 2014 14:01:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:message-id:subject:from:to:content-type; bh=KiermWVouk+nGTHWfhKwRRb2k8gYaeEmsJpHy8K/494=; b=mwtFZLdgtoiFJiG1iBq0ta6lOtmRMxHAy7uY6YUGy2T4i8l3Th5Rvn7oqpsnwnCu1L 7ZzuguuWTyr6bRnn6Y/D7W+DgcvzxaKhdXqeedenC7wu0gdgH9XPlN0S+a/4LADiyBwU PNOR5SzrhUUbU5f4jXUkwPyzTF57Ok5Ma50z0VxygA4aAbCkcc9efG5NrszAiWfooq+V x1wOy3GP9lXM37aFWKkr0UrzO5bGMAkmHHNLwrtADvTAvraw1m1TYCiKm0QFhQ/aLXl7 QBJnNXwMbShmZt2iTa488b9WvczvbYY47Iu7CW88lqi0xpUJdadi3S/d7le+0qhm5SPL QYPQ== MIME-Version: 1.0 X-Received: by 10.224.3.10 with SMTP id 10mr49808550qal.58.1389045685354; Mon, 06 Jan 2014 14:01:25 -0800 (PST) Sender: benlaurie@gmail.com Received: by 10.96.72.196 with HTTP; Mon, 6 Jan 2014 14:01:25 -0800 (PST) Date: Mon, 6 Jan 2014 22:01:25 +0000 X-Google-Sender-Auth: Z11f52OYjrEvPL-G7Dh5VM5FyDY Message-ID: Subject: Dropped interrupts From: Ben Laurie To: freebsd-scsi@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jan 2014 22:01:26 -0000 Not subscribed to the list, so please cc on replies. I'm using Bacula with an LTO-2 SCSI drive. With increasing frequency lately, I've been getting errors like this from bacula: backup-sd JobId 13092: Error: block.c:608 Write error at 23:6772 on device "Ultrium" (/dev/nsa0). ERR=Operation not permitted. Associated with this, I see in dmesg: ahc0: Recovery Initiated [a lot of dump info, including...] ahc0: Timedout SCBs already complete. Interrupts may not be functioning. Interrupts must be kinda functioning, because it can go for weeks at a time without problems. I've looked into it a bit, and the SCSI board is sharing an interrupt with one of the USB hubs. My first thought was to configure it to not use a shared interrupt, but this turns out to be impossible. Could that be the cause? Any suggestions on how to fix this? I believe backups are not actually compromised, but tapes are not filled, which causes some pretty serious problems. Obviously happy to provide any more info that would be helpful. From owner-freebsd-scsi@FreeBSD.ORG Tue Jan 7 05:46:36 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 70875737 for ; Tue, 7 Jan 2014 05:46:36 +0000 (UTC) Received: from aslan.scsiguy.com (mail.scsiguy.com [70.89.174.89]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 3A0C31309 for ; Tue, 7 Jan 2014 05:46:34 +0000 (UTC) Received: from jt-mbp.home.scsiguy.org (jt-mbp.home.scsiguy.org [192.168.0.61]) (authenticated bits=0) by aslan.scsiguy.com (8.14.7/8.14.5) with ESMTP id s075kRpa049119 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Mon, 6 Jan 2014 22:46:33 -0700 (MST) (envelope-from gibbs@scsiguy.com) Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 7.1 \(1827\)) Subject: Re: Dropped interrupts From: "Justin T. Gibbs" In-Reply-To: Date: Mon, 6 Jan 2014 22:46:28 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: References: To: Ben Laurie X-Mailer: Apple Mail (2.1827) X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.4.3 (aslan.scsiguy.com [70.89.174.89]); Mon, 06 Jan 2014 22:46:33 -0700 (MST) Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Jan 2014 05:46:36 -0000 On Jan 6, 2014, at 3:01 PM, Ben Laurie wrote: > Not subscribed to the list, so please cc on replies. >=20 > I'm using Bacula with an LTO-2 SCSI drive. >=20 > With increasing frequency lately, I've been getting errors like this > from bacula: >=20 > backup-sd JobId 13092: Error: block.c:608 Write error at 23:6772 on > device "Ultrium" (/dev/nsa0). ERR=3DOperation not permitted. >=20 > Associated with this, I see in dmesg: >=20 > ahc0: Recovery Initiated >=20 > [a lot of dump info, including=85] If you provide the dump info, I may be able to tell you why recovery is = starting. The dmesg information from a boot of the system would be good to have = too. =97 Justin= From owner-freebsd-scsi@FreeBSD.ORG Tue Jan 7 07:36:46 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4C22E989 for ; Tue, 7 Jan 2014 07:36:46 +0000 (UTC) Received: from mail-qc0-x22c.google.com (mail-qc0-x22c.google.com [IPv6:2607:f8b0:400d:c01::22c]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 0230319FA for ; Tue, 7 Jan 2014 07:36:45 +0000 (UTC) Received: by mail-qc0-f172.google.com with SMTP id e16so18843775qcx.31 for ; Mon, 06 Jan 2014 23:36:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=ZERa958gG8GuiSzUGLNJ+Bwm0q3IvOkv6WZhEEg6BNc=; b=fuZHYvNO9B9X6aTBS6spGcyiPa/wNtKlUCaHyS1JJ1Ih55fnn4PmhEieZUz5GWtBbu YGN2bm8JKLrpPcClhiak37UNKdsU9PpOjbxjLd3qYMiPB1BdYSL6R0SAokeldvg3Qmc2 PGFJ6xiMhxUizzzGukLGZ8ofDNtm0RVjPYrvzF6ZqoWXWckiKVkIYv2fyVnAit0jJkSA bUGvMwqqVO8M3bHIYBYJI+yK1r9Olm8HWBOo/rPvyCKEBvsKi7/wihX581d9qdIzL4qY O5DeiocW/gNHzPlbWZcSiJxLO6Zfls6gd27ndugEAphW6JQBRdEam3Stc79JKCZ4yc+J ZKHg== MIME-Version: 1.0 X-Received: by 10.224.97.67 with SMTP id k3mr64297651qan.17.1389080205160; Mon, 06 Jan 2014 23:36:45 -0800 (PST) Sender: benlaurie@gmail.com Received: by 10.96.72.196 with HTTP; Mon, 6 Jan 2014 23:36:44 -0800 (PST) In-Reply-To: References: Date: Tue, 7 Jan 2014 07:36:44 +0000 X-Google-Sender-Auth: 2OF9n32b6SsSge2nsQ-ljXi6oK0 Message-ID: Subject: Re: Dropped interrupts From: Ben Laurie To: "Justin T. Gibbs" Content-Type: multipart/mixed; boundary=001a11c3f2b431916e04ef5c7039 Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Jan 2014 07:36:46 -0000 --001a11c3f2b431916e04ef5c7039 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Attached. On 7 January 2014 05:46, Justin T. Gibbs wrote: > On Jan 6, 2014, at 3:01 PM, Ben Laurie wrote: > >> Not subscribed to the list, so please cc on replies. >> >> I'm using Bacula with an LTO-2 SCSI drive. >> >> With increasing frequency lately, I've been getting errors like this >> from bacula: >> >> backup-sd JobId 13092: Error: block.c:608 Write error at 23:6772 on >> device "Ultrium" (/dev/nsa0). ERR=3DOperation not permitted. >> >> Associated with this, I see in dmesg: >> >> ahc0: Recovery Initiated >> >> [a lot of dump info, including=85] > > If you provide the dump info, I may be able to tell you why recovery is s= tarting. > > The dmesg information from a boot of the system would be good to have too= . > > =97 > Justin --001a11c3f2b431916e04ef5c7039 Content-Type: application/octet-stream; name=dmesg Content-Disposition: attachment; filename=dmesg Content-Transfer-Encoding: base64 X-Attachment-Id: f_hq4uh2cx0 Q29weXJpZ2h0IChjKSAxOTkyLTIwMTQgVGhlIEZyZWVCU0QgUHJvamVjdC4KQ29weXJpZ2h0IChj KSAxOTc5LCAxOTgwLCAxOTgzLCAxOTg2LCAxOTg4LCAxOTg5LCAxOTkxLCAxOTkyLCAxOTkzLCAx OTk0CglUaGUgUmVnZW50cyBvZiB0aGUgVW5pdmVyc2l0eSBvZiBDYWxpZm9ybmlhLiBBbGwgcmln aHRzIHJlc2VydmVkLgpGcmVlQlNEIGlzIGEgcmVnaXN0ZXJlZCB0cmFkZW1hcmsgb2YgVGhlIEZy ZWVCU0QgRm91bmRhdGlvbi4KRnJlZUJTRCA5LjItU1RBQkxFICM1IHIyNjAyNTc6IFNhdCBKYW4g IDQgMjI6MDk6MDggR01UIDIwMTQKICAgIHJvb3RAYmFja3VwLmhvbWUuOi91c3Ivb2JqL3Vzci9z cmMvc3lzL0dFTkVSSUMgaTM4NgpnY2MgdmVyc2lvbiA0LjIuMSAyMDA3MDgzMSBwYXRjaGVkIFtG cmVlQlNEXQpDUFU6IEludGVsKFIpIFBlbnRpdW0oUikgNCBDUFUgMi44MEdIeiAoMjc5My4xNi1N SHogNjg2LWNsYXNzIENQVSkKICBPcmlnaW4gPSAiR2VudWluZUludGVsIiAgSWQgPSAweGY0OSAg RmFtaWx5ID0gMHhmICBNb2RlbCA9IDB4NCAgU3RlcHBpbmcgPSA5CiAgRmVhdHVyZXM9MHhiZmVi ZmJmZjxGUFUsVk1FLERFLFBTRSxUU0MsTVNSLFBBRSxNQ0UsQ1g4LEFQSUMsU0VQLE1UUlIsUEdF LE1DQSxDTU9WLFBBVCxQU0UzNixDTEZMVVNILERUUyxBQ1BJLE1NWCxGWFNSLFNTRSxTU0UyLFNT LEhUVCxUTSxQQkU+CiAgRmVhdHVyZXMyPTB4NjQxZDxTU0UzLERURVM2NCxNT04sRFNfQ1BMLENO WFQtSUQsQ1gxNix4VFBSPgogIEFNRCBGZWF0dXJlcz0weDIwMTAwMDAwPE5YLExNPgogIEFNRCBG ZWF0dXJlczI9MHgxPExBSEY+CiAgVFNDOiBQLXN0YXRlIGludmFyaWFudApyZWFsIG1lbW9yeSAg PSA1MzY4NzA5MTIgKDUxMiBNQikKYXZhaWwgbWVtb3J5ID0gNDkxNzI0ODAwICg0NjggTUIpCkV2 ZW50IHRpbWVyICJMQVBJQyIgcXVhbGl0eSA0MDAKQUNQSSBBUElDIFRhYmxlOiA8REVMTCAgIEdY NTIwICA+CkZyZWVCU0QvU01QOiBNdWx0aXByb2Nlc3NvciBTeXN0ZW0gRGV0ZWN0ZWQ6IDIgQ1BV cwpGcmVlQlNEL1NNUDogMSBwYWNrYWdlKHMpIHggMSBjb3JlKHMpIHggMiBIVFQgdGhyZWFkcwog Y3B1MCAoQlNQKTogQVBJQyBJRDogIDAKIGNwdTEgKEFQL0hUKTogQVBJQyBJRDogIDEKaW9hcGlj MDogQ2hhbmdpbmcgQVBJQyBJRCB0byA4CmlvYXBpYzAgPFZlcnNpb24gMi4wPiBpcnFzIDAtMjMg b24gbW90aGVyYm9hcmQKbGFwaWMwOiBGb3JjaW5nIExJTlQxIHRvIGVkZ2UgdHJpZ2dlcgprYmQx IGF0IGtiZG11eDAKYWNwaTA6IDxERUxMIEdYNTIwICA+IG9uIG1vdGhlcmJvYXJkCmFjcGkwOiBQ b3dlciBCdXR0b24gKGZpeGVkKQphY3BpMDogcmVzZXJ2YXRpb24gb2YgMCwgYTAwMDAgKDMpIGZh aWxlZAphY3BpMDogcmVzZXJ2YXRpb24gb2YgMTAwMDAwLCBmMDAwMDAgKDMpIGZhaWxlZAphY3Bp MDogcmVzZXJ2YXRpb24gb2YgMTAwMDAwMCwgMWU2ODZjMDAgKDMpIGZhaWxlZApjcHUwOiA8QUNQ SSBDUFU+IG9uIGFjcGkwCmNwdTE6IDxBQ1BJIENQVT4gb24gYWNwaTAKYXRydGMwOiA8QVQgcmVh bHRpbWUgY2xvY2s+IHBvcnQgMHg3MC0weDdmIGlycSA4IG9uIGFjcGkwCkV2ZW50IHRpbWVyICJS VEMiIGZyZXF1ZW5jeSAzMjc2OCBIeiBxdWFsaXR5IDAKYXR0aW1lcjA6IDxBVCB0aW1lcj4gcG9y dCAweDQwLTB4NWYgaXJxIDAgb24gYWNwaTAKVGltZWNvdW50ZXIgImk4MjU0IiBmcmVxdWVuY3kg MTE5MzE4MiBIeiBxdWFsaXR5IDAKRXZlbnQgdGltZXIgImk4MjU0IiBmcmVxdWVuY3kgMTE5MzE4 MiBIeiBxdWFsaXR5IDEwMApUaW1lY291bnRlciAiQUNQSS1mYXN0IiBmcmVxdWVuY3kgMzU3OTU0 NSBIeiBxdWFsaXR5IDkwMAphY3BpX3RpbWVyMDogPDI0LWJpdCB0aW1lciBhdCAzLjU3OTU0NU1I ej4gcG9ydCAweDgwOC0weDgwYiBvbiBhY3BpMApocGV0MDogPEhpZ2ggUHJlY2lzaW9uIEV2ZW50 IFRpbWVyPiBpb21lbSAweGZlZDAwMDAwLTB4ZmVkMDAzZmYgb24gYWNwaTAKVGltZWNvdW50ZXIg IkhQRVQiIGZyZXF1ZW5jeSAxNDMxODE4MCBIeiBxdWFsaXR5IDk1MApFdmVudCB0aW1lciAiSFBF VCIgZnJlcXVlbmN5IDE0MzE4MTgwIEh6IHF1YWxpdHkgNDUwCkV2ZW50IHRpbWVyICJIUEVUMSIg ZnJlcXVlbmN5IDE0MzE4MTgwIEh6IHF1YWxpdHkgNDQwCkV2ZW50IHRpbWVyICJIUEVUMiIgZnJl cXVlbmN5IDE0MzE4MTgwIEh6IHF1YWxpdHkgNDQwCmFjcGlfYnV0dG9uMDogPFBvd2VyIEJ1dHRv bj4gb24gYWNwaTAKcGNpYjA6IDxBQ1BJIEhvc3QtUENJIGJyaWRnZT4gcG9ydCAweGNmOC0weGNm ZiBvbiBhY3BpMApwY2kwOiA8QUNQSSBQQ0kgYnVzPiBvbiBwY2liMApwY2liMTogPEFDUEkgUENJ LVBDSSBicmlkZ2U+IGlycSAxNiBhdCBkZXZpY2UgMS4wIG9uIHBjaTAKcGNpMTogPEFDUEkgUENJ IGJ1cz4gb24gcGNpYjEKdmdhcGNpMDogPFZHQS1jb21wYXRpYmxlIGRpc3BsYXk+IHBvcnQgMHhl Y2Q4LTB4ZWNkZiBtZW0gMHhmZWIwMDAwMC0weGZlYjdmZmZmLDB4ZTAwMDAwMDAtMHhlZmZmZmZm ZiwweGZlYWMwMDAwLTB4ZmVhZmZmZmYgaXJxIDE2IGF0IGRldmljZSAyLjAgb24gcGNpMAphZ3Aw OiA8SW50ZWwgODI5NDVHICg5NDVHIEdNQ0gpIFNWR0EgY29udHJvbGxlcj4gb24gdmdhcGNpMAph Z3AwOiBhcGVydHVyZSBzaXplIGlzIDI1Nk0sIGRldGVjdGVkIDc5MzJrIHN0b2xlbiBtZW1vcnkK dmdhcGNpMTogPFZHQS1jb21wYXRpYmxlIGRpc3BsYXk+IG1lbSAweGZlYjgwMDAwLTB4ZmViZmZm ZmYgYXQgZGV2aWNlIDIuMSBvbiBwY2kwCnBjaWIyOiA8QUNQSSBQQ0ktUENJIGJyaWRnZT4gaXJx IDE2IGF0IGRldmljZSAyOC4wIG9uIHBjaTAKcGNpMjogPEFDUEkgUENJIGJ1cz4gb24gcGNpYjIK YmdlMDogPEJyb2FkY29tIEdpZ2FiaXQgRXRoZXJuZXQgQ29udHJvbGxlciwgQVNJQyByZXYuIDB4 MDA0MDAxPiBtZW0gMHhmZThmMDAwMC0weGZlOGZmZmZmIGlycSAxNiBhdCBkZXZpY2UgMC4wIG9u IHBjaTIKYmdlMDogQ0hJUCBJRCAweDAwMDA0MDAxOyBBU0lDIFJFViAweDA0OyBDSElQIFJFViAw eDQwOyBQQ0ktRQptaWlidXMwOiA8TUlJIGJ1cz4gb24gYmdlMApicmdwaHkwOiA8QkNNNTc1MCAx MDAwQkFTRS1UIG1lZGlhIGludGVyZmFjZT4gUEhZIDEgb24gbWlpYnVzMApicmdwaHkwOiAgMTBi YXNlVCwgMTBiYXNlVC1GRFgsIDEwMGJhc2VUWCwgMTAwYmFzZVRYLUZEWCwgMTAwMGJhc2VULCAx MDAwYmFzZVQtbWFzdGVyLCAxMDAwYmFzZVQtRkRYLCAxMDAwYmFzZVQtRkRYLW1hc3RlciwgYXV0 bywgYXV0by1mbG93CmJnZTA6IEV0aGVybmV0IGFkZHJlc3M6IDAwOjEzOjcyOjg4OmI0OjFlCnBj aWIzOiA8QUNQSSBQQ0ktUENJIGJyaWRnZT4gaXJxIDE3IGF0IGRldmljZSAyOC4xIG9uIHBjaTAK cGNpMzogPEFDUEkgUENJIGJ1cz4gb24gcGNpYjMKdWhjaTA6IDxJbnRlbCA4MjgwMUcgKElDSDcp IFVTQiBjb250cm9sbGVyIFVTQi1BPiBwb3J0IDB4ZmY4MC0weGZmOWYgaXJxIDIxIGF0IGRldmlj ZSAyOS4wIG9uIHBjaTAKdWhjaTA6IExlZ1N1cCA9IDB4MzAwMAp1c2J1czAgb24gdWhjaTAKdWhj aTE6IDxJbnRlbCA4MjgwMUcgKElDSDcpIFVTQiBjb250cm9sbGVyIFVTQi1CPiBwb3J0IDB4ZmY2 MC0weGZmN2YgaXJxIDIyIGF0IGRldmljZSAyOS4xIG9uIHBjaTAKdXNidXMxIG9uIHVoY2kxCnVo Y2kyOiA8SW50ZWwgODI4MDFHIChJQ0g3KSBVU0IgY29udHJvbGxlciBVU0ItQz4gcG9ydCAweGZm NDAtMHhmZjVmIGlycSAxOCBhdCBkZXZpY2UgMjkuMiBvbiBwY2kwCnVzYnVzMiBvbiB1aGNpMgp1 aGNpMzogPEludGVsIDgyODAxRyAoSUNINykgVVNCIGNvbnRyb2xsZXIgVVNCLUQ+IHBvcnQgMHhm ZjIwLTB4ZmYzZiBpcnEgMjMgYXQgZGV2aWNlIDI5LjMgb24gcGNpMAp1c2J1czMgb24gdWhjaTMK ZWhjaTA6IDxJbnRlbCA4MjgwMUdCL1IgKElDSDcpIFVTQiAyLjAgY29udHJvbGxlcj4gbWVtIDB4 ZmZhODA4MDAtMHhmZmE4MGJmZiBpcnEgMjEgYXQgZGV2aWNlIDI5Ljcgb24gcGNpMAp1c2J1czQ6 IEVIQ0kgdmVyc2lvbiAxLjAKdXNidXM0IG9uIGVoY2kwCnBjaWI0OiA8QUNQSSBQQ0ktUENJIGJy aWRnZT4gYXQgZGV2aWNlIDMwLjAgb24gcGNpMApwY2k0OiA8QUNQSSBQQ0kgYnVzPiBvbiBwY2li NAphaGMwOiA8QWRhcHRlYyAyOTE2MCBVbHRyYTE2MCBTQ1NJIGFkYXB0ZXI+IHBvcnQgMHhkYzAw LTB4ZGNmZiBtZW0gMHhmZTVmZjAwMC0weGZlNWZmZmZmIGlycSAxOCBhdCBkZXZpY2UgMi4wIG9u IHBjaTQKYWljNzg5MjogVWx0cmExNjAgV2lkZSBDaGFubmVsIEEsIFNDU0kgSWQ9NywgMzIvMjUz IFNDQnMKaXNhYjA6IDxQQ0ktSVNBIGJyaWRnZT4gYXQgZGV2aWNlIDMxLjAgb24gcGNpMAppc2Ew OiA8SVNBIGJ1cz4gb24gaXNhYjAKYXRhcGNpMDogPEludGVsIElDSDcgVURNQTEwMCBjb250cm9s bGVyPiBwb3J0IDB4MWYwLTB4MWY3LDB4M2Y2LDB4MTcwLTB4MTc3LDB4Mzc2LDB4ZmZhMC0weGZm YWYgaXJxIDE2IGF0IGRldmljZSAzMS4xIG9uIHBjaTAKYXRhMDogPEFUQSBjaGFubmVsPiBhdCBj aGFubmVsIDAgb24gYXRhcGNpMAphdGFwY2kxOiA8SW50ZWwgSUNINyBTQVRBMzAwIGNvbnRyb2xs ZXI+IHBvcnQgMHhmZTAwLTB4ZmUwNywweGZlMTAtMHhmZTEzLDB4ZmUyMC0weGZlMjcsMHhmZTMw LTB4ZmUzMywweGZlYTAtMHhmZWFmIGlycSAyMCBhdCBkZXZpY2UgMzEuMiBvbiBwY2kwCmF0YTI6 IDxBVEEgY2hhbm5lbD4gYXQgY2hhbm5lbCAwIG9uIGF0YXBjaTEKYXRhMzogPEFUQSBjaGFubmVs PiBhdCBjaGFubmVsIDEgb24gYXRhcGNpMQpwY2kwOiA8c2VyaWFsIGJ1cywgU01CdXM+IGF0IGRl dmljZSAzMS4zIChubyBkcml2ZXIgYXR0YWNoZWQpCnBtdGltZXIwIG9uIGlzYTAKb3JtMDogPElT QSBPcHRpb24gUk9Ncz4gYXQgaW9tZW0gMHhjMDAwMC0weGNhN2ZmLDB4Y2E4MDAtMHhjYzdmZiww eGNjODAwLTB4Y2ZmZmYgcG5waWQgT1JNMDAwMCBvbiBpc2EwCnNjMDogPFN5c3RlbSBjb25zb2xl PiBhdCBmbGFncyAweDEwMCBvbiBpc2EwCnNjMDogVkdBIDwxNiB2aXJ0dWFsIGNvbnNvbGVzLCBm bGFncz0weDMwMD4KdmdhMDogPEdlbmVyaWMgSVNBIFZHQT4gYXQgcG9ydCAweDNjMC0weDNkZiBp b21lbSAweGEwMDAwLTB4YmZmZmYgb24gaXNhMAphdGtiZGMwOiA8S2V5Ym9hcmQgY29udHJvbGxl ciAoaTgwNDIpPiBhdCBwb3J0IDB4NjAsMHg2NCBvbiBpc2EwCmF0a2JkMDogPEFUIEtleWJvYXJk PiBpcnEgMSBvbiBhdGtiZGMwCmtiZDAgYXQgYXRrYmQwCmF0a2JkMDogW0dJQU5ULUxPQ0tFRF0K cHBjMDogcGFyYWxsZWwgcG9ydCBub3QgZm91bmQuCnA0dGNjMDogPENQVSBGcmVxdWVuY3kgVGhl cm1hbCBDb250cm9sPiBvbiBjcHUwCnA0dGNjMTogPENQVSBGcmVxdWVuY3kgVGhlcm1hbCBDb250 cm9sPiBvbiBjcHUxClRpbWVjb3VudGVycyB0aWNrIGV2ZXJ5IDEuMDAwIG1zZWMKdXNidXMwOiAx Mk1icHMgRnVsbCBTcGVlZCBVU0IgdjEuMAp1c2J1czE6IDEyTWJwcyBGdWxsIFNwZWVkIFVTQiB2 MS4wCnVzYnVzMjogMTJNYnBzIEZ1bGwgU3BlZWQgVVNCIHYxLjAKdXNidXMzOiAxMk1icHMgRnVs bCBTcGVlZCBVU0IgdjEuMAp1c2J1czQ6IDQ4ME1icHMgSGlnaCBTcGVlZCBVU0IgdjIuMAp1Z2Vu MC4xOiA8SW50ZWw+IGF0IHVzYnVzMAp1aHViMDogPEludGVsIFVIQ0kgcm9vdCBIVUIsIGNsYXNz IDkvMCwgcmV2IDEuMDAvMS4wMCwgYWRkciAxPiBvbiB1c2J1czAKdWdlbjEuMTogPEludGVsPiBh dCB1c2J1czEKdWh1YjE6IDxJbnRlbCBVSENJIHJvb3QgSFVCLCBjbGFzcyA5LzAsIHJldiAxLjAw LzEuMDAsIGFkZHIgMT4gb24gdXNidXMxCnVnZW4yLjE6IDxJbnRlbD4gYXQgdXNidXMyCnVodWIy OiA8SW50ZWwgVUhDSSByb290IEhVQiwgY2xhc3MgOS8wLCByZXYgMS4wMC8xLjAwLCBhZGRyIDE+ IG9uIHVzYnVzMgp1Z2VuMy4xOiA8SW50ZWw+IGF0IHVzYnVzMwp1aHViMzogPEludGVsIFVIQ0kg cm9vdCBIVUIsIGNsYXNzIDkvMCwgcmV2IDEuMDAvMS4wMCwgYWRkciAxPiBvbiB1c2J1czMKdWdl bjQuMTogPEludGVsPiBhdCB1c2J1czQKdWh1YjQ6IDxJbnRlbCBFSENJIHJvb3QgSFVCLCBjbGFz cyA5LzAsIHJldiAyLjAwLzEuMDAsIGFkZHIgMT4gb24gdXNidXM0CmF0YTA6IERNQSBsaW1pdGVk IHRvIFVETUEzMywgY29udHJvbGxlciBmb3VuZCBub24tQVRBNjYgY2FibGUKdWh1YjA6IDIgcG9y dHMgd2l0aCAyIHJlbW92YWJsZSwgc2VsZiBwb3dlcmVkCnVodWIxOiAyIHBvcnRzIHdpdGggMiBy ZW1vdmFibGUsIHNlbGYgcG93ZXJlZAp1aHViMjogMiBwb3J0cyB3aXRoIDIgcmVtb3ZhYmxlLCBz ZWxmIHBvd2VyZWQKdWh1YjM6IDIgcG9ydHMgd2l0aCAyIHJlbW92YWJsZSwgc2VsZiBwb3dlcmVk CnVodWI0OiA4IHBvcnRzIHdpdGggOCByZW1vdmFibGUsIHNlbGYgcG93ZXJlZApzYTAgYXQgYWhj MCBidXMgMCBzY2J1czAgdGFyZ2V0IDIgbHVuIDAKc2EwOiA8SUJNIFVMVFJJVU0tVEQyIDNBWUM+ IFJlbW92YWJsZSBTZXF1ZW50aWFsIEFjY2VzcyBTQ1NJLTMgZGV2aWNlIApzYTA6IFNlcmlhbCBO dW1iZXIgMTExMDE4NzI0OQpzYTA6IDE2MC4wMDBNQi9zIHRyYW5zZmVycyAoODAuMDAwTUh6IERU LCBvZmZzZXQgMzEsIDE2Yml0KQphZGEwIGF0IGF0YTAgYnVzIDAgc2NidXMxIHRhcmdldCAwIGx1 biAwCmFkYTA6IDxXREMgV0QzMjAwU0ItMDFLTUEwIDA4LjA1SjA4PiBBVEEtNiBkZXZpY2UKYWRh MDogU2VyaWFsIE51bWJlciBXRC1XQ0FNUjMzMzAxNjMKYWRhMDogMTAwLjAwME1CL3MgdHJhbnNm ZXJzIChVRE1BNSwgUElPIDgxOTJieXRlcykKYWRhMDogMzA1MjQ0TUIgKDYyNTE0MDMzNSA1MTIg Ynl0ZSBzZWN0b3JzOiAxNkggNjNTL1QgMTYzODNDKQphZGEwOiBQcmV2aW91c2x5IHdhcyBrbm93 biBhcyBhZDAKYWRhMSBhdCBhdGEwIGJ1cyAwIHNjYnVzMSB0YXJnZXQgMSBsdW4gMAphZGExOiA8 U1QzMzIwNjIwQSAzLkFBRD4gQVRBLTcgZGV2aWNlCmFkYTE6IFNlcmlhbCBOdW1iZXIgOVFGMDZR Nk0KYWRhMTogMzMuMzAwTUIvcyB0cmFuc2ZlcnMgKFVETUEyLCBQSU8gODE5MmJ5dGVzKQphZGEx OiAzMDUyNDVNQiAoNjI1MTQyNDQ4IDUxMiBieXRlIHNlY3RvcnM6IDE2SCA2M1MvVCAxNjM4M0Mp CmFkYTE6IFByZXZpb3VzbHkgd2FzIGtub3duIGFzIGFkMQpsYXBpYzE6IEZvcmNpbmcgTElOVDEg dG8gZWRnZSB0cmlnZ2VyClNNUDogQVAgQ1BVICMxIExhdW5jaGVkIQpUaW1lY291bnRlciAiVFND LWxvdyIgZnJlcXVlbmN5IDEzOTY1Nzc3MjggSHogcXVhbGl0eSAxMDAwClRyeWluZyB0byBtb3Vu dCByb290IGZyb20gdWZzOi9kZXYvYWQwczFhIFtyd10uLi4KYWhjMDogUmVjb3ZlcnkgSW5pdGlh dGVkCj4+Pj4+Pj4+Pj4+Pj4+Pj4+PiBEdW1wIENhcmQgU3RhdGUgQmVnaW5zIDw8PDw8PDw8PDw8 PDw8PDw8CmFoYzA6IER1bXBpbmcgQ2FyZCBTdGF0ZSBpbiBDb21tYW5kIHBoYXNlLCBhdCBTRVFB RERSIDB4MTZjCkNhcmQgd2FzIHBhdXNlZApBQ0NVTSA9IDB4ODAsIFNJTkRFWCA9IDB4YTAsIERJ TkRFWCA9IDB4ZTQsIEFSR18yID0gMHgzYgpIQ05UID0gMHgwIFNDQlBUUiA9IDB4MApTQ1NJUEhB U0VbMHgwXSBTQ1NJU0lHSVsweDg0XTooQlNZSXxDREkpIEVSUk9SWzB4MF0gClNDU0lCVVNMWzB4 ODBdIExBU1RQSEFTRVsweDgwXTooQ0RJKSBTQ1NJU0VRWzB4MTJdOihFTkFVVE9BVE5QfEVOUlNF TEkpIApTQkxLQ1RMWzB4YV06KFNFTFdJREV8U0VMQlVTQikgU0NTSVJBVEVbMHhjMl06KEVOQUJM RV9DUkN8V0lERVhGRVIpIApTRVFDVExbMHgxMF06KEZBU1RNT0RFKSBTRVFfRkxBR1NbMHgwXSBT U1RBVDBbMHg3XTooRE1BRE9ORXxTUElPUkRZfFNET05FKSAKU1NUQVQxWzB4MF0gU1NUQVQyWzB4 MF0gU1NUQVQzWzB4MF0gU0lNT0RFMFsweDhdOihFTlNXUkFQKSAKU0lNT0RFMVsweGFjXTooRU5T Q1NJUEVSUnxFTkJVU0ZSRUV8RU5TQ1NJUlNUfEVOU0VMVElNTykgClNYRlJDVEwwWzB4ODhdOihT UElPRU58REZPTikgREZDTlRSTFsweDRdOihESVJFQ1RJT04pIApERlNUQVRVU1sweDg5XTooRklG T0VNUHxIRE9ORXxQUkVMT0FEX0FWQUlMKSAKU1RBQ0s6IDB4MzQgMHgwIDB4MTY0IDB4MTc5ClND QiBjb3VudCA9IDI1NApLZXJuZWwgTkVYVFFTQ0IgPSAyNTEKQ2FyZCBORVhUUVNDQiA9IDI1MQpR SU5GSUZPIGVudHJpZXM6IApXYWl0aW5nIFF1ZXVlIGVudHJpZXM6IApEaXNjb25uZWN0ZWQgUXVl dWUgZW50cmllczogClFPVVRGSUZPIGVudHJpZXM6IApTZXF1ZW5jZXIgRnJlZSBTQ0IgTGlzdDog MSAyIDMgNCA1IDYgNyA4IDkgMTAgMTEgMTIgMTMgMTQgMTUgMTYgMTcgMTggMTkgMjAgMjEgMjIg MjMgMjQgMjUgMjYgMjcgMjggMjkgMzAgMzEgClNlcXVlbmNlciBTQ0IgSW5mbzogCiAgMCBTQ0Jf Q09OVFJPTFsweDBdIFNDQl9TQ1NJSURbMHgyN10gU0NCX0xVTlsweDBdIFNDQl9UQUdbMHhlZV0g CiAgMSBTQ0JfQ09OVFJPTFsweDBdIFNDQl9TQ1NJSURbMHhmZl06KFRXSU5fQ0hOTEJ8T0lEfFRX SU5fVElEKSAKU0NCX0xVTlsweGZmXTooU0NCX1hGRVJMRU5fT0REfExJRCkgU0NCX1RBR1sweGZm XSAKICAyIFNDQl9DT05UUk9MWzB4MF0gU0NCX1NDU0lJRFsweGZmXTooVFdJTl9DSE5MQnxPSUR8 VFdJTl9USUQpIApTQ0JfTFVOWzB4ZmZdOihTQ0JfWEZFUkxFTl9PRER8TElEKSBTQ0JfVEFHWzB4 ZmZdIAogIDMgU0NCX0NPTlRST0xbMHgwXSBTQ0JfU0NTSUlEWzB4ZmZdOihUV0lOX0NITkxCfE9J RHxUV0lOX1RJRCkgClNDQl9MVU5bMHhmZl06KFNDQl9YRkVSTEVOX09ERHxMSUQpIFNDQl9UQUdb MHhmZl0gCiAgNCBTQ0JfQ09OVFJPTFsweDBdIFNDQl9TQ1NJSURbMHhmZl06KFRXSU5fQ0hOTEJ8 T0lEfFRXSU5fVElEKSAKU0NCX0xVTlsweGZmXTooU0NCX1hGRVJMRU5fT0REfExJRCkgU0NCX1RB R1sweGZmXSAKICA1IFNDQl9DT05UUk9MWzB4MF0gU0NCX1NDU0lJRFsweGZmXTooVFdJTl9DSE5M QnxPSUR8VFdJTl9USUQpIApTQ0JfTFVOWzB4ZmZdOihTQ0JfWEZFUkxFTl9PRER8TElEKSBTQ0Jf VEFHWzB4ZmZdIAogIDYgU0NCX0NPTlRST0xbMHgwXSBTQ0JfU0NTSUlEWzB4ZmZdOihUV0lOX0NI TkxCfE9JRHxUV0lOX1RJRCkgClNDQl9MVU5bMHhmZl06KFNDQl9YRkVSTEVOX09ERHxMSUQpIFND Ql9UQUdbMHhmZl0gCiAgNyBTQ0JfQ09OVFJPTFsweDBdIFNDQl9TQ1NJSURbMHhmZl06KFRXSU5f Q0hOTEJ8T0lEfFRXSU5fVElEKSAKU0NCX0xVTlsweGZmXTooU0NCX1hGRVJMRU5fT0REfExJRCkg U0NCX1RBR1sweGZmXSAKICA4IFNDQl9DT05UUk9MWzB4MF0gU0NCX1NDU0lJRFsweGZmXTooVFdJ Tl9DSE5MQnxPSUR8VFdJTl9USUQpIApTQ0JfTFVOWzB4ZmZdOihTQ0JfWEZFUkxFTl9PRER8TElE KSBTQ0JfVEFHWzB4ZmZdIAogIDkgU0NCX0NPTlRST0xbMHgwXSBTQ0JfU0NTSUlEWzB4ZmZdOihU V0lOX0NITkxCfE9JRHxUV0lOX1RJRCkgClNDQl9MVU5bMHhmZl06KFNDQl9YRkVSTEVOX09ERHxM SUQpIFNDQl9UQUdbMHhmZl0gCiAxMCBTQ0JfQ09OVFJPTFsweDBdIFNDQl9TQ1NJSURbMHhmZl06 KFRXSU5fQ0hOTEJ8T0lEfFRXSU5fVElEKSAKU0NCX0xVTlsweGZmXTooU0NCX1hGRVJMRU5fT0RE fExJRCkgU0NCX1RBR1sweGZmXSAKIDExIFNDQl9DT05UUk9MWzB4MF0gU0NCX1NDU0lJRFsweGZm XTooVFdJTl9DSE5MQnxPSUR8VFdJTl9USUQpIApTQ0JfTFVOWzB4ZmZdOihTQ0JfWEZFUkxFTl9P RER8TElEKSBTQ0JfVEFHWzB4ZmZdIAogMTIgU0NCX0NPTlRST0xbMHgwXSBTQ0JfU0NTSUlEWzB4 ZmZdOihUV0lOX0NITkxCfE9JRHxUV0lOX1RJRCkgClNDQl9MVU5bMHhmZl06KFNDQl9YRkVSTEVO X09ERHxMSUQpIFNDQl9UQUdbMHhmZl0gCiAxMyBTQ0JfQ09OVFJPTFsweDBdIFNDQl9TQ1NJSURb MHhmZl06KFRXSU5fQ0hOTEJ8T0lEfFRXSU5fVElEKSAKU0NCX0xVTlsweGZmXTooU0NCX1hGRVJM RU5fT0REfExJRCkgU0NCX1RBR1sweGZmXSAKIDE0IFNDQl9DT05UUk9MWzB4MF0gU0NCX1NDU0lJ RFsweGZmXTooVFdJTl9DSE5MQnxPSUR8VFdJTl9USUQpIApTQ0JfTFVOWzB4ZmZdOihTQ0JfWEZF UkxFTl9PRER8TElEKSBTQ0JfVEFHWzB4ZmZdIAogMTUgU0NCX0NPTlRST0xbMHgwXSBTQ0JfU0NT SUlEWzB4ZmZdOihUV0lOX0NITkxCfE9JRHxUV0lOX1RJRCkgClNDQl9MVU5bMHhmZl06KFNDQl9Y RkVSTEVOX09ERHxMSUQpIFNDQl9UQUdbMHhmZl0gCiAxNiBTQ0JfQ09OVFJPTFsweDBdIFNDQl9T Q1NJSURbMHhmZl06KFRXSU5fQ0hOTEJ8T0lEfFRXSU5fVElEKSAKU0NCX0xVTlsweGZmXTooU0NC X1hGRVJMRU5fT0REfExJRCkgU0NCX1RBR1sweGZmXSAKIDE3IFNDQl9DT05UUk9MWzB4MF0gU0NC X1NDU0lJRFsweGZmXTooVFdJTl9DSE5MQnxPSUR8VFdJTl9USUQpIApTQ0JfTFVOWzB4ZmZdOihT Q0JfWEZFUkxFTl9PRER8TElEKSBTQ0JfVEFHWzB4ZmZdIAogMTggU0NCX0NPTlRST0xbMHgwXSBT Q0JfU0NTSUlEWzB4ZmZdOihUV0lOX0NITkxCfE9JRHxUV0lOX1RJRCkgClNDQl9MVU5bMHhmZl06 KFNDQl9YRkVSTEVOX09ERHxMSUQpIFNDQl9UQUdbMHhmZl0gCiAxOSBTQ0JfQ09OVFJPTFsweDBd IFNDQl9TQ1NJSURbMHhmZl06KFRXSU5fQ0hOTEJ8T0lEfFRXSU5fVElEKSAKU0NCX0xVTlsweGZm XTooU0NCX1hGRVJMRU5fT0REfExJRCkgU0NCX1RBR1sweGZmXSAKIDIwIFNDQl9DT05UUk9MWzB4 MF0gU0NCX1NDU0lJRFsweGZmXTooVFdJTl9DSE5MQnxPSUR8VFdJTl9USUQpIApTQ0JfTFVOWzB4 ZmZdOihTQ0JfWEZFUkxFTl9PRER8TElEKSBTQ0JfVEFHWzB4ZmZdIAogMjEgU0NCX0NPTlRST0xb MHgwXSBTQ0JfU0NTSUlEWzB4ZmZdOihUV0lOX0NITkxCfE9JRHxUV0lOX1RJRCkgClNDQl9MVU5b MHhmZl06KFNDQl9YRkVSTEVOX09ERHxMSUQpIFNDQl9UQUdbMHhmZl0gCiAyMiBTQ0JfQ09OVFJP TFsweDBdIFNDQl9TQ1NJSURbMHhmZl06KFRXSU5fQ0hOTEJ8T0lEfFRXSU5fVElEKSAKU0NCX0xV TlsweGZmXTooU0NCX1hGRVJMRU5fT0REfExJRCkgU0NCX1RBR1sweGZmXSAKIDIzIFNDQl9DT05U Uk9MWzB4MF0gU0NCX1NDU0lJRFsweGZmXTooVFdJTl9DSE5MQnxPSUR8VFdJTl9USUQpIApTQ0Jf TFVOWzB4ZmZdOihTQ0JfWEZFUkxFTl9PRER8TElEKSBTQ0JfVEFHWzB4ZmZdIAogMjQgU0NCX0NP TlRST0xbMHgwXSBTQ0JfU0NTSUlEWzB4ZmZdOihUV0lOX0NITkxCfE9JRHxUV0lOX1RJRCkgClND Ql9MVU5bMHhmZl06KFNDQl9YRkVSTEVOX09ERHxMSUQpIFNDQl9UQUdbMHhmZl0gCiAyNSBTQ0Jf Q09OVFJPTFsweDBdIFNDQl9TQ1NJSURbMHhmZl06KFRXSU5fQ0hOTEJ8T0lEfFRXSU5fVElEKSAK U0NCX0xVTlsweGZmXTooU0NCX1hGRVJMRU5fT0REfExJRCkgU0NCX1RBR1sweGZmXSAKIDI2IFND Ql9DT05UUk9MWzB4MF0gU0NCX1NDU0lJRFsweGZmXTooVFdJTl9DSE5MQnxPSUR8VFdJTl9USUQp IApTQ0JfTFVOWzB4ZmZdOihTQ0JfWEZFUkxFTl9PRER8TElEKSBTQ0JfVEFHWzB4ZmZdIAogMjcg U0NCX0NPTlRST0xbMHgwXSBTQ0JfU0NTSUlEWzB4ZmZdOihUV0lOX0NITkxCfE9JRHxUV0lOX1RJ RCkgClNDQl9MVU5bMHhmZl06KFNDQl9YRkVSTEVOX09ERHxMSUQpIFNDQl9UQUdbMHhmZl0gCiAy OCBTQ0JfQ09OVFJPTFsweDBdIFNDQl9TQ1NJSURbMHhmZl06KFRXSU5fQ0hOTEJ8T0lEfFRXSU5f VElEKSAKU0NCX0xVTlsweGZmXTooU0NCX1hGRVJMRU5fT0REfExJRCkgU0NCX1RBR1sweGZmXSAK IDI5IFNDQl9DT05UUk9MWzB4MF0gU0NCX1NDU0lJRFsweGZmXTooVFdJTl9DSE5MQnxPSUR8VFdJ Tl9USUQpIApTQ0JfTFVOWzB4ZmZdOihTQ0JfWEZFUkxFTl9PRER8TElEKSBTQ0JfVEFHWzB4ZmZd IAogMzAgU0NCX0NPTlRST0xbMHgwXSBTQ0JfU0NTSUlEWzB4ZmZdOihUV0lOX0NITkxCfE9JRHxU V0lOX1RJRCkgClNDQl9MVU5bMHhmZl06KFNDQl9YRkVSTEVOX09ERHxMSUQpIFNDQl9UQUdbMHhm Zl0gCiAzMSBTQ0JfQ09OVFJPTFsweDBdIFNDQl9TQ1NJSURbMHhmZl06KFRXSU5fQ0hOTEJ8T0lE fFRXSU5fVElEKSAKU0NCX0xVTlsweGZmXTooU0NCX1hGRVJMRU5fT0REfExJRCkgU0NCX1RBR1sw eGZmXSAKUGVuZGluZyBsaXN0OiAKMjM4IFNDQl9DT05UUk9MWzB4MF0gU0NCX1NDU0lJRFsweDI3 XSBTQ0JfTFVOWzB4MF0gCktlcm5lbCBGcmVlIFNDQiBsaXN0OiAyMzkgMjQwIDI0MSAyNDIgMjQz IDI0NCAyNDUgMjQ2IDI0NyAyNDggMjQ5IDI1MCAyNTIgMjUzIDIzNyAyMzYgMjM1IDIzNCAyMzMg MjMyIDIzMSAyMzAgMjI5IDIyOCAyMjcgMjI2IDIyNSAyMjQgMjIzIDIyMiAyMjEgMjIwIDIxOSAy MTggMjE3IDIxNiAyMTUgMjE0IDIxMyAyMTIgMjExIDIxMCAyMDkgMjA4IDIwNyAyMDYgMjA1IDIw NCAyMDMgMjAyIDIwMSAyMDAgMTk5IDE5OCAxOTcgMTk2IDE5NSAxOTQgMTkzIDE5MiAxOTEgMTkw IDE4OSAxODggMTg3IDE4NiAxODUgMTg0IDE4MyAxODIgMTgxIDE4MCAxNzkgMTc4IDE3NyAxNzYg MTc1IDE3NCAxNzMgMTcyIDE3MSAxNzAgMTY5IDE2OCAxNjcgMTY2IDE2NSAxNjQgMTYzIDE2MiAx NjEgMTYwIDE1OSAxNTggMTU3IDE1NiAxNTUgMTU0IDE1MyAxNTIgMTUxIDE1MCAxNDkgMTQ4IDE0 NyAxNDYgMTQ1IDE0NCAxNDMgMTQyIDE0MSAxNDAgMTM5IDEzOCAxMzcgMTM2IDEzNSAxMzQgMTMz IDEzMiAxMzEgMTMwIDEyOSAxMjggMTI3IDEyNiAxMjUgMTI0IDEyMyAxMjIgMTIxIDEyMCAxMTkg MTE4IDExNyAxMTYgMTE1IDExNCAxMTMgMTEyIDExMSAxMTAgMTA5IDEwOCAxMDcgMTA2IDEwNSAx MDQgMTAzIDEwMiAxMDEgMTAwIDk5IDk4IDk3IDk2IDk1IDk0IDkzIDkyIDkxIDkwIDg5IDg4IDg3 IDg2IDg1IDg0IDgzIDgyIDgxIDgwIDc5IDc4IDc3IDc2IDc1IDc0IDczIDcyIDcxIDcwIDY5IDY4 IDY3IDY2IDY1IDY0IDYzIDYyIDYxIDYwIDU5IDU4IDU3IDU2IDU1IDU0IDUzIDUyIDUxIDUwIDQ5 IDQ4IDQ3IDQ2IDQ1IDQ0IDQzIDQyIDQxIDQwIDM5IDM4IDM3IDM2IDM1IDM0IDMzIDMyIDMxIDMw IDI5IDI4IDI3IDI2IDI1IDI0IDIzIDIyIDIxIDIwIDE5IDE4IDE3IDE2IDE1IDE0IDEzIDEyIDEx IDEwIDkgOCA3IDYgNSA0IDMgMiAxIDAgClVudGFnZ2VkIFEoMik6IDIzOCAKCjw8PDw8PDw8PDw8 PDw8PDw8IER1bXAgQ2FyZCBTdGF0ZSBFbmRzID4+Pj4+Pj4+Pj4+Pj4+Pj4+Pgooc2EwOmFoYzA6 MDoyOjApOiBTQ0IgMHhlZSAtIHRpbWVkIG91dApzZ1swXSAtIEFkZHIgMHgxNjdmMTAyOCA6IExl bmd0aCA0MDU2CnNnWzFdIC0gQWRkciAweDZkNTEwMDAgOiBMZW5ndGggNDA5NgpzZ1syXSAtIEFk ZHIgMHhjMjEyMDAwIDogTGVuZ3RoIDgxOTIKc2dbM10gLSBBZGRyIDB4MWFiMGUwMDAgOiBMZW5n dGggODE5MgpzZ1s0XSAtIEFkZHIgMHhmMGIwMDAwIDogTGVuZ3RoIDgxOTIKc2dbNV0gLSBBZGRy IDB4MTdmOGUwMDAgOiBMZW5ndGggODE5MgpzZ1s2XSAtIEFkZHIgMHhjZGUyMDAwIDogTGVuZ3Ro IDgxOTIKc2dbN10gLSBBZGRyIDB4MTMwNjIwMDAgOiBMZW5ndGggODE5MgpzZ1s4XSAtIEFkZHIg MHgxMDI0YzAwMCA6IExlbmd0aCA0MDk2CnNnWzldIC0gQWRkciAweDZkNTAwMDAgOiBMZW5ndGgg MzExMgooc2EwOmFoYzA6MDoyOjApOiBCRFIgbWVzc2FnZSBpbiBtZXNzYWdlIGJ1ZmZlcgphaGMw OiBUaW1lZG91dCBTQ0JzIGFscmVhZHkgY29tcGxldGUuIEludGVycnVwdHMgbWF5IG5vdCBiZSBm dW5jdGlvbmluZy4KYWhjMDogUmVjb3ZlcnkgSW5pdGlhdGVkCj4+Pj4+Pj4+Pj4+Pj4+Pj4+PiBE dW1wIENhcmQgU3RhdGUgQmVnaW5zIDw8PDw8PDw8PDw8PDw8PDw8CmFoYzA6IER1bXBpbmcgQ2Fy ZCBTdGF0ZSBpbiBDb21tYW5kIHBoYXNlLCBhdCBTRVFBRERSIDB4MTZjCkNhcmQgd2FzIHBhdXNl ZApBQ0NVTSA9IDB4ODAsIFNJTkRFWCA9IDB4YTAsIERJTkRFWCA9IDB4ZTQsIEFSR18yID0gMHgz YgpIQ05UID0gMHgwIFNDQlBUUiA9IDB4MApTQ1NJUEhBU0VbMHgwXSBTQ1NJU0lHSVsweDk0XToo QlNZSXxBVE5JfENESSkgRVJST1JbMHgwXSAKU0NTSUJVU0xbMHg4MF0gTEFTVFBIQVNFWzB4ODBd OihDREkpIFNDU0lTRVFbMHgxMl06KEVOQVVUT0FUTlB8RU5SU0VMSSkgClNCTEtDVExbMHhhXToo U0VMV0lERXxTRUxCVVNCKSBTQ1NJUkFURVsweGMyXTooRU5BQkxFX0NSQ3xXSURFWEZFUikgClNF UUNUTFsweDEwXTooRkFTVE1PREUpIFNFUV9GTEFHU1sweDBdIFNTVEFUMFsweDddOihETUFET05F fFNQSU9SRFl8U0RPTkUpIApTU1RBVDFbMHgwXSBTU1RBVDJbMHgwXSBTU1RBVDNbMHgwXSBTSU1P REUwWzB4OF06KEVOU1dSQVApIApTSU1PREUxWzB4YWNdOihFTlNDU0lQRVJSfEVOQlVTRlJFRXxF TlNDU0lSU1R8RU5TRUxUSU1PKSAKU1hGUkNUTDBbMHg4OF06KFNQSU9FTnxERk9OKSBERkNOVFJM WzB4NF06KERJUkVDVElPTikgCkRGU1RBVFVTWzB4ODldOihGSUZPRU1QfEhET05FfFBSRUxPQURf QVZBSUwpIApTVEFDSzogMHgzNCAweDAgMHgxNjQgMHgxNzkKU0NCIGNvdW50ID0gMjU0Cktlcm5l bCBORVhUUVNDQiA9IDI1MQpDYXJkIE5FWFRRU0NCID0gMjUxClFJTkZJRk8gZW50cmllczogCldh aXRpbmcgUXVldWUgZW50cmllczogCkRpc2Nvbm5lY3RlZCBRdWV1ZSBlbnRyaWVzOiAKUU9VVEZJ Rk8gZW50cmllczogClNlcXVlbmNlciBGcmVlIFNDQiBMaXN0OiAxIDIgMyA0IDUgNiA3IDggOSAx MCAxMSAxMiAxMyAxNCAxNSAxNiAxNyAxOCAxOSAyMCAyMSAyMiAyMyAyNCAyNSAyNiAyNyAyOCAy OSAzMCAzMSAKU2VxdWVuY2VyIFNDQiBJbmZvOiAKICAwIFNDQl9DT05UUk9MWzB4MF0gU0NCX1ND U0lJRFsweDI3XSBTQ0JfTFVOWzB4MF0gU0NCX1RBR1sweGVlXSAKICAxIFNDQl9DT05UUk9MWzB4 MF0gU0NCX1NDU0lJRFsweGZmXTooVFdJTl9DSE5MQnxPSUR8VFdJTl9USUQpIApTQ0JfTFVOWzB4 ZmZdOihTQ0JfWEZFUkxFTl9PRER8TElEKSBTQ0JfVEFHWzB4ZmZdIAogIDIgU0NCX0NPTlRST0xb MHgwXSBTQ0JfU0NTSUlEWzB4ZmZdOihUV0lOX0NITkxCfE9JRHxUV0lOX1RJRCkgClNDQl9MVU5b MHhmZl06KFNDQl9YRkVSTEVOX09ERHxMSUQpIFNDQl9UQUdbMHhmZl0gCiAgMyBTQ0JfQ09OVFJP TFsweDBdIFNDQl9TQ1NJSURbMHhmZl06KFRXSU5fQ0hOTEJ8T0lEfFRXSU5fVElEKSAKU0NCX0xV TlsweGZmXTooU0NCX1hGRVJMRU5fT0REfExJRCkgU0NCX1RBR1sweGZmXSAKICA0IFNDQl9DT05U Uk9MWzB4MF0gU0NCX1NDU0lJRFsweGZmXTooVFdJTl9DSE5MQnxPSUR8VFdJTl9USUQpIApTQ0Jf TFVOWzB4ZmZdOihTQ0JfWEZFUkxFTl9PRER8TElEKSBTQ0JfVEFHWzB4ZmZdIAogIDUgU0NCX0NP TlRST0xbMHgwXSBTQ0JfU0NTSUlEWzB4ZmZdOihUV0lOX0NITkxCfE9JRHxUV0lOX1RJRCkgClND Ql9MVU5bMHhmZl06KFNDQl9YRkVSTEVOX09ERHxMSUQpIFNDQl9UQUdbMHhmZl0gCiAgNiBTQ0Jf Q09OVFJPTFsweDBdIFNDQl9TQ1NJSURbMHhmZl06KFRXSU5fQ0hOTEJ8T0lEfFRXSU5fVElEKSAK U0NCX0xVTlsweGZmXTooU0NCX1hGRVJMRU5fT0REfExJRCkgU0NCX1RBR1sweGZmXSAKICA3IFND Ql9DT05UUk9MWzB4MF0gU0NCX1NDU0lJRFsweGZmXTooVFdJTl9DSE5MQnxPSUR8VFdJTl9USUQp IApTQ0JfTFVOWzB4ZmZdOihTQ0JfWEZFUkxFTl9PRER8TElEKSBTQ0JfVEFHWzB4ZmZdIAogIDgg U0NCX0NPTlRST0xbMHgwXSBTQ0JfU0NTSUlEWzB4ZmZdOihUV0lOX0NITkxCfE9JRHxUV0lOX1RJ RCkgClNDQl9MVU5bMHhmZl06KFNDQl9YRkVSTEVOX09ERHxMSUQpIFNDQl9UQUdbMHhmZl0gCiAg OSBTQ0JfQ09OVFJPTFsweDBdIFNDQl9TQ1NJSURbMHhmZl06KFRXSU5fQ0hOTEJ8T0lEfFRXSU5f VElEKSAKU0NCX0xVTlsweGZmXTooU0NCX1hGRVJMRU5fT0REfExJRCkgU0NCX1RBR1sweGZmXSAK IDEwIFNDQl9DT05UUk9MWzB4MF0gU0NCX1NDU0lJRFsweGZmXTooVFdJTl9DSE5MQnxPSUR8VFdJ Tl9USUQpIApTQ0JfTFVOWzB4ZmZdOihTQ0JfWEZFUkxFTl9PRER8TElEKSBTQ0JfVEFHWzB4ZmZd IAogMTEgU0NCX0NPTlRST0xbMHgwXSBTQ0JfU0NTSUlEWzB4ZmZdOihUV0lOX0NITkxCfE9JRHxU V0lOX1RJRCkgClNDQl9MVU5bMHhmZl06KFNDQl9YRkVSTEVOX09ERHxMSUQpIFNDQl9UQUdbMHhm Zl0gCiAxMiBTQ0JfQ09OVFJPTFsweDBdIFNDQl9TQ1NJSURbMHhmZl06KFRXSU5fQ0hOTEJ8T0lE fFRXSU5fVElEKSAKU0NCX0xVTlsweGZmXTooU0NCX1hGRVJMRU5fT0REfExJRCkgU0NCX1RBR1sw eGZmXSAKIDEzIFNDQl9DT05UUk9MWzB4MF0gU0NCX1NDU0lJRFsweGZmXTooVFdJTl9DSE5MQnxP SUR8VFdJTl9USUQpIApTQ0JfTFVOWzB4ZmZdOihTQ0JfWEZFUkxFTl9PRER8TElEKSBTQ0JfVEFH WzB4ZmZdIAogMTQgU0NCX0NPTlRST0xbMHgwXSBTQ0JfU0NTSUlEWzB4ZmZdOihUV0lOX0NITkxC fE9JRHxUV0lOX1RJRCkgClNDQl9MVU5bMHhmZl06KFNDQl9YRkVSTEVOX09ERHxMSUQpIFNDQl9U QUdbMHhmZl0gCiAxNSBTQ0JfQ09OVFJPTFsweDBdIFNDQl9TQ1NJSURbMHhmZl06KFRXSU5fQ0hO TEJ8T0lEfFRXSU5fVElEKSAKU0NCX0xVTlsweGZmXTooU0NCX1hGRVJMRU5fT0REfExJRCkgU0NC X1RBR1sweGZmXSAKIDE2IFNDQl9DT05UUk9MWzB4MF0gU0NCX1NDU0lJRFsweGZmXTooVFdJTl9D SE5MQnxPSUR8VFdJTl9USUQpIApTQ0JfTFVOWzB4ZmZdOihTQ0JfWEZFUkxFTl9PRER8TElEKSBT Q0JfVEFHWzB4ZmZdIAogMTcgU0NCX0NPTlRST0xbMHgwXSBTQ0JfU0NTSUlEWzB4ZmZdOihUV0lO X0NITkxCfE9JRHxUV0lOX1RJRCkgClNDQl9MVU5bMHhmZl06KFNDQl9YRkVSTEVOX09ERHxMSUQp IFNDQl9UQUdbMHhmZl0gCiAxOCBTQ0JfQ09OVFJPTFsweDBdIFNDQl9TQ1NJSURbMHhmZl06KFRX SU5fQ0hOTEJ8T0lEfFRXSU5fVElEKSAKU0NCX0xVTlsweGZmXTooU0NCX1hGRVJMRU5fT0REfExJ RCkgU0NCX1RBR1sweGZmXSAKIDE5IFNDQl9DT05UUk9MWzB4MF0gU0NCX1NDU0lJRFsweGZmXToo VFdJTl9DSE5MQnxPSUR8VFdJTl9USUQpIApTQ0JfTFVOWzB4ZmZdOihTQ0JfWEZFUkxFTl9PRER8 TElEKSBTQ0JfVEFHWzB4ZmZdIAogMjAgU0NCX0NPTlRST0xbMHgwXSBTQ0JfU0NTSUlEWzB4ZmZd OihUV0lOX0NITkxCfE9JRHxUV0lOX1RJRCkgClNDQl9MVU5bMHhmZl06KFNDQl9YRkVSTEVOX09E RHxMSUQpIFNDQl9UQUdbMHhmZl0gCiAyMSBTQ0JfQ09OVFJPTFsweDBdIFNDQl9TQ1NJSURbMHhm Zl06KFRXSU5fQ0hOTEJ8T0lEfFRXSU5fVElEKSAKU0NCX0xVTlsweGZmXTooU0NCX1hGRVJMRU5f T0REfExJRCkgU0NCX1RBR1sweGZmXSAKIDIyIFNDQl9DT05UUk9MWzB4MF0gU0NCX1NDU0lJRFsw eGZmXTooVFdJTl9DSE5MQnxPSUR8VFdJTl9USUQpIApTQ0JfTFVOWzB4ZmZdOihTQ0JfWEZFUkxF Tl9PRER8TElEKSBTQ0JfVEFHWzB4ZmZdIAogMjMgU0NCX0NPTlRST0xbMHgwXSBTQ0JfU0NTSUlE WzB4ZmZdOihUV0lOX0NITkxCfE9JRHxUV0lOX1RJRCkgClNDQl9MVU5bMHhmZl06KFNDQl9YRkVS TEVOX09ERHxMSUQpIFNDQl9UQUdbMHhmZl0gCiAyNCBTQ0JfQ09OVFJPTFsweDBdIFNDQl9TQ1NJ SURbMHhmZl06KFRXSU5fQ0hOTEJ8T0lEfFRXSU5fVElEKSAKU0NCX0xVTlsweGZmXTooU0NCX1hG RVJMRU5fT0REfExJRCkgU0NCX1RBR1sweGZmXSAKIDI1IFNDQl9DT05UUk9MWzB4MF0gU0NCX1ND U0lJRFsweGZmXTooVFdJTl9DSE5MQnxPSUR8VFdJTl9USUQpIApTQ0JfTFVOWzB4ZmZdOihTQ0Jf WEZFUkxFTl9PRER8TElEKSBTQ0JfVEFHWzB4ZmZdIAogMjYgU0NCX0NPTlRST0xbMHgwXSBTQ0Jf U0NTSUlEWzB4ZmZdOihUV0lOX0NITkxCfE9JRHxUV0lOX1RJRCkgClNDQl9MVU5bMHhmZl06KFND Ql9YRkVSTEVOX09ERHxMSUQpIFNDQl9UQUdbMHhmZl0gCiAyNyBTQ0JfQ09OVFJPTFsweDBdIFND Ql9TQ1NJSURbMHhmZl06KFRXSU5fQ0hOTEJ8T0lEfFRXSU5fVElEKSAKU0NCX0xVTlsweGZmXToo U0NCX1hGRVJMRU5fT0REfExJRCkgU0NCX1RBR1sweGZmXSAKIDI4IFNDQl9DT05UUk9MWzB4MF0g U0NCX1NDU0lJRFsweGZmXTooVFdJTl9DSE5MQnxPSUR8VFdJTl9USUQpIApTQ0JfTFVOWzB4ZmZd OihTQ0JfWEZFUkxFTl9PRER8TElEKSBTQ0JfVEFHWzB4ZmZdIAogMjkgU0NCX0NPTlRST0xbMHgw XSBTQ0JfU0NTSUlEWzB4ZmZdOihUV0lOX0NITkxCfE9JRHxUV0lOX1RJRCkgClNDQl9MVU5bMHhm Zl06KFNDQl9YRkVSTEVOX09ERHxMSUQpIFNDQl9UQUdbMHhmZl0gCiAzMCBTQ0JfQ09OVFJPTFsw eDBdIFNDQl9TQ1NJSURbMHhmZl06KFRXSU5fQ0hOTEJ8T0lEfFRXSU5fVElEKSAKU0NCX0xVTlsw eGZmXTooU0NCX1hGRVJMRU5fT0REfExJRCkgU0NCX1RBR1sweGZmXSAKIDMxIFNDQl9DT05UUk9M WzB4MF0gU0NCX1NDU0lJRFsweGZmXTooVFdJTl9DSE5MQnxPSUR8VFdJTl9USUQpIApTQ0JfTFVO WzB4ZmZdOihTQ0JfWEZFUkxFTl9PRER8TElEKSBTQ0JfVEFHWzB4ZmZdIApQZW5kaW5nIGxpc3Q6 IAoyMzggU0NCX0NPTlRST0xbMHgwXSBTQ0JfU0NTSUlEWzB4MjddIFNDQl9MVU5bMHgwXSAKS2Vy bmVsIEZyZWUgU0NCIGxpc3Q6IDIzOSAyNDAgMjQxIDI0MiAyNDMgMjQ0IDI0NSAyNDYgMjQ3IDI0 OCAyNDkgMjUwIDI1MiAyNTMgMjM3IDIzNiAyMzUgMjM0IDIzMyAyMzIgMjMxIDIzMCAyMjkgMjI4 IDIyNyAyMjYgMjI1IDIyNCAyMjMgMjIyIDIyMSAyMjAgMjE5IDIxOCAyMTcgMjE2IDIxNSAyMTQg MjEzIDIxMiAyMTEgMjEwIDIwOSAyMDggMjA3IDIwNiAyMDUgMjA0IDIwMyAyMDIgMjAxIDIwMCAx OTkgMTk4IDE5NyAxOTYgMTk1IDE5NCAxOTMgMTkyIDE5MSAxOTAgMTg5IDE4OCAxODcgMTg2IDE4 NSAxODQgMTgzIDE4MiAxODEgMTgwIDE3OSAxNzggMTc3IDE3NiAxNzUgMTc0IDE3MyAxNzIgMTcx IDE3MCAxNjkgMTY4IDE2NyAxNjYgMTY1IDE2NCAxNjMgMTYyIDE2MSAxNjAgMTU5IDE1OCAxNTcg MTU2IDE1NSAxNTQgMTUzIDE1MiAxNTEgMTUwIDE0OSAxNDggMTQ3IDE0NiAxNDUgMTQ0IDE0MyAx NDIgMTQxIDE0MCAxMzkgMTM4IDEzNyAxMzYgMTM1IDEzNCAxMzMgMTMyIDEzMSAxMzAgMTI5IDEy OCAxMjcgMTI2IDEyNSAxMjQgMTIzIDEyMiAxMjEgMTIwIDExOSAxMTggMTE3IDExNiAxMTUgMTE0 IDExMyAxMTIgMTExIDExMCAxMDkgMTA4IDEwNyAxMDYgMTA1IDEwNCAxMDMgMTAyIDEwMSAxMDAg OTkgOTggOTcgOTYgOTUgOTQgOTMgOTIgOTEgOTAgODkgODggODcgODYgODUgODQgODMgODIgODEg ODAgNzkgNzggNzcgNzYgNzUgNzQgNzMgNzIgNzEgNzAgNjkgNjggNjcgNjYgNjUgNjQgNjMgNjIg NjEgNjAgNTkgNTggNTcgNTYgNTUgNTQgNTMgNTIgNTEgNTAgNDkgNDggNDcgNDYgNDUgNDQgNDMg NDIgNDEgNDAgMzkgMzggMzcgMzYgMzUgMzQgMzMgMzIgMzEgMzAgMjkgMjggMjcgMjYgMjUgMjQg MjMgMjIgMjEgMjAgMTkgMTggMTcgMTYgMTUgMTQgMTMgMTIgMTEgMTAgOSA4IDcgNiA1IDQgMyAy IDEgMCAKVW50YWdnZWQgUSgyKTogMjM4IAoKPDw8PDw8PDw8PDw8PDw8PDwgRHVtcCBDYXJkIFN0 YXRlIEVuZHMgPj4+Pj4+Pj4+Pj4+Pj4+Pj4+CihzYTA6YWhjMDowOjI6MCk6IFNDQiAweGVlIC0g dGltZWQgb3V0CnNnWzBdIC0gQWRkciAweDE2N2YxMDI4IDogTGVuZ3RoIDQwNTYKc2dbMV0gLSBB ZGRyIDB4NmQ1MTAwMCA6IExlbmd0aCA0MDk2CnNnWzJdIC0gQWRkciAweGMyMTIwMDAgOiBMZW5n dGggODE5MgpzZ1szXSAtIEFkZHIgMHgxYWIwZTAwMCA6IExlbmd0aCA4MTkyCnNnWzRdIC0gQWRk ciAweGYwYjAwMDAgOiBMZW5ndGggODE5MgpzZ1s1XSAtIEFkZHIgMHgxN2Y4ZTAwMCA6IExlbmd0 aCA4MTkyCnNnWzZdIC0gQWRkciAweGNkZTIwMDAgOiBMZW5ndGggODE5MgpzZ1s3XSAtIEFkZHIg MHgxMzA2MjAwMCA6IExlbmd0aCA4MTkyCnNnWzhdIC0gQWRkciAweDEwMjRjMDAwIDogTGVuZ3Ro IDQwOTYKc2dbOV0gLSBBZGRyIDB4NmQ1MDAwMCA6IExlbmd0aCAzMTEyCihzYTA6YWhjMDowOjI6 MCk6IG5vIGxvbmdlciBpbiB0aW1lb3V0LCBzdGF0dXMgPSAyNGIKYWhjMDogSXNzdWVkIENoYW5u ZWwgQSBCdXMgUmVzZXQuIDEgU0NCcyBhYm9ydGVkCihzYTA6YWhjMDowOjI6MCk6IFdSSVRFKDYp LiBDREI6IDBhIDAwIDAwIGZjIDAwIDAwIAooc2EwOmFoYzA6MDoyOjApOiBDQU0gc3RhdHVzOiBD b21tYW5kIHRpbWVvdXQKKHNhMDphaGMwOjA6MjowKTogRXJyb3IgNSwgUmV0cmllcyBleGhhdXN0 ZWQKYWhjMDogVGltZWRvdXQgU0NCcyBhbHJlYWR5IGNvbXBsZXRlLiBJbnRlcnJ1cHRzIG1heSBu b3QgYmUgZnVuY3Rpb25pbmcuCihzYTA6YWhjMDowOjI6MCk6IE1PREUgU0VOU0UoNikuIENEQjog MWEgMDAgMGYgMDAgMWMgMDAgCihzYTA6YWhjMDowOjI6MCk6IFNDU0kgc2Vuc2U6IFVOSVQgQVRU RU5USU9OIGFzYzoyOSwwIChQb3dlciBvbiwgcmVzZXQsIG9yIGJ1cyBkZXZpY2UgcmVzZXQgb2Nj dXJyZWQpCihzYTA6YWhjMDowOjI6MCk6IEZpZWxkIFJlcGxhY2VhYmxlIFVuaXQ6IDQ4Cg== --001a11c3f2b431916e04ef5c7039-- From owner-freebsd-scsi@FreeBSD.ORG Tue Jan 7 05:52:42 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 02A7B992; Tue, 7 Jan 2014 05:52:42 +0000 (UTC) Received: from na01-by2-obe.outbound.protection.outlook.com (mail-by2lp0244.outbound.protection.outlook.com [207.46.163.244]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 9B1C81390; Tue, 7 Jan 2014 05:52:41 +0000 (UTC) Received: from BN1PR07MB247.namprd07.prod.outlook.com (10.141.64.146) by BN1PR07MB264.namprd07.prod.outlook.com (10.141.64.27) with Microsoft SMTP Server (TLS) id 15.0.842.7; Tue, 7 Jan 2014 05:37:18 +0000 Received: from BN1PR07MB247.namprd07.prod.outlook.com ([169.254.7.143]) by BN1PR07MB247.namprd07.prod.outlook.com ([169.254.7.143]) with mapi id 15.00.0842.003; Tue, 7 Jan 2014 05:37:18 +0000 From: "Desai, Kashyap" To: Doug Ambrisko Subject: RE: LSI - MR-Fusion controller driver patch and man page Thread-Topic: LSI - MR-Fusion controller driver patch and man page Thread-Index: Ac7ykKZzGPABTpK8R3+D5UAv3TD3WQWOC9SAAHVVybAAG8S5gAAWATMA Date: Tue, 7 Jan 2014 05:37:17 +0000 Message-ID: References: <08ba2a262fba45f687cdd3225f325110@BN1PR07MB247.namprd07.prod.outlook.com> <20140103211449.GA69721@cisco.com> <8c423414ecc2421fbace3eb9f386be91@BN1PR07MB247.namprd07.prod.outlook.com> <20140106182935.GC93278@cisco.com> In-Reply-To: <20140106182935.GC93278@cisco.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.19.239.250] x-forefront-prvs: 008421A8FF x-forefront-antispam-report: SFV:NSPM; SFS:(10019001)(199002)(189002)(69234005)(24454002)(53754006)(51704005)(164054003)(377454003)(13464003)(66066001)(76786001)(19580405001)(56776001)(31966008)(19580395003)(81542001)(69226001)(47736001)(51856001)(85306002)(74706001)(56816005)(81686001)(46102001)(76796001)(90146001)(33646001)(81342001)(50986001)(74316001)(80976001)(76576001)(76482001)(47976001)(4396001)(77096001)(54356001)(81816001)(74366001)(85852003)(83072002)(15202345003)(59766001)(53806001)(15975445006)(2656002)(74662001)(65816001)(80022001)(74876001)(47446002)(83322001)(74502001)(49866001)(87266001)(87936001)(63696002)(79102001)(54316002)(77982001)(24736002); DIR:OUT; SFP:1102; SCL:1; SRVR:BN1PR07MB264; H:BN1PR07MB247.namprd07.prod.outlook.com; CLIP:192.19.239.250; FPR:; RD:InfoNoRecords; A:1; MX:1; LANG:en; Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: lsi.com X-Mailman-Approved-At: Tue, 07 Jan 2014 12:50:59 +0000 Cc: "scottl@netflix.com" , "Radford, Adam" , "Kenneth D. Merry" , "sean_bruno@yahoo.com" , "Mankani, Krishnaraddi" , "dwhite@ixsystems.com" , "Maloy, Joe" , "jpaetzel@freebsd.org" , "freebsd-scsi@freebsd.org" , "McConnell, Stephen" X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Jan 2014 05:52:42 -0000 Doug: I have replied to your mail at very high level and not going for inline rep= ly to your respond to capture key issue and possible resolution.=20 Hi All, There are two biggest challenge for drivers. 1. conflict with for Thunderbolt, Invader and Fury controller= s.=20 As we discussed, we have resolve this giving priority to controller a= nd only when customer wants use they need proper hint support. Currently, there are customer who is using posted from LSI external= site and able to configure using storcli and driver is working well for th= eir need. Definitely, we are doing lots of enhancement and defect fixes in those area= s, but note that before we work on actual issue, lots of customer complain = that is not loaded due to and PnP ID conflict. Please= understand that, this is bit critical for customers who want to use . To this issue, we need svn commits in driver (from submitted patch s= ee only related changes) to understand device.hints at least, so that= customer can use driver from lsi external site. Anything related to / interopt can be considered a different de= sign goal and requirement. =20 E.a "We should be fine not have alias of /dev/daX to /dev/mfidX for initia= l drop. Probably we should document this constraint for initial commit of = mrsas" E.a "Please consider that customer is not using to configure wh= en driver is used."=20 Till now customer is not having any issue using with . = We may go for good amount of interopt development to support configuration = utils as , but that may need time. 2. is only available via LSI external site. We need upstre= am, so that customer should not wait for LSI external site downloads for mo= re generic use case. This can be done in different phases, as LSI is doing for driver. Th= ere are good comment posted by Doug and we are working on those. We do not wanted to change code without posting for internal Q/A testing. W= e should take advantage of LSI internal Q/A testing to fix most of the defe= ct internally. To resolve above issue, we should find middle way to post the check= -in in mainline. I mean, keep few know limitation as documented and do not= wait for gathering of all feature in single drop.=20 E.a If we start changing code to meet style(9) coding standard, we may add= some manual mistakes in code. We can take each action item as Enhancement= request (Including style(9) change as well) for LSI and we will be doing a= ll those step by step.( As we are doing for driver). I am not aware of project method to do svn check, since I worked with Ken D= Merry earlier to do actual commit in FreeBSD head repo. I thought it is go= ing to be same way for . Sorry, but doing lots of discussion in single mail thread is prone to diver= t actual agenda and we may hang forever without any conclusion. So, let's open defect in as soon as it get upstream, LSI will work = for all the defect against .=20 At least immediate check in diver to understand device.hints, will al= low customer to use from LSI external site. Thanks, Kashyap > -----Original Message----- > From: Doug Ambrisko [mailto:ambrisko@cisco.com] > Sent: Tuesday, January 07, 2014 12:00 AM > To: Desai, Kashyap > Cc: freebsd-scsi@freebsd.org; scottl@netflix.com; sean_bruno@yahoo.com; > dwhite@ixsystems.com; jpaetzel@freebsd.org; Maloy, Joe; Mankani, > Krishnaraddi; McConnell, Stephen; Saxena, Sumit; Radford, Adam; Kenneth > D. Merry > Subject: Re: LSI - MR-Fusion controller driver patch and man page >=20 > On Mon, Jan 06, 2014 at 07:43:27AM +0000, Desai, Kashyap wrote: > | Doug: > | > | I have replied inline for your comment. For most of the comment you > | posted below can be work out and we can prioritize completion of those > | based on nature of issue. > | I will work on READ/WRITE comparison w.r.t and as a > | priority item. >=20 > IO performance is the highest priority. >=20 > | Please consider that it will be really helpful for further > | development, if we have common code check-in in Upstream FreeBSD. We > | will actively work on each comment posted for and address > | those before we have in-box for any FreeBSD release. >=20 > What we have done in the past is to create a project branch to do this an= d it > has worked well. It would be nice if you could check in directly to a th= is > project branch so we could use svn to resolve differences. >=20 > I poked around at adding the ioctl compatibility support and noticed that > mrsas seems to lack a general method to send a dcmd to the controller. > What Scott did in mfi was to create a generic function to send dcmd's wit= h a > simple data buffer. That data buffer was then put into the sg list in th= e > generic function. This made it easier to translate the various ioctl's s= ince we > could convert the sg lists passed down from the user land into a simple > malloced data base and then covert back out. > In the mfituil ioctl method it just passes down the data buffer and then > let's the driver convert it to a sg list. To make ioctl emulation > easier in these cases: > 32 bit FreeBSD LSI ioctl on 64 bit > 32 bit Linux LSI ioctl on 32 bit > 32 bit Linux LSI ioctl on 64 bit > 32 bit mfiutil ioctl on 32 bit > 32 bit mfiutil ioctl on 64 bit >=20 > Of high priority is style, please make it style(9) compliant. There are = a trailing > spaces, spaces instead of tabs, I think lack of tab aligned indent. I no= ticed > functions being externally defined in .c files instead of .h. There shou= ld be .h > files that can be included in user land applications for ioctls and there= can be > .h files just for the kernel. > These files will need to be installed as appropriate. I found this with = the ioctl > function. >=20 > | > -----Original Message----- > | > From: Doug Ambrisko [mailto:ambrisko@cisco.com] > | > Sent: Saturday, January 04, 2014 2:45 AM > | > To: Desai, Kashyap > | > Cc: freebsd-scsi@freebsd.org; scottl@netflix.com; > | > sean_bruno@yahoo.com; dwhite@ixsystems.com; > jpaetzel@freebsd.org; > | > Maloy, Joe; Mankani, Krishnaraddi; McConnell, Stephen; Saxena, > | > Sumit; Radford, Adam; Kenneth D. Merry > | > Subject: Re: LSI - MR-Fusion controller driver patch and man > | > page > | > > | > On Fri, Dec 06, 2013 at 02:37:21PM +0000, Desai, Kashyap wrote: > | > | Hi, > | > | > | > | Please consider attached patch for FreeBSD upstream check-in. > | > | > | > | Please find attached patch for driver for LSI's MegaRaid > | > Controllers. This driver supports Thunderbolt onwards Device IDs of > | > MR controllers. > | > | Currently it supports 0x005B and 0x005D Device IDs. > | > | > | > | NOTE : This driver will not eliminate or by pass any functionality > | > | of > | > driver which already support above to Device IDs to keep existing > | > user experience unchanged. > | > | > | > | Driver will be always given priority over driver and > | > | only if customer/user wants to use/migrate from to , > | > | it will hook up into kernel via device.hint rules. (Attached is > | > | mrsas man page for more info.) > | > | > | > | LSI will continue to update driver in future in timely base= s. > | > | We have another set of patch in pipeline to be submitted for > | > | , but need first go ahead for attached patch and later we > | > | will continue to keep up-to-date (In sync with LSI > | > | released driver which is available at lsi's external site) > | > | > | > | Apply patch with "patch -p0 < patchname.patch" from head directory. > | > | > | > | -- Few notes for user-- > | > | LSI recommends using fusion_force bit In hint settings at start of > | > | the day, if > | > they want to use . ( will be a default choice for > | > MR-Fusion HBA), if will be changed only with fusion_force hint > | > settings. (See mrsas man > | > page) Changing any default behavior is well tested for most of the > condition. > | > | Switching from to for MR-Fusion options is designed > | > | to > | > allow user as one time choice, though multiple time switch from > | > to is possible, it is not recommended. So, user needs > | > to decide from start of the day, which driver they want to use for MR= - > Fusion card. > | > | > | > | -- Implementation details -- > | > | To support this feature, we have modify code to change > | > | default > | > return type from probe. Currently driver return > | > "BUS_PROBE_DEFAULT". driver has been be changed to return > | > "BUS_PROBE_LOW_PRIORITY" if fusion_force hint from device.hints is > set. > | > | Please notice, above mentioned implementation in driver is > | > | only > | > applicable in case of MR-Fusion controller detection. For any other > | > controllers, supported by driver, the behavior of probe return > | > will remain same as before. > | > | > | > | > | > | -- High level feature list of -- 1. Supports Fast Path > | > | feature of LSI controllers. > | > | 2. Supports 4K sector Drives. > | > | 3. CAM layer based interface. All VDs will be attached to CAM > | > | layer (Expected storage will be visible in "camcontroll") 4. > | > | Complete support of Online Controller Reset. (OCR) 5. OCR on > | > | Fimrware fault and IO > | > timeout case. > | > | 6. Work well with management application which is > | > | generic > | > application provided by LSI for all other Operating system. > | > | 7. Supporst DIF enabled VDs (Same support as provided in Linux and > | > | other OSes in FreeBSD) 8. Fast Path Load balance support. > | > | > | > | - In summary, this driver is in part with Linux based MR drivers > | > | and all other features will be available to as planned > | > | activity from LSI > | > | > | > | This code is well tested by LSI Q/A team on 32 bit and 64 bit > | > | FreeBSD > | > Released OSes. > | > > | > Sorry it is has taken so long to start playing with this. I found > | > one critical issue with read performance. I added a printf to the > | > driver to make sure the fast path is not being used and it claims it = isn't. > | > Doing either a dd or using the > | > dt(http://www.scsifaq.org/RMiller_Tools/dt.html) > | > I see good write performance with both mrsas and mfi but on read > | > mrsas is backing up and appears to be single threaded. With mrsas > | > doing a dd of=3D/dev/zero if=3D/dev/da0 bs=3D1m I'm seeing 45685 kBps= and > | > with mfi seeing > | > 597072 kBps via gstat. Using dt via: > | > dt of=3D/dev/da0 bs=3D64k limit=3D1g aios=3D32 I see writes at 53897= 7 > | > Kbytes/sec and reads at bouncing around between > | > 20000 - 8000 Kbytes/sec, mfi reports 244744 Kbytes/sec. The > | > interesting piece is seeing the L(q) via gstat being around 30 when > | > doing mrsas reads, writes are 1 or less. With mfi both read and writ= es are > 1 or less. > | > > | Thanks for for providing feedback. This is very important observation. > | We will work on this and get back to you. Meanwhile, if we have high > | level acceptance(considering few defect fixes and optimization as a > | follow up activity) of current code, can have commit > | in current FreeBSD driver. This will definitely help us to manage > | source code from single destination. Currently there are multiple > | flavors/version of driver code is floated due to Upstream > | commit is pending. We can work for each observation/optimization > | proposed from upstream community on priority bases. >=20 > Yes a project branch can help with this. >=20 > | > This is with a GENERIC kernel on -current/amd64. I updated my > | > source to > | > 260231 and see the same. Witness, etc. are all on. I don't recall > | > seeing this when I tested mrsas a long time ago. > | > | Thanks for additional info. > | > | > > | > The card I'm using is: > | > LSI MegaRAID SAS 9266-8i > | > Firmware BIOS 5.38.00_4.12.05.00_0x05180000 > | > Firmware BCON 6.1-49-e_49-Rel > | > Firmware PCLI 05.05-03:#%00011 > | > Firmware APP 3.220.75-2196 > | > Firmware NVDT 2.1209.03-0117 > | > Firmware BTBL 2.05.00.00-0010 > | > Firmware BOOT 07.26.13.219 > | > > | > Some other issues I ran into when I kldunload mrsas I got a: > | > sysctl_unregister_oid: failed to unregister sysctl and then when I > | > | I do not recall if we have seen this issue, but will try this on our se= tup. >=20 > It think this and the panic might have been an artifact of my inition tes= ting. I > followed the instruction of kldload mrsas from /boot/loader.conf and saw > this. However, when I started working the emulation I modified my > GERNERIC config to not have mfi in it so I could switch between mfi and > mrsas via kldload/kldunload to test ioctl emulation. When I did this I n= oticed > that the patch included mrsas in the GENERIC configuration. > That might have caused this issue having both the module and static drive= r in > the same system. So this is probably a bad bug report. > Doing kldload and kldunload without mrsas and mfi static in the kernel I = didn't > run into this again. >=20 > | > kldload mrsas I got a panic: > | > mrsas0: port 0x7000-0x70ff mem > | > 0xdf060000-0xdf0 63fff,0xdf000000-0xdf03ffff irq 32 at device 0.0 on > | > pci4 > ######################################################### > | > Driver has detected MR-Fusion Controller.If you wish to > | > switch to driver for MR-Fusion,read manual of mrsas. Once you > | > chose to use Driverf or MR-Fusion Controllers, you should > | > not switch to .LSI recommend to use | > rsas> driver for MR-Fusion.Use if you are not sure about > | > rsas> choice.Quick he > | > lp to use for MR-Fustion Card: > | > Remove hint.mrsas.0.fusion_force=3D1 from /boot/device.hints. > | > > ######################################################### > | > mrsas0: Waiting for FW to come to ready state > | > > | > > | > > | > db> tr > | > Tracing pid 534 tid 100245 td 0xfffff80015035490 > | > malloc() at malloc+0x1c8/frame 0xfffffe03fbab3200 > | > cpuctl_devs() at cpuctl_devs+0x61df/frame 0xfffffe03fbab3290 > | > cpuctl_devs() at cpuctl_devs+0x6189/frame 0xfffffe03fbab32f0 > | > cpuctl_devs() at cpuctl_devs+0x990a/frame 0xfffffe03fbab34b0 > | > device_attach() at device_attach+0x3a2/frame 0xfffffe03fbab3510 > | > pci_driver_added() at pci_driver_added+0xfa/frame 0xfffffe03fbab3550 > | > devclass_driver_added() at devclass_driver_added+0x7d/frame > | > 0xfffffe03fbab3590 > | > devclass_add_driver() at devclass_add_driver+0x127/frame > | > 0xfffffe03fbab35d0 > | > module_register_init() at module_register_init+0xb0/frame > | > 0xfffffe03fbab3600 > | > linker_load_module() at linker_load_module+0xbda/frame > | > 0xfffffe03fbab3930 > | > kern_kldload() at kern_kldload+0xad/frame 0xfffffe03fbab3970 > | > sys_kldload() at sys_kldload+0x5b/frame 0xfffffe03fbab39a0 > | > amd64_syscall() at amd64_syscall+0x265/frame 0xfffffe03fbab3ab0 > | > Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe03fbab3ab0 > | > > | > I played with storcli/storcli64 and MegaCli/MegaCli64 from LSI's web = site. > | > I noticed that the 32 bit versions are built for FreeBSD 8.1 and the > | > 64 bit FreeBSD 7.4. It might not be a critical problem but some > | > companies might still be on earlier releases. > | > > | > I need to look at adding the ioctl translation support for mfiutil > | > and Linux to mrsas. This means we should have a 32 bit to 64 bit > | > ioctl translator in the driver since our Linux emulation is 32 bit. > | We can pick this work as next action item. >=20 > I started working on this and mentioned some things that would make this > easier. >=20 > | > It's also handy to run 32 bit tools on the 64 bit kernel. > | > > | > It would be good to add alias support for /dev/mfid* such as was > | > done via ATA CAM sys/cam/ata/ata_da.c. Search for > | > ada_legacy_aliases and legacy_id. The helped the migration from > /dev/ad* to /dev/ada*. > | > > | Let me check this. You can help me to understand what is expected here > | and we can provide solution for this. > | Again, we can mark this as next to-do for mrsas and will provide patch > | as an when we have code ready. >=20 > The idea is that when /dev/da node is created and alias of /dev/mfi= is > created point to /dev/da. This way existing fstab that reference > /dev/mfid0 for example will continue to work. It just makes it easier fo= r > migration. >=20 > A trickier part will be the feature to allow both LD and PD of an LD to b= e > available in a system. It is a feature that people use. With mfi it is = easy to see > RAID controlled devices via /dev/mfid and mfisyspd. > The physical device access come up as /dev/da. So people should know > be very carefull when accessing /dev/da since it could upset the RAID > controller, such as flashing firmware and when the disk goes offline it c= an kick > out the disk from the RAID. >=20 > I haven't looked at how mrsas deals with deleting and recreating RAID on = the > fly. mfi can do it with the proper sysctl/tunables set. >=20 > | > I'm not sure hint.mrsas..fusion_force=3D"1" is a good toggle to > | > enable/disable mrsas. I can see why you did that but it might be > | > more obvious to make hint.mfi..disabled=3D"1" to work since that > | > follows more of what people are used to. > | > | Either way is fine, since hint.mfi..disabled may miss lead that we > | are going to disable completely. > | We may have driver working for Falcon and other old MFI based > | controllers. >=20 > Since it is a hint.mfi..disabled, the x refers to card x, ie. > hint.mfi.1.disabled would disable the 2nd card in the system what would b= e > the ThunderBolt based can and leave the others to be mfi. We should get > more input from FreeBSD folks. fusion_force seems pretty non-obvious to > me. I'd suggest force_mrsas to be more obvious and it should probably be= a > generic tunabled and device instance specific. >=20 > Adding printing events to the kernel messages seems to be missing. > This needs to be controlled by sysctl/tunable but I might not have seen t= his > yet. >=20 > I'd say the highest priority is on performance and style. >=20 > Getting a review from Scott would be good but I'm not sure he has had tim= e > to do that. It might be good to look at mfi to see how things are laid o= ut. >=20 > FYI, my biggest problem working on FreeBSD is time. My day job prioritie= s > end up eating into time to work on "less critical" issues. Unfortunately= a > bunch of FreeBSD things fall into that catagory. I need to really resume= on > my mount path changes. I know what I need to do, I just need to carve ou= t > some time to work on it. >=20 > Thanks, >=20 > Doug A. From owner-freebsd-scsi@FreeBSD.ORG Tue Jan 7 18:11:37 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 82CB9E27 for ; Tue, 7 Jan 2014 18:11:37 +0000 (UTC) Received: from aslan.scsiguy.com (aslan.scsiguy.com [70.89.174.89]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 594A1109B for ; Tue, 7 Jan 2014 18:11:37 +0000 (UTC) Received: from ryanguthrielt.sldomain.com (207-225-98-3.dia.static.qwest.net [207.225.98.3]) (authenticated bits=0) by aslan.scsiguy.com (8.14.7/8.14.5) with ESMTP id s07IBXqn053557 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Tue, 7 Jan 2014 11:11:35 -0700 (MST) (envelope-from gibbs@scsiguy.com) Content-Type: multipart/mixed; boundary="Apple-Mail=_144B057E-CE36-465F-8836-FE3D78F47F9A" Mime-Version: 1.0 (Mac OS X Mail 7.1 \(1827\)) Subject: Re: Dropped interrupts From: "Justin T. Gibbs" In-Reply-To: Date: Tue, 7 Jan 2014 11:11:28 -0700 Message-Id: <84D23688-DDC6-421E-9D21-3DA646229038@scsiguy.com> References: To: Ben Laurie X-Mailer: Apple Mail (2.1827) X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.4.3 (aslan.scsiguy.com [70.89.174.89]); Tue, 07 Jan 2014 11:11:36 -0700 (MST) Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Jan 2014 18:11:37 -0000 --Apple-Mail=_144B057E-CE36-465F-8836-FE3D78F47F9A Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 On Jan 7, 2014, at 12:36 AM, Ben Laurie wrote: > Attached. >=20 > On 7 January 2014 05:46, Justin T. Gibbs wrote: >> On Jan 6, 2014, at 3:01 PM, Ben Laurie wrote: >>=20 >>> Not subscribed to the list, so please cc on replies. >>>=20 >>> I'm using Bacula with an LTO-2 SCSI drive. >>>=20 >>> With increasing frequency lately, I've been getting errors like this >>> from bacula: >>>=20 >>> backup-sd JobId 13092: Error: block.c:608 Write error at 23:6772 on >>> device "Ultrium" (/dev/nsa0). ERR=3DOperation not permitted. >>>=20 >>> Associated with this, I see in dmesg: >>>=20 >>> ahc0: Recovery Initiated >>>=20 >>> [a lot of dump info, including=85] >>=20 >> If you provide the dump info, I may be able to tell you why recovery = is starting. >>=20 >> The dmesg information from a boot of the system would be good to have = too. >>=20 >> =97 >> Justin The target is keeping us in command phase for some reason. No parity or = other errors are being reported. My guess is that the tape drive does not = like the command that was issued for some reason. Attached are two totally untested/uncompiled changes for you to try out. = The first should give more information about the command that timed out so we can = better determine if it is well formed. The second is an attempted fix for = spurious=20 =93Interrupts may not be functioning=94 warnings. Can you attempt to = replicate this again with these changes? Thanks, Justin --Apple-Mail=_144B057E-CE36-465F-8836-FE3D78F47F9A Content-Disposition: attachment; filename=aic7xxx.diff Content-Type: application/octet-stream; x-unix-mode=0644; name="aic7xxx.diff" Content-Transfer-Encoding: 7bit --- //SpectraBSD/stable/sys/dev/aic7xxx/aic7xxx.c 2012-08-29 22:35:04.000000000 -0600 +++ /usr/home/justing/perforce/SpectraBSD/sys/dev/aic7xxx/aic7xxx.c 2012-08-29 22:35:04.000000000 -0600 @@ -7043,7 +7043,6 @@ u_int saved_scbptr; int target; int lun; - int i; char channel; target = SCB_GET_TARGET(ahc, scb); @@ -7051,15 +7050,9 @@ lun = SCB_GET_LUN(scb); ahc_print_path(ahc, scb); - printf("SCB 0x%x - timed out\n", scb->hscb->tag); - if (scb->sg_count > 0) { - for (i = 0; i < scb->sg_count; i++) { - printf("sg[%d] - Addr 0x%x : Length %d\n", - i, - scb->sg_list[i].addr, - scb->sg_list[i].len & AHC_SG_LEN_MASK); - } - } + printf("Timed out "); + ahc_print_scb(scb) + if (scb->flags & (SCB_DEVICE_RESET|SCB_ABORT)) { /* * Been down this road before. --- //SpectraBSD/stable/sys/dev/aic7xxx/aic_osm_lib.c 2012-06-29 17:23:35.000000000 -0600 +++ /usr/home/justing/perforce/SpectraBSD/sys/dev/aic7xxx/aic_osm_lib.c 2012-06-29 17:23:35.000000000 -0600 @@ -113,13 +113,14 @@ aic_lock(aic); for (;;) { - if (LIST_EMPTY(&aic->timedout_scbs) != 0 - && (aic->flags & AIC_SHUTDOWN_RECOVERY) == 0) - msleep(aic, &aic->platform_data->mtx, PUSER, "idle", 0); - if ((aic->flags & AIC_SHUTDOWN_RECOVERY) != 0) break; + if (LIST_EMPTY(&aic->timedout_scbs) != 0) { + msleep(aic, &aic->platform_data->mtx, PUSER, "idle", 0); + continue; + } + aic_recover_commands(aic); } aic->platform_data->recovery_thread = NULL; --Apple-Mail=_144B057E-CE36-465F-8836-FE3D78F47F9A Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=windows-1252 --Apple-Mail=_144B057E-CE36-465F-8836-FE3D78F47F9A-- From owner-freebsd-scsi@FreeBSD.ORG Tue Jan 7 18:15:20 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 10B29F70; Tue, 7 Jan 2014 18:15:20 +0000 (UTC) Received: from mtv-iport-3.cisco.com (mtv-iport-3.cisco.com [173.36.130.14]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id CB4C110C1; Tue, 7 Jan 2014 18:15:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=24615; q=dns/txt; s=iport; t=1389118519; x=1390328119; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=IQn/qbFttRfAzFxgdtvwYvW8oMmr40J3MskHCqy7OWE=; b=a0I0EMD9+TMz7quZn+8+O6Hw4nz41j2MmYdcW6utTdfWrm63zPmyLUb7 dhlLhccSMRY0hz8I9SUAWD131O11qU5Vm5NzU7huIL3D9G37fo1P+PlmA HFXOpdFbsvZgYswclDWLz+z4lPplALPxJnJPfzYn9VVUrwMbrkaneRWX1 U=; X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ag4FAEJDzFKrRDoJ/2dsb2JhbABZgws4qCGSG4ESFnSCJQEBAQQODCArFAwECxEEAQEKHgcPBTEBCQ4TCQsHA4dSAxAOxBwXjHGBJxQBBgEBBkkHBoMegRMEiUOENog4gWUBjFqFO4NOgUcBCBc X-IronPort-AV: E=Sophos;i="4.95,620,1384300800"; d="scan'208";a="99752817" Received: from mtv-core-4.cisco.com ([171.68.58.9]) by mtv-iport-3.cisco.com with ESMTP; 07 Jan 2014 18:14:10 +0000 Received: from dambriskodt.cisco.com (dambriskodt.cisco.com [171.71.230.115]) by mtv-core-4.cisco.com (8.14.5/8.14.5) with ESMTP id s07IE5Z3017907 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 7 Jan 2014 18:14:09 GMT Received: from dambriskodt.cisco.com (localhost [127.0.0.1]) by dambriskodt.cisco.com (8.14.7/8.14.3) with ESMTP id s07IBeOP002623; Tue, 7 Jan 2014 10:11:40 -0800 (PST) (envelope-from ambrisko@dambriskodt.cisco.com) Received: (from ambrisko@localhost) by dambriskodt.cisco.com (8.14.7/8.14.3/Submit) id s07IBdjn002622; Tue, 7 Jan 2014 10:11:39 -0800 (PST) (envelope-from ambrisko) Date: Tue, 7 Jan 2014 10:11:39 -0800 From: Doug Ambrisko To: "Desai, Kashyap" Subject: Re: LSI - MR-Fusion controller driver patch and man page Message-ID: <20140107181139.GC2080@cisco.com> References: <08ba2a262fba45f687cdd3225f325110@BN1PR07MB247.namprd07.prod.outlook.com> <20140103211449.GA69721@cisco.com> <8c423414ecc2421fbace3eb9f386be91@BN1PR07MB247.namprd07.prod.outlook.com> <20140106182935.GC93278@cisco.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-Mailman-Approved-At: Wed, 08 Jan 2014 05:24:56 +0000 Cc: "scottl@netflix.com" , "Radford, Adam" , "Kenneth D. Merry" , "sean_bruno@yahoo.com" , "Mankani, Krishnaraddi" , "dwhite@ixsystems.com" , "Maloy, Joe" , "jpaetzel@freebsd.org" , "freebsd-scsi@freebsd.org" , "McConnell, Stephen" X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Jan 2014 18:15:20 -0000 On Tue, Jan 07, 2014 at 05:37:17AM +0000, Desai, Kashyap wrote: | Doug: | | I have replied to your mail at very high level and not going for inline | reply to your respond to capture key issue and possible resolution. We might want to take this discussion into our meeting on the 20th. However, I'll give you my view on your concerns. I understand LSI concerns and LSI needs to understand FreeBSD's concerns since we deal with with them a lot. | Hi All, | | There are two biggest challenge for drivers. | | 1. conflict with for Thunderbolt, Invader and Fury controllers. | As we discussed, we have resolve this giving priority to controller | and only when customer wants use they need proper hint support. | Currently, there are customer who is using posted from LSI | external site and able to configure using storcli and driver is working | well for their need. | Definitely, we are doing lots of enhancement and defect fixes in those | areas, but note that before we work on actual issue, lots of customer | complain that is not loaded due to and PnP ID | conflict. Please understand that, this is bit critical for customers who | want to use . LSI also have a lot of customers/machines using mfi on Thunderbolt cards that depend on various features. We have found in the past that transitioning to a new an improved infrastructure impacts a lot of users so based on our last meeting I thought it was decided that we'd work adding support to make that transition as smooth as possible and FreeBSD people would help LSI with that effort. This work is a win for all since then there is a clear migration path. | To this issue, we need svn commits in driver (from submitted | patch see only related changes) to understand device.hints at | least, so that customer can use driver from lsi external site. Yes, we can probably make the minimal change to mfi to allow mrsas to optionally take over. That can probably be done the quickest. | Anything related to / interopt can be considered a different | design goal and requirement. | E.a "We should be fine not have alias of /dev/daX to /dev/mfidX for | initial drop. Probably we should document this constraint for initial | commit of mrsas" We have found this to be a problem in the past. People will flip things to test the waters so adding this fairly simple feature will address that. | E.a "Please consider that customer is not using to configure | when driver is used." | Till now customer is not having any issue using with . | We may go for good amount of interopt development to support configuration | utils as , but that may need time. As a long time user of LSI MegaRAID SAS cards when debugging issues we had to run Linux binaries supplied by our system vendor to get support. Being able to run MegaCli/storcli is only part of what people depend on. | 2. is only available via LSI external site. We need xi | upstream, so that customer should not wait for LSI external site downloads | for more generic use case. | This can be done in different phases, as LSI is doing for driver. | There are good comment posted by Doug and we are working on those. | We do not wanted to change code without posting for internal Q/A testing. | We should take advantage of LSI internal Q/A testing to fix most of the | defect internally. That is the goal we are working towards and we had agreed on things to do to get mrsas into FreeBSD. Getting mrsas into FreeBSD will also give LSI additional real work testing and bug fixes from the wild. So that is a win/win. | To resolve above issue, we should find middle way to post the | check-in in mainline. I mean, keep few know limitation as documented and | do not wait for gathering of all feature in single drop. | E.a If we start changing code to meet style(9) coding standard, we may | add some manual mistakes in code. We can take each action item as | Enhancement request (Including style(9) change as well) for LSI and we | will be doing all those step by step.( As we are doing for driver). The problem with ignoring style causes a bunch of issues. The white space style issues should be delt with right away. That doesn't change the code and the resulting code can be checked to make sure the resulting code doesn't change. Once people start using ioctl's and what are in the ioctl's then changing style in headers later becomes a problem. Going out on a limb a bit here, I'm as concerned about things that can be changed in the future and have no impact outside the /sys/dev/mrsas area (again, ioctl's are outside that that is a problem). However, once code gets in then sometimes style issues don't get fixed. | I am not aware of project method to do svn check, since I worked with | Ken D Merry earlier to do actual commit in FreeBSD head repo. I thought | it is going to be same way for . I know we have GSoC (Google Summer of Code) projects checked in by folks that were not committers but I'm not sure if we can do that with subversion. I need to check or someone else can chime in. It would be nice if LSI could directly check into a project and then we can have people test that and merge that into head. I think things are more complicated for mrsas just due to the requirements people have for LSI MegaRAID SAS. As it we can't use mrsas for any of our products since we have an infrastructure depending on features that mfi has. Making the transition smoother is a win for everyone so that if some say mfi can do this and mrsas can do it, it just makes things easier. For me I know I did something pretty well if people use things I've done but never ask me questions about it. | Sorry, but doing lots of discussion in single mail thread is prone to | divert actual agenda and we may hang forever without any conclusion. | So, let's open defect in as soon as it get upstream, LSI will | work for all the defect against . | At least immediate check in diver to understand device.hints, | will allow customer to use from LSI external site. Again, this isn't typically how it works unless you can find someone that will take the responsibility of all of the issues and make it right. I for one do not want to check it in and take the back lash. I try to work within the FreeBSD guidelines for code and style is pretty important. I don't want to rehash old issues but working better with FreeBSD from the start could have avoided a bunch of issues. LSI did a bunch of good work but things could have been done more optimal so mrsas could have been in FreeBSD sooner. Of course I'm only one person in FreeBSD so other people might have other views and if they see an easier way to help LSI then I'm all for that. I'm just stating what I see needs to be done before I check it in. So to me gating issues to check it in are: - performance - style - mfiutil ioctl support - alias support - Scott's code review - man page review by doc's team - testing I know this is additional work on everyone's side but it really does make things better. I have WIP code that I'd love to check into FreeBSD but it is not ready for FreeBSD standards. People have given me suggestions of which I've gone back to make things better. So that is a good thing the only problem I have is limited time to do the changes and test the changes. So I understand the pain. Having the driver available in a SCM repository would be great since various people can help make simple changes. Maybe putting it up in git might help if we can't make a subversion project work? Thanks, Doug A. | > -----Original Message----- | > From: Doug Ambrisko [mailto:ambrisko@cisco.com] | > Sent: Tuesday, January 07, 2014 12:00 AM | > To: Desai, Kashyap | > Cc: freebsd-scsi@freebsd.org; scottl@netflix.com; sean_bruno@yahoo.com; | > dwhite@ixsystems.com; jpaetzel@freebsd.org; Maloy, Joe; Mankani, | > Krishnaraddi; McConnell, Stephen; Saxena, Sumit; Radford, Adam; Kenneth | > D. Merry | > Subject: Re: LSI - MR-Fusion controller driver patch and man page | > | > On Mon, Jan 06, 2014 at 07:43:27AM +0000, Desai, Kashyap wrote: | > | Doug: | > | | > | I have replied inline for your comment. For most of the comment you | > | posted below can be work out and we can prioritize completion of those | > | based on nature of issue. | > | I will work on READ/WRITE comparison w.r.t and as a | > | priority item. | > | > IO performance is the highest priority. | > | > | Please consider that it will be really helpful for further | > | development, if we have common code check-in in Upstream FreeBSD. We | > | will actively work on each comment posted for and address | > | those before we have in-box for any FreeBSD release. | > | > What we have done in the past is to create a project branch to do this and it | > has worked well. It would be nice if you could check in directly to a this | > project branch so we could use svn to resolve differences. | > | > I poked around at adding the ioctl compatibility support and noticed that | > mrsas seems to lack a general method to send a dcmd to the controller. | > What Scott did in mfi was to create a generic function to send dcmd's with a | > simple data buffer. That data buffer was then put into the sg list in the | > generic function. This made it easier to translate the various ioctl's since we | > could convert the sg lists passed down from the user land into a simple | > malloced data base and then covert back out. | > In the mfituil ioctl method it just passes down the data buffer and then | > let's the driver convert it to a sg list. To make ioctl emulation | > easier in these cases: | > 32 bit FreeBSD LSI ioctl on 64 bit | > 32 bit Linux LSI ioctl on 32 bit | > 32 bit Linux LSI ioctl on 64 bit | > 32 bit mfiutil ioctl on 32 bit | > 32 bit mfiutil ioctl on 64 bit | > | > Of high priority is style, please make it style(9) compliant. There are a trailing | > spaces, spaces instead of tabs, I think lack of tab aligned indent. I noticed | > functions being externally defined in .c files instead of .h. There should be .h | > files that can be included in user land applications for ioctls and there can be | > .h files just for the kernel. | > These files will need to be installed as appropriate. I found this with the ioctl | > function. | > | > | > -----Original Message----- | > | > From: Doug Ambrisko [mailto:ambrisko@cisco.com] | > | > Sent: Saturday, January 04, 2014 2:45 AM | > | > To: Desai, Kashyap | > | > Cc: freebsd-scsi@freebsd.org; scottl@netflix.com; | > | > sean_bruno@yahoo.com; dwhite@ixsystems.com; | > jpaetzel@freebsd.org; | > | > Maloy, Joe; Mankani, Krishnaraddi; McConnell, Stephen; Saxena, | > | > Sumit; Radford, Adam; Kenneth D. Merry | > | > Subject: Re: LSI - MR-Fusion controller driver patch and man | > | > page | > | > | > | > On Fri, Dec 06, 2013 at 02:37:21PM +0000, Desai, Kashyap wrote: | > | > | Hi, | > | > | | > | > | Please consider attached patch for FreeBSD upstream check-in. | > | > | | > | > | Please find attached patch for driver for LSI's MegaRaid | > | > Controllers. This driver supports Thunderbolt onwards Device IDs of | > | > MR controllers. | > | > | Currently it supports 0x005B and 0x005D Device IDs. | > | > | | > | > | NOTE : This driver will not eliminate or by pass any functionality | > | > | of | > | > driver which already support above to Device IDs to keep existing | > | > user experience unchanged. | > | > | | > | > | Driver will be always given priority over driver and | > | > | only if customer/user wants to use/migrate from to , | > | > | it will hook up into kernel via device.hint rules. (Attached is | > | > | mrsas man page for more info.) | > | > | | > | > | LSI will continue to update driver in future in timely bases. | > | > | We have another set of patch in pipeline to be submitted for | > | > | , but need first go ahead for attached patch and later we | > | > | will continue to keep up-to-date (In sync with LSI | > | > | released driver which is available at lsi's external site) | > | > | | > | > | Apply patch with "patch -p0 < patchname.patch" from head directory. | > | > | | > | > | -- Few notes for user-- | > | > | LSI recommends using fusion_force bit In hint settings at start of | > | > | the day, if | > | > they want to use . ( will be a default choice for | > | > MR-Fusion HBA), if will be changed only with fusion_force hint | > | > settings. (See mrsas man | > | > page) Changing any default behavior is well tested for most of the | > condition. | > | > | Switching from to for MR-Fusion options is designed | > | > | to | > | > allow user as one time choice, though multiple time switch from | > | > to is possible, it is not recommended. So, user needs | > | > to decide from start of the day, which driver they want to use for MR- | > Fusion card. | > | > | | > | > | -- Implementation details -- | > | > | To support this feature, we have modify code to change | > | > | default | > | > return type from probe. Currently driver return | > | > "BUS_PROBE_DEFAULT". driver has been be changed to return | > | > "BUS_PROBE_LOW_PRIORITY" if fusion_force hint from device.hints is | > set. | > | > | Please notice, above mentioned implementation in driver is | > | > | only | > | > applicable in case of MR-Fusion controller detection. For any other | > | > controllers, supported by driver, the behavior of probe return | > | > will remain same as before. | > | > | | > | > | | > | > | -- High level feature list of -- 1. Supports Fast Path | > | > | feature of LSI controllers. | > | > | 2. Supports 4K sector Drives. | > | > | 3. CAM layer based interface. All VDs will be attached to CAM | > | > | layer (Expected storage will be visible in "camcontroll") 4. | > | > | Complete support of Online Controller Reset. (OCR) 5. OCR on | > | > | Fimrware fault and IO | > | > timeout case. | > | > | 6. Work well with management application which is | > | > | generic | > | > application provided by LSI for all other Operating system. | > | > | 7. Supporst DIF enabled VDs (Same support as provided in Linux and | > | > | other OSes in FreeBSD) 8. Fast Path Load balance support. | > | > | | > | > | - In summary, this driver is in part with Linux based MR drivers | > | > | and all other features will be available to as planned | > | > | activity from LSI | > | > | | > | > | This code is well tested by LSI Q/A team on 32 bit and 64 bit | > | > | FreeBSD | > | > Released OSes. | > | > | > | > Sorry it is has taken so long to start playing with this. I found | > | > one critical issue with read performance. I added a printf to the | > | > driver to make sure the fast path is not being used and it claims it isn't. | > | > Doing either a dd or using the | > | > dt(http://www.scsifaq.org/RMiller_Tools/dt.html) | > | > I see good write performance with both mrsas and mfi but on read | > | > mrsas is backing up and appears to be single threaded. With mrsas | > | > doing a dd of=/dev/zero if=/dev/da0 bs=1m I'm seeing 45685 kBps and | > | > with mfi seeing | > | > 597072 kBps via gstat. Using dt via: | > | > dt of=/dev/da0 bs=64k limit=1g aios=32 I see writes at 538977 | > | > Kbytes/sec and reads at bouncing around between | > | > 20000 - 8000 Kbytes/sec, mfi reports 244744 Kbytes/sec. The | > | > interesting piece is seeing the L(q) via gstat being around 30 when | > | > doing mrsas reads, writes are 1 or less. With mfi both read and writes are | > 1 or less. | > | > | > | Thanks for for providing feedback. This is very important observation. | > | We will work on this and get back to you. Meanwhile, if we have high | > | level acceptance(considering few defect fixes and optimization as a | > | follow up activity) of current code, can have commit | > | in current FreeBSD driver. This will definitely help us to manage | > | source code from single destination. Currently there are multiple | > | flavors/version of driver code is floated due to Upstream | > | commit is pending. We can work for each observation/optimization | > | proposed from upstream community on priority bases. | > | > Yes a project branch can help with this. | > | > | > This is with a GENERIC kernel on -current/amd64. I updated my | > | > source to | > | > 260231 and see the same. Witness, etc. are all on. I don't recall | > | > seeing this when I tested mrsas a long time ago. | > | | > | Thanks for additional info. | > | | > | > | > | > The card I'm using is: | > | > LSI MegaRAID SAS 9266-8i | > | > Firmware BIOS 5.38.00_4.12.05.00_0x05180000 | > | > Firmware BCON 6.1-49-e_49-Rel | > | > Firmware PCLI 05.05-03:#%00011 | > | > Firmware APP 3.220.75-2196 | > | > Firmware NVDT 2.1209.03-0117 | > | > Firmware BTBL 2.05.00.00-0010 | > | > Firmware BOOT 07.26.13.219 | > | > | > | > Some other issues I ran into when I kldunload mrsas I got a: | > | > sysctl_unregister_oid: failed to unregister sysctl and then when I | > | | > | I do not recall if we have seen this issue, but will try this on our setup. | > | > It think this and the panic might have been an artifact of my inition testing. I | > followed the instruction of kldload mrsas from /boot/loader.conf and saw | > this. However, when I started working the emulation I modified my | > GERNERIC config to not have mfi in it so I could switch between mfi and | > mrsas via kldload/kldunload to test ioctl emulation. When I did this I noticed | > that the patch included mrsas in the GENERIC configuration. | > That might have caused this issue having both the module and static driver in | > the same system. So this is probably a bad bug report. | > Doing kldload and kldunload without mrsas and mfi static in the kernel I didn't | > run into this again. | > | > | > kldload mrsas I got a panic: | > | > mrsas0: port 0x7000-0x70ff mem | > | > 0xdf060000-0xdf0 63fff,0xdf000000-0xdf03ffff irq 32 at device 0.0 on | > | > pci4 | > ######################################################### | > | > Driver has detected MR-Fusion Controller.If you wish to | > | > switch to driver for MR-Fusion,read manual of mrsas. Once you | > | > chose to use Driverf or MR-Fusion Controllers, you should | > | > not switch to .LSI recommend to use | > rsas> driver for MR-Fusion.Use if you are not sure about | > | > rsas> choice.Quick he | > | > lp to use for MR-Fustion Card: | > | > Remove hint.mrsas.0.fusion_force=1 from /boot/device.hints. | > | > | > ######################################################### | > | > mrsas0: Waiting for FW to come to ready state | > | > | > | > | > | > | > | > db> tr | > | > Tracing pid 534 tid 100245 td 0xfffff80015035490 | > | > malloc() at malloc+0x1c8/frame 0xfffffe03fbab3200 | > | > cpuctl_devs() at cpuctl_devs+0x61df/frame 0xfffffe03fbab3290 | > | > cpuctl_devs() at cpuctl_devs+0x6189/frame 0xfffffe03fbab32f0 | > | > cpuctl_devs() at cpuctl_devs+0x990a/frame 0xfffffe03fbab34b0 | > | > device_attach() at device_attach+0x3a2/frame 0xfffffe03fbab3510 | > | > pci_driver_added() at pci_driver_added+0xfa/frame 0xfffffe03fbab3550 | > | > devclass_driver_added() at devclass_driver_added+0x7d/frame | > | > 0xfffffe03fbab3590 | > | > devclass_add_driver() at devclass_add_driver+0x127/frame | > | > 0xfffffe03fbab35d0 | > | > module_register_init() at module_register_init+0xb0/frame | > | > 0xfffffe03fbab3600 | > | > linker_load_module() at linker_load_module+0xbda/frame | > | > 0xfffffe03fbab3930 | > | > kern_kldload() at kern_kldload+0xad/frame 0xfffffe03fbab3970 | > | > sys_kldload() at sys_kldload+0x5b/frame 0xfffffe03fbab39a0 | > | > amd64_syscall() at amd64_syscall+0x265/frame 0xfffffe03fbab3ab0 | > | > Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe03fbab3ab0 | > | > | > | > I played with storcli/storcli64 and MegaCli/MegaCli64 from LSI's web site. | > | > I noticed that the 32 bit versions are built for FreeBSD 8.1 and the | > | > 64 bit FreeBSD 7.4. It might not be a critical problem but some | > | > companies might still be on earlier releases. | > | > | > | > I need to look at adding the ioctl translation support for mfiutil | > | > and Linux to mrsas. This means we should have a 32 bit to 64 bit | > | > ioctl translator in the driver since our Linux emulation is 32 bit. | > | We can pick this work as next action item. | > | > I started working on this and mentioned some things that would make this | > easier. | > | > | > It's also handy to run 32 bit tools on the 64 bit kernel. | > | > | > | > It would be good to add alias support for /dev/mfid* such as was | > | > done via ATA CAM sys/cam/ata/ata_da.c. Search for | > | > ada_legacy_aliases and legacy_id. The helped the migration from | > /dev/ad* to /dev/ada*. | > | > | > | Let me check this. You can help me to understand what is expected here | > | and we can provide solution for this. | > | Again, we can mark this as next to-do for mrsas and will provide patch | > | as an when we have code ready. | > | > The idea is that when /dev/da node is created and alias of /dev/mfi is | > created point to /dev/da. This way existing fstab that reference | > /dev/mfid0 for example will continue to work. It just makes it easier for | > migration. | > | > A trickier part will be the feature to allow both LD and PD of an LD to be | > available in a system. It is a feature that people use. With mfi it is easy to see | > RAID controlled devices via /dev/mfid and mfisyspd. | > The physical device access come up as /dev/da. So people should know | > be very carefull when accessing /dev/da since it could upset the RAID | > controller, such as flashing firmware and when the disk goes offline it can kick | > out the disk from the RAID. | > | > I haven't looked at how mrsas deals with deleting and recreating RAID on the | > fly. mfi can do it with the proper sysctl/tunables set. | > | > | > I'm not sure hint.mrsas..fusion_force="1" is a good toggle to | > | > enable/disable mrsas. I can see why you did that but it might be | > | > more obvious to make hint.mfi..disabled="1" to work since that | > | > follows more of what people are used to. | > | | > | Either way is fine, since hint.mfi..disabled may miss lead that we | > | are going to disable completely. | > | We may have driver working for Falcon and other old MFI based | > | controllers. | > | > Since it is a hint.mfi..disabled, the x refers to card x, ie. | > hint.mfi.1.disabled would disable the 2nd card in the system what would be | > the ThunderBolt based can and leave the others to be mfi. We should get | > more input from FreeBSD folks. fusion_force seems pretty non-obvious to | > me. I'd suggest force_mrsas to be more obvious and it should probably be a | > generic tunabled and device instance specific. | > | > Adding printing events to the kernel messages seems to be missing. | > This needs to be controlled by sysctl/tunable but I might not have seen this | > yet. | > | > I'd say the highest priority is on performance and style. | > | > Getting a review from Scott would be good but I'm not sure he has had time | > to do that. It might be good to look at mfi to see how things are laid out. | > | > FYI, my biggest problem working on FreeBSD is time. My day job priorities | > end up eating into time to work on "less critical" issues. Unfortunately a | > bunch of FreeBSD things fall into that catagory. I need to really resume on | > my mount path changes. I know what I need to do, I just need to carve out | > some time to work on it. | > | > Thanks, | > | > Doug A. From owner-freebsd-scsi@FreeBSD.ORG Wed Jan 8 06:44:40 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6EC297AA for ; Wed, 8 Jan 2014 06:44:40 +0000 (UTC) Received: from mail-qa0-x236.google.com (mail-qa0-x236.google.com [IPv6:2607:f8b0:400d:c00::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 2BD8A150A for ; Wed, 8 Jan 2014 06:44:40 +0000 (UTC) Received: by mail-qa0-f54.google.com with SMTP id j7so1497748qaq.27 for ; Tue, 07 Jan 2014 22:44:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type:content-transfer-encoding; bh=P8qBMFjvY6ft08jt+d3kwqcT+bLhrchwwifmwop+3Vo=; b=JHV6TbduIrUUO2CaYxaT9i85KwkzHsf3q4o46IrCV1u2ig2/RsmBXb8QYSeL7KJSDv Mc5D/ME3krNvvvM+NjgSy5ImfyT/DNhqXxmzv0aiVFmQA/c9Xt3wI6wZrO7Aq8XcgIiL itBEpKtqFeFMQv4fRCgeWwgPpNw77JmAaWGudoRe1SiwLKCKaQmQb48D94MKOK8Tk13m dMOKbjwuWoJp/Kj4n1Kh041VpakXpeMW7WmFY93LUW9zrJAWFiNpb0+T2VAuQ7lfw9cN 3+oQ6YjwIrmGPeaNvi3AfjXRs+KZ2WcPOu+I3hPnQo7Bg88VnlOeddiLP7ByGfoTxqZK Om5w== MIME-Version: 1.0 X-Received: by 10.49.59.83 with SMTP id x19mr10837717qeq.47.1389163479422; Tue, 07 Jan 2014 22:44:39 -0800 (PST) Sender: benlaurie@gmail.com Received: by 10.96.72.196 with HTTP; Tue, 7 Jan 2014 22:44:39 -0800 (PST) In-Reply-To: <84D23688-DDC6-421E-9D21-3DA646229038@scsiguy.com> References: <84D23688-DDC6-421E-9D21-3DA646229038@scsiguy.com> Date: Wed, 8 Jan 2014 06:44:39 +0000 X-Google-Sender-Auth: 56hwxsa3MWCOd786M4ZTICXNaR4 Message-ID: Subject: Re: Dropped interrupts From: Ben Laurie To: "Justin T. Gibbs" Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Jan 2014 06:44:40 -0000 On 7 January 2014 18:11, Justin T. Gibbs wrote: > On Jan 7, 2014, at 12:36 AM, Ben Laurie wrote: > >> Attached. >> >> On 7 January 2014 05:46, Justin T. Gibbs wrote: >>> On Jan 6, 2014, at 3:01 PM, Ben Laurie wrote: >>> >>>> Not subscribed to the list, so please cc on replies. >>>> >>>> I'm using Bacula with an LTO-2 SCSI drive. >>>> >>>> With increasing frequency lately, I've been getting errors like this >>>> from bacula: >>>> >>>> backup-sd JobId 13092: Error: block.c:608 Write error at 23:6772 on >>>> device "Ultrium" (/dev/nsa0). ERR=3DOperation not permitted. >>>> >>>> Associated with this, I see in dmesg: >>>> >>>> ahc0: Recovery Initiated >>>> >>>> [a lot of dump info, including=85] >>> >>> If you provide the dump info, I may be able to tell you why recovery is= starting. >>> >>> The dmesg information from a boot of the system would be good to have t= oo. >>> >>> =97 >>> Justin > > The target is keeping us in command phase for some reason. No parity or = other > errors are being reported. My guess is that the tape drive does not like= the command > that was issued for some reason. > > Attached are two totally untested/uncompiled changes for you to try out. = The first > should give more information about the command that timed out so we can b= etter > determine if it is well formed. The second is an attempted fix for spuri= ous > =93Interrupts may not be functioning=94 warnings. Can you attempt to rep= licate this > again with these changes? Rebuilding now - you had a ; missing in the patch :-) Of course, now I've done this, it'll not fail for a month (its been failing multiple times per day recently, but on average its a lot rarer than that!). Will let you know when I get a fresh failure. From owner-freebsd-scsi@FreeBSD.ORG Wed Jan 8 18:37:50 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id AED853E3 for ; Wed, 8 Jan 2014 18:37:50 +0000 (UTC) Received: from khavrinen.csail.mit.edu (khavrinen.csail.mit.edu [IPv6:2001:470:8b2d:1e1c:21b:21ff:feb8:d7b0]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 78F0F150E for ; Wed, 8 Jan 2014 18:37:50 +0000 (UTC) Received: from khavrinen.csail.mit.edu (localhost [127.0.0.1]) by khavrinen.csail.mit.edu (8.14.7/8.14.7) with ESMTP id s08Ibm2N009776 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL CN=khavrinen.csail.mit.edu issuer=Client+20CA) for ; Wed, 8 Jan 2014 13:37:48 -0500 (EST) (envelope-from wollman@khavrinen.csail.mit.edu) Received: (from wollman@localhost) by khavrinen.csail.mit.edu (8.14.7/8.14.7/Submit) id s08Ibmem009773; Wed, 8 Jan 2014 13:37:48 -0500 (EST) (envelope-from wollman) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <21197.39676.138433.937002@khavrinen.csail.mit.edu> Date: Wed, 8 Jan 2014 13:37:48 -0500 From: Garrett Wollman To: freebsd-scsi@freebsd.org Subject: Attempting ATA TRIM on SAS devices? X-Mailer: VM 7.17 under 21.4 (patch 22) "Instant Classic" XEmacs Lucid X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (khavrinen.csail.mit.edu [127.0.0.1]); Wed, 08 Jan 2014 13:37:48 -0500 (EST) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Jan 2014 18:37:50 -0000 On a freshly-installed 9.2 system, I see the following: (da12:mps0:0:59:0): ATA TRIM failed, switching to UNMAP BIO_DELETE (da12:mps0:0:59:0): ATA COMMAND PASS THROUGH(16). CDB: 85 0d 06 00 01 00 01 00 00 00 00 00 00 40 06 00 (da12:mps0:0:59:0): CAM status: SCSI Status Error (da12:mps0:0:59:0): SCSI status: Check Condition (da12:mps0:0:59:0): SCSI sense: ILLEGAL REQUEST asc:24,0 (Invalid field in CDB) (da12:mps0:0:59:0): Error 22, Unretryable error (da35:mps0:0:82:0): ATA TRIM failed, switching to UNMAP BIO_DELETE (da35:mps0:0:82:0): ATA COMMAND PASS THROUGH(16). CDB: 85 0d 06 00 01 00 01 00 00 00 00 00 00 40 06 00 (da35:mps0:0:82:0): CAM status: SCSI Status Error (da35:mps0:0:82:0): SCSI status: Check Condition (da35:mps0:0:82:0): SCSI sense: ILLEGAL REQUEST asc:24,0 (Invalid field in CDB) (da35:mps0:0:82:0): Error 22, Unretryable error da12 and da35 are identified as follows: [root@nfs-prod-3 /export]# camcontrol inquiry da12 pass13: Fixed Direct Access SCSI-5 device pass13: Serial Number A179E011337000251 pass13: 600.000MB/s transfers, Command Queueing Enabled [root@nfs-prod-3 /export]# camcontrol inquiry da35 pass36: Fixed Direct Access SCSI-5 device pass36: Serial Number A179E011337000234 pass36: 600.000MB/s transfers, Command Queueing Enabled Why would it be trying ATA commands here? These are SAS devices. -GAWollman From owner-freebsd-scsi@FreeBSD.ORG Wed Jan 8 20:36:07 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B7C2EB3D for ; Wed, 8 Jan 2014 20:36:07 +0000 (UTC) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 541DD1F94 for ; Wed, 8 Jan 2014 20:36:07 +0000 (UTC) Received: from r2d2 ([82.69.141.170]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50007475910.msg for ; Wed, 08 Jan 2014 20:36:03 +0000 X-Spam-Processed: mail1.multiplay.co.uk, Wed, 08 Jan 2014 20:36:03 +0000 (not processed: message from valid local sender) X-MDDKIM-Result: neutral (mail1.multiplay.co.uk) X-MDRemoteIP: 82.69.141.170 X-Return-Path: prvs=10857851f7=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-scsi@freebsd.org Message-ID: <430F34AB88D44C4BAF31DDC7AD214137@multiplay.co.uk> From: "Steven Hartland" To: "Garrett Wollman" , References: <21197.39676.138433.937002@khavrinen.csail.mit.edu> Subject: Re: Attempting ATA TRIM on SAS devices? Date: Wed, 8 Jan 2014 20:36:13 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Jan 2014 20:36:07 -0000 It should only do this if your drive reports it supports ATA_TRIM by setting the ATA_SUPPORT_DSM_TRIM bit in the response to an ATA_IDENTIFY command sent to it using ATA passthrough. What does the following report: camcontrol identify da12 Regards Steve ----- Original Message ----- From: "Garrett Wollman" To: Sent: Wednesday, January 08, 2014 6:37 PM Subject: Attempting ATA TRIM on SAS devices? > On a freshly-installed 9.2 system, I see the following: > > (da12:mps0:0:59:0): ATA TRIM failed, switching to UNMAP BIO_DELETE > (da12:mps0:0:59:0): ATA COMMAND PASS THROUGH(16). CDB: 85 0d 06 00 01 00 01 00 00 00 00 00 00 40 06 00 > (da12:mps0:0:59:0): CAM status: SCSI Status Error > (da12:mps0:0:59:0): SCSI status: Check Condition > (da12:mps0:0:59:0): SCSI sense: ILLEGAL REQUEST asc:24,0 (Invalid field in CDB) > (da12:mps0:0:59:0): Error 22, Unretryable error > (da35:mps0:0:82:0): ATA TRIM failed, switching to UNMAP BIO_DELETE > (da35:mps0:0:82:0): ATA COMMAND PASS THROUGH(16). CDB: 85 0d 06 00 01 00 01 00 00 00 00 00 00 40 06 00 > (da35:mps0:0:82:0): CAM status: SCSI Status Error > (da35:mps0:0:82:0): SCSI status: Check Condition > (da35:mps0:0:82:0): SCSI sense: ILLEGAL REQUEST asc:24,0 (Invalid field in CDB) > (da35:mps0:0:82:0): Error 22, Unretryable error > > da12 and da35 are identified as follows: > > [root@nfs-prod-3 /export]# camcontrol inquiry da12 > pass13: Fixed Direct Access SCSI-5 device > pass13: Serial Number A179E011337000251 > pass13: 600.000MB/s transfers, Command Queueing Enabled > [root@nfs-prod-3 /export]# camcontrol inquiry da35 > pass36: Fixed Direct Access SCSI-5 device > pass36: Serial Number A179E011337000234 > pass36: 600.000MB/s transfers, Command Queueing Enabled > > Why would it be trying ATA commands here? These are SAS devices. > > -GAWollman > _______________________________________________ > freebsd-scsi@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-scsi > To unsubscribe, send any mail to "freebsd-scsi-unsubscribe@freebsd.org" > ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-scsi@FreeBSD.ORG Wed Jan 8 20:48:17 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D372CB9 for ; Wed, 8 Jan 2014 20:48:17 +0000 (UTC) Received: from khavrinen.csail.mit.edu (khavrinen.csail.mit.edu [IPv6:2001:470:8b2d:1e1c:21b:21ff:feb8:d7b0]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 9FA4C10C0 for ; Wed, 8 Jan 2014 20:48:17 +0000 (UTC) Received: from khavrinen.csail.mit.edu (localhost [127.0.0.1]) by khavrinen.csail.mit.edu (8.14.7/8.14.7) with ESMTP id s08KmG4h011212 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL CN=khavrinen.csail.mit.edu issuer=Client+20CA); Wed, 8 Jan 2014 15:48:16 -0500 (EST) (envelope-from wollman@khavrinen.csail.mit.edu) Received: (from wollman@localhost) by khavrinen.csail.mit.edu (8.14.7/8.14.7/Submit) id s08KmGPe011209; Wed, 8 Jan 2014 15:48:16 -0500 (EST) (envelope-from wollman) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <21197.47504.93295.771468@khavrinen.csail.mit.edu> Date: Wed, 8 Jan 2014 15:48:16 -0500 From: Garrett Wollman To: "Steven Hartland" Subject: Re: Attempting ATA TRIM on SAS devices? In-Reply-To: <430F34AB88D44C4BAF31DDC7AD214137@multiplay.co.uk> References: <21197.39676.138433.937002@khavrinen.csail.mit.edu> <430F34AB88D44C4BAF31DDC7AD214137@multiplay.co.uk> X-Mailer: VM 7.17 under 21.4 (patch 22) "Instant Classic" XEmacs Lucid X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (khavrinen.csail.mit.edu [127.0.0.1]); Wed, 08 Jan 2014 15:48:16 -0500 (EST) Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Jan 2014 20:48:17 -0000 < said: > It should only do this if your drive reports it supports > ATA_TRIM by setting the ATA_SUPPORT_DSM_TRIM bit in the > response to an ATA_IDENTIFY command sent to it using ATA > passthrough. > What does the following report: > camcontrol identify da12 It says: pass13: ATA-8 SATA 3.x device pass13: 600.000MB/s transfers, Command Queueing Enabled protocol ATA/ATAPI-8 SATA 3.x device model TALOS2 firmware revision 2.25 serial number A179E011337000251 WWN 5e83a97a101b6024 cylinders 16383 heads 16 sectors/track 63 sector size logical 512, physical 512, offset 0 LBA supported 468883199 sectors LBA48 supported 468883199 sectors PIO supported PIO4 DMA supported WDMA2 UDMA6 media RPM non-rotating Feature Support Enabled Value Vendor read ahead yes no write cache yes no flush cache yes yes overlap no Tagged Command Queuing (TCQ) no no Native Command Queuing (NCQ) yes 32 tags SMART yes yes microcode download yes yes security yes no power management yes yes advanced power management yes yes 254/0xFE automatic acoustic management no no media status notification no no power-up in Standby yes no write-read-verify yes no 0/0x0 unload yes yes free-fall no no Data Set Management (DSM/TRIM) yes DSM - max 512byte blocks yes 1 DSM - deterministic read yes any value Host Protected Area (HPA) yes no 468883199/1 HPA - Security no A very similar device (firmware 2.15 instead of 2.25) in a 9.1 system returns no results to this command. Not sure if that's 9.1/9.2 difference of a firmware difference. -GAWollman From owner-freebsd-scsi@FreeBSD.ORG Wed Jan 8 21:16:04 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3BA6AC53 for ; Wed, 8 Jan 2014 21:16:04 +0000 (UTC) Received: from smtp.infotech.no (smtp.infotech.no [82.134.31.41]) by mx1.freebsd.org (Postfix) with ESMTP id B5BC212DA for ; Wed, 8 Jan 2014 21:16:03 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp.infotech.no (Postfix) with ESMTP id B072C2041CC; Wed, 8 Jan 2014 22:08:25 +0100 (CET) X-Virus-Scanned: by amavisd-new-2.6.6 (20110518) (Debian) at infotech.no Received: from smtp.infotech.no ([127.0.0.1]) by localhost (smtp.infotech.no [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ZYYO53AdLlhy; Wed, 8 Jan 2014 22:08:23 +0100 (CET) Received: from [192.168.48.66] (unknown [216.99.48.99]) by smtp.infotech.no (Postfix) with ESMTPA id 5EB74204162; Wed, 8 Jan 2014 22:08:21 +0100 (CET) Message-ID: <52CDBE2F.5030500@interlog.com> Date: Wed, 08 Jan 2014 16:07:59 -0500 From: Douglas Gilbert User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: Steven Hartland , Garrett Wollman , freebsd-scsi@freebsd.org Subject: Re: Attempting ATA TRIM on SAS devices? References: <21197.39676.138433.937002@khavrinen.csail.mit.edu> <430F34AB88D44C4BAF31DDC7AD214137@multiplay.co.uk> In-Reply-To: <430F34AB88D44C4BAF31DDC7AD214137@multiplay.co.uk> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list Reply-To: dgilbert@interlog.com List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Jan 2014 21:16:04 -0000 A SATA disk with a SCSI to ATA Translation Layer (SATL) in front of it can look like a SCSI disk. If the SATL is well written such as the ones found in LSI SAS HBAs then VPD page 0x89 ("ATA Information") will be present. Rather than try to read that VPD page, just try to read VPD page 0x0 which is the "Supported VPD pages" page. Then look for 0x89 in the response. Also any "real" SCSI disk should not support either ATA COMMAND PASS THROUGH command (12 or 16 byte). And they really should yield sense data of ILLEGAL REQUEST, Invalid command operation code (0x20,0). Below the log reports "Invalid field in CDB" (0x24,0) which many real SCSI devices and SATLs sloppily report. Also any device that reports such an error should also report the "field pointer sense key specific information". That pinpoints _which_ field the device doesn't like. In the unlikely event that the device does the "right thing" and reports that, FreeBSD should output that in its log. There is no sign of that below. Also if it is a SCSI disk then ATA's trim corresponds to either the SCSI UNMAP command or WRITE SAME command with the UNMAP bit set. Also ATA's trim command is broken with respect to SATA's NCQ (queuing) **, no such problem with SCSI disks. Doug Gilbert ** I think a fix for that is in the pipeline. On 14-01-08 03:36 PM, Steven Hartland wrote: > It should only do this if your drive reports it supports > ATA_TRIM by setting the ATA_SUPPORT_DSM_TRIM bit in the > response to an ATA_IDENTIFY command sent to it using ATA > passthrough. > > What does the following report: > camcontrol identify da12 > > Regards > Steve > > ----- Original Message ----- From: "Garrett Wollman" > To: > Sent: Wednesday, January 08, 2014 6:37 PM > Subject: Attempting ATA TRIM on SAS devices? > > >> On a freshly-installed 9.2 system, I see the following: >> >> (da12:mps0:0:59:0): ATA TRIM failed, switching to UNMAP BIO_DELETE >> (da12:mps0:0:59:0): ATA COMMAND PASS THROUGH(16). CDB: 85 0d 06 00 01 00 01 00 >> 00 00 00 00 00 40 06 00 (da12:mps0:0:59:0): CAM status: SCSI Status Error >> (da12:mps0:0:59:0): SCSI status: Check Condition >> (da12:mps0:0:59:0): SCSI sense: ILLEGAL REQUEST asc:24,0 (Invalid field in CDB) >> (da12:mps0:0:59:0): Error 22, Unretryable error >> (da35:mps0:0:82:0): ATA TRIM failed, switching to UNMAP BIO_DELETE >> (da35:mps0:0:82:0): ATA COMMAND PASS THROUGH(16). CDB: 85 0d 06 00 01 00 01 00 >> 00 00 00 00 00 40 06 00 (da35:mps0:0:82:0): CAM status: SCSI Status Error >> (da35:mps0:0:82:0): SCSI status: Check Condition >> (da35:mps0:0:82:0): SCSI sense: ILLEGAL REQUEST asc:24,0 (Invalid field in CDB) >> (da35:mps0:0:82:0): Error 22, Unretryable error >> >> da12 and da35 are identified as follows: >> >> [root@nfs-prod-3 /export]# camcontrol inquiry da12 >> pass13: Fixed Direct Access SCSI-5 device pass13: Serial >> Number A179E011337000251 pass13: 600.000MB/s transfers, Command Queueing Enabled >> [root@nfs-prod-3 /export]# camcontrol inquiry da35 >> pass36: Fixed Direct Access SCSI-5 device pass36: Serial >> Number A179E011337000234 pass36: 600.000MB/s transfers, Command Queueing Enabled >> >> Why would it be trying ATA commands here? These are SAS devices. >> >> -GAWollman >> _______________________________________________ >> freebsd-scsi@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-scsi >> To unsubscribe, send any mail to "freebsd-scsi-unsubscribe@freebsd.org" >> > > ================================================ > This e.mail is private and confidential between Multiplay (UK) Ltd. and the > person or entity to whom it is addressed. In the event of misdirection, the > recipient is prohibited from using, copying, printing or otherwise disseminating > it or any information contained in it. > In the event of misdirection, illegible or incomplete transmission please > telephone +44 845 868 1337 > or return the E.mail to postmaster@multiplay.co.uk. > > _______________________________________________ > freebsd-scsi@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-scsi > To unsubscribe, send any mail to "freebsd-scsi-unsubscribe@freebsd.org" > From owner-freebsd-scsi@FreeBSD.ORG Wed Jan 8 21:27:26 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A6006125 for ; Wed, 8 Jan 2014 21:27:26 +0000 (UTC) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 40A2D139F for ; Wed, 8 Jan 2014 21:27:25 +0000 (UTC) Received: from r2d2 ([82.69.141.170]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50007476548.msg for ; Wed, 08 Jan 2014 21:27:23 +0000 X-Spam-Processed: mail1.multiplay.co.uk, Wed, 08 Jan 2014 21:27:23 +0000 (not processed: message from valid local sender) X-MDDKIM-Result: neutral (mail1.multiplay.co.uk) X-MDRemoteIP: 82.69.141.170 X-Return-Path: prvs=10857851f7=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-scsi@freebsd.org Message-ID: From: "Steven Hartland" To: "Garrett Wollman" References: <21197.39676.138433.937002@khavrinen.csail.mit.edu><430F34AB88D44C4BAF31DDC7AD214137@multiplay.co.uk> <21197.47504.93295.771468@khavrinen.csail.mit.edu> Subject: Re: Attempting ATA TRIM on SAS devices? Date: Wed, 8 Jan 2014 21:27:34 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Jan 2014 21:27:26 -0000 ----- Original Message ----- From: "Garrett Wollman" To: "Steven Hartland" Cc: Sent: Wednesday, January 08, 2014 8:48 PM Subject: Re: Attempting ATA TRIM on SAS devices? > < said: > >> It should only do this if your drive reports it supports >> ATA_TRIM by setting the ATA_SUPPORT_DSM_TRIM bit in the >> response to an ATA_IDENTIFY command sent to it using ATA >> passthrough. > >> What does the following report: >> camcontrol identify da12 > > It says: > > pass13: ATA-8 SATA 3.x device > pass13: 600.000MB/s transfers, Command Queueing Enabled > > protocol ATA/ATAPI-8 SATA 3.x > device model TALOS2 > firmware revision 2.25 > serial number A179E011337000251 > WWN 5e83a97a101b6024 > cylinders 16383 > heads 16 > sectors/track 63 > sector size logical 512, physical 512, offset 0 > LBA supported 468883199 sectors > LBA48 supported 468883199 sectors > PIO supported PIO4 > DMA supported WDMA2 UDMA6 > media RPM non-rotating > > Feature Support Enabled Value Vendor > read ahead yes no > write cache yes no > flush cache yes yes > overlap no > Tagged Command Queuing (TCQ) no no > Native Command Queuing (NCQ) yes 32 tags > SMART yes yes > microcode download yes yes > security yes no > power management yes yes > advanced power management yes yes 254/0xFE > automatic acoustic management no no > media status notification no no > power-up in Standby yes no > write-read-verify yes no 0/0x0 > unload yes yes > free-fall no no > Data Set Management (DSM/TRIM) yes > DSM - max 512byte blocks yes 1 > DSM - deterministic read yes any value > Host Protected Area (HPA) yes no 468883199/1 > HPA - Security no > > A very similar device (firmware 2.15 instead of 2.25) in a 9.1 system > returns no results to this command. Not sure if that's 9.1/9.2 > difference of a firmware difference. That will be 9.1 -> 9.2 differences. So as suspected your drive is reporting it supports ATA_TRIM but then refusing the command. Now the question is why? The most likely cause is a firmware bug. I know its spec says its SAS but can you plug this drive into a SATA controller such as on board Intel controller? If so does work and then it correctly support delete's? Note: The CAM SCSI layer currently prefers ATA_TRIM over UNMAP as in all tests I've done it performs better. Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-scsi@FreeBSD.ORG Wed Jan 8 23:17:58 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 39119108 for ; Wed, 8 Jan 2014 23:17:58 +0000 (UTC) Received: from khavrinen.csail.mit.edu (khavrinen.csail.mit.edu [IPv6:2001:470:8b2d:1e1c:21b:21ff:feb8:d7b0]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 034621B7C for ; Wed, 8 Jan 2014 23:17:57 +0000 (UTC) Received: from khavrinen.csail.mit.edu (localhost [127.0.0.1]) by khavrinen.csail.mit.edu (8.14.7/8.14.7) with ESMTP id s08NHu6U013143 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL CN=khavrinen.csail.mit.edu issuer=Client+20CA); Wed, 8 Jan 2014 18:17:56 -0500 (EST) (envelope-from wollman@khavrinen.csail.mit.edu) Received: (from wollman@localhost) by khavrinen.csail.mit.edu (8.14.7/8.14.7/Submit) id s08NHuj6013140; Wed, 8 Jan 2014 18:17:56 -0500 (EST) (envelope-from wollman) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <21197.56484.67992.739053@khavrinen.csail.mit.edu> Date: Wed, 8 Jan 2014 18:17:56 -0500 From: Garrett Wollman To: "Steven Hartland" Subject: Re: Attempting ATA TRIM on SAS devices? In-Reply-To: References: <21197.39676.138433.937002@khavrinen.csail.mit.edu> <430F34AB88D44C4BAF31DDC7AD214137@multiplay.co.uk> <21197.47504.93295.771468@khavrinen.csail.mit.edu> X-Mailer: VM 7.17 under 21.4 (patch 22) "Instant Classic" XEmacs Lucid X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (khavrinen.csail.mit.edu [127.0.0.1]); Wed, 08 Jan 2014 18:17:56 -0500 (EST) Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Jan 2014 23:17:58 -0000 < said: > I know its spec says its SAS but can you plug this drive into a > SATA controller such as on board Intel controller? Not easily; it's in a separate enclosure. It does have the standard dual-port SAS connector on it, and we are running it dual-ported (active/passive, not active/active, at least for the moment). > If so does work and then it correctly support delete's? I don't have any way to confirm this. We're just using these drives as ZFS cache, so it doesn't make a huge difference to us one way or the other. -GAWollman From owner-freebsd-scsi@FreeBSD.ORG Wed Jan 8 23:38:39 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3C04D631 for ; Wed, 8 Jan 2014 23:38:39 +0000 (UTC) Received: from khavrinen.csail.mit.edu (khavrinen.csail.mit.edu [IPv6:2001:470:8b2d:1e1c:21b:21ff:feb8:d7b0]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id B5B7C1DD3 for ; Wed, 8 Jan 2014 23:38:38 +0000 (UTC) Received: from khavrinen.csail.mit.edu (localhost [127.0.0.1]) by khavrinen.csail.mit.edu (8.14.7/8.14.7) with ESMTP id s08Ncb7U013218 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL CN=khavrinen.csail.mit.edu issuer=Client+20CA); Wed, 8 Jan 2014 18:38:37 -0500 (EST) (envelope-from wollman@khavrinen.csail.mit.edu) Received: (from wollman@localhost) by khavrinen.csail.mit.edu (8.14.7/8.14.7/Submit) id s08NcbrF013215; Wed, 8 Jan 2014 18:38:37 -0500 (EST) (envelope-from wollman) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <21197.57725.503538.512849@khavrinen.csail.mit.edu> Date: Wed, 8 Jan 2014 18:38:37 -0500 From: Garrett Wollman To: dgilbert@interlog.com Subject: Re: Attempting ATA TRIM on SAS devices? In-Reply-To: <52CDBE2F.5030500@interlog.com> References: <21197.39676.138433.937002@khavrinen.csail.mit.edu> <430F34AB88D44C4BAF31DDC7AD214137@multiplay.co.uk> <52CDBE2F.5030500@interlog.com> X-Mailer: VM 7.17 under 21.4 (patch 22) "Instant Classic" XEmacs Lucid X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (khavrinen.csail.mit.edu [127.0.0.1]); Wed, 08 Jan 2014 18:38:37 -0500 (EST) Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Jan 2014 23:38:39 -0000 < said: > A SATA disk with a SCSI to ATA Translation Layer (SATL) > in front of it can look like a SCSI disk. If the > SATL is well written such as the ones found in > LSI SAS HBAs then VPD page 0x89 ("ATA Information") will > be present. This is not a SATA disk, although the underlying flash controller may well be a SATA emulator. (I believe the hardware is mostly shared between Talos 2 and Deneva 2 platforms.) This drive reports: [root@nfs-prod-3 /export]# sg_vpd -l -p sn /dev/da12 Unit serial number VPD page: [PQual=0 Peripheral device type: disk] Unit serial number: A179E011337000251 [root@nfs-prod-3 /export]# sg_vpd -l -p di /dev/da12 Device Identification VPD page: [PQual=0 Peripheral device type: disk] Addressed logical unit: designator type: NAA, code set: Binary NAA 5, IEEE Company_id: 0xe83a97 Vendor Specific Identifier: 0xa101b6024 [0x5e83a97a101b6024] Target port: designator type: NAA, code set: Binary transport: Serial Attached SCSI Protocol (SPL-2) NAA 5, IEEE Company_id: 0xe83a97 Vendor Specific Identifier: 0xa101b6025 [0x5e83a97a101b6025] designator type: Relative target port, code set: Binary transport: Serial Attached SCSI Protocol (SPL-2) Relative target port: 0x1 Target device that contains addressed lu: designator type: NAA, code set: Binary transport: Serial Attached SCSI Protocol (SPL-2) NAA 5, IEEE Company_id: 0xe83a97 Vendor Specific Identifier: 0xa101b6024 [0x5e83a97a101b6024] [root@nfs-prod-3 /export]# sg_vpd -l -p ai /dev/da12 ATA information VPD page: [PQual=0 Peripheral device type: disk] SAT Vendor identification: OCZ SAT Product identification: LUIGI_V2_MGT SAT Product revision level: ST00 Signature (Device to host FIS): 00 34 00 50 01 01 00 00 00 00 00 00 00 01 00 00 00 4.P............. 10 00 00 00 00 .... ATA command IDENTIFY DEVICE response summary: model: TALOS2 serial number: A179E011337000251 firmware revision: 2.25 ATA command IDENTIFY DEVICE response in hex: 00 0c5a 3fff c837 0010 0000 0000 003f 0000 .Z ?. .7 .. .. .. .? .. 08 0000 0000 4131 3739 4530 3131 3333 3730 .. .. A1 79 E0 11 33 70 10 3030 3235 3120 2020 0000 0000 0004 322e 00 25 1 .. .. .. 2. 18 3235 2020 2020 5441 4c4f 5332 2020 2020 25 TA LO S2 20 2020 2020 2020 2020 2020 2020 2020 2020 28 2020 2020 2020 2020 2020 2020 2020 8010 .. 30 4000 2f00 4000 0200 0200 0007 3fff 0010 @. /. @. .. .. .. ?. .. 38 003f fc10 00fb 0110 4bb0 0df9 0000 0007 .? .. .. .. K. .. .. .. 40 0003 0078 0078 0078 0078 4200 0000 0000 .. .x .x .x .x B. .. .. 48 0000 0000 0000 001f c70e 0006 0044 0040 .. .. .. .. .. .. .D .@ 50 01fc 0110 746b 7469 6163 7409 b449 6163 .. .. tk ti ac t. .I ac 58 207f 0001 0000 00fe fffe 0000 0000 0000 . .. .. .. .. .. .. .. 60 0000 0000 0000 0000 4bb0 0df9 0000 0000 .. .. .. .. K. .. .. .. 68 0000 0001 4000 0000 5e83 a97a 101b 6024 .. .. @. .. ^. .z .. `$ 70 0000 0000 0000 0000 0000 0000 0000 401a .. .. .. .. .. .. .. @. 78 4018 0000 0000 0000 0000 0000 0000 0000 @. .. .. .. .. .. .. .. 80 0001 4f24 4e89 0400 0102 0002 000e 0000 .. O$ N. .. .. .. .. .. 88 0000 0104 0100 0000 0000 0000 0000 0000 .. .. .. .. .. .. .. .. 90 0000 0000 0000 0000 0000 0000 0000 0000 .. .. .. .. .. .. .. .. 98 0000 0000 0000 0000 0000 0000 0000 3600 .. .. .. .. .. .. .. 6. a0 0000 0000 0000 0000 0000 0000 0000 0000 .. .. .. .. .. .. .. .. a8 0000 0001 0000 0000 0000 0000 0000 0000 .. .. .. .. .. .. .. .. b0 0000 0000 0000 0000 0000 0000 0000 0000 .. .. .. .. .. .. .. .. b8 0000 0000 0000 0000 0000 0000 0000 0000 .. .. .. .. .. .. .. .. c0 0000 0000 0000 0000 0000 0000 0000 0000 .. .. .. .. .. .. .. .. c8 0000 0000 0000 0000 0000 0000 0021 0000 .. .. .. .. .. .. .! .. d0 0000 4000 0000 0000 0100 0000 0000 0000 .. @. .. .. .. .. .. .. d8 0000 0001 0000 0000 0000 0000 103f 0000 .. .. .. .. .. .. .? .. e0 0000 0000 0000 0000 0000 0000 0000 0000 .. .. .. .. .. .. .. .. e8 0000 0000 0000 0000 0000 0000 0000 0000 .. .. .. .. .. .. .. .. f0 0000 0000 0000 0000 0000 0000 0000 0000 .. .. .. .. .. .. .. .. f8 0000 0000 0000 0000 0000 0000 0000 32a5 .. .. .. .. .. .. .. 2. [root@nfs-prod-3 /export]# sg_vpd -l -p bl /dev/da12 Block limits VPD page (SBC): [PQual=0 Peripheral device type: disk] Write same no zero (WSNZ): 0 Maximum compare and write length: 0 blocks Optimal transfer length granularity: 0 blocks Maximum transfer length: 0 blocks Optimal transfer length: 0 blocks Maximum prefetch length: 0 blocks Maximum unmap LBA count: 4294967295 Maximum unmap block descriptor count: 256 Optimal unmap granularity: 32 Unmap granularity alignment valid: 0 Unmap granularity alignment: 0 Maximum write same length: 0x0 blocks [root@nfs-prod-3 /export]# sg_vpd -l -p bdc /dev/da12 Block device characteristics VPD page (SBC): [PQual=0 Peripheral device type: disk] Non-rotating medium (e.g. solid state) Product type: Not specified WABEREQ=0 WACEREQ=0 Nominal form factor not reported FUAB=0 VBULS=0 [root@nfs-prod-3 /export]# sg_vpd -l -p lbpv /dev/da12 Logical block provisioning VPD page (SBC): [PQual=0 Peripheral device type: disk] Unmap command supported (LBPU): 1 Write same (16) with unmap bit supported (LBWS): 0 Write same (10) with unmap bit supported (LBWS10): 0 Logical block provisioning read zeros (LBPRZ): 0 Anchored LBAs supported (ANC_SUP): 0 Threshold exponent: 1 Descriptor present (DP): 0 Provisioning type: 0 My older drives (with the 2.15 firmware) are identical except that they does not support VPD page 0xb2 (logical block provisioning). In contrast, a different SSD (STec ZeusRAM) supports the following VPD pages: Supported VPD pages VPD page: [PQual=0 Peripheral device type: disk] 0x00 Supported VPD pages [sv] 0x3 0x80 Unit serial number [sn] 0x83 Device identification [di] 0x86 Extended inquiry data [ei] 0x87 Mode page policy [mpp] 0x88 SCSI ports [sp] 0xb1 Block device characteristics (SBC) [bdc] 0xc0 0xdb (However, sg_vpd returns "page length too short" for 0xb1.) -GAWollman From owner-freebsd-scsi@FreeBSD.ORG Wed Jan 8 23:57:51 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C6E01A05 for ; Wed, 8 Jan 2014 23:57:51 +0000 (UTC) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 60E8D1F03 for ; Wed, 8 Jan 2014 23:57:50 +0000 (UTC) Received: from r2d2 ([82.69.141.170]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50007478206.msg for ; Wed, 08 Jan 2014 23:57:48 +0000 X-Spam-Processed: mail1.multiplay.co.uk, Wed, 08 Jan 2014 23:57:48 +0000 (not processed: message from valid local sender) X-MDDKIM-Result: neutral (mail1.multiplay.co.uk) X-MDRemoteIP: 82.69.141.170 X-Return-Path: prvs=10857851f7=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-scsi@freebsd.org Message-ID: From: "Steven Hartland" To: "Garrett Wollman" References: <21197.39676.138433.937002@khavrinen.csail.mit.edu><430F34AB88D44C4BAF31DDC7AD214137@multiplay.co.uk><21197.47504.93295.771468@khavrinen.csail.mit.edu> <21197.56484.67992.739053@khavrinen.csail.mit.edu> Subject: Re: Attempting ATA TRIM on SAS devices? Date: Wed, 8 Jan 2014 23:57:57 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Jan 2014 23:57:51 -0000 ----- Original Message ----- From: "Garrett Wollman" > < said: > >> I know its spec says its SAS but can you plug this drive into a >> SATA controller such as on board Intel controller? > > Not easily; it's in a separate enclosure. It does have the standard > dual-port SAS connector on it, and we are running it dual-ported > (active/passive, not active/active, at least for the moment). > >> If so does work and then it correctly support delete's? > > I don't have any way to confirm this. We're just using these drives > as ZFS cache, so it doesn't make a huge difference to us one way or > the other. You could just force the delete_method to UNMAP then: kern.cam.da.X.delete_method=UNMAP Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-scsi@FreeBSD.ORG Thu Jan 9 00:05:44 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9E420C00 for ; Thu, 9 Jan 2014 00:05:44 +0000 (UTC) Received: from khavrinen.csail.mit.edu (khavrinen.csail.mit.edu [IPv6:2001:470:8b2d:1e1c:21b:21ff:feb8:d7b0]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 4FCEF1F9B for ; Thu, 9 Jan 2014 00:05:44 +0000 (UTC) Received: from khavrinen.csail.mit.edu (localhost [127.0.0.1]) by khavrinen.csail.mit.edu (8.14.7/8.14.7) with ESMTP id s0905gIJ013489 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL CN=khavrinen.csail.mit.edu issuer=Client+20CA); Wed, 8 Jan 2014 19:05:42 -0500 (EST) (envelope-from wollman@khavrinen.csail.mit.edu) Received: (from wollman@localhost) by khavrinen.csail.mit.edu (8.14.7/8.14.7/Submit) id s0905gua013486; Wed, 8 Jan 2014 19:05:42 -0500 (EST) (envelope-from wollman) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <21197.59350.752519.254639@khavrinen.csail.mit.edu> Date: Wed, 8 Jan 2014 19:05:42 -0500 From: Garrett Wollman To: "Steven Hartland" Subject: Re: Attempting ATA TRIM on SAS devices? In-Reply-To: References: <21197.39676.138433.937002@khavrinen.csail.mit.edu> <430F34AB88D44C4BAF31DDC7AD214137@multiplay.co.uk> <21197.47504.93295.771468@khavrinen.csail.mit.edu> <21197.56484.67992.739053@khavrinen.csail.mit.edu> X-Mailer: VM 7.17 under 21.4 (patch 22) "Instant Classic" XEmacs Lucid X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (khavrinen.csail.mit.edu [127.0.0.1]); Wed, 08 Jan 2014 19:05:42 -0500 (EST) Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Jan 2014 00:05:44 -0000 < said: > You could just force the delete_method to UNMAP then: > kern.cam.da.X.delete_method=UNMAP I'm not sure that that works, given that the zpool is auto-imported at boot time and I don't know which da units will be assigned to the SSDs in question (no way to pin them so far as I can tell). Does ZFS generate BIO_DELETE on the cache devices at any time other than pool import? -GAWollman From owner-freebsd-scsi@FreeBSD.ORG Thu Jan 9 00:13:54 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 52BE3171 for ; Thu, 9 Jan 2014 00:13:54 +0000 (UTC) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id E08311077 for ; Thu, 9 Jan 2014 00:13:53 +0000 (UTC) Received: from r2d2 ([82.69.141.170]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50007478370.msg for ; Thu, 09 Jan 2014 00:13:50 +0000 X-Spam-Processed: mail1.multiplay.co.uk, Thu, 09 Jan 2014 00:13:50 +0000 (not processed: message from valid local sender) X-MDDKIM-Result: neutral (mail1.multiplay.co.uk) X-MDRemoteIP: 82.69.141.170 X-Return-Path: prvs=1086c5d11c=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-scsi@freebsd.org Message-ID: From: "Steven Hartland" To: "Garrett Wollman" References: <21197.39676.138433.937002@khavrinen.csail.mit.edu><430F34AB88D44C4BAF31DDC7AD214137@multiplay.co.uk><21197.47504.93295.771468@khavrinen.csail.mit.edu><21197.56484.67992.739053@khavrinen.csail.mit.edu> <21197.59350.752519.254639@khavrinen.csail.mit.edu> Subject: Re: Attempting ATA TRIM on SAS devices? Date: Thu, 9 Jan 2014 00:14:02 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Jan 2014 00:13:54 -0000 ----- Original Message ----- From: "Garrett Wollman" > < said: > >> You could just force the delete_method to UNMAP then: >> kern.cam.da.X.delete_method=UNMAP > > I'm not sure that that works, given that the zpool is auto-imported at > boot time and I don't know which da units will be assigned to the SSDs > in question (no way to pin them so far as I can tell). Does ZFS > generate BIO_DELETE on the cache devices at any time other than pool > import? Yes on every delete, once a delete method has failed though it wont be tried again so the only real side effect is that the first delete will be attempted twice, so if you can live with that and cant get a fixed FW for the disk then... Regarsd Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-scsi@FreeBSD.ORG Thu Jan 9 00:25:00 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 048C4687 for ; Thu, 9 Jan 2014 00:25:00 +0000 (UTC) Received: from smtp.infotech.no (smtp.infotech.no [82.134.31.41]) by mx1.freebsd.org (Postfix) with ESMTP id 7D7E81152 for ; Thu, 9 Jan 2014 00:24:59 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp.infotech.no (Postfix) with ESMTP id CF7932041CC; Thu, 9 Jan 2014 01:24:56 +0100 (CET) X-Virus-Scanned: by amavisd-new-2.6.6 (20110518) (Debian) at infotech.no Received: from smtp.infotech.no ([127.0.0.1]) by localhost (smtp.infotech.no [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9mUybyHGQIHJ; Thu, 9 Jan 2014 01:24:54 +0100 (CET) Received: from [192.168.48.66] (unknown [216.99.48.99]) by smtp.infotech.no (Postfix) with ESMTPA id 023EA204169; Thu, 9 Jan 2014 01:24:53 +0100 (CET) Message-ID: <52CDEC3F.9000001@interlog.com> Date: Wed, 08 Jan 2014 19:24:31 -0500 From: Douglas Gilbert User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: Garrett Wollman Subject: Re: Attempting ATA TRIM on SAS devices? References: <21197.39676.138433.937002@khavrinen.csail.mit.edu> <430F34AB88D44C4BAF31DDC7AD214137@multiplay.co.uk> <52CDBE2F.5030500@interlog.com> <21197.57725.503538.512849@khavrinen.csail.mit.edu> In-Reply-To: <21197.57725.503538.512849@khavrinen.csail.mit.edu> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list Reply-To: dgilbert@interlog.com List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Jan 2014 00:25:00 -0000 On 14-01-08 06:38 PM, Garrett Wollman wrote: > < said: > >> A SATA disk with a SCSI to ATA Translation Layer (SATL) >> in front of it can look like a SCSI disk. If the >> SATL is well written such as the ones found in >> LSI SAS HBAs then VPD page 0x89 ("ATA Information") will >> be present. > > This is not a SATA disk, although the underlying flash controller may > well be a SATA emulator. (I believe the hardware is mostly shared > between Talos 2 and Deneva 2 platforms.) Interesting. > This drive reports: > > [root@nfs-prod-3 /export]# sg_vpd -l -p sn /dev/da12 > Unit serial number VPD page: > [PQual=0 Peripheral device type: disk] > Unit serial number: A179E011337000251 > [root@nfs-prod-3 /export]# sg_vpd -l -p di /dev/da12 > Device Identification VPD page: > [PQual=0 Peripheral device type: disk] > Addressed logical unit: > designator type: NAA, code set: Binary > NAA 5, IEEE Company_id: 0xe83a97 IEEE OUI database reports: E8-3A-97 (hex) OCZ Technology Group > Vendor Specific Identifier: 0xa101b6024 > [0x5e83a97a101b6024] > Target port: > designator type: NAA, code set: Binary > transport: Serial Attached SCSI Protocol (SPL-2) > NAA 5, IEEE Company_id: 0xe83a97 > Vendor Specific Identifier: 0xa101b6025 > [0x5e83a97a101b6025] > designator type: Relative target port, code set: Binary > transport: Serial Attached SCSI Protocol (SPL-2) > Relative target port: 0x1 > Target device that contains addressed lu: > designator type: NAA, code set: Binary > transport: Serial Attached SCSI Protocol (SPL-2) > NAA 5, IEEE Company_id: 0xe83a97 > Vendor Specific Identifier: 0xa101b6024 > [0x5e83a97a101b6024] Target device NAA and LU NAA are the same. That is a no-no in SAS. Sloppy. > [root@nfs-prod-3 /export]# sg_vpd -l -p ai /dev/da12 > ATA information VPD page: > [PQual=0 Peripheral device type: disk] > SAT Vendor identification: OCZ > SAT Product identification: LUIGI_V2_MGT > SAT Product revision level: ST00 > Signature (Device to host FIS): > 00 34 00 50 01 01 00 00 00 00 00 00 00 01 00 00 00 4.P............. > 10 00 00 00 00 .... > ATA command IDENTIFY DEVICE response summary: > model: TALOS2 > serial number: A179E011337000251 > firmware revision: 2.25 > ATA command IDENTIFY DEVICE response in hex: > 00 0c5a 3fff c837 0010 0000 0000 003f 0000 .Z ?. .7 .. .. .. .? .. > 08 0000 0000 4131 3739 4530 3131 3333 3730 .. .. A1 79 E0 11 33 70 > 10 3030 3235 3120 2020 0000 0000 0004 322e 00 25 1 .. .. .. 2. > 18 3235 2020 2020 5441 4c4f 5332 2020 2020 25 TA LO S2 > 20 2020 2020 2020 2020 2020 2020 2020 2020 > 28 2020 2020 2020 2020 2020 2020 2020 8010 .. > 30 4000 2f00 4000 0200 0200 0007 3fff 0010 @. /. @. .. .. .. ?. .. > 38 003f fc10 00fb 0110 4bb0 0df9 0000 0007 .? .. .. .. K. .. .. .. > 40 0003 0078 0078 0078 0078 4200 0000 0000 .. .x .x .x .x B. .. .. > 48 0000 0000 0000 001f c70e 0006 0044 0040 .. .. .. .. .. .. .D .@ > 50 01fc 0110 746b 7469 6163 7409 b449 6163 .. .. tk ti ac t. .I ac > 58 207f 0001 0000 00fe fffe 0000 0000 0000 . .. .. .. .. .. .. .. > 60 0000 0000 0000 0000 4bb0 0df9 0000 0000 .. .. .. .. K. .. .. .. > 68 0000 0001 4000 0000 5e83 a97a 101b 6024 .. .. @. .. ^. .z .. `$ > 70 0000 0000 0000 0000 0000 0000 0000 401a .. .. .. .. .. .. .. @. > 78 4018 0000 0000 0000 0000 0000 0000 0000 @. .. .. .. .. .. .. .. > 80 0001 4f24 4e89 0400 0102 0002 000e 0000 .. O$ N. .. .. .. .. .. > 88 0000 0104 0100 0000 0000 0000 0000 0000 .. .. .. .. .. .. .. .. > 90 0000 0000 0000 0000 0000 0000 0000 0000 .. .. .. .. .. .. .. .. > 98 0000 0000 0000 0000 0000 0000 0000 3600 .. .. .. .. .. .. .. 6. > a0 0000 0000 0000 0000 0000 0000 0000 0000 .. .. .. .. .. .. .. .. > a8 0000 0001 0000 0000 0000 0000 0000 0000 .. .. .. .. .. .. .. .. > b0 0000 0000 0000 0000 0000 0000 0000 0000 .. .. .. .. .. .. .. .. > b8 0000 0000 0000 0000 0000 0000 0000 0000 .. .. .. .. .. .. .. .. > c0 0000 0000 0000 0000 0000 0000 0000 0000 .. .. .. .. .. .. .. .. > c8 0000 0000 0000 0000 0000 0000 0021 0000 .. .. .. .. .. .. .! .. > d0 0000 4000 0000 0000 0100 0000 0000 0000 .. @. .. .. .. .. .. .. > d8 0000 0001 0000 0000 0000 0000 103f 0000 .. .. .. .. .. .. .? .. > e0 0000 0000 0000 0000 0000 0000 0000 0000 .. .. .. .. .. .. .. .. > e8 0000 0000 0000 0000 0000 0000 0000 0000 .. .. .. .. .. .. .. .. > f0 0000 0000 0000 0000 0000 0000 0000 0000 .. .. .. .. .. .. .. .. > f8 0000 0000 0000 0000 0000 0000 0000 32a5 .. .. .. .. .. .. .. 2. There is the WWN on line 0x68: 5e83 a97a 101b 6024 > [root@nfs-prod-3 /export]# sg_vpd -l -p bl /dev/da12 > Block limits VPD page (SBC): > [PQual=0 Peripheral device type: disk] > Write same no zero (WSNZ): 0 > Maximum compare and write length: 0 blocks > Optimal transfer length granularity: 0 blocks > Maximum transfer length: 0 blocks > Optimal transfer length: 0 blocks > Maximum prefetch length: 0 blocks > Maximum unmap LBA count: 4294967295 > Maximum unmap block descriptor count: 256 > Optimal unmap granularity: 32 > Unmap granularity alignment valid: 0 > Unmap granularity alignment: 0 > Maximum write same length: 0x0 blocks Many of those "0 blocks" should probably read "unlimited" Might look at that. > [root@nfs-prod-3 /export]# sg_vpd -l -p bdc /dev/da12 > Block device characteristics VPD page (SBC): > [PQual=0 Peripheral device type: disk] > Non-rotating medium (e.g. solid state) > Product type: Not specified > WABEREQ=0 > WACEREQ=0 > Nominal form factor not reported > FUAB=0 > VBULS=0 > [root@nfs-prod-3 /export]# sg_vpd -l -p lbpv /dev/da12 > Logical block provisioning VPD page (SBC): > [PQual=0 Peripheral device type: disk] > Unmap command supported (LBPU): 1 > Write same (16) with unmap bit supported (LBWS): 0 > Write same (10) with unmap bit supported (LBWS10): 0 > Logical block provisioning read zeros (LBPRZ): 0 > Anchored LBAs supported (ANC_SUP): 0 > Threshold exponent: 1 > Descriptor present (DP): 0 > Provisioning type: 0 Grrr, LBPRZ=0 which implies read rubbish from trimmed (unmapped) blocks. > My older drives (with the 2.15 firmware) are identical except that > they does not support VPD page 0xb2 (logical block provisioning). So it looks like they have put a SAS target chip directly in front of what is otherwise a SATA SSD. > In contrast, a different SSD (STec ZeusRAM) supports the following VPD > pages: > Supported VPD pages VPD page: > [PQual=0 Peripheral device type: disk] > 0x00 Supported VPD pages [sv] > 0x3 > 0x80 Unit serial number [sn] > 0x83 Device identification [di] > 0x86 Extended inquiry data [ei] > 0x87 Mode page policy [mpp] > 0x88 SCSI ports [sp] > 0xb1 Block device characteristics (SBC) [bdc] > 0xc0 > 0xdb > > (However, sg_vpd returns "page length too short" for 0xb1.) sg_vpd -H -vvv -p 0xb1 might show what is returned (in hex). At least Hitachi is breaking away from these silly games with a 12 Gbps SAS-3 SSD whose maximum read rate is around 1100 MB/sec. SATA won't be going to 12 Gbps, SAS is already there and looking at 24 Gbps (or maybe 20 Gbps ...). A SAS-3 SSD could theoretically read 2400 MB/sec aggregate with its two phys doing separate READs on its 2-phy, wide port. So it's between PCIe (SOP ?) and SAS for high end SSDs. Doug Gilbert From owner-freebsd-scsi@FreeBSD.ORG Thu Jan 9 07:37:03 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CD1855ED for ; Thu, 9 Jan 2014 07:37:03 +0000 (UTC) Received: from mail-ie0-x22a.google.com (mail-ie0-x22a.google.com [IPv6:2607:f8b0:4001:c03::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 9888C1628 for ; Thu, 9 Jan 2014 07:37:03 +0000 (UTC) Received: by mail-ie0-f170.google.com with SMTP id tq11so2456341ieb.1 for ; Wed, 08 Jan 2014 23:37:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type:content-transfer-encoding; bh=r2NUFq/YEoDlwOqEie7qEQOKrIHbLjJEugIVLc9Mptg=; b=cCMKg40qWqkSGo0nMBgENgSIm5G/ipERZxk+SUGIwpufI48AvckxG6mX4q/Fz7gA40 rDNyViHFnAi+8T+h/xsYdtstFSG0e/m283RY55wUHM/KDUsN9w8jlLyXTHOcpIXNWZm8 RxJdTQTLKWCS5dQovfuQJBLJj1YWvOuhWrEQkywWmLa7m0If4LnFhYlE5WcMXRduULs/ UBp8Da3yZZ5mzu7F8PoNwO8BzJP9hMDJknt2tJGb1GtBrPiw7l9pXHoYh/sQoNj1Bn2+ /59swc0pse3Rzgc7tCvq9g4mhWk07V8ReiPDZl3s6OWkwfG9O6hBQ+oc0hXZKltplYfN N+Nw== MIME-Version: 1.0 X-Received: by 10.43.74.198 with SMTP id yx6mr1161315icb.40.1389253023064; Wed, 08 Jan 2014 23:37:03 -0800 (PST) Sender: benlaurie@gmail.com Received: by 10.64.225.103 with HTTP; Wed, 8 Jan 2014 23:37:02 -0800 (PST) In-Reply-To: References: <84D23688-DDC6-421E-9D21-3DA646229038@scsiguy.com> Date: Thu, 9 Jan 2014 07:37:02 +0000 X-Google-Sender-Auth: rQ5jM0A_6Ad3du4O9LMe01tC9uY Message-ID: Subject: Re: Dropped interrupts From: Ben Laurie To: "Justin T. Gibbs" Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Jan 2014 07:37:04 -0000 On 8 January 2014 06:44, Ben Laurie wrote: > On 7 January 2014 18:11, Justin T. Gibbs wrote: >> On Jan 7, 2014, at 12:36 AM, Ben Laurie wrote: >> >>> Attached. >>> >>> On 7 January 2014 05:46, Justin T. Gibbs wrote: >>>> On Jan 6, 2014, at 3:01 PM, Ben Laurie wrote: >>>> >>>>> Not subscribed to the list, so please cc on replies. >>>>> >>>>> I'm using Bacula with an LTO-2 SCSI drive. >>>>> >>>>> With increasing frequency lately, I've been getting errors like this >>>>> from bacula: >>>>> >>>>> backup-sd JobId 13092: Error: block.c:608 Write error at 23:6772 on >>>>> device "Ultrium" (/dev/nsa0). ERR=3DOperation not permitted. >>>>> >>>>> Associated with this, I see in dmesg: >>>>> >>>>> ahc0: Recovery Initiated >>>>> >>>>> [a lot of dump info, including=85] >>>> >>>> If you provide the dump info, I may be able to tell you why recovery i= s starting. >>>> >>>> The dmesg information from a boot of the system would be good to have = too. >>>> >>>> =97 >>>> Justin >> >> The target is keeping us in command phase for some reason. No parity or= other >> errors are being reported. My guess is that the tape drive does not lik= e the command >> that was issued for some reason. >> >> Attached are two totally untested/uncompiled changes for you to try out.= The first >> should give more information about the command that timed out so we can = better >> determine if it is well formed. The second is an attempted fix for spur= ious >> =93Interrupts may not be functioning=94 warnings. Can you attempt to re= plicate this >> again with these changes? > > Rebuilding now - you had a ; missing in the patch :-) > > Of course, now I've done this, it'll not fail for a month (its been > failing multiple times per day recently, but on average its a lot > rarer than that!). > > Will let you know when I get a fresh failure. As predicted, it has now done 3 complete tapes with no problems, and is on the fourth. From owner-freebsd-scsi@FreeBSD.ORG Thu Jan 9 18:38:07 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BFE66471 for ; Thu, 9 Jan 2014 18:38:07 +0000 (UTC) Received: from khavrinen.csail.mit.edu (khavrinen.csail.mit.edu [IPv6:2001:470:8b2d:1e1c:21b:21ff:feb8:d7b0]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 8987D118F for ; Thu, 9 Jan 2014 18:38:07 +0000 (UTC) Received: from khavrinen.csail.mit.edu (localhost [127.0.0.1]) by khavrinen.csail.mit.edu (8.14.7/8.14.7) with ESMTP id s09Ic5sa018491 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL CN=khavrinen.csail.mit.edu issuer=Client+20CA); Thu, 9 Jan 2014 13:38:05 -0500 (EST) (envelope-from wollman@khavrinen.csail.mit.edu) Received: (from wollman@localhost) by khavrinen.csail.mit.edu (8.14.7/8.14.7/Submit) id s09Ic5DQ018488; Thu, 9 Jan 2014 13:38:05 -0500 (EST) (envelope-from wollman) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <21198.60557.126129.808095@khavrinen.csail.mit.edu> Date: Thu, 9 Jan 2014 13:38:05 -0500 From: Garrett Wollman To: "Steven Hartland" Subject: Re: Attempting ATA TRIM on SAS devices? In-Reply-To: References: <21197.39676.138433.937002@khavrinen.csail.mit.edu> <430F34AB88D44C4BAF31DDC7AD214137@multiplay.co.uk> <21197.47504.93295.771468@khavrinen.csail.mit.edu> <21197.56484.67992.739053@khavrinen.csail.mit.edu> <21197.59350.752519.254639@khavrinen.csail.mit.edu> X-Mailer: VM 7.17 under 21.4 (patch 22) "Instant Classic" XEmacs Lucid X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (khavrinen.csail.mit.edu [127.0.0.1]); Thu, 09 Jan 2014 13:38:05 -0500 (EST) Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Jan 2014 18:38:07 -0000 < said: >> I'm not sure that that works, given that the zpool is auto-imported at >> boot time and I don't know which da units will be assigned to the SSDs >> in question (no way to pin them so far as I can tell). Does ZFS >> generate BIO_DELETE on the cache devices at any time other than pool >> import? > Yes on every delete, What do you mean by "on every delete"? Do you mean whenever a block is evicted from the L2ARC? -GAWollman From owner-freebsd-scsi@FreeBSD.ORG Thu Jan 9 19:05:20 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 56CD61F8 for ; Thu, 9 Jan 2014 19:05:20 +0000 (UTC) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id E139813E2 for ; Thu, 9 Jan 2014 19:05:19 +0000 (UTC) Received: from r2d2 ([82.69.141.170]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50007489945.msg for ; Thu, 09 Jan 2014 19:05:15 +0000 X-Spam-Processed: mail1.multiplay.co.uk, Thu, 09 Jan 2014 19:05:15 +0000 (not processed: message from valid local sender) X-MDDKIM-Result: neutral (mail1.multiplay.co.uk) X-MDRemoteIP: 82.69.141.170 X-Return-Path: prvs=1086c5d11c=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-scsi@freebsd.org Message-ID: <8A5A4C9BC59E434289AF83B025F3380A@multiplay.co.uk> From: "Steven Hartland" To: "Garrett Wollman" References: <21197.39676.138433.937002@khavrinen.csail.mit.edu><430F34AB88D44C4BAF31DDC7AD214137@multiplay.co.uk><21197.47504.93295.771468@khavrinen.csail.mit.edu><21197.56484.67992.739053@khavrinen.csail.mit.edu><21197.59350.752519.254639@khavrinen.csail.mit.edu> <21198.60557.126129.808095@khavrinen.csail.mit.edu> Subject: Re: Attempting ATA TRIM on SAS devices? Date: Thu, 9 Jan 2014 19:05:10 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Jan 2014 19:05:20 -0000 ----- Original Message ----- From: "Garrett Wollman" > < said: > >>> I'm not sure that that works, given that the zpool is auto-imported at >>> boot time and I don't know which da units will be assigned to the SSDs >>> in question (no way to pin them so far as I can tell). Does ZFS >>> generate BIO_DELETE on the cache devices at any time other than pool >>> import? > >> Yes on every delete, > > What do you mean by "on every delete"? Do you mean whenever a block > is evicted from the L2ARC? Only when space is freed, as a trim map is maintained and processed after a period, so if the same block has already been reused it will have been removed from the map and the trim doesn't take place. This means L2ARC is processed slightly differently from normal devices as ZFS tries to keep it as full as possible. Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk.