From owner-freebsd-scsi@freebsd.org Sun Feb 28 20:21:13 2016 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 75714AB6B91 for ; Sun, 28 Feb 2016 20:21:13 +0000 (UTC) (envelope-from dgilbert@interlog.com) Received: from smtp.infotech.no (smtp.infotech.no [82.134.31.41]) by mx1.freebsd.org (Postfix) with ESMTP id 041A01797 for ; Sun, 28 Feb 2016 20:21:12 +0000 (UTC) (envelope-from dgilbert@interlog.com) Received: from localhost (localhost [127.0.0.1]) by smtp.infotech.no (Postfix) with ESMTP id 2B90C20418E for ; Sun, 28 Feb 2016 21:11:57 +0100 (CET) X-Virus-Scanned: by amavisd-new-2.6.6 (20110518) (Debian) at infotech.no Received: from smtp.infotech.no ([127.0.0.1]) by localhost (smtp.infotech.no [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ngBKHNrGorIB for ; Sun, 28 Feb 2016 21:11:54 +0100 (CET) Received: from [192.168.48.86] (host-45-58-210-241.dyn.295.ca [45.58.210.241]) by smtp.infotech.no (Postfix) with ESMTPA id 8A2AE204165 for ; Sun, 28 Feb 2016 21:11:54 +0100 (CET) Reply-To: dgilbert@interlog.com To: freebsd-scsi@freebsd.org From: Douglas Gilbert Subject: [Announce] sg3_utils 1.42 and sdparm 1.10 Message-ID: <56D35488.9070507@interlog.com> Date: Sun, 28 Feb 2016 15:11:52 -0500 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.5.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 28 Feb 2016 20:21:13 -0000 I recently released these two SCSI support packages, information and downloads on these pages: http://sg.danny.cz/sg/sg3_utils.html http://sg.danny.cz/sg/sdparm.html Recent betas of some of my other packages (e.g. smp_utils and ddpt) have links at the top of this page: http://sg.danny.cz/sg/index.html Note that lsscsi is Linux only and is roughly equivalent to 'camcontrol devlist'. Changelog for sg3_utils-1.42 [20160217] [svn: r663] - sg_timestamp: new, to report or set timestamp - sg_read_attr: new, supported by tape drives - sg_stpg: fix truncation of target port field - sg_inq: cope with unicode strings, udev fixes - update version descriptor list to 20160125 - '--export': new entries for UUID descriptor - sg_ses: add more field acronyms (ses3r11) - sg_logs: add Utilization lpage (sbc4r07) - add Background operation lpage - add Pending defects lpage - add LPS misalignment lpage (sbc4r10) - document '--All' ('-A') option - rework lto tape vendor lpages - sg_vpd: add Block limits extension VPD page - add Device constituents VPD page - add LB Protection VPD page (ssc 15-296r1) - LB provisioning VPD page: expand LBPRZ, add Minimum and Threshold percentage fields - rework lto tape vendor VPD pages - sg_inq+sg_vpd+sg_xcopy: add support for locally assigned UUIDs in VPD page 0x83 (15-267r2) - sg_sanitize: add --znr option (sbc4r07) - sg_rep_zones: add --partial option (zbc-r04) - sg_format: add ffmt option (sbc4r10) - add support for FORMAT MEDIUM (for tape) - sg_raw: document length relationships - rescan-scsi-bus.sh: updates from Suse - sg_lib_data: sync asc/ascq codes with T10 20151126 - sg_lib: add 'sense' categories for SCSI statuses: condition met, busy, task set full, ACA active and task aborted - add pr2serr() extern - change sg_get_sense_str() and dStrHexStr(), return chars written (returned void previously) - add sg_get_sense_descriptors_str() function - add sg_get_designation_descriptor_str() function - sg_get_desig_type_str()+sg_get_desig_assoc_str() and sg_get_desig_code_set_str() added - sg_get_opcode_sa_name() break out zoning in/out, read attribute and read position service actions - sg_cmds_extra: add sg_ll_format_unit2() for FFMT - sg_pr2serr.h: new, to shorten fprintf(stderr, ...) - sg_io_linux, sg_pt_linux: drop SUGGEST_* decoding - sg_unaligned.h: add 48 bit support and gets for variable length unsigned integers Changelog for sg3_utils-1.41 [20150511] [svn: r644] ... ChangeLog for sdparm-1.10 [20160222] [svn: r279] - add --inhex=FN option for decoding without device present, FN is interpreted as response to mode sense(10) command - add --raw option to interpret FN as binary (def: ASCII hex) - add --pdt=PDT option for use with --inhex=FN - --quiet used twice hides changeable, default + saved - add IO advice hints grouping mode page (sbc4r06, 8) - add Unit serial number VPD page specific sanity check - add NO_PI_CHK to Supported block lengths and protection types VPD page - add Background operation control mpage (sbc4r07) - Read-write error recovery mpage: add Misaligned writes reporting field (MWR) - sync tape mpages with ssc5r02 - add Block limits extension VPD page - add Device constituents VPD page - add LB protection VPD page (ssc5r02a) - LB provisioning VPD page: expand LBPRZ, add Minimum and Threshold percentage fields - device identification VPD page: add decoding for locally assigned UUIDs (spc5r08) - the --inhex=FN option together with --inquiry decodes FN as a single VPD page - improve lto5 and lto6 vendor mpage support - sync to spc5r08 and sbc4r10 - add SAS G5 (22.5 Gbps) settings (spl4r06) - point svn:externals to rev 663 of sg3_utils - upgrade automake to version 1.15 (U15.10) - autogen.sh: upgrade to buildconf 20091223 version ChangeLog for sdparm-1.09 [20141226] [svn: r257] ... Doug Gilbert From owner-freebsd-scsi@freebsd.org Mon Feb 29 21:45:06 2016 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DDFDBAB88FB for ; Mon, 29 Feb 2016 21:45:05 +0000 (UTC) (envelope-from ken@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id CE69A1DC1 for ; Mon, 29 Feb 2016 21:45:05 +0000 (UTC) (envelope-from ken@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id CC97DAB88F7; Mon, 29 Feb 2016 21:45:05 +0000 (UTC) Delivered-To: scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CC191AB88F6; Mon, 29 Feb 2016 21:45:05 +0000 (UTC) (envelope-from ken@freebsd.org) Received: from mithlond.kdm.org (mithlond.kdm.org [96.89.93.250]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "A1-33714", Issuer "A1-33714" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 845D91DC0; Mon, 29 Feb 2016 21:45:05 +0000 (UTC) (envelope-from ken@freebsd.org) Received: from [10.0.0.27] (mbp2013-wired.int.kdm.org [10.0.0.27]) (authenticated bits=0) by mithlond.kdm.org (8.15.2/8.14.9) with ESMTPSA id u1TLj3Uq067615 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 29 Feb 2016 16:45:04 -0500 (EST) (envelope-from ken@freebsd.org) From: Ken Merry Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Subject: FUSE extended attribute patches available Date: Mon, 29 Feb 2016 16:45:03 -0500 Message-Id: Cc: scsi@freebsd.org To: fs@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 9.2 \(3112\)) X-Mailer: Apple Mail (2.3112) X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.4.3 (mithlond.kdm.org [96.89.93.250]); Mon, 29 Feb 2016 16:45:04 -0500 (EST) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Feb 2016 21:45:06 -0000 I have patches for FreeBSD=E2=80=99s FUSE filesystem kernel module to = support extended attributes: https://people.freebsd.org/~ken/fuse_extattr.20160229.1.txt The patch implements the get/set/delete/list extended attribute methods. = The listing code also converts extended attribute lists from the = Linux/FUSE format to the FreeBSD format. For example: # touch foo # ls -la foo -rwxrwxrwx 1 root wheel 0 Feb 29 21:40 foo # lsextattr user foo foo # setextattr user testattr1 "12345678" foo # lsextattr user foo foo testattr1 # getextattr user testattr1 foo foo 12345678 # setextattr user testattr2 "87654321" foo # lsextattr user foo foo testattr2 testattr1 # rmextattr user testattr1 foo # lsextattr user foo foo testattr2 # getextattr user testattr1 foo getextattr: foo: failed: Attribute not found # getextattr user testattr2 foo foo 87654321 Just to be clear on what this does, it only provides extended attribute = support to FreeBSD applications if the underlying FUSE filesystem = implements FUSE extended attribute support. Many FUSE filesystems = don=E2=80=99t support the extended attribute VFS operations. I have tested this out on IBM=E2=80=99s LTFS implementation, but I have = not yet found another FUSE filesystem that supports extended attributes. = If anyone knows of one, please let me know so I can try it out. (I = looked through a number of the filesystems in sysutils/fusefs* in the = ports tree.) Any feedback is welcome. I=E2=80=99m planning to check this into = FreeBSD/head in the next week or so. Obviously, I=E2=80=99ve also ported IBM=E2=80=99s LTFS implementation to = FreeBSD. It works in the standard FUSE mode, and you can also link it = into an application as a library if you don=E2=80=99t want to incur the = overhead of running through FUSE. I haven=E2=80=99t gotten around to = packaging it up to go out for testing / review. If anyone has IBM LTO-5 or newer tape drives, or IBM TS1140 or newer = tape drives, and wants to try it out, let me know. I=E2=80=99ll send = you the code when I=E2=80=99ve got it at least somewhat ready. This is = IBM-specific, and won=E2=80=99t work on HP tape drives. Ken =E2=80=94=20 Ken Merry ken@FreeBSD.ORG From owner-freebsd-scsi@freebsd.org Tue Mar 1 00:02:39 2016 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ECD9CAB9D76 for ; Tue, 1 Mar 2016 00:02:38 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id D0648EC9 for ; Tue, 1 Mar 2016 00:02:38 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: by mailman.ysv.freebsd.org (Postfix) id CB467AB9D74; Tue, 1 Mar 2016 00:02:38 +0000 (UTC) Delivered-To: scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B18D9AB9D72; Tue, 1 Mar 2016 00:02:38 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 26C41EC6; Tue, 1 Mar 2016 00:02:37 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) IronPort-PHdr: 9a23:xXC7FBV3sOwopxkcFA90GP1FxxXV8LGtZVwlr6E/grcLSJyIuqrYZhCBt8tkgFKBZ4jH8fUM07OQ6PC/HzJRqszY+Fk5M7VyFDY9wf0MmAIhBMPXQWbaF9XNKxIAIcJZSVV+9Gu6O0UGUOz3ZlnVv2HgpWVKQka3CwN5K6zPF5LIiIzvjqbpq8KVPVQD3mP1SIgxBSv1hD2ZjtMRj4pmJ/R54TryiVwMRd5rw3h1L0mYhRf265T41pdi9yNNp6BprJYYAu2pN5g/GJ9VCnwDPnov9YW/thTFZQWV63YWSWlQlQBHVVvr9hb/C63wuSiyk+N22y2XOIWiV7U9Ujem4qJDVRjnlSoDLz5/+2iB2Z84t75SvB/0/083+IXTeozAbPc= X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2DQAQAT29RW/61jaINeFoN2bQa6ZwENgWcXCoUoSgKBchQBAQEBAQEBAWMngi2CFAEBAQMBAQEBICsgCwULAgEIGAICDRkCAicBCSYCBAgHBAEcAgKHdggOsQKPBgEBAQEBAQEDAQEBAQEBAQEYe4UXgXSCRoQFEAIBBRaCSjgTgScFh1SGSz2IMIVZhSGERkuDeYhSjkgCHgEBQoIDGYFmHi4BAQEEh0d+AQEB X-IronPort-AV: E=Sophos;i="5.22,521,1449550800"; d="scan'208";a="268423735" Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 29 Feb 2016 19:02:30 -0500 Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 86C5F15F574; Mon, 29 Feb 2016 19:02:30 -0500 (EST) Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id uY9p308SZ875; Mon, 29 Feb 2016 19:02:29 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id E981C15F578; Mon, 29 Feb 2016 19:02:28 -0500 (EST) X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 7Nv56U13dYVU; Mon, 29 Feb 2016 19:02:28 -0500 (EST) Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id C3E3E15F574; Mon, 29 Feb 2016 19:02:28 -0500 (EST) Date: Mon, 29 Feb 2016 19:02:28 -0500 (EST) From: Rick Macklem To: Ken Merry Cc: fs@freebsd.org, scsi@freebsd.org Message-ID: <1740288370.14765302.1456790548437.JavaMail.zimbra@uoguelph.ca> In-Reply-To: References: Subject: Re: FUSE extended attribute patches available MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [172.17.95.11] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - FF44 (Win)/8.0.9_GA_6191) Thread-Topic: FUSE extended attribute patches available Thread-Index: L3zcHd+nBH3JTTOVXhgbDnAZGiFZUA== X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Mar 2016 00:02:39 -0000 Ken Merry wrote: > I have patches for FreeBSD=E2=80=99s FUSE filesystem kernel module to sup= port > extended attributes: >=20 > https://people.freebsd.org/~ken/fuse_extattr.20160229.1.txt >=20 > The patch implements the get/set/delete/list extended attribute methods. = The > listing code also converts extended attribute lists from the Linux/FUSE > format to the FreeBSD format. I also have patches, although my list didn't work. (I didn't know that ther= e was a difference between what Linux/FUSE returns vs what FreeBSD wanted. So now= I know why my "list" didn't work.) Btw, when I discussed what to do w.r.t. extended attribute namespace, he se= emed to think just considering all the fuse ones as User was ok. I also have patched FreeBSD's Fuse for the other stuff needed to export the= fuse mount via an NFS server. (VFS_FHTOVP() etc) These aren't quite ready for "p= rime time" but if anyone wants to try them, just email me and I'll send you a copy. Btw, I have a couple of patches related to direct I/O and buffered I/O. You= can find 2 of these at PR#206238. I also patched fuse_vnop_inactive() to flush/= write dirty pages. (I think this is needed when an application mmaps a file and t= he modifies it after closing the file descriptor. I haven't actually tested to= see if this fix is needed, so I haven't put it anywhere yet.) > For example: >=20 > # touch foo > # ls -la foo > -rwxrwxrwx 1 root wheel 0 Feb 29 21:40 foo > # lsextattr user foo > foo > # setextattr user testattr1 "12345678" foo > # lsextattr user foo > foo testattr1 > # getextattr user testattr1 foo > foo 12345678 > # setextattr user testattr2 "87654321" foo > # lsextattr user foo > foo testattr2 testattr1 > # rmextattr user testattr1 foo > # lsextattr user foo > foo testattr2 > # getextattr user testattr1 foo > getextattr: foo: failed: Attribute not found > # getextattr user testattr2 foo > foo 87654321 >=20 >=20 > Just to be clear on what this does, it only provides extended attribute > support to FreeBSD applications if the underlying FUSE filesystem impleme= nts > FUSE extended attribute support. Many FUSE filesystems don=E2=80=99t sup= port the > extended attribute VFS operations. >=20 > I have tested this out on IBM=E2=80=99s LTFS implementation, but I have n= ot yet found > another FUSE filesystem that supports extended attributes. If anyone kno= ws > of one, please let me know so I can try it out. (I looked through a numb= er > of the filesystems in sysutils/fusefs* in the ports tree.) >=20 The FreeBSD GlusterFS port includes a fuse interface that supports extended attributes. That is how I tested what I have. (I think the port is now in svn, but if not you can find the GlusterFS port at PR#194409.) It glusterfs.org doc doesn't mention that you can create/test a volume of only one brick, but that works for trivial testing. If you decide to use it for testing and have trouble getting it going, just email. I know diddly ab= out GlusterFS, but I have fired it up a few times. > Any feedback is welcome. I=E2=80=99m planning to check this into FreeBSD= /head in the > next week or so. >=20 I'll try to get around and taking a look to see if there is anything differ= ent than what I did (other than the above "list" case I didn't get right;-). rick > Obviously, I=E2=80=99ve also ported IBM=E2=80=99s LTFS implementation to = FreeBSD. It works > in the standard FUSE mode, and you can also link it into an application a= s a > library if you don=E2=80=99t want to incur the overhead of running throug= h FUSE. I > haven=E2=80=99t gotten around to packaging it up to go out for testing / = review. >=20 > If anyone has IBM LTO-5 or newer tape drives, or IBM TS1140 or newer tape > drives, and wants to try it out, let me know. I=E2=80=99ll send you the = code when > I=E2=80=99ve got it at least somewhat ready. This is IBM-specific, and w= on=E2=80=99t work > on HP tape drives. >=20 > Ken > =E2=80=94 > Ken Merry > ken@FreeBSD.ORG >=20 >=20 >=20 > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-scsi@freebsd.org Tue Mar 1 15:11:19 2016 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C7050ABE4A2 for ; Tue, 1 Mar 2016 15:11:19 +0000 (UTC) (envelope-from ken@kdm.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id B0DD71585 for ; Tue, 1 Mar 2016 15:11:19 +0000 (UTC) (envelope-from ken@kdm.org) Received: by mailman.ysv.freebsd.org (Postfix) id B0773ABE4A0; Tue, 1 Mar 2016 15:11:19 +0000 (UTC) Delivered-To: scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AFF57ABE49F; Tue, 1 Mar 2016 15:11:19 +0000 (UTC) (envelope-from ken@kdm.org) Received: from mithlond.kdm.org (mithlond.kdm.org [96.89.93.250]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "A1-33714", Issuer "A1-33714" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 896B91584; Tue, 1 Mar 2016 15:11:19 +0000 (UTC) (envelope-from ken@kdm.org) Received: from mithlond.kdm.org (localhost [127.0.0.1]) by mithlond.kdm.org (8.15.2/8.14.9) with ESMTPS id u21FBAQs080918 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 1 Mar 2016 10:11:10 -0500 (EST) (envelope-from ken@mithlond.kdm.org) Received: (from ken@localhost) by mithlond.kdm.org (8.15.2/8.14.9/Submit) id u21FBAbi080917; Tue, 1 Mar 2016 10:11:10 -0500 (EST) (envelope-from ken) Date: Tue, 1 Mar 2016 10:11:10 -0500 From: "Kenneth D. Merry" To: Rick Macklem Cc: fs@freebsd.org, scsi@freebsd.org Subject: Re: FUSE extended attribute patches available Message-ID: <20160301151109.GA79912@mithlond.kdm.org> References: <1740288370.14765302.1456790548437.JavaMail.zimbra@uoguelph.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1740288370.14765302.1456790548437.JavaMail.zimbra@uoguelph.ca> User-Agent: Mutt/1.5.23 (2014-03-12) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (mithlond.kdm.org [127.0.0.1]); Tue, 01 Mar 2016 10:11:10 -0500 (EST) X-Spam-Status: No, score=-1.2 required=5.0 tests=ALL_TRUSTED,BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,URIBL_BLACK autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on mithlond.kdm.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Mar 2016 15:11:19 -0000 On Mon, Feb 29, 2016 at 19:02:28 -0500, Rick Macklem wrote: > Ken Merry wrote: > > I have patches for FreeBSD???s FUSE filesystem kernel module to support > > extended attributes: > > > > https://people.freebsd.org/~ken/fuse_extattr.20160229.1.txt > > > > The patch implements the get/set/delete/list extended attribute methods. The > > listing code also converts extended attribute lists from the Linux/FUSE > > format to the FreeBSD format. > I also have patches, although my list didn't work. (I didn't know that there was > a difference between what Linux/FUSE returns vs what FreeBSD wanted. So now I know > why my "list" didn't work.) > Btw, when I discussed what to do w.r.t. extended attribute namespace, he seemed to > think just considering all the fuse ones as User was ok. Ahh. Who is "he" in this context? It was easy enough to allow access to the user and system namespace. FUSE and Linux use the same names, so might as well pass them through. The other issue as far as FUSE goes, is that it is expecting the "user." or "system." prefix on the attribute name. FreeBSD passes those as a separate, numeric, argument in the VFS layer at least, and expects to not see the namespace as a prefix when listing attributes. > I also have patched FreeBSD's Fuse for the other stuff needed to export the fuse > mount via an NFS server. (VFS_FHTOVP() etc) These aren't quite ready for "prime time" > but if anyone wants to try them, just email me and I'll send you a copy. > > Btw, I have a couple of patches related to direct I/O and buffered I/O. You can > find 2 of these at PR#206238. I also patched fuse_vnop_inactive() to flush/write > dirty pages. (I think this is needed when an application mmaps a file and the > modifies it after closing the file descriptor. I haven't actually tested to see > if this fix is needed, so I haven't put it anywhere yet.) Ahh, cool! Now I can export a tape via NFS! :) Seriously, though, that will be helpful for some filesystems. > > For example: > > > > # touch foo > > # ls -la foo > > -rwxrwxrwx 1 root wheel 0 Feb 29 21:40 foo > > # lsextattr user foo > > foo > > # setextattr user testattr1 "12345678" foo > > # lsextattr user foo > > foo testattr1 > > # getextattr user testattr1 foo > > foo 12345678 > > # setextattr user testattr2 "87654321" foo > > # lsextattr user foo > > foo testattr2 testattr1 > > # rmextattr user testattr1 foo > > # lsextattr user foo > > foo testattr2 > > # getextattr user testattr1 foo > > getextattr: foo: failed: Attribute not found > > # getextattr user testattr2 foo > > foo 87654321 > > > > > > Just to be clear on what this does, it only provides extended attribute > > support to FreeBSD applications if the underlying FUSE filesystem implements > > FUSE extended attribute support. Many FUSE filesystems don???t support the > > extended attribute VFS operations. > > > > I have tested this out on IBM???s LTFS implementation, but I have not yet found > > another FUSE filesystem that supports extended attributes. If anyone knows > > of one, please let me know so I can try it out. (I looked through a number > > of the filesystems in sysutils/fusefs* in the ports tree.) > > > The FreeBSD GlusterFS port includes a fuse interface that supports extended > attributes. That is how I tested what I have. (I think the port is now in > svn, but if not you can find the GlusterFS port at PR#194409.) > It glusterfs.org doc doesn't mention that you can create/test a volume of > only one brick, but that works for trivial testing. If you decide to use it > for testing and have trouble getting it going, just email. I know diddly about > GlusterFS, but I have fired it up a few times. Ahh, that would be helpful. I'll give it a try. > > Any feedback is welcome. I???m planning to check this into FreeBSD/head in the > > next week or so. > > > I'll try to get around and taking a look to see if there is anything different > than what I did (other than the above "list" case I didn't get right;-). Yes, it would be good to get another set of eyes. Perhaps the namespace handling (user versus system) is different as well. Ken -- Kenneth Merry ken@FreeBSD.ORG From owner-freebsd-scsi@freebsd.org Tue Mar 1 20:38:18 2016 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D0713ABE344 for ; Tue, 1 Mar 2016 20:38:18 +0000 (UTC) (envelope-from sbruno@freebsd.org) Received: from mail.ignoranthack.me (ignoranthack.me [199.102.79.106]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BA1EF1070 for ; Tue, 1 Mar 2016 20:38:18 +0000 (UTC) (envelope-from sbruno@freebsd.org) Received: from [192.168.200.206] (unknown [50.136.155.142]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: sbruno@ignoranthack.me) by mail.ignoranthack.me (Postfix) with ESMTPSA id 97D5A192993 for ; Tue, 1 Mar 2016 20:38:17 +0000 (UTC) To: freebsd-scsi@freebsd.org From: Sean Bruno Subject: mpr(4) SAS3008 Repeated Crashing Message-ID: <56D5FDB8.8040402@freebsd.org> Date: Tue, 1 Mar 2016 12:38:16 -0800 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.5.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Mar 2016 20:38:18 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 We're starting to look into cabling and whatnot on a group of Supermicro hosts we're using after repeatedly running into a crash/timeout that more or less looks like the following capture from the serial console: https://people.freebsd.org/~sbruno/mpr_stable10.txt sean -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQF8BAEBCgBmBQJW1f20XxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRCQUFENDYzMkU3MTIxREU4RDIwOTk3REQx MjAxRUZDQTFFNzI3RTY0AAoJEBIB78oecn5k9tEH/3nasxb6yWshKzskzOr5KxwY dBV2RkoDlq0GKc/oYBfOOsvOAegnrHCFRDQTlXIL3nvBMDyZgtEalcbsnF2jWMaA Ff4Lt5ZJeg03Xc+Ge/wbAIAThzsCrbH2isLS1jnZi3sc047WtN+HjhAxfozUL/Dj 1beCTgda36A4K7DBBvPz0gcRS2e/a8Xy1mhlIOr4n3eG+kZrAUSTYrAVos6JybkN YipRQeHeUDJlO2vSxYITK9kzJn887eg/iwk0udtIaranzJ3CCTmZn3gHuyVDjKJy eq9DsxrtCvcAMbw4SGrJqPi0BwjfejLgD/+LXAs9Dh6L3BFqZ9Z1BsjlJvfHLXI= =q6ub -----END PGP SIGNATURE----- From owner-freebsd-scsi@freebsd.org Tue Mar 1 22:08:47 2016 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CC9B3AC0FEF for ; Tue, 1 Mar 2016 22:08:47 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wm0-x22b.google.com (mail-wm0-x22b.google.com [IPv6:2a00:1450:400c:c09::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8A9481AEE for ; Tue, 1 Mar 2016 22:08:47 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wm0-x22b.google.com with SMTP id n186so58775082wmn.1 for ; Tue, 01 Mar 2016 14:08:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-transfer-encoding; bh=Nn1e7v22XYIJa3IhzkcfxeDYJoHgeTuLqy/8ot9SmxM=; b=K9IW9RZ4rCFEylm30evhWD+w13Dv+VO05MGRDeZAfH2SAb/x+gFUh4jDTY20ty9GTs atmoFPSzZbTZH9wgRkmr8qrjcbGvKmeNTmHpf1AUXMSZEYH+AYTIL3EYMPUkuFEKkGKi rn3KLKRnG5adv99qVoyfnKgtzO0xiwCd+FuOSuRWNV99cP6C6wXnVYC5gy3BKtpsRHdm vc7eP+vDHQDGiCE6gUEiDfo+Y09HRx1b8AV4XVty5STaXeNDcEH5m4iI0f8BvPeHxIJG VV5VfHImlj1ZjE/FnXhBLEBl4p+ekfHk3bW3dUfnNvJWnh5q1Z4Tn8TtO0E3/PWAoeGr Rk5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=Nn1e7v22XYIJa3IhzkcfxeDYJoHgeTuLqy/8ot9SmxM=; b=V2FlbgL34L9IU9ElIkH9UoIGNlPthUhjEzb0IY/K8IQCHjZQ9Gbgs0hZFpePUb7DZE QbacNmUP2P9IbWnxcOaTgkCE5k+XYyUSqdPTJXez31SfDyBwIfP9aRZ4PXeYZXp1XG3W FRx6oJDAHSI/0jCLSMR8Zbf3KSpLkFTcIArpl8AaOBP9oqPCamIpQ8x7wS6DT2XMsYHR QgXG1guKH13eC7vAGtM7gdcSFtnrHwfJHvaDylaR6upkujnRx+A466pAwLQ24y5I2XOw MrMgifmlZclAvHaPHZSvfvqmrChSH+uu5gPF1WlTbm0IqeZadPslel8n2LyrDRTy5Sra gUEw== X-Gm-Message-State: AD7BkJLIUzLdHFai5LmKRjIhOUN5ahWccnMcGX9N6dwlBISU0AemuVZm83iA0cU5hvG3JKk+ X-Received: by 10.28.133.14 with SMTP id h14mr1258831wmd.100.1456870125883; Tue, 01 Mar 2016 14:08:45 -0800 (PST) Received: from [10.10.1.58] (liv3d.labs.multiplay.co.uk. [82.69.141.171]) by smtp.gmail.com with ESMTPSA id za6sm32808885wjc.18.2016.03.01.14.08.44 for (version=TLSv1/SSLv3 cipher=OTHER); Tue, 01 Mar 2016 14:08:44 -0800 (PST) Subject: Re: mpr(4) SAS3008 Repeated Crashing To: freebsd-scsi@freebsd.org References: <56D5FDB8.8040402@freebsd.org> From: Steven Hartland Message-ID: <56D612FA.6090909@multiplay.co.uk> Date: Tue, 1 Mar 2016 22:08:58 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.6.0 MIME-Version: 1.0 In-Reply-To: <56D5FDB8.8040402@freebsd.org> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Mar 2016 22:08:48 -0000 Initial ideas would be bad signalling. If you have the option to drop the speeds down and that helps then almost certainly the case. The original mfi driver was very bad at recovering from issues like this too, I spent over a month fixing and patching it to get it working reliably when there where hardware related issues. In my case it turned out the be a dodge CPU causing memory corruption but you'll get similar behaviour from badly designed installs, particularly with expanders in play for high speed devices (6-12Gbps) link speed. Regards Steve On 01/03/2016 20:38, Sean Bruno wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 > > We're starting to look into cabling and whatnot on a group of > Supermicro hosts we're using after repeatedly running into a > crash/timeout that more or less looks like the following capture from > the serial console: > > https://people.freebsd.org/~sbruno/mpr_stable10.txt > > sean > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v2 > > iQF8BAEBCgBmBQJW1f20XxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w > ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRCQUFENDYzMkU3MTIxREU4RDIwOTk3REQx > MjAxRUZDQTFFNzI3RTY0AAoJEBIB78oecn5k9tEH/3nasxb6yWshKzskzOr5KxwY > dBV2RkoDlq0GKc/oYBfOOsvOAegnrHCFRDQTlXIL3nvBMDyZgtEalcbsnF2jWMaA > Ff4Lt5ZJeg03Xc+Ge/wbAIAThzsCrbH2isLS1jnZi3sc047WtN+HjhAxfozUL/Dj > 1beCTgda36A4K7DBBvPz0gcRS2e/a8Xy1mhlIOr4n3eG+kZrAUSTYrAVos6JybkN > YipRQeHeUDJlO2vSxYITK9kzJn887eg/iwk0udtIaranzJ3CCTmZn3gHuyVDjKJy > eq9DsxrtCvcAMbw4SGrJqPi0BwjfejLgD/+LXAs9Dh6L3BFqZ9Z1BsjlJvfHLXI= > =q6ub > -----END PGP SIGNATURE----- > _______________________________________________ > freebsd-scsi@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-scsi > To unsubscribe, send any mail to "freebsd-scsi-unsubscribe@freebsd.org" From owner-freebsd-scsi@freebsd.org Tue Mar 1 22:48:02 2016 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 898DEAC08C8 for ; Tue, 1 Mar 2016 22:48:02 +0000 (UTC) (envelope-from ken@kdm.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 748CE1971 for ; Tue, 1 Mar 2016 22:48:02 +0000 (UTC) (envelope-from ken@kdm.org) Received: by mailman.ysv.freebsd.org (Postfix) id 6FAD8AC08C6; Tue, 1 Mar 2016 22:48:02 +0000 (UTC) Delivered-To: scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6EC5EAC08C4; Tue, 1 Mar 2016 22:48:02 +0000 (UTC) (envelope-from ken@kdm.org) Received: from mithlond.kdm.org (mithlond.kdm.org [96.89.93.250]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "A1-33714", Issuer "A1-33714" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 34396196F; Tue, 1 Mar 2016 22:48:01 +0000 (UTC) (envelope-from ken@kdm.org) Received: from mithlond.kdm.org (localhost [127.0.0.1]) by mithlond.kdm.org (8.15.2/8.14.9) with ESMTPS id u21MlwhB087148 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 1 Mar 2016 17:47:58 -0500 (EST) (envelope-from ken@mithlond.kdm.org) Received: (from ken@localhost) by mithlond.kdm.org (8.15.2/8.14.9/Submit) id u21Mlwff087147; Tue, 1 Mar 2016 17:47:58 -0500 (EST) (envelope-from ken) Date: Tue, 1 Mar 2016 17:47:58 -0500 From: "Kenneth D. Merry" To: scsi@freebsd.org, current@freebsd.org Subject: Re: CAM Shingled Disk support patches available Message-ID: <20160301224758.GA86834@mithlond.kdm.org> References: <20151118171309.GA3564@mithlond.kdm.org> <20160118223704.GA87506@mithlond.kdm.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160118223704.GA87506@mithlond.kdm.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (mithlond.kdm.org [127.0.0.1]); Tue, 01 Mar 2016 17:47:58 -0500 (EST) X-Spam-Status: No, score=-0.4 required=5.0 tests=ALL_TRUSTED,BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,TO_NO_BRKTS_PCNT autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on mithlond.kdm.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Mar 2016 22:48:02 -0000 I have a new set of SMR patches available. (See the original message below for a more detailed description of what these patches do.) The primary change is to add library versioning to libcam so that we can change the function prototype of scsi_ata_pass_16() in a way that won't break existing binaries. If someone more familiar with library versioning wants to review this, I'd appreciate it. The patches are here: FreeBSD/head, as of SVN revision 296278 https://people.freebsd.org/~ken/cam_smr.head.20160301.1.txt stable/10, as of SVN revision 296248 https://people.freebsd.org/~ken/cam_smr.stable10.20160301.1.txt (Note that although there is a stable/10 version of the patches, I'm not planning to merge them to stable/10 because of the change to struct bio. I can't really figure out a good way to make that backward compatible. If there is consensus that breaking it is fine because it isn't a user API, then that may be another story.) The problem is that the existing, in-tree version of scsi_ata_pass_16() has a dxfer_len argument that is a uint16_t. That restricts transfer sizes to 64KB. So, we need to update it to allow larger than 64K transfers. I could just create a new function, but I'd rather just retire the broken version. The intent here is that: 1. Binaries built against the old version of libcam, before versioning was turned on, will get the old version of the scsi_ata_pass_16() function with a uint16_t dxfer_len. 2. Binaries built against the new version of libcam will get the new version of the scsi_ata_pass_16() function with a uint32_t dxfer_len. I've tested this, and it appears to work, but I'm not 100% certain this is all correct. I looked at Dan Eischen's description of symbol versioning here: https://people.freebsd.org/~deischen/symver/freebsd_versioning.txt And it looks like the actual implementation is a little different than what is described there. I looked around the tree, and didn't see anything that is obviously exactly like what I'm trying to do here. So, what I did is as follows: 1. For the kernel, the only change is to switch the dxfer_len argument from a uint16_t to a uint32_t. 2. For userland, in scsi_all.c, there are now two versions of scsi_ata_pass_16 -- _ver1 and _ver2. _ver1 is aliased to scsi_ata_pass_16() for FBSD_1.3 using __sym_compat(). _ver2 is aliased to scsi_ata_pass_16() for FBSD_1.4 using __sym_default(). 3. In lib/libcam/Versions.def, I defined FBSD_1.3 and FBSD_1.4, which depends on FBSD_1.3. 4. In lib/libcam/Symbol.map, I pulled out all of the functions defined in libcam, sorted them, and defined them in FBSD_1.3. I moved scsi_ata_pass_16() to FBSD_1.4. (According to the freebsd_versioning.txt paper linked above, I should have been able to have scsi_ata_pass_16() in both FBSD_1.3 and FBSD_1.4, but that isn't the case in practice.) In testing an old binary (linked against libcam without symbol versioning) against a new libcam (with symbol versioning), the old version of the function appears to be used. With a new binary, the new version of the function appears to be used. So it looks like things work as intended, but I don't fully trust my understanding here. So, if someone could take a look at the changes, I'd appreciate it. In particular, I have a few questions: 1. If this change to scsi_ata_pass_16() gets merged to stable/10 (apart from the larger SMR changes), what should be done with the libcam library version? 2. Are 1.3 and 1.4 the proper versions to use? 3. If we make additional CAM helper function library changes, when do the versions get bumped? i.e., is this an opportunity to look for other library functions with issues and make changes if possible? 4. When you're going from an unversioned library to a versioned library, which version of a function gets linked in to a binary linked to the unversioned library when you run it against a versioned library? In other words, what is supposed to happen in the test scenario I tried above, and am I really seeing what is supposed to happen? Thanks, Ken On Mon, Jan 18, 2016 at 17:37:04 -0500, Kenneth D. Merry wrote: > I have a new set of SMR patches available. See below for the full > explanation. > > The primary change here is that I have added SMR support to the ada(4) > driver. I spent some time considering whether to try to make the da(4) and > ada(4) probe infrastructure somewhat common, but in the end concluded it > would be too involved with not enough code reduction (if any) in the end. > > So, although the ideas are similar, the probe logic is separate. > > Note that NCQ support for SMR commands (Report Zones, Reset Write Pointer, > etc.) for SATA protocol shingled drives isn't active. For both the da(4) > and ada(4) driver this is for lack of a way to plumb the ATA Auxiliary > register down to the drive. > > In the ada(4) case, we need to add the register to struct ccb_ataio and > add support in one or more of the underlying SATA drivers, e.g. ahci(4). > > In the da(4) case, it will require an update of the T-10 SAT spec to > provide a way to pass the Auxiliary register down via the SCSI ATA > PASS-THROUGH command, and then a subsquent update of the SAT layer in > various vendors' SAS controller firmware. At that point, there may be > an official mapping of the SCSI ZBC commands to the ATA ZAC commands, and > we may be able to just issue the SCSI version of the commands instead of > composing ATA commands in the da(4) driver. (We'll still need to keep the > ATA passthrough version for a while at least to support controllers that > don't have the updated translation code.) > > FreeBSD/head as of SVN revision 294105: > > https://people.freebsd.org/~ken/cam_smr.head.20160118.1.txt > > FreeBSD stable/10 as of SVN revision 294100: > > https://people.freebsd.org/~ken/cam_smr.stable10.20160118.1.txt > > Testing and comments are welcome. > > Ken > > On Wed, Nov 18, 2015 at 12:13:09 -0500, Kenneth D. Merry wrote: > > > > I have work in progress patches to add SMR (Shingled Magnetic Recording) > > support to CAM and GEOM here: > > > > FreeBSD/head as of SVN revision 290997: > > > > https://people.freebsd.org/~ken/cam_smr.head.20151117.1.txt > > > > FreeBSD stable/10 as of SVN revision 290995: > > > > https://people.freebsd.org/~ken/cam_smr.stable10.20151117.1.txt > > > > This includes support for Host Managed, Host Aware and Drive Managed SMR > > drives that are either SCSI (ZBC) or ATA (ZAC) attached via a SAS > > controller. This does not include support for SMR ATA drives attched via > > an ATA controller. Also, I have not yet figured out how to properly detect > > a Host Managed ATA drive, so this code won't do that. > > > > The big drive vendors are moving to SMR for at least some of their drives. > > The primary challenge with SMR is that it requires writing a relatively > > large zone sequentially starting at the beginning of the zone. The usual > > zone size is 256MB. It is conceptually almost like having a 256MB sector > > size. > > > > We (Spectra Logic) are working on ZFS changes that will use this CAM and > > GEOM infrastructure to make ZFS play well with SMR drives. Those changes > > aren't yet done. > > > > The patches linked above include: > > o A new 'camcontrol zone' command that allows displaying and managing > > drive zones via SCSI/ATA passthrough. > > o A new zonectl(8) utility that uses the new DIOCZONECMD ioctl to display > > and manage zones via the da(4) (and later ada(4)) driver. > > o Changes to diskinfo -v to display the zone mode of a drive. > > o A new disk zone API, sys/sys/disk_zone.h. > > o A new bio type, BIO_ZONE, and modifications to GEOM to support it. This > > new bio will allow filesystems to query zone support in a drive and > > manage zoned drives. > > o Extensive modifications to the da(4) driver to handle probing SCSI and > > SATA behind SAS SMR drives. > > o Additional CAM CDB building functions for zone commands. > > > > The current issues that need to be addressed are: > > o The da(4) driver now has 6 additional probe states, 5 of which are > > needed for probing ATA drives behind SAS controllers. I have not yet > > added support for BIO_ZONE bios to ada(4), but it will be very similar > > to the da(4) driver version. The ATA probe code needs to be pulled > > out of the da(4) driver and changed into a form that will allow it to > > work for either the ada(4) or da(4) driver. Otherwise we'll have a fair > > amount of code duplication between the two drivers. > > > > o There is a reasonable amount of code duplication between 'camcontrol zone' > > and zonectl(8). This was done for speed / expediency's sake, but it may > > be possible to make more of the code common there. > > > > o In order to add the new BIO_ZONE bio command, I had to change the bio_cmd > > field in struct bio from a uint8_t to a uint16_t. This will cause > > binary compatibility problems with any 3rd party loadable modules. > > Advice on how to handle this would be welcome. > > > > o In the process of developing these changes, I discovered that the > > dxfer_len paramter for scsi_ata_pass_16() was too small (uint16_t, and > > it needed to be uint32_t). I increased it, but that will potentially > > cause a binary incompatibility problem with any existing applications > > that use the current API via libcam. Advice on how to handle that > > would be welcome. > > > > If you look through the code, you'll notice that the disk_zone.h API is > > separate from the SCSI and ATA APIs. The intent is to allow filesystems > > and other consumers of the API to just talk to the disk zone API without > > dealing with the SCSI and ATA specifics. Another reason behind all of this > > is that even though the SCSI ZBC and ATA ZAC specs were developed in > > concert, and are intended to be functionally identical, they are still SCSI > > and ATA. As usual, SCSI is big endian and ATA is little endian. So to > > present a common API to the filesystem, we give all of the zone data back > > in native byte order, regardless of the underlying device protocol. > > > > Another thing to note is the extensive use of ATA passthrough in the da(4) > > driver. This is necessary because the SCSI SAT (SCSI to ATA Translation) > > specification has not yet caught up with translating SCSI zone commands > > (ZBC) to ATA zone commands (ZAC). So, until the spec is updated and LSI > > and other vendors update their SCSI to ATA translation layers, we'll have > > to use the ATA version of the commands when talking to ATA drives via SAS > > controllers. > > > > I have only tested the code so far with Seagate SATA Drive Managed and Host > > Aware drives. I would appreciate testing with any drives. (And testing to > > make sure that the patches don't cause problems with existing hardware.) > > Right now, all you can really do is manage the zones manually using > > camcontrol(8) or zonectl(8). Automatic management will come with the ZFS > > changes. (Or changes to other filesysems if people want to do it.) > > > > If you have a SATA Host Aware drive, in theory camcontrol(8) should allow > > you to manage the drive if you have it attached to a SATA controller. > > > > Here is an example of some of the commands. > > > > Get general zoning information: > > > > [root@sm4u-1 ~]# zonectl -c params -d /dev/da21 > > Zone Mode: Host Aware > > Command support: Report Zones, Open, Close, Finish, Reset Write Pointer > > Unrestricted Read in Sequential Write Required Zone (URSWRZ): No > > Optimal Number of Open Sequential Write Preferred Zones: 128 > > Optimal Number of Non-Sequentially Written Sequential Write Preferred Zones: 8 > > Maximum Number of Open Sequential Write Required Zones: Unlimited > > > > Look at information from the da(4) driver: > > > > [root@sm4u-1 ~]# sysctl kern.cam.da.21 > > kern.cam.da.21.delete_method: NONE > > kern.cam.da.21.delete_max: 1081344 > > kern.cam.da.21.minimum_cmd_size: 6 > > kern.cam.da.21.sort_io_queue: -1 > > kern.cam.da.21.zone_mode: Host Aware > > kern.cam.da.21.zone_support: Report Zones, Open, Close, Finish, Reset Write Pointer > > kern.cam.da.21.optimal_seq_zones: 128 > > kern.cam.da.21.optimal_nonseq_zones: 8 > > kern.cam.da.21.max_seq_zones: 4294967295 > > kern.cam.da.21.error_inject: 0 > > > > Display all of the zones with zonectl(8): > > > > [root@sm4u-1 ~]# zonectl -d /dev/da21 -c rz > > 29809 zones, Maximum LBA 0x3a3812aaf (15628053167) > > Zone lengths and types may vary > > Start LBA Length WP LBA Zone Type Condition Sequential Reset > > 0, 524288, 0x80000, Conventional, NWP, Sequential, No Reset Needed > > 0x80000, 524288, 0x100000, Conventional, NWP, Sequential, No Reset Needed > > 0x100000, 524288, 0x180000, Conventional, NWP, Sequential, No Reset Needed > > 0x180000, 524288, 0x200000, Conventional, NWP, Sequential, No Reset Needed > > 0x200000, 524288, 0x280000, Conventional, NWP, Sequential, No Reset Needed > > 0x280000, 524288, 0x300000, Conventional, NWP, Sequential, No Reset Needed > > 0x300000, 524288, 0x380000, Conventional, NWP, Sequential, No Reset Needed > > 0x380000, 524288, 0x400000, Conventional, NWP, Sequential, No Reset Needed > > 0x400000, 524288, 0x480000, Conventional, NWP, Sequential, No Reset Needed > > 0x480000, 524288, 0x500000, Conventional, NWP, Sequential, No Reset Needed > > 0x500000, 524288, 0x580000, Conventional, NWP, Sequential, No Reset Needed > > 0x580000, 524288, 0x600000, Conventional, NWP, Sequential, No Reset Needed > > 0x600000, 524288, 0x680000, Conventional, NWP, Sequential, No Reset Needed > > 0x680000, 524288, 0x700000, Conventional, NWP, Sequential, No Reset Needed > > 0x700000, 524288, 0x780000, Conventional, NWP, Sequential, No Reset Needed > > [ ... ] > > 0x1f00000, 524288, 0x1f80000, Conventional, NWP, Sequential, No Reset Needed > > 0x1f80000, 524288, 0x2000000, Conventional, NWP, Sequential, No Reset Needed > > 0x2000000, 524288, 0x2080000, Seq Preferred, Full, Sequential, No Reset Needed > > 0x2080000, 524288, 0x2100000, Seq Preferred, Full, Sequential, No Reset Needed > > 0x2100000, 524288, 0x2180000, Seq Preferred, Full, Sequential, No Reset Needed > > 0x2180000, 524288, 0x2200000, Seq Preferred, Full, Sequential, No Reset Needed > > 0x2200000, 524288, 0x2280000, Seq Preferred, Full, Sequential, No Reset Needed > > 0x2280000, 524288, 0x2300000, Seq Preferred, Full, Sequential, No Reset Needed > > [ ... ] > > > > Use camcontrol zone to reset the write pointer for the first shingled zone > > listed above: > > > > [root@sm4u-1 ~]# camcontrol zone da21 -v -c rwp -l 0x2000000 > > > > Use camcontrol zone to ask the drive to report empty zones: > > > > [root@sm4u-1 ~]# camcontrol zone da21 -v -c rz -o empty > > 1 zones, Maximum LBA 0x3a3812aaf (15628053167) > > Zone lengths and types may vary > > Start LBA Length WP LBA Zone Type Condition Sequential Reset > > 0x2000000, 524288, 0x2000000, Seq Preferred, Empty, Sequential, No Reset Needed > > > > Get information on a Host Aware drive: > > > > root@sm4u-1 ~]# diskinfo -v da21 > > da21 > > 512 # sectorsize > > 8001563222016 # mediasize in bytes (7.3T) > > 15628053168 # mediasize in sectors > > 4096 # stripesize > > 0 # stripeoffset > > 972801 # Cylinders according to firmware. > > 255 # Heads according to firmware. > > 63 # Sectors according to firmware. > > Z84008NY # Disk ident. > > enc@5003048001f311fd/elmtype@array_device/slot@22 # Physical path > > Host Aware # Zone Mode > > > > Get information on a drive managed drive: > > > > [root@sm4u-1 ~]# diskinfo -v da20 > > da20 > > 512 # sectorsize > > 8001563222016 # mediasize in bytes (7.3T) > > 15628053168 # mediasize in sectors > > 4096 # stripesize > > 0 # stripeoffset > > 972801 # Cylinders according to firmware. > > 255 # Heads according to firmware. > > 63 # Sectors according to firmware. > > Z8405938 # Disk ident. > > enc@5003048001f311fd/elmtype@array_device/slot@21 # Physical path > > Drive Managed # Zone Mode > > > > Get information on a non-zoned drive: > > > > [root@sm4u-1 ~]# diskinfo -v da4 > > da4 > > 512 # sectorsize > > 100030242816 # mediasize in bytes (93G) > > 195371568 # mediasize in sectors > > 0 # stripesize > > 0 # stripeoffset > > 12161 # Cylinders according to firmware. > > 255 # Heads according to firmware. > > 63 # Sectors according to firmware. > > 124903574F36 # Disk ident. > > enc@5003048001f311fd/elmtype@array_device/slot@5 # Physical path > > Not Zoned # Zone Mode > > > > Testing and comments are welcome. > > > > Thanks, > > > > Ken > > -- > > Kenneth Merry > > ken@FreeBSD.ORG > > -- > Kenneth Merry > ken@FreeBSD.ORG -- Kenneth Merry ken@FreeBSD.ORG From owner-freebsd-scsi@freebsd.org Wed Mar 2 01:04:01 2016 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 10374ABE6BE for ; Wed, 2 Mar 2016 01:04:01 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id E49F01C17 for ; Wed, 2 Mar 2016 01:04:00 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: by mailman.ysv.freebsd.org (Postfix) id E09B5ABE6BC; Wed, 2 Mar 2016 01:04:00 +0000 (UTC) Delivered-To: scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C62F4ABE6BA; Wed, 2 Mar 2016 01:04:00 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 396111C14; Wed, 2 Mar 2016 01:03:59 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) IronPort-PHdr: 9a23:VQBJoxHYPQl88CXGEufR0p1GYnF86YWxBRYc798ds5kLTJ75osmwAkXT6L1XgUPTWs2DsrQf27WQ7PmrAzxIyK3CmU5BWaQEbwUCh8QSkl5oK+++Imq/EsTXaTcnFt9JTl5v8iLzG0FUHMHjew+a+SXqvnYsExnyfTB4Ov7yUtaLyZ/niKbtotaJM01hv3mUX/BbFF2OtwLft80b08NJC50a7V/3mEZOYPlc3mhyJFiezF7W78a0+4N/oWwL46pyv50IbaKvNYc1S7pVEDRuHyZ9wcDxrwiJBV+M6300fH8bnzBzL07i1j6sDbnrtS6vjOt222G/NMb1Sb0xEWC46q5gSxvljQ8aMDEk/WXPiop7hfQI81qauxVjztuMM8muP/1kc/aFcA== X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2DPAQBaO9ZW/61jaINcFoN2bQa4GoITAQ2BZiGFcgKCBhQBAQEBAQEBAWMngi2CFAEBAQMBI1YFCwIBCBgCAg0ZAgJXAgQTiBcIDq8MjxEBAQEBAQEBAQEBAQEBAQEBGnuFF4F0gkaEBREBBoMYgToFh1SFV3Q9iDKFWYlpS4N5iFKOSgIeAQFCggMZgWYeLgEBAYcMNH4BAQE X-IronPort-AV: E=Sophos;i="5.22,524,1449550800"; d="scan'208";a="270296052" Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-annu.net.uoguelph.ca with ESMTP; 01 Mar 2016 20:03:52 -0500 Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id DCAB315F55D; Tue, 1 Mar 2016 20:03:52 -0500 (EST) Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id ecdwE9WP9mWa; Tue, 1 Mar 2016 20:03:52 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 1AC9D15F565; Tue, 1 Mar 2016 20:03:52 -0500 (EST) X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id In4kA1axQUpe; Tue, 1 Mar 2016 20:03:52 -0500 (EST) Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id F379B15F55D; Tue, 1 Mar 2016 20:03:51 -0500 (EST) Date: Tue, 1 Mar 2016 20:03:51 -0500 (EST) From: Rick Macklem To: "Kenneth D. Merry" Cc: fs@freebsd.org, scsi@freebsd.org Message-ID: <2079334495.1177596.1456880631768.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <20160301151109.GA79912@mithlond.kdm.org> References: <1740288370.14765302.1456790548437.JavaMail.zimbra@uoguelph.ca> <20160301151109.GA79912@mithlond.kdm.org> Subject: Re: FUSE extended attribute patches available MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.95.12] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - IE7 (Win)/8.0.9_GA_6191) Thread-Topic: FUSE extended attribute patches available Thread-Index: f0Y98rCs8CXfE8GZlh7LylfdvF2/qg== X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Mar 2016 01:04:01 -0000 Kenneth D. Merry wrote: > On Mon, Feb 29, 2016 at 19:02:28 -0500, Rick Macklem wrote: > > Ken Merry wrote: > > > I have patches for FreeBSD???s FUSE filesystem kernel module to support > > > extended attributes: > > > > > > https://people.freebsd.org/~ken/fuse_extattr.20160229.1.txt > > > > > > The patch implements the get/set/delete/list extended attribute methods. > > > The > > > listing code also converts extended attribute lists from the Linux/FUSE > > > format to the FreeBSD format. > > I also have patches, although my list didn't work. (I didn't know that > > there was > > a difference between what Linux/FUSE returns vs what FreeBSD wanted. So now > > I know > > why my "list" didn't work.) > > Btw, when I discussed what to do w.r.t. extended attribute namespace, he > > seemed to > > think just considering all the fuse ones as User was ok. > > Ahh. Who is "he" in this context? > rwatson@ > It was easy enough to allow access to the user and system namespace. FUSE > and Linux use the same names, so might as well pass them through. The > other issue as far as FUSE goes, is that it is expecting the "user." or > "system." prefix on the attribute name. > Well, the extended attributes I am interested in are generated automagically by GlusterFS and they don't have a "user." or "system." prefix. For example: glusterfs.gfid This wouldn't work if fuse prepended the "user." or "system.". > FreeBSD passes those as a separate, numeric, argument in the VFS layer at > least, and expects to not see the namespace as a prefix when listing > attributes. > > > I also have patched FreeBSD's Fuse for the other stuff needed to export the > > fuse > > mount via an NFS server. (VFS_FHTOVP() etc) These aren't quite ready for > > "prime time" > > but if anyone wants to try them, just email me and I'll send you a copy. > > > > Btw, I have a couple of patches related to direct I/O and buffered I/O. You > > can > > find 2 of these at PR#206238. I also patched fuse_vnop_inactive() to > > flush/write > > dirty pages. (I think this is needed when an application mmaps a file and > > the > > modifies it after closing the file descriptor. I haven't actually tested to > > see > > if this fix is needed, so I haven't put it anywhere yet.) > > Ahh, cool! Now I can export a tape via NFS! :) > > Seriously, though, that will be helpful for some filesystems. > Btw, I have also done VOP_ADVLOCK(), since GlusterFS supports that. > > > For example: > > > > > > # touch foo > > > # ls -la foo > > > -rwxrwxrwx 1 root wheel 0 Feb 29 21:40 foo > > > # lsextattr user foo > > > foo > > > # setextattr user testattr1 "12345678" foo > > > # lsextattr user foo > > > foo testattr1 > > > # getextattr user testattr1 foo > > > foo 12345678 > > > # setextattr user testattr2 "87654321" foo > > > # lsextattr user foo > > > foo testattr2 testattr1 > > > # rmextattr user testattr1 foo > > > # lsextattr user foo > > > foo testattr2 > > > # getextattr user testattr1 foo > > > getextattr: foo: failed: Attribute not found > > > # getextattr user testattr2 foo > > > foo 87654321 > > > > > > > > > Just to be clear on what this does, it only provides extended attribute > > > support to FreeBSD applications if the underlying FUSE filesystem > > > implements > > > FUSE extended attribute support. Many FUSE filesystems don???t support > > > the > > > extended attribute VFS operations. > > > > > > I have tested this out on IBM???s LTFS implementation, but I have not yet > > > found > > > another FUSE filesystem that supports extended attributes. If anyone > > > knows > > > of one, please let me know so I can try it out. (I looked through a > > > number > > > of the filesystems in sysutils/fusefs* in the ports tree.) > > > > > The FreeBSD GlusterFS port includes a fuse interface that supports extended > > attributes. That is how I tested what I have. (I think the port is now in > > svn, but if not you can find the GlusterFS port at PR#194409.) > > It glusterfs.org doc doesn't mention that you can create/test a volume of > > only one brick, but that works for trivial testing. If you decide to use it > > for testing and have trouble getting it going, just email. I know diddly > > about > > GlusterFS, but I have fired it up a few times. > > Ahh, that would be helpful. I'll give it a try. > > > > Any feedback is welcome. I???m planning to check this into FreeBSD/head > > > in the > > > next week or so. > > > > > I'll try to get around and taking a look to see if there is anything > > different > > than what I did (other than the above "list" case I didn't get right;-). > > Yes, it would be good to get another set of eyes. Perhaps the namespace > handling (user versus system) is different as well. > Yep, as above, the "namespace" is an issue. Have fun with it, rick > Ken > -- > Kenneth Merry > ken@FreeBSD.ORG > From owner-freebsd-scsi@freebsd.org Wed Mar 2 03:13:49 2016 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E422CAC0940 for ; Wed, 2 Mar 2016 03:13:49 +0000 (UTC) (envelope-from scott4long@yahoo.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id C69DB1C44 for ; Wed, 2 Mar 2016 03:13:49 +0000 (UTC) (envelope-from scott4long@yahoo.com) Received: by mailman.ysv.freebsd.org (Postfix) id C4718AC093E; Wed, 2 Mar 2016 03:13:49 +0000 (UTC) Delivered-To: scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ABA48AC093C for ; Wed, 2 Mar 2016 03:13:49 +0000 (UTC) (envelope-from scott4long@yahoo.com) Received: from nm6-vm3.bullet.mail.gq1.yahoo.com (nm6-vm3.bullet.mail.gq1.yahoo.com [98.136.218.194]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 851541C42 for ; Wed, 2 Mar 2016 03:13:48 +0000 (UTC) (envelope-from scott4long@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1456888040; bh=UCZKjbPdIVUwMTzDRBF6t2TsyAPQThiGsCYj4EcMn3M=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject; b=cFjVE2PbLhuX60/cEUKGYYbhIEQeJBJ/vqovNwqEefJeg3gLQBkV8urRxuHO3Q3RMzr1HWe1VXQMYq2uf71H57++kqBLoqWq+tbu0rhw905zomvNMvF6EJUdwt569Qm4mwThnSCzJ2Qjias1EkWfu2CLFWnoM/VhYy8qnL3/67GcRqswUIfZju0WVlsxMyzX1aeUHiMEzzx8D9UdTSOBbZtB8QBAudALTTLfJEFdtRS042lz5i5okImiu9YewgYUsXei8rskA8/epiRb0uPrz1VkdgiTD3iOOm4eQI96ePQ6lz3Za8xy4S2ksBNmhs7gFRq/FAmfOFLvRimTEPzVow== Received: from [98.137.12.59] by nm6.bullet.mail.gq1.yahoo.com with NNFMP; 02 Mar 2016 03:07:20 -0000 Received: from [98.136.164.76] by tm4.bullet.mail.gq1.yahoo.com with NNFMP; 02 Mar 2016 03:07:20 -0000 Received: from [127.0.0.1] by smtp238.mail.gq1.yahoo.com with NNFMP; 02 Mar 2016 03:07:20 -0000 X-Yahoo-Newman-Id: 398030.1184.bm@smtp238.mail.gq1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: nwGW3xoVM1ljTQPYibjrm1mnX6yOxVOs5c3.d6oU5AeNIMO T87POG1p9CluxHr61TK5Th31_EeFuw.iVCcHZhxHn74GewwUBSB97YjXDIuQ GpWnZPmGnVPGZvVk.umJTHmZYOMAKHua1TfcLG.HfKNpcRyVJ1ujZ6napN53 12gMeqr_2_TBRC7N2p2yT5_Z6LvuDaeYCta5hvKjgzDcKoFJ9.gvZ0mhtc0w wJAycRY8UAg97uefUmgDF_an_ju2YHuDJqWTCQZaLCnKI6UPjOLUdyFkGmJl Pp6Y0ryOSyi1ReI.5nkHI6Hsbn3JbY7i_WivphrUh1PxuSaVoxnQER2oSX2L UP4K3_exRbWQEYbFyzzdASMFGdPQgyMwkEfs.vFP7s6LaquLLX6YHCHC6PkR MSePIrwM6mTHJqLOkADIAwRjCge091jAcW4qg6EzRYKL.HSuvWtVytrLO.Y. rDOofDTWZUXzKKtQ1OoO1wroNgPh_Pbc_P2FHHbY.bIvuf3BHwkR5sMqcfm. ZwrGKTPSdwN_gRkAQHr0crWizQQZKk0J5v78.2g0E X-Yahoo-SMTP: clhABp.swBB7fs.LwIJpv3jkWgo2NU8- Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.2 \(3112\)) Subject: Re: CAM Shingled Disk support patches available From: Scott Long In-Reply-To: <20160301224758.GA86834@mithlond.kdm.org> Date: Tue, 1 Mar 2016 20:07:19 -0700 Cc: scsi@freebsd.org, current@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <20151118171309.GA3564@mithlond.kdm.org> <20160118223704.GA87506@mithlond.kdm.org> <20160301224758.GA86834@mithlond.kdm.org> To: "Kenneth D. Merry" X-Mailer: Apple Mail (2.3112) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Mar 2016 03:13:50 -0000 Hi Ken, I=E2=80=99m against changing the function signature of = scsi_ata_pass_16(). Even if you manage to get things right with symbol versioning, it still leads = to problems of code compatibility. Maybe pre-existing binaries will work, = but source code will forever have to include an #if __FreeBSD_version < xxxxxx bit of nonsense. I agree that it was incorrect for dxferlen to be declared as a uint16_t. However, the function already contains a sector count argument pair. In theory the sector count multiplied by the sector length, both of which = the application should know in order to arrive at a sensible dxferlen, can substitute for the dxferlen argument. If so, then we can just ignore = that argument and declare that sector_count has logical priority. Really though, I think that scsi_ata_pass_16() is a crummy function. If = its purpose is to implement SAT-3 12.2.2, it does an incredibly poor job at = it: - By my count, it only covers 12 of the available 13 registers. - It has no 12 byte, opcode 0xa1 variant. - It doesn=E2=80=99t make any allowance for providing the response = registers to the caller on completion. Well, maybe it kinda does through a sense = descriptor, but=E2=80=A6. it=E2=80=99s kinda open to vague interpretation. - Its use of the registers is clunky, assuming for example that you=E2=80=99= ll only want to fill the six LBA registers with a host-ordered 64-bit number. There = are plenty of commands that re-use sub-parts of the LBA, features, and/or = sector count registers for different things. =20 I know you stated that you didn=E2=80=99t want to do this, but I think = it=E2=80=99s better to start over with a better function that has a better signature and a new name. = In fact, I think it=E2=80=99s better to use the existing ata_cmd and ata_res = structures from=20 sys/cam/ata/ata_all.h, provide accessors for the multi-byte registers if = needed, provide a 12-byte compatibility, and simply the signature. Something = like this: void scsi_ata_pass(struct ccb_scsiio *csio, u_int32_t retries, void (*cbfcnp)(struct cam_periph *, union ccb *), u_int32_t flags, u_int8_t tag_action, struct ata_cmd *cmd, struct ata_res *res, u_int8_t *data_ptr, u_int32_t dxfer_len, u_int8_t *data_ptr, u_int16_t dxfer_len, u_int8_t sense_len, u_int32_t timeout); To differentiate between the 12 and 16 byte variants, you=E2=80=99d look = at the AP_EXTEND flag in the protocol field. Btw, the handling of that flag is inconsistent in the implementation of the existing scsi_ata_pass_16(). = If the caller providse an ata_res pointer then it gets filled on = completion, otherwise the caller does its best to look at 12.2.2.6 and extract what = it can from the sense descriptor. So my proposal is to create a new scsi_ata_pass and deprecate but not = remove scsi_ata_pass_16. Tell people that if they need to use it, dxfer_len is = going to have lower priority than sector_count/sector_count_exp if the latter = multiply to more than 65535. Scott > On Mar 1, 2016, at 3:47 PM, Kenneth D. Merry wrote: >=20 > I have a new set of SMR patches available. (See the original message = below > for a more detailed description of what these patches do.) >=20 > The primary change is to add library versioning to libcam so that we = can > change the function prototype of scsi_ata_pass_16() in a way that = won't > break existing binaries. >=20 > If someone more familiar with library versioning wants to review this, = I'd > appreciate it. >=20 > The patches are here: >=20 > FreeBSD/head, as of SVN revision 296278 >=20 > https://people.freebsd.org/~ken/cam_smr.head.20160301.1.txt >=20 > stable/10, as of SVN revision 296248 >=20 > https://people.freebsd.org/~ken/cam_smr.stable10.20160301.1.txt >=20 > (Note that although there is a stable/10 version of the patches, I'm = not > planning to merge them to stable/10 because of the change to struct = bio. I > can't really figure out a good way to make that backward compatible. = If > there is consensus that breaking it is fine because it isn't a user = API, > then that may be another story.) >=20 > The problem is that the existing, in-tree version of = scsi_ata_pass_16() has > a dxfer_len argument that is a uint16_t. That restricts transfer = sizes to > 64KB. So, we need to update it to allow larger than 64K transfers. I > could just create a new function, but I'd rather just retire the = broken > version. >=20 > The intent here is that: >=20 > 1. Binaries built against the old version of libcam, before versioning = was > turned on, will get the old version of the scsi_ata_pass_16() function = with > a uint16_t dxfer_len. >=20 > 2. Binaries built against the new version of libcam will get the new > version of the scsi_ata_pass_16() function with a uint32_t dxfer_len. >=20 > I've tested this, and it appears to work, but I'm not 100% certain = this is > all correct. I looked at Dan Eischen's description of symbol = versioning > here: >=20 > https://people.freebsd.org/~deischen/symver/freebsd_versioning.txt >=20 > And it looks like the actual implementation is a little different than = what > is described there. I looked around the tree, and didn't see anything = that > is obviously exactly like what I'm trying to do here. >=20 > So, what I did is as follows: >=20 > 1. For the kernel, the only change is to switch the dxfer_len argument = from > a uint16_t to a uint32_t. >=20 > 2. For userland, in scsi_all.c, there are now two versions of > scsi_ata_pass_16 -- _ver1 and _ver2. _ver1 is aliased to > scsi_ata_pass_16() for FBSD_1.3 using __sym_compat(). _ver2 is = aliased to > scsi_ata_pass_16() for FBSD_1.4 using __sym_default(). >=20 > 3. In lib/libcam/Versions.def, I defined FBSD_1.3 and FBSD_1.4, which > depends on FBSD_1.3. >=20 > 4. In lib/libcam/Symbol.map, I pulled out all of the functions defined = in > libcam, sorted them, and defined them in FBSD_1.3. I moved > scsi_ata_pass_16() to FBSD_1.4. (According to the = freebsd_versioning.txt > paper linked above, I should have been able to have scsi_ata_pass_16() = in > both FBSD_1.3 and FBSD_1.4, but that isn't the case in practice.) >=20 > In testing an old binary (linked against libcam without symbol = versioning) > against a new libcam (with symbol versioning), the old version of the > function appears to be used. With a new binary, the new version of = the > function appears to be used. >=20 > So it looks like things work as intended, but I don't fully trust my > understanding here. So, if someone could take a look at the changes, = I'd > appreciate it. >=20 > In particular, I have a few questions: >=20 > 1. If this change to scsi_ata_pass_16() gets merged to stable/10 = (apart > from the larger SMR changes), what should be done with the libcam = library > version? >=20 > 2. Are 1.3 and 1.4 the proper versions to use? >=20 > 3. If we make additional CAM helper function library changes, when do = the > versions get bumped? i.e., is this an opportunity to look for other > library functions with issues and make changes if possible? >=20 > 4. When you're going from an unversioned library to a versioned = library, > which version of a function gets linked in to a binary linked to the > unversioned library when you run it against a versioned library? In = other > words, what is supposed to happen in the test scenario I tried above, = and > am I really seeing what is supposed to happen? >=20 > Thanks, >=20 > Ken >=20 > On Mon, Jan 18, 2016 at 17:37:04 -0500, Kenneth D. Merry wrote: >> I have a new set of SMR patches available. See below for the full >> explanation. >>=20 >> The primary change here is that I have added SMR support to the = ada(4) >> driver. I spent some time considering whether to try to make the = da(4) and >> ada(4) probe infrastructure somewhat common, but in the end concluded = it >> would be too involved with not enough code reduction (if any) in the = end. >>=20 >> So, although the ideas are similar, the probe logic is separate. >>=20 >> Note that NCQ support for SMR commands (Report Zones, Reset Write = Pointer, >> etc.) for SATA protocol shingled drives isn't active. For both the = da(4) >> and ada(4) driver this is for lack of a way to plumb the ATA = Auxiliary >> register down to the drive. >>=20 >> In the ada(4) case, we need to add the register to struct ccb_ataio = and >> add support in one or more of the underlying SATA drivers, e.g. = ahci(4). >>=20 >> In the da(4) case, it will require an update of the T-10 SAT spec to >> provide a way to pass the Auxiliary register down via the SCSI ATA >> PASS-THROUGH command, and then a subsquent update of the SAT layer in >> various vendors' SAS controller firmware. At that point, there may = be=20 >> an official mapping of the SCSI ZBC commands to the ATA ZAC commands, = and >> we may be able to just issue the SCSI version of the commands instead = of >> composing ATA commands in the da(4) driver. (We'll still need to = keep the >> ATA passthrough version for a while at least to support controllers = that >> don't have the updated translation code.) >>=20 >> FreeBSD/head as of SVN revision 294105: >>=20 >> https://people.freebsd.org/~ken/cam_smr.head.20160118.1.txt >>=20 >> FreeBSD stable/10 as of SVN revision 294100: >>=20 >> https://people.freebsd.org/~ken/cam_smr.stable10.20160118.1.txt >>=20 >> Testing and comments are welcome. >>=20 >> Ken >>=20 >> On Wed, Nov 18, 2015 at 12:13:09 -0500, Kenneth D. Merry wrote: >>>=20 >>> I have work in progress patches to add SMR (Shingled Magnetic = Recording) >>> support to CAM and GEOM here: >>>=20 >>> FreeBSD/head as of SVN revision 290997: >>>=20 >>> https://people.freebsd.org/~ken/cam_smr.head.20151117.1.txt >>>=20 >>> FreeBSD stable/10 as of SVN revision 290995: >>>=20 >>> https://people.freebsd.org/~ken/cam_smr.stable10.20151117.1.txt >>>=20 >>> This includes support for Host Managed, Host Aware and Drive Managed = SMR >>> drives that are either SCSI (ZBC) or ATA (ZAC) attached via a SAS >>> controller. This does not include support for SMR ATA drives = attched via >>> an ATA controller. Also, I have not yet figured out how to properly = detect >>> a Host Managed ATA drive, so this code won't do that. >>>=20 >>> The big drive vendors are moving to SMR for at least some of their = drives. >>> The primary challenge with SMR is that it requires writing a = relatively >>> large zone sequentially starting at the beginning of the zone. The = usual >>> zone size is 256MB. It is conceptually almost like having a 256MB = sector >>> size. >>>=20 >>> We (Spectra Logic) are working on ZFS changes that will use this CAM = and >>> GEOM infrastructure to make ZFS play well with SMR drives. Those = changes >>> aren't yet done. >>>=20 >>> The patches linked above include: >>> o A new 'camcontrol zone' command that allows displaying and = managing >>> drive zones via SCSI/ATA passthrough. >>> o A new zonectl(8) utility that uses the new DIOCZONECMD ioctl to = display >>> and manage zones via the da(4) (and later ada(4)) driver. >>> o Changes to diskinfo -v to display the zone mode of a drive. >>> o A new disk zone API, sys/sys/disk_zone.h. >>> o A new bio type, BIO_ZONE, and modifications to GEOM to support it. = This >>> new bio will allow filesystems to query zone support in a drive = and >>> manage zoned drives. >>> o Extensive modifications to the da(4) driver to handle probing SCSI = and >>> SATA behind SAS SMR drives. >>> o Additional CAM CDB building functions for zone commands. >>>=20 >>> The current issues that need to be addressed are: >>> o The da(4) driver now has 6 additional probe states, 5 of which are >>> needed for probing ATA drives behind SAS controllers. I have not = yet >>> added support for BIO_ZONE bios to ada(4), but it will be very = similar >>> to the da(4) driver version. The ATA probe code needs to be = pulled >>> out of the da(4) driver and changed into a form that will allow it = to >>> work for either the ada(4) or da(4) driver. Otherwise we'll have = a fair >>> amount of code duplication between the two drivers. >>>=20 >>> o There is a reasonable amount of code duplication between = 'camcontrol zone' >>> and zonectl(8). This was done for speed / expediency's sake, but = it may >>> be possible to make more of the code common there. >>>=20 >>> o In order to add the new BIO_ZONE bio command, I had to change the = bio_cmd >>> field in struct bio from a uint8_t to a uint16_t. This will cause >>> binary compatibility problems with any 3rd party loadable modules. >>> Advice on how to handle this would be welcome. >>>=20 >>> o In the process of developing these changes, I discovered that the >>> dxfer_len paramter for scsi_ata_pass_16() was too small (uint16_t, = and >>> it needed to be uint32_t). I increased it, but that will = potentially >>> cause a binary incompatibility problem with any existing = applications >>> that use the current API via libcam. Advice on how to handle that >>> would be welcome. >>>=20 >>> If you look through the code, you'll notice that the disk_zone.h API = is >>> separate from the SCSI and ATA APIs. The intent is to allow = filesystems >>> and other consumers of the API to just talk to the disk zone API = without >>> dealing with the SCSI and ATA specifics. Another reason behind all = of this >>> is that even though the SCSI ZBC and ATA ZAC specs were developed in >>> concert, and are intended to be functionally identical, they are = still SCSI >>> and ATA. As usual, SCSI is big endian and ATA is little endian. So = to >>> present a common API to the filesystem, we give all of the zone data = back >>> in native byte order, regardless of the underlying device protocol. >>>=20 >>> Another thing to note is the extensive use of ATA passthrough in the = da(4) >>> driver. This is necessary because the SCSI SAT (SCSI to ATA = Translation) >>> specification has not yet caught up with translating SCSI zone = commands >>> (ZBC) to ATA zone commands (ZAC). So, until the spec is updated and = LSI >>> and other vendors update their SCSI to ATA translation layers, we'll = have >>> to use the ATA version of the commands when talking to ATA drives = via SAS >>> controllers. >>>=20 >>> I have only tested the code so far with Seagate SATA Drive Managed = and Host >>> Aware drives. I would appreciate testing with any drives. (And = testing to >>> make sure that the patches don't cause problems with existing = hardware.) >>> Right now, all you can really do is manage the zones manually using >>> camcontrol(8) or zonectl(8). Automatic management will come with = the ZFS >>> changes. (Or changes to other filesysems if people want to do it.) >>>=20 >>> If you have a SATA Host Aware drive, in theory camcontrol(8) should = allow >>> you to manage the drive if you have it attached to a SATA = controller. >>>=20 >>> Here is an example of some of the commands. >>>=20 >>> Get general zoning information: >>>=20 >>> [root@sm4u-1 ~]# zonectl -c params -d /dev/da21 >>> Zone Mode: Host Aware >>> Command support: Report Zones, Open, Close, Finish, Reset Write = Pointer >>> Unrestricted Read in Sequential Write Required Zone (URSWRZ): No >>> Optimal Number of Open Sequential Write Preferred Zones: 128 >>> Optimal Number of Non-Sequentially Written Sequential Write = Preferred Zones: 8 >>> Maximum Number of Open Sequential Write Required Zones: Unlimited >>>=20 >>> Look at information from the da(4) driver: >>>=20 >>> [root@sm4u-1 ~]# sysctl kern.cam.da.21 >>> kern.cam.da.21.delete_method: NONE >>> kern.cam.da.21.delete_max: 1081344 >>> kern.cam.da.21.minimum_cmd_size: 6 >>> kern.cam.da.21.sort_io_queue: -1 >>> kern.cam.da.21.zone_mode: Host Aware >>> kern.cam.da.21.zone_support: Report Zones, Open, Close, Finish, = Reset Write Pointer >>> kern.cam.da.21.optimal_seq_zones: 128 >>> kern.cam.da.21.optimal_nonseq_zones: 8 >>> kern.cam.da.21.max_seq_zones: 4294967295 >>> kern.cam.da.21.error_inject: 0 >>>=20 >>> Display all of the zones with zonectl(8): >>>=20 >>> [root@sm4u-1 ~]# zonectl -d /dev/da21 -c rz >>> 29809 zones, Maximum LBA 0x3a3812aaf (15628053167) >>> Zone lengths and types may vary >>> Start LBA Length WP LBA Zone Type Condition = Sequential Reset >>> 0, 524288, 0x80000, Conventional, NWP, = Sequential, No Reset Needed >>> 0x80000, 524288, 0x100000, Conventional, NWP, = Sequential, No Reset Needed >>> 0x100000, 524288, 0x180000, Conventional, NWP, = Sequential, No Reset Needed >>> 0x180000, 524288, 0x200000, Conventional, NWP, = Sequential, No Reset Needed >>> 0x200000, 524288, 0x280000, Conventional, NWP, = Sequential, No Reset Needed >>> 0x280000, 524288, 0x300000, Conventional, NWP, = Sequential, No Reset Needed >>> 0x300000, 524288, 0x380000, Conventional, NWP, = Sequential, No Reset Needed >>> 0x380000, 524288, 0x400000, Conventional, NWP, = Sequential, No Reset Needed >>> 0x400000, 524288, 0x480000, Conventional, NWP, = Sequential, No Reset Needed >>> 0x480000, 524288, 0x500000, Conventional, NWP, = Sequential, No Reset Needed >>> 0x500000, 524288, 0x580000, Conventional, NWP, = Sequential, No Reset Needed >>> 0x580000, 524288, 0x600000, Conventional, NWP, = Sequential, No Reset Needed >>> 0x600000, 524288, 0x680000, Conventional, NWP, = Sequential, No Reset Needed >>> 0x680000, 524288, 0x700000, Conventional, NWP, = Sequential, No Reset Needed >>> 0x700000, 524288, 0x780000, Conventional, NWP, = Sequential, No Reset Needed >>> [ ... ] >>> 0x1f00000, 524288, 0x1f80000, Conventional, NWP, = Sequential, No Reset Needed >>> 0x1f80000, 524288, 0x2000000, Conventional, NWP, = Sequential, No Reset Needed >>> 0x2000000, 524288, 0x2080000, Seq Preferred, Full, = Sequential, No Reset Needed >>> 0x2080000, 524288, 0x2100000, Seq Preferred, Full, = Sequential, No Reset Needed >>> 0x2100000, 524288, 0x2180000, Seq Preferred, Full, = Sequential, No Reset Needed >>> 0x2180000, 524288, 0x2200000, Seq Preferred, Full, = Sequential, No Reset Needed >>> 0x2200000, 524288, 0x2280000, Seq Preferred, Full, = Sequential, No Reset Needed >>> 0x2280000, 524288, 0x2300000, Seq Preferred, Full, = Sequential, No Reset Needed >>> [ ... ] >>>=20 >>> Use camcontrol zone to reset the write pointer for the first = shingled zone >>> listed above: >>>=20 >>> [root@sm4u-1 ~]# camcontrol zone da21 -v -c rwp -l 0x2000000 >>>=20 >>> Use camcontrol zone to ask the drive to report empty zones: >>>=20 >>> [root@sm4u-1 ~]# camcontrol zone da21 -v -c rz -o empty >>> 1 zones, Maximum LBA 0x3a3812aaf (15628053167) >>> Zone lengths and types may vary >>> Start LBA Length WP LBA Zone Type Condition = Sequential Reset >>> 0x2000000, 524288, 0x2000000, Seq Preferred, Empty, = Sequential, No Reset Needed >>>=20 >>> Get information on a Host Aware drive: >>>=20 >>> root@sm4u-1 ~]# diskinfo -v da21 >>> da21 >>> 512 # sectorsize >>> 8001563222016 # mediasize in bytes (7.3T) >>> 15628053168 # mediasize in sectors >>> 4096 # stripesize >>> 0 # stripeoffset >>> 972801 # Cylinders according to firmware. >>> 255 # Heads according to firmware. >>> 63 # Sectors according to firmware. >>> Z84008NY # Disk ident. >>> enc@5003048001f311fd/elmtype@array_device/slot@22 # = Physical path >>> Host Aware # Zone Mode >>>=20 >>> Get information on a drive managed drive: >>>=20 >>> [root@sm4u-1 ~]# diskinfo -v da20 >>> da20 >>> 512 # sectorsize >>> 8001563222016 # mediasize in bytes (7.3T) >>> 15628053168 # mediasize in sectors >>> 4096 # stripesize >>> 0 # stripeoffset >>> 972801 # Cylinders according to firmware. >>> 255 # Heads according to firmware. >>> 63 # Sectors according to firmware. >>> Z8405938 # Disk ident. >>> enc@5003048001f311fd/elmtype@array_device/slot@21 # = Physical path >>> Drive Managed # Zone Mode >>>=20 >>> Get information on a non-zoned drive: >>>=20 >>> [root@sm4u-1 ~]# diskinfo -v da4 >>> da4 >>> 512 # sectorsize >>> 100030242816 # mediasize in bytes (93G) >>> 195371568 # mediasize in sectors >>> 0 # stripesize >>> 0 # stripeoffset >>> 12161 # Cylinders according to firmware. >>> 255 # Heads according to firmware. >>> 63 # Sectors according to firmware. >>> 124903574F36 # Disk ident. >>> enc@5003048001f311fd/elmtype@array_device/slot@5 # = Physical path >>> Not Zoned # Zone Mode >>>=20 >>> Testing and comments are welcome. >>>=20 >>> Thanks, >>>=20 >>> Ken >>> --=20 >>> Kenneth Merry >>> ken@FreeBSD.ORG >>=20 >> --=20 >> Kenneth Merry >> ken@FreeBSD.ORG >=20 > --=20 > Kenneth Merry > ken@FreeBSD.ORG > _______________________________________________ > freebsd-scsi@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-scsi > To unsubscribe, send any mail to = "freebsd-scsi-unsubscribe@freebsd.org" From owner-freebsd-scsi@freebsd.org Wed Mar 2 07:23:17 2016 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F2F05AC0ABD for ; Wed, 2 Mar 2016 07:23:17 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from cu01176a.smtpx.saremail.com (cu01176a.smtpx.saremail.com [195.16.150.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 71FB51CEB for ; Wed, 2 Mar 2016 07:23:16 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from [172.16.8.96] (izaro.sarenet.es [192.148.167.11]) by proxypop03.sare.net (Postfix) with ESMTPSA id 101CD9DC95E; Wed, 2 Mar 2016 08:23:07 +0100 (CET) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.2 \(3112\)) Subject: Re: mpr(4) SAS3008 Repeated Crashing From: Borja Marcos In-Reply-To: <56D612FA.6090909@multiplay.co.uk> Date: Wed, 2 Mar 2016 08:23:07 +0100 Cc: freebsd-scsi@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <56D5FDB8.8040402@freebsd.org> <56D612FA.6090909@multiplay.co.uk> To: Steven Hartland X-Mailer: Apple Mail (2.3112) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Mar 2016 07:23:18 -0000 > On 01 Mar 2016, at 23:08, Steven Hartland = wrote: >=20 > Initial ideas would be bad signalling. >=20 > If you have the option to drop the speeds down and that helps then = almost certainly the case. >=20 > The original mfi driver was very bad at recovering from issues like = this too, I spent over a month fixing and patching it to get it working = reliably when there where hardware related issues. In my case it turned = out the be a dodge CPU causing memory corruption but you'll get similar = behaviour from badly designed installs, particularly with expanders in = play for high speed devices (6-12Gbps) link speed. I=E2=80=99ve suffered similar problems, although not as severe, on one = of my storage servers. It=E2=80=99s an IBM X Series with a LSI 3008 HBA=20= connected to the backplane, using SATA SSDs. But mine are almost = certainly hardware problems. An identical system is working without issues. The symptom: with high I/O activity, for example, running Bonnie++, some = commands abort with the disks returning a unit attention (power on/reset) asc 0,29. it definitely this looks like a hardware problem to me. Might be the = backplane (it doesn=E2=80=99t affect the same disk every time, it=E2=80=99s = completely random) or maybe a power supply problem making the disks = reset? And it hasn=E2=80=99t caused serious data corruption. (It=E2=80=99s = decomissioned for now, of coursw!) Now and then ZFS will complain of a = checksum failure, but a scrub fixes it. Now I=E2=80=99m fighting with IBM (now Lenovo) because all the = components were sourced from them and it=E2=80=99s their call to debug = it. Maybe I=E2=80=99ll hook an oscilloscope to the power rails to check for suspicious transients or something like = that, though. So far their response has been absolutely unacceptable. = They ask for the =E2=80=9CRAID vendor=E2=80=9D, and they seem unable to understand that = someone might want to run these things with an OS different than = Windows, and without creating RAID volumes with the built in controller. Sigh. Maybe I could bribe someone to pose as =E2=80=9CRAID vendor=E2=80=9D ;) Feb 12 07:43:59 clientes-ssd8 kernel: (noperiph:mpr0:0:4294967295:0): = SMID 33 Aborting command 0xfffffe0000c7baf0 Feb 12 07:43:59 clientes-ssd8 kernel: (da14:mpr0:0:40:0): WRITE(10). = CDB: 2a 00 39 a1 fe f0 00 00 20 00 length 16384 SMID 989 terminated ioc = 804b scsi 0 state c xfer 0 Feb 12 07:43:59 clientes-ssd8 kernel: (da14:mpr0:0:40:0): READ(10). CDB: = 28 00 31 40 ea 20 00 00 18 00 length 12288 SMID 953 terminated ioc 804b = scsi 0 state c xfe(da14:mpr0:0:40:0): READ(10). CDB: 28 00 31 40 ea 40 = 00 00 20 00=20 Feb 12 07:43:59 clientes-ssd8 kernel: (da14:mpr0:0:40:0): CAM status: = Command timeout Feb 12 07:43:59 clientes-ssd8 kernel: (da14:mpr0:0:40:0): READ(10). CDB: = 28 00 31 40 ea 00 00 00 20 00 length 16384 SMID 571 terminated ioc 804b = scsi 0 state c xfe(da14:r 0 Feb 12 07:43:59 clientes-ssd8 kernel: mpr0:0: (da14:mpr0:0:40:0): ATA = COMMAND PASS THROUGH(16). CDB: 85 0d 06 00 01 00 01 00 00 00 00 00 00 40 = 06 00 length 512 SMID 638 te40:rminated ioc 804b scsi 0 state c xfer 0 Feb 12 07:44:00 clientes-ssd8 kernel: mpr0: log_info(0x31120440): = originator(PL), code(0x12), sub_code(0x0440) Feb 12 07:44:00 clientes-ssd8 kernel: (da14:mpr0:0:40:0): WRITE(10). = CDB: 2a 00 39 a1 fe f0 00 00 20 00 length 16384 SMID 818 terminated ioc = 804b scsi 0 state c xfer 0 Feb 12 07:44:00 clientes-ssd8 kernel: mpr0: log_info(0x31120440): = originator(PL), code(0x12), sub_code(0x0440) Feb 12 07:44:00 clientes-ssd8 kernel: (da14:mpr0:0:40:0): READ(10). CDB: = 28 00 31 40 ea 40 00 00 20 00 length 16384 SMID 952 terminated ioc 804b = scsi 0 state c xfer 0 Feb 12 07:44:00 clientes-ssd8 kernel: mpr0: log_info(0x31120440): = originator(PL), code(0x12), sub_code(0x0440) Feb 12 07:44:00 clientes-ssd8 kernel: (da14:mpr0:0:40:0): READ(10). CDB: = 28 00 31 40 ea 20 00 00 18 00 length 12288 SMID 922 terminated ioc 804b = scsi 0 state c xfer 0 Feb 12 07:44:00 clientes-ssd8 kernel: mpr0: log_info(0x31120440): = originator(PL), code(0x12), sub_code(0x0440) Feb 12 07:44:00 clientes-ssd8 kernel: (da14:mpr0:0:40:0): READ(10). CDB: = 28 00 31 40 ea 00 00 00 20 00 length 16384 SMID 823 terminated ioc 804b = scsi 0 state c xfer 0 Feb 12 07:44:00 clientes-ssd8 kernel: (da14:mpr0:0:40:0): WRITE(10). = CDB: 2a 00 39 a1 fe f0 00 00 20 00=20 Feb 12 07:44:00 clientes-ssd8 kernel: (da14:mpr0:0:40:0): CAM status: = SCSI Status Error Feb 12 07:44:00 clientes-ssd8 kernel: (da14:mpr0:0:40:0): SCSI status: = Check Condition Feb 12 07:44:00 clientes-ssd8 kernel: (da14:mpr0:0:40:0): SCSI sense: = UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred) Feb 12 07:44:00 clientes-ssd8 kernel: (da14:mpr0:0:40:0): Retrying = command (per sense data) Borja. From owner-freebsd-scsi@freebsd.org Wed Mar 2 10:04:12 2016 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7A71AAC07EE for ; Wed, 2 Mar 2016 10:04:12 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wm0-x236.google.com (mail-wm0-x236.google.com [IPv6:2a00:1450:400c:c09::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 26A9911EC for ; Wed, 2 Mar 2016 10:04:11 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wm0-x236.google.com with SMTP id n186so77170310wmn.1 for ; Wed, 02 Mar 2016 02:04:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-transfer-encoding; bh=9lnKMwGvjV/2anbvlOQDeWWSSMbLsW3punbGoZ6iAC8=; b=Zf4cDVLMrCh6JQh+pfBbDaHcprkIWDLNhDhnblcByTen4w8Q1lE5+/OHogBdZgcgfF WV69jtO1zezGiKeJST7mvb7fAElHnurVD9OFis+DL5bgqvkvPsC3YijxGjR1MKR4vI/3 rd5cu9wl6mjuXI2uLefzvzENe7/7+x48gKOQDxPwppPAsse6JsqOs2unrL7ygueG3Xl6 sOG99tQT4MgvboiAk4YHiEMODQUd9VCZa+MIR1crbwta9OvMninNaPspkaj4fu8+b4au +LGhxwfOask9THfKdeM9woG0FAa3lus4yxWGoRnYejy4xjSdW9OCWbeiy3T7y6vf2bml 8Gwg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=9lnKMwGvjV/2anbvlOQDeWWSSMbLsW3punbGoZ6iAC8=; b=H7CQ8BWH/gn1C0jCS1cLTykrDfteBv3bQffvSCaRha/dtTjS9WHGuJS2aAd0OHTOZp iQI8Lz7JFWvAggHZbyliDNeQcQtabpYFfgye3LGYCcsVNVBw7YP9vBSqUCLCYjBzB1VN VnPgjk/WAxuihUSrCM70IXNX8QpIQ3J8juhqJPZSpJJhC2AxptGG+yIMieBvwfa06rIJ 8/A/yhDDaMmXrYYwvtZl2tovRs5Jf1A26sCnbRYAf/bquDOKGq21bzTJfPlv9EfgeEX/ tKwE6YbMMGXL2QEiB+pLlm7xJicM8ROVonaf+dW2ka6ZhuuNGmOqSmDPWKw+1hpLbnSA 5pLQ== X-Gm-Message-State: AD7BkJJ31gnM4A0m7761PbzliqmNLd7ssfqW8zUefSEZ+UfT1Tm+lhJv1raeuCuol+4s4PcB X-Received: by 10.28.132.17 with SMTP id g17mr3850543wmd.63.1456913050221; Wed, 02 Mar 2016 02:04:10 -0800 (PST) Received: from [10.10.1.58] (liv3d.labs.multiplay.co.uk. [82.69.141.171]) by smtp.gmail.com with ESMTPSA id t7sm5477097wjf.39.2016.03.02.02.04.08 for (version=TLSv1/SSLv3 cipher=OTHER); Wed, 02 Mar 2016 02:04:08 -0800 (PST) Subject: Re: CAM Shingled Disk support patches available To: freebsd-scsi@freebsd.org References: <20151118171309.GA3564@mithlond.kdm.org> <20160118223704.GA87506@mithlond.kdm.org> <20160301224758.GA86834@mithlond.kdm.org> From: Steven Hartland Message-ID: <56D6BAA7.1050004@multiplay.co.uk> Date: Wed, 2 Mar 2016 10:04:23 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.6.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Mar 2016 10:04:12 -0000 scsi_ata_pass_16 was added to support TRIM and implements 12.2.3 (not 12.2.2) providing access to all features as described by Table 104. I'll admit I didn't take into account callers which make non LBA use of the LBA fields, but I'm sure you'll agree that's an experience thing i.e. knowing about about the existence of features which abuse structure fields for other purposes ;-) Its use of u_int16_t dxfer_len is a bug, surprised the compiler didn't flag this, not sure this can be fixed without breaking backwards compatibility though? Not sure I understand the 12 vs 13 registers comment, I'm guessing you mean the 12 bytes of 12.2.2, however it actually exposes all 16 bytes of 12.2.3. For reference it has two callers in the kernel scsi_ata_identify used to probe TRIM capability and scsi_ata_trim used to actually perform TRIM operations. In addition to this its also used by camcontrol to support passthrough from ata_do_cmd and ata_do_28bit_cmd again supporting TRIM, in this case the ATA security feature set as well as identify. I would be very surprised if its used elsewhere but in order to maintain compatibility I'd definitely agree with Scott on creating an scsi_ata_pass which can be used to implement and scsi_ata_pass_16 which can then be deprecated. With regards to providing 12.2.2 and 12.2.3 by looking at AP_EXTEND I don't believe that's possible as you need this to tell the difference between 28bit and 48bit ATA within 12.2.3. Regards Steve On 02/03/2016 03:07, Scott Long via freebsd-scsi wrote: > Hi Ken, > > I’m against changing the function signature of scsi_ata_pass_16(). Even > if you manage to get things right with symbol versioning, it still leads to > problems of code compatibility. Maybe pre-existing binaries will work, but > source code will forever have to include an #if __FreeBSD_version < > xxxxxx bit of nonsense. > > I agree that it was incorrect for dxferlen to be declared as a uint16_t. > However, the function already contains a sector count argument pair. In > theory the sector count multiplied by the sector length, both of which the > application should know in order to arrive at a sensible dxferlen, can > substitute for the dxferlen argument. If so, then we can just ignore that > argument and declare that sector_count has logical priority. > > Really though, I think that scsi_ata_pass_16() is a crummy function. If its > purpose is to implement SAT-3 12.2.2, it does an incredibly poor job at it: > > - By my count, it only covers 12 of the available 13 registers. > > - It has no 12 byte, opcode 0xa1 variant. > > - It doesn’t make any allowance for providing the response registers to the > caller on completion. Well, maybe it kinda does through a sense descriptor, > but…. it’s kinda open to vague interpretation. > > - Its use of the registers is clunky, assuming for example that you’ll only want > to fill the six LBA registers with a host-ordered 64-bit number. There are > plenty of commands that re-use sub-parts of the LBA, features, and/or sector > count registers for different things. > > I know you stated that you didn’t want to do this, but I think it’s better to start > over with a better function that has a better signature and a new name. In fact, > I think it’s better to use the existing ata_cmd and ata_res structures from > sys/cam/ata/ata_all.h, provide accessors for the multi-byte registers if needed, > provide a 12-byte compatibility, and simply the signature. Something like this: > > void scsi_ata_pass(struct ccb_scsiio *csio, u_int32_t retries, > void (*cbfcnp)(struct cam_periph *, union ccb *), > u_int32_t flags, u_int8_t tag_action, > struct ata_cmd *cmd, struct ata_res *res, > u_int8_t *data_ptr, u_int32_t dxfer_len, > u_int8_t *data_ptr, u_int16_t dxfer_len, > u_int8_t sense_len, u_int32_t timeout); > > To differentiate between the 12 and 16 byte variants, you’d look at the > AP_EXTEND flag in the protocol field. Btw, the handling of that flag is > inconsistent in the implementation of the existing scsi_ata_pass_16(). If > the caller providse an ata_res pointer then it gets filled on completion, > otherwise the caller does its best to look at 12.2.2.6 and extract what it > can from the sense descriptor. > > So my proposal is to create a new scsi_ata_pass and deprecate but not remove > scsi_ata_pass_16. Tell people that if they need to use it, dxfer_len is going to > have lower priority than sector_count/sector_count_exp if the latter multiply to > more than 65535. > > Scott > > > >> On Mar 1, 2016, at 3:47 PM, Kenneth D. Merry wrote: >> >> I have a new set of SMR patches available. (See the original message below >> for a more detailed description of what these patches do.) >> >> The primary change is to add library versioning to libcam so that we can >> change the function prototype of scsi_ata_pass_16() in a way that won't >> break existing binaries. >> >> If someone more familiar with library versioning wants to review this, I'd >> appreciate it. >> >> The patches are here: >> >> FreeBSD/head, as of SVN revision 296278 >> >> https://people.freebsd.org/~ken/cam_smr.head.20160301.1.txt >> >> stable/10, as of SVN revision 296248 >> >> https://people.freebsd.org/~ken/cam_smr.stable10.20160301.1.txt >> >> (Note that although there is a stable/10 version of the patches, I'm not >> planning to merge them to stable/10 because of the change to struct bio. I >> can't really figure out a good way to make that backward compatible. If >> there is consensus that breaking it is fine because it isn't a user API, >> then that may be another story.) >> >> The problem is that the existing, in-tree version of scsi_ata_pass_16() has >> a dxfer_len argument that is a uint16_t. That restricts transfer sizes to >> 64KB. So, we need to update it to allow larger than 64K transfers. I >> could just create a new function, but I'd rather just retire the broken >> version. >> >> The intent here is that: >> >> 1. Binaries built against the old version of libcam, before versioning was >> turned on, will get the old version of the scsi_ata_pass_16() function with >> a uint16_t dxfer_len. >> >> 2. Binaries built against the new version of libcam will get the new >> version of the scsi_ata_pass_16() function with a uint32_t dxfer_len. >> >> I've tested this, and it appears to work, but I'm not 100% certain this is >> all correct. I looked at Dan Eischen's description of symbol versioning >> here: >> >> https://people.freebsd.org/~deischen/symver/freebsd_versioning.txt >> >> And it looks like the actual implementation is a little different than what >> is described there. I looked around the tree, and didn't see anything that >> is obviously exactly like what I'm trying to do here. >> >> So, what I did is as follows: >> >> 1. For the kernel, the only change is to switch the dxfer_len argument from >> a uint16_t to a uint32_t. >> >> 2. For userland, in scsi_all.c, there are now two versions of >> scsi_ata_pass_16 -- _ver1 and _ver2. _ver1 is aliased to >> scsi_ata_pass_16() for FBSD_1.3 using __sym_compat(). _ver2 is aliased to >> scsi_ata_pass_16() for FBSD_1.4 using __sym_default(). >> >> 3. In lib/libcam/Versions.def, I defined FBSD_1.3 and FBSD_1.4, which >> depends on FBSD_1.3. >> >> 4. In lib/libcam/Symbol.map, I pulled out all of the functions defined in >> libcam, sorted them, and defined them in FBSD_1.3. I moved >> scsi_ata_pass_16() to FBSD_1.4. (According to the freebsd_versioning.txt >> paper linked above, I should have been able to have scsi_ata_pass_16() in >> both FBSD_1.3 and FBSD_1.4, but that isn't the case in practice.) >> >> In testing an old binary (linked against libcam without symbol versioning) >> against a new libcam (with symbol versioning), the old version of the >> function appears to be used. With a new binary, the new version of the >> function appears to be used. >> >> So it looks like things work as intended, but I don't fully trust my >> understanding here. So, if someone could take a look at the changes, I'd >> appreciate it. >> >> In particular, I have a few questions: >> >> 1. If this change to scsi_ata_pass_16() gets merged to stable/10 (apart >> from the larger SMR changes), what should be done with the libcam library >> version? >> >> 2. Are 1.3 and 1.4 the proper versions to use? >> >> 3. If we make additional CAM helper function library changes, when do the >> versions get bumped? i.e., is this an opportunity to look for other >> library functions with issues and make changes if possible? >> >> 4. When you're going from an unversioned library to a versioned library, >> which version of a function gets linked in to a binary linked to the >> unversioned library when you run it against a versioned library? In other >> words, what is supposed to happen in the test scenario I tried above, and >> am I really seeing what is supposed to happen? >> >> Thanks, >> >> Ken >> >> On Mon, Jan 18, 2016 at 17:37:04 -0500, Kenneth D. Merry wrote: >>> I have a new set of SMR patches available. See below for the full >>> explanation. >>> >>> The primary change here is that I have added SMR support to the ada(4) >>> driver. I spent some time considering whether to try to make the da(4) and >>> ada(4) probe infrastructure somewhat common, but in the end concluded it >>> would be too involved with not enough code reduction (if any) in the end. >>> >>> So, although the ideas are similar, the probe logic is separate. >>> >>> Note that NCQ support for SMR commands (Report Zones, Reset Write Pointer, >>> etc.) for SATA protocol shingled drives isn't active. For both the da(4) >>> and ada(4) driver this is for lack of a way to plumb the ATA Auxiliary >>> register down to the drive. >>> >>> In the ada(4) case, we need to add the register to struct ccb_ataio and >>> add support in one or more of the underlying SATA drivers, e.g. ahci(4). >>> >>> In the da(4) case, it will require an update of the T-10 SAT spec to >>> provide a way to pass the Auxiliary register down via the SCSI ATA >>> PASS-THROUGH command, and then a subsquent update of the SAT layer in >>> various vendors' SAS controller firmware. At that point, there may be >>> an official mapping of the SCSI ZBC commands to the ATA ZAC commands, and >>> we may be able to just issue the SCSI version of the commands instead of >>> composing ATA commands in the da(4) driver. (We'll still need to keep the >>> ATA passthrough version for a while at least to support controllers that >>> don't have the updated translation code.) >>> >>> FreeBSD/head as of SVN revision 294105: >>> >>> https://people.freebsd.org/~ken/cam_smr.head.20160118.1.txt >>> >>> FreeBSD stable/10 as of SVN revision 294100: >>> >>> https://people.freebsd.org/~ken/cam_smr.stable10.20160118.1.txt >>> >>> Testing and comments are welcome. >>> >>> Ken >>> >>> On Wed, Nov 18, 2015 at 12:13:09 -0500, Kenneth D. Merry wrote: >>>> I have work in progress patches to add SMR (Shingled Magnetic Recording) >>>> support to CAM and GEOM here: >>>> >>>> FreeBSD/head as of SVN revision 290997: >>>> >>>> https://people.freebsd.org/~ken/cam_smr.head.20151117.1.txt >>>> >>>> FreeBSD stable/10 as of SVN revision 290995: >>>> >>>> https://people.freebsd.org/~ken/cam_smr.stable10.20151117.1.txt >>>> >>>> This includes support for Host Managed, Host Aware and Drive Managed SMR >>>> drives that are either SCSI (ZBC) or ATA (ZAC) attached via a SAS >>>> controller. This does not include support for SMR ATA drives attched via >>>> an ATA controller. Also, I have not yet figured out how to properly detect >>>> a Host Managed ATA drive, so this code won't do that. >>>> >>>> The big drive vendors are moving to SMR for at least some of their drives. >>>> The primary challenge with SMR is that it requires writing a relatively >>>> large zone sequentially starting at the beginning of the zone. The usual >>>> zone size is 256MB. It is conceptually almost like having a 256MB sector >>>> size. >>>> >>>> We (Spectra Logic) are working on ZFS changes that will use this CAM and >>>> GEOM infrastructure to make ZFS play well with SMR drives. Those changes >>>> aren't yet done. >>>> >>>> The patches linked above include: >>>> o A new 'camcontrol zone' command that allows displaying and managing >>>> drive zones via SCSI/ATA passthrough. >>>> o A new zonectl(8) utility that uses the new DIOCZONECMD ioctl to display >>>> and manage zones via the da(4) (and later ada(4)) driver. >>>> o Changes to diskinfo -v to display the zone mode of a drive. >>>> o A new disk zone API, sys/sys/disk_zone.h. >>>> o A new bio type, BIO_ZONE, and modifications to GEOM to support it. This >>>> new bio will allow filesystems to query zone support in a drive and >>>> manage zoned drives. >>>> o Extensive modifications to the da(4) driver to handle probing SCSI and >>>> SATA behind SAS SMR drives. >>>> o Additional CAM CDB building functions for zone commands. >>>> >>>> The current issues that need to be addressed are: >>>> o The da(4) driver now has 6 additional probe states, 5 of which are >>>> needed for probing ATA drives behind SAS controllers. I have not yet >>>> added support for BIO_ZONE bios to ada(4), but it will be very similar >>>> to the da(4) driver version. The ATA probe code needs to be pulled >>>> out of the da(4) driver and changed into a form that will allow it to >>>> work for either the ada(4) or da(4) driver. Otherwise we'll have a fair >>>> amount of code duplication between the two drivers. >>>> >>>> o There is a reasonable amount of code duplication between 'camcontrol zone' >>>> and zonectl(8). This was done for speed / expediency's sake, but it may >>>> be possible to make more of the code common there. >>>> >>>> o In order to add the new BIO_ZONE bio command, I had to change the bio_cmd >>>> field in struct bio from a uint8_t to a uint16_t. This will cause >>>> binary compatibility problems with any 3rd party loadable modules. >>>> Advice on how to handle this would be welcome. >>>> >>>> o In the process of developing these changes, I discovered that the >>>> dxfer_len paramter for scsi_ata_pass_16() was too small (uint16_t, and >>>> it needed to be uint32_t). I increased it, but that will potentially >>>> cause a binary incompatibility problem with any existing applications >>>> that use the current API via libcam. Advice on how to handle that >>>> would be welcome. >>>> >>>> If you look through the code, you'll notice that the disk_zone.h API is >>>> separate from the SCSI and ATA APIs. The intent is to allow filesystems >>>> and other consumers of the API to just talk to the disk zone API without >>>> dealing with the SCSI and ATA specifics. Another reason behind all of this >>>> is that even though the SCSI ZBC and ATA ZAC specs were developed in >>>> concert, and are intended to be functionally identical, they are still SCSI >>>> and ATA. As usual, SCSI is big endian and ATA is little endian. So to >>>> present a common API to the filesystem, we give all of the zone data back >>>> in native byte order, regardless of the underlying device protocol. >>>> >>>> Another thing to note is the extensive use of ATA passthrough in the da(4) >>>> driver. This is necessary because the SCSI SAT (SCSI to ATA Translation) >>>> specification has not yet caught up with translating SCSI zone commands >>>> (ZBC) to ATA zone commands (ZAC). So, until the spec is updated and LSI >>>> and other vendors update their SCSI to ATA translation layers, we'll have >>>> to use the ATA version of the commands when talking to ATA drives via SAS >>>> controllers. >>>> >>>> I have only tested the code so far with Seagate SATA Drive Managed and Host >>>> Aware drives. I would appreciate testing with any drives. (And testing to >>>> make sure that the patches don't cause problems with existing hardware.) >>>> Right now, all you can really do is manage the zones manually using >>>> camcontrol(8) or zonectl(8). Automatic management will come with the ZFS >>>> changes. (Or changes to other filesysems if people want to do it.) >>>> >>>> If you have a SATA Host Aware drive, in theory camcontrol(8) should allow >>>> you to manage the drive if you have it attached to a SATA controller. >>>> >>>> Here is an example of some of the commands. >>>> >>>> Get general zoning information: >>>> >>>> [root@sm4u-1 ~]# zonectl -c params -d /dev/da21 >>>> Zone Mode: Host Aware >>>> Command support: Report Zones, Open, Close, Finish, Reset Write Pointer >>>> Unrestricted Read in Sequential Write Required Zone (URSWRZ): No >>>> Optimal Number of Open Sequential Write Preferred Zones: 128 >>>> Optimal Number of Non-Sequentially Written Sequential Write Preferred Zones: 8 >>>> Maximum Number of Open Sequential Write Required Zones: Unlimited >>>> >>>> Look at information from the da(4) driver: >>>> >>>> [root@sm4u-1 ~]# sysctl kern.cam.da.21 >>>> kern.cam.da.21.delete_method: NONE >>>> kern.cam.da.21.delete_max: 1081344 >>>> kern.cam.da.21.minimum_cmd_size: 6 >>>> kern.cam.da.21.sort_io_queue: -1 >>>> kern.cam.da.21.zone_mode: Host Aware >>>> kern.cam.da.21.zone_support: Report Zones, Open, Close, Finish, Reset Write Pointer >>>> kern.cam.da.21.optimal_seq_zones: 128 >>>> kern.cam.da.21.optimal_nonseq_zones: 8 >>>> kern.cam.da.21.max_seq_zones: 4294967295 >>>> kern.cam.da.21.error_inject: 0 >>>> >>>> Display all of the zones with zonectl(8): >>>> >>>> [root@sm4u-1 ~]# zonectl -d /dev/da21 -c rz >>>> 29809 zones, Maximum LBA 0x3a3812aaf (15628053167) >>>> Zone lengths and types may vary >>>> Start LBA Length WP LBA Zone Type Condition Sequential Reset >>>> 0, 524288, 0x80000, Conventional, NWP, Sequential, No Reset Needed >>>> 0x80000, 524288, 0x100000, Conventional, NWP, Sequential, No Reset Needed >>>> 0x100000, 524288, 0x180000, Conventional, NWP, Sequential, No Reset Needed >>>> 0x180000, 524288, 0x200000, Conventional, NWP, Sequential, No Reset Needed >>>> 0x200000, 524288, 0x280000, Conventional, NWP, Sequential, No Reset Needed >>>> 0x280000, 524288, 0x300000, Conventional, NWP, Sequential, No Reset Needed >>>> 0x300000, 524288, 0x380000, Conventional, NWP, Sequential, No Reset Needed >>>> 0x380000, 524288, 0x400000, Conventional, NWP, Sequential, No Reset Needed >>>> 0x400000, 524288, 0x480000, Conventional, NWP, Sequential, No Reset Needed >>>> 0x480000, 524288, 0x500000, Conventional, NWP, Sequential, No Reset Needed >>>> 0x500000, 524288, 0x580000, Conventional, NWP, Sequential, No Reset Needed >>>> 0x580000, 524288, 0x600000, Conventional, NWP, Sequential, No Reset Needed >>>> 0x600000, 524288, 0x680000, Conventional, NWP, Sequential, No Reset Needed >>>> 0x680000, 524288, 0x700000, Conventional, NWP, Sequential, No Reset Needed >>>> 0x700000, 524288, 0x780000, Conventional, NWP, Sequential, No Reset Needed >>>> [ ... ] >>>> 0x1f00000, 524288, 0x1f80000, Conventional, NWP, Sequential, No Reset Needed >>>> 0x1f80000, 524288, 0x2000000, Conventional, NWP, Sequential, No Reset Needed >>>> 0x2000000, 524288, 0x2080000, Seq Preferred, Full, Sequential, No Reset Needed >>>> 0x2080000, 524288, 0x2100000, Seq Preferred, Full, Sequential, No Reset Needed >>>> 0x2100000, 524288, 0x2180000, Seq Preferred, Full, Sequential, No Reset Needed >>>> 0x2180000, 524288, 0x2200000, Seq Preferred, Full, Sequential, No Reset Needed >>>> 0x2200000, 524288, 0x2280000, Seq Preferred, Full, Sequential, No Reset Needed >>>> 0x2280000, 524288, 0x2300000, Seq Preferred, Full, Sequential, No Reset Needed >>>> [ ... ] >>>> >>>> Use camcontrol zone to reset the write pointer for the first shingled zone >>>> listed above: >>>> >>>> [root@sm4u-1 ~]# camcontrol zone da21 -v -c rwp -l 0x2000000 >>>> >>>> Use camcontrol zone to ask the drive to report empty zones: >>>> >>>> [root@sm4u-1 ~]# camcontrol zone da21 -v -c rz -o empty >>>> 1 zones, Maximum LBA 0x3a3812aaf (15628053167) >>>> Zone lengths and types may vary >>>> Start LBA Length WP LBA Zone Type Condition Sequential Reset >>>> 0x2000000, 524288, 0x2000000, Seq Preferred, Empty, Sequential, No Reset Needed >>>> >>>> Get information on a Host Aware drive: >>>> >>>> root@sm4u-1 ~]# diskinfo -v da21 >>>> da21 >>>> 512 # sectorsize >>>> 8001563222016 # mediasize in bytes (7.3T) >>>> 15628053168 # mediasize in sectors >>>> 4096 # stripesize >>>> 0 # stripeoffset >>>> 972801 # Cylinders according to firmware. >>>> 255 # Heads according to firmware. >>>> 63 # Sectors according to firmware. >>>> Z84008NY # Disk ident. >>>> enc@5003048001f311fd/elmtype@array_device/slot@22 # Physical path >>>> Host Aware # Zone Mode >>>> >>>> Get information on a drive managed drive: >>>> >>>> [root@sm4u-1 ~]# diskinfo -v da20 >>>> da20 >>>> 512 # sectorsize >>>> 8001563222016 # mediasize in bytes (7.3T) >>>> 15628053168 # mediasize in sectors >>>> 4096 # stripesize >>>> 0 # stripeoffset >>>> 972801 # Cylinders according to firmware. >>>> 255 # Heads according to firmware. >>>> 63 # Sectors according to firmware. >>>> Z8405938 # Disk ident. >>>> enc@5003048001f311fd/elmtype@array_device/slot@21 # Physical path >>>> Drive Managed # Zone Mode >>>> >>>> Get information on a non-zoned drive: >>>> >>>> [root@sm4u-1 ~]# diskinfo -v da4 >>>> da4 >>>> 512 # sectorsize >>>> 100030242816 # mediasize in bytes (93G) >>>> 195371568 # mediasize in sectors >>>> 0 # stripesize >>>> 0 # stripeoffset >>>> 12161 # Cylinders according to firmware. >>>> 255 # Heads according to firmware. >>>> 63 # Sectors according to firmware. >>>> 124903574F36 # Disk ident. >>>> enc@5003048001f311fd/elmtype@array_device/slot@5 # Physical path >>>> Not Zoned # Zone Mode >>>> >>>> Testing and comments are welcome. >>>> >>>> Thanks, >>>> >>>> Ken >>>> -- >>>> Kenneth Merry >>>> ken@FreeBSD.ORG >>> -- >>> Kenneth Merry >>> ken@FreeBSD.ORG >> -- >> Kenneth Merry >> ken@FreeBSD.ORG >> _______________________________________________ >> freebsd-scsi@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-scsi >> To unsubscribe, send any mail to "freebsd-scsi-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-scsi@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-scsi > To unsubscribe, send any mail to "freebsd-scsi-unsubscribe@freebsd.org" From owner-freebsd-scsi@freebsd.org Wed Mar 2 16:15:42 2016 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B1F82AC13D1 for ; Wed, 2 Mar 2016 16:15:42 +0000 (UTC) (envelope-from scott4long@yahoo.com) Received: from nm7-vm4.bullet.mail.gq1.yahoo.com (nm7-vm4.bullet.mail.gq1.yahoo.com [98.136.218.211]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8F7D71C7E for ; Wed, 2 Mar 2016 16:15:42 +0000 (UTC) (envelope-from scott4long@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1456935148; bh=+heFZYjM/DlmXP2giZ8PsKIBw5ITWqNZwBdQ2xzFlYY=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject; b=Uway9YV57XHiS5O2Bkbox10XRcJfEgkoeD0xeNq4cOblgJHtem75mDJcJ2cYzhniB/L6dddrJ6dSdM1pMnm1umG90oXkD2L5DpOFkUNxE4NrtfTylNbyNB37l5gIfl/TQFdILh3O30J9TGnMQijhPPjLXRbIrsZOGy2Exl36BhUCgUJOcJDgV91BJ9DLzPU72BTUswPyPhZ5xjwQQFBTtvVxpFzx4xN+CDE3SeUAGe4hO4eDqX/OvgsIXULIQPqhsla/xJg2maDbfTHjaK/ozDqGbAf4M4fR1oZWTYn0xMM0FQED3V3MYzmlqdrn2s6eUVZp5X84OZ4iPqZJVtgPWQ== Received: from [216.39.60.180] by nm7.bullet.mail.gq1.yahoo.com with NNFMP; 02 Mar 2016 16:12:28 -0000 Received: from [208.71.42.199] by tm16.bullet.mail.gq1.yahoo.com with NNFMP; 02 Mar 2016 16:12:28 -0000 Received: from [127.0.0.1] by smtp210.mail.gq1.yahoo.com with NNFMP; 02 Mar 2016 16:12:28 -0000 X-Yahoo-Newman-Id: 148981.87868.bm@smtp210.mail.gq1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: SwtqhlwVM1mK8KeL9g5sB39KARXzkz7mz5W0ZagByQwPBUu 7cTEEy9cW5pzZMcaoLzfs4UO8pn4I5GDpH9xCpKw8L63VcqObo0ySGlX0_5t Xr34jkliOXNu_UQBsq4WBOYnuUVCH0iTStWivt.1ijvjMzjbhc3V0NIZ7ZBY VOw5Fbdoci8nbYPY2R1drNiLG0JUZOpQbOf28s.afuKu5FxknoakyNd3nTQG FNY8KdpCXe4xmQVU3IESfeLP1HUHR3ehTdsPDq92A0KsIkIGS3C8t31vSmUK 2dbSQ.V2byHcXo3VRl2JIcT4jHPjXecfZJYQZTncMlRG6bpVDbk9NyG6fGEY 4PSJo87CeKFvoCFAnLW7qgSQoHcUYsaRcvOVLeIkijSNSf0AH7.Ee65JEkmK I0BFy65sDoXFnJrMjUkIWM_uR8yFx8gGH_9XCtORGoz_f1HF.VcRsg7vvcUg rdNIb6Rpp4blR4t3vvayQHRLzoJOm2nK.5TOdPepFBFVZJIBfVEEWxFJaQBl 2uid5pC2gOecAZeG.nw_o_usyZ8YPxYdHUkeEP3PRW14NoA-- X-Yahoo-SMTP: clhABp.swBB7fs.LwIJpv3jkWgo2NU8- Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.2 \(3112\)) Subject: Re: CAM Shingled Disk support patches available From: Scott Long In-Reply-To: <56D6BAA7.1050004@multiplay.co.uk> Date: Wed, 2 Mar 2016 09:12:25 -0700 Cc: freebsd-scsi@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <20151118171309.GA3564@mithlond.kdm.org> <20160118223704.GA87506@mithlond.kdm.org> <20160301224758.GA86834@mithlond.kdm.org> <56D6BAA7.1050004@multiplay.co.uk> To: Steven Hartland X-Mailer: Apple Mail (2.3112) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Mar 2016 16:15:42 -0000 > On Mar 2, 2016, at 3:04 AM, Steven Hartland = wrote: >=20 > scsi_ata_pass_16 was added to support TRIM and implements 12.2.3 (not = 12.2.2) providing access to all features as described by Table 104. I'll = admit I didn't take into account callers which make non LBA use of the = LBA fields, but I'm sure you'll agree that's an experience thing i.e. = knowing about about the existence of features which abuse structure = fields for other purposes ;-) >=20 I=E2=80=99m going from SAT-3r5, which unfortunately is the only rev I = have since T10 makes it hard to get SAT-3r7. Maybe the numbers changed? = I meant 12.2.2.x, which covers 12.2.2.1, 12.2.2.2, and 12.2.2.3 in the = draft that I have. There is no 12.2.3 in my draft. > Its use of u_int16_t dxfer_len is a bug, surprised the compiler didn't = flag this, not sure this can be fixed without breaking backwards = compatibility though? >=20 > Not sure I understand the 12 vs 13 registers comment, I'm guessing you = mean the 12 bytes of 12.2.2, however it actually exposes all 16 bytes of = 12.2.3. >=20 scsi_ata_pass_16 doesn=E2=80=99t accept a device argument and assumes it = to be ATA_DEV_LBA, 0x40. There are two commands in ACS-3, = READ_FPDMA_QUEUED and WRITE_FPDMA_QUEUED, that could use a different = value for bit 7. Maybe that=E2=80=99s an unlikely case for userland = passthrough, but not impossible. However bit 4 is also a wildcard that = is transport-dependent for many commands. SATA is pretty clear about = the bit always being zero. Technically in PATA you might want to = control bit 4 to specify the master vs slave device, but I=E2=80=99m = guessing that FreeBSD would be more likely to override that than support = it. Still, I haven=E2=80=99t done a comprehensive review of all command = sets and transports, and I think it would be prudent to leave the device = register exposed to the API rather than hard-coded. > For reference it has two callers in the kernel scsi_ata_identify used = to probe TRIM capability and scsi_ata_trim used to actually perform TRIM = operations. >=20 > In addition to this its also used by camcontrol to support passthrough = from ata_do_cmd and ata_do_28bit_cmd again supporting TRIM, in this case = the ATA security feature set as well as identify. >=20 > I would be very surprised if its used elsewhere but in order to = maintain compatibility I'd definitely agree with Scott on creating an = scsi_ata_pass which can be used to implement and scsi_ata_pass_16 which = can then be deprecated. >=20 It=E2=80=99s exposed in libcam.so, and it sounds like Ken is also = expanding its use, so it=E2=80=99s fair to say that it might have a = wider audience than just those TRIM uses. > With regards to providing 12.2.2 and 12.2.3 by looking at AP_EXTEND I = don't believe that's possible as you need this to tell the difference = between 28bit and 48bit ATA within 12.2.3. >=20 12.2.2.3 states the following: If the EXTEND bit is set to zero, then the FEATURES (15:8) field, the = SECTOR_COUNT (15:8) field, the LBA_LOW (15:8) field, then the LBA_MID = (15:8) field, and the LBA_HIGH (15:8) field shall be ignored by the = SATL, and the SATL shall process this command as specified in 12.2.2.2. My proposal isn=E2=80=99t completely correct, it would be abusing the = EXTEND bit to force the use of opcode A1 vs 85, and really the spec says = that 85 can be used with the bit being zero. One might want to send an = 0x85 opcode with EXTEND set to zero to test that the receiver will do = the right thing with ignoring the 5 extended fields, or you might have a = receiver that only accepts 0x85 but you need to send a 28 bit command. = So yeah, I concede it=E2=80=99s not great. Might need to either add an = extra flag, or split the function into two. Scott From owner-freebsd-scsi@freebsd.org Wed Mar 2 18:12:12 2016 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3650FAC2204 for ; Wed, 2 Mar 2016 18:12:12 +0000 (UTC) (envelope-from dgilbert@interlog.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 210C41BF4 for ; Wed, 2 Mar 2016 18:12:12 +0000 (UTC) (envelope-from dgilbert@interlog.com) Received: by mailman.ysv.freebsd.org (Postfix) id 20930AC2203; Wed, 2 Mar 2016 18:12:12 +0000 (UTC) Delivered-To: scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 06112AC2201; Wed, 2 Mar 2016 18:12:12 +0000 (UTC) (envelope-from dgilbert@interlog.com) Received: from smtp.infotech.no (smtp.infotech.no [82.134.31.41]) by mx1.freebsd.org (Postfix) with ESMTP id 1D8841BF2; Wed, 2 Mar 2016 18:12:10 +0000 (UTC) (envelope-from dgilbert@interlog.com) Received: from localhost (localhost [127.0.0.1]) by smtp.infotech.no (Postfix) with ESMTP id D2E7C204195; Wed, 2 Mar 2016 19:12:09 +0100 (CET) X-Virus-Scanned: by amavisd-new-2.6.6 (20110518) (Debian) at infotech.no Received: from smtp.infotech.no ([127.0.0.1]) by localhost (smtp.infotech.no [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id zP-pPFar928I; Wed, 2 Mar 2016 19:12:09 +0100 (CET) Received: from [10.7.0.18] (unknown [10.7.0.18]) by smtp.infotech.no (Postfix) with ESMTPA id A846F204192; Wed, 2 Mar 2016 19:12:08 +0100 (CET) Reply-To: dgilbert@interlog.com Subject: Re: CAM Shingled Disk support patches available References: <20151118171309.GA3564@mithlond.kdm.org> <20160118223704.GA87506@mithlond.kdm.org> <20160301224758.GA86834@mithlond.kdm.org> To: Scott Long , "Kenneth D. Merry" Cc: current@freebsd.org, scsi@freebsd.org From: Douglas Gilbert Message-ID: <56D72CF7.2040806@interlog.com> Date: Wed, 2 Mar 2016 13:12:07 -0500 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.5.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Mar 2016 18:12:12 -0000 On 16-03-01 10:07 PM, Scott Long via freebsd-scsi wrote: > Hi Ken, > > I’m against changing the function signature of scsi_ata_pass_16(). Even > if you manage to get things right with symbol versioning, it still leads to > problems of code compatibility. Maybe pre-existing binaries will work, but > source code will forever have to include an #if __FreeBSD_version < > xxxxxx bit of nonsense. > > I agree that it was incorrect for dxferlen to be declared as a uint16_t. > However, the function already contains a sector count argument pair. In > theory the sector count multiplied by the sector length, both of which the > application should know in order to arrive at a sensible dxferlen, can > substitute for the dxferlen argument. If so, then we can just ignore that > argument and declare that sector_count has logical priority. > > Really though, I think that scsi_ata_pass_16() is a crummy function. If its > purpose is to implement SAT-3 12.2.2, it does an incredibly poor job at it: > > - By my count, it only covers 12 of the available 13 registers. > > - It has no 12 byte, opcode 0xa1 variant. > > - It doesn’t make any allowance for providing the response registers to the > caller on completion. Well, maybe it kinda does through a sense descriptor, > but…. it’s kinda open to vague interpretation. > > - Its use of the registers is clunky, assuming for example that you’ll only want > to fill the six LBA registers with a host-ordered 64-bit number. There are > plenty of commands that re-use sub-parts of the LBA, features, and/or sector > count registers for different things. > > I know you stated that you didn’t want to do this, but I think it’s better to start > over with a better function that has a better signature and a new name. In fact, > I think it’s better to use the existing ata_cmd and ata_res structures from > sys/cam/ata/ata_all.h, provide accessors for the multi-byte registers if needed, > provide a 12-byte compatibility, and simply the signature. Something like this: > > void scsi_ata_pass(struct ccb_scsiio *csio, u_int32_t retries, > void (*cbfcnp)(struct cam_periph *, union ccb *), > u_int32_t flags, u_int8_t tag_action, > struct ata_cmd *cmd, struct ata_res *res, > u_int8_t *data_ptr, u_int32_t dxfer_len, > u_int8_t *data_ptr, u_int16_t dxfer_len, > u_int8_t sense_len, u_int32_t timeout); uint32_t and uint8_t please :-) For the pendants: https://www.freebsd.org/cgi/man.cgi?query=style&sektion=9 "The project is slowly moving to use the ISO/IEC 9899:1999 (``ISO C99'') unsigned integer identifiers of the form uintXX_t in preference to the older BSD-style integer identifiers of the form u_intXX_t. New code should use the former, and old code should be converted to the new form if other major work is being done in that area and there is no overriding reason to prefer the older BSD-style. Like white-space commits, care should be taken in making uintXX_t only commits." The Linux kernel ain't much better with u8, u16 and u32 typedefs everywhere. Doug Gilbert > To differentiate between the 12 and 16 byte variants, you’d look at the > AP_EXTEND flag in the protocol field. Btw, the handling of that flag is > inconsistent in the implementation of the existing scsi_ata_pass_16(). If > the caller providse an ata_res pointer then it gets filled on completion, > otherwise the caller does its best to look at 12.2.2.6 and extract what it > can from the sense descriptor. > > So my proposal is to create a new scsi_ata_pass and deprecate but not remove > scsi_ata_pass_16. Tell people that if they need to use it, dxfer_len is going to > have lower priority than sector_count/sector_count_exp if the latter multiply to > more than 65535. > > Scott > > > >> On Mar 1, 2016, at 3:47 PM, Kenneth D. Merry wrote: >> >> I have a new set of SMR patches available. (See the original message below >> for a more detailed description of what these patches do.) >> >> The primary change is to add library versioning to libcam so that we can >> change the function prototype of scsi_ata_pass_16() in a way that won't >> break existing binaries. >> >> If someone more familiar with library versioning wants to review this, I'd >> appreciate it. >> >> The patches are here: >> >> FreeBSD/head, as of SVN revision 296278 >> >> https://people.freebsd.org/~ken/cam_smr.head.20160301.1.txt >> >> stable/10, as of SVN revision 296248 >> >> https://people.freebsd.org/~ken/cam_smr.stable10.20160301.1.txt >> >> (Note that although there is a stable/10 version of the patches, I'm not >> planning to merge them to stable/10 because of the change to struct bio. I >> can't really figure out a good way to make that backward compatible. If >> there is consensus that breaking it is fine because it isn't a user API, >> then that may be another story.) >> >> The problem is that the existing, in-tree version of scsi_ata_pass_16() has >> a dxfer_len argument that is a uint16_t. That restricts transfer sizes to >> 64KB. So, we need to update it to allow larger than 64K transfers. I >> could just create a new function, but I'd rather just retire the broken >> version. >> >> The intent here is that: >> >> 1. Binaries built against the old version of libcam, before versioning was >> turned on, will get the old version of the scsi_ata_pass_16() function with >> a uint16_t dxfer_len. >> >> 2. Binaries built against the new version of libcam will get the new >> version of the scsi_ata_pass_16() function with a uint32_t dxfer_len. >> >> I've tested this, and it appears to work, but I'm not 100% certain this is >> all correct. I looked at Dan Eischen's description of symbol versioning >> here: >> >> https://people.freebsd.org/~deischen/symver/freebsd_versioning.txt >> >> And it looks like the actual implementation is a little different than what >> is described there. I looked around the tree, and didn't see anything that >> is obviously exactly like what I'm trying to do here. >> >> So, what I did is as follows: >> >> 1. For the kernel, the only change is to switch the dxfer_len argument from >> a uint16_t to a uint32_t. >> >> 2. For userland, in scsi_all.c, there are now two versions of >> scsi_ata_pass_16 -- _ver1 and _ver2. _ver1 is aliased to >> scsi_ata_pass_16() for FBSD_1.3 using __sym_compat(). _ver2 is aliased to >> scsi_ata_pass_16() for FBSD_1.4 using __sym_default(). >> >> 3. In lib/libcam/Versions.def, I defined FBSD_1.3 and FBSD_1.4, which >> depends on FBSD_1.3. >> >> 4. In lib/libcam/Symbol.map, I pulled out all of the functions defined in >> libcam, sorted them, and defined them in FBSD_1.3. I moved >> scsi_ata_pass_16() to FBSD_1.4. (According to the freebsd_versioning.txt >> paper linked above, I should have been able to have scsi_ata_pass_16() in >> both FBSD_1.3 and FBSD_1.4, but that isn't the case in practice.) >> >> In testing an old binary (linked against libcam without symbol versioning) >> against a new libcam (with symbol versioning), the old version of the >> function appears to be used. With a new binary, the new version of the >> function appears to be used. >> >> So it looks like things work as intended, but I don't fully trust my >> understanding here. So, if someone could take a look at the changes, I'd >> appreciate it. >> >> In particular, I have a few questions: >> >> 1. If this change to scsi_ata_pass_16() gets merged to stable/10 (apart >> from the larger SMR changes), what should be done with the libcam library >> version? >> >> 2. Are 1.3 and 1.4 the proper versions to use? >> >> 3. If we make additional CAM helper function library changes, when do the >> versions get bumped? i.e., is this an opportunity to look for other >> library functions with issues and make changes if possible? >> >> 4. When you're going from an unversioned library to a versioned library, >> which version of a function gets linked in to a binary linked to the >> unversioned library when you run it against a versioned library? In other >> words, what is supposed to happen in the test scenario I tried above, and >> am I really seeing what is supposed to happen? >> >> Thanks, >> >> Ken >> >> On Mon, Jan 18, 2016 at 17:37:04 -0500, Kenneth D. Merry wrote: >>> I have a new set of SMR patches available. See below for the full >>> explanation. >>> >>> The primary change here is that I have added SMR support to the ada(4) >>> driver. I spent some time considering whether to try to make the da(4) and >>> ada(4) probe infrastructure somewhat common, but in the end concluded it >>> would be too involved with not enough code reduction (if any) in the end. >>> >>> So, although the ideas are similar, the probe logic is separate. >>> >>> Note that NCQ support for SMR commands (Report Zones, Reset Write Pointer, >>> etc.) for SATA protocol shingled drives isn't active. For both the da(4) >>> and ada(4) driver this is for lack of a way to plumb the ATA Auxiliary >>> register down to the drive. >>> >>> In the ada(4) case, we need to add the register to struct ccb_ataio and >>> add support in one or more of the underlying SATA drivers, e.g. ahci(4). >>> >>> In the da(4) case, it will require an update of the T-10 SAT spec to >>> provide a way to pass the Auxiliary register down via the SCSI ATA >>> PASS-THROUGH command, and then a subsquent update of the SAT layer in >>> various vendors' SAS controller firmware. At that point, there may be >>> an official mapping of the SCSI ZBC commands to the ATA ZAC commands, and >>> we may be able to just issue the SCSI version of the commands instead of >>> composing ATA commands in the da(4) driver. (We'll still need to keep the >>> ATA passthrough version for a while at least to support controllers that >>> don't have the updated translation code.) >>> >>> FreeBSD/head as of SVN revision 294105: >>> >>> https://people.freebsd.org/~ken/cam_smr.head.20160118.1.txt >>> >>> FreeBSD stable/10 as of SVN revision 294100: >>> >>> https://people.freebsd.org/~ken/cam_smr.stable10.20160118.1.txt >>> >>> Testing and comments are welcome. >>> >>> Ken >>> >>> On Wed, Nov 18, 2015 at 12:13:09 -0500, Kenneth D. Merry wrote: >>>> >>>> I have work in progress patches to add SMR (Shingled Magnetic Recording) >>>> support to CAM and GEOM here: >>>> >>>> FreeBSD/head as of SVN revision 290997: >>>> >>>> https://people.freebsd.org/~ken/cam_smr.head.20151117.1.txt >>>> >>>> FreeBSD stable/10 as of SVN revision 290995: >>>> >>>> https://people.freebsd.org/~ken/cam_smr.stable10.20151117.1.txt >>>> >>>> This includes support for Host Managed, Host Aware and Drive Managed SMR >>>> drives that are either SCSI (ZBC) or ATA (ZAC) attached via a SAS >>>> controller. This does not include support for SMR ATA drives attched via >>>> an ATA controller. Also, I have not yet figured out how to properly detect >>>> a Host Managed ATA drive, so this code won't do that. >>>> >>>> The big drive vendors are moving to SMR for at least some of their drives. >>>> The primary challenge with SMR is that it requires writing a relatively >>>> large zone sequentially starting at the beginning of the zone. The usual >>>> zone size is 256MB. It is conceptually almost like having a 256MB sector >>>> size. >>>> >>>> We (Spectra Logic) are working on ZFS changes that will use this CAM and >>>> GEOM infrastructure to make ZFS play well with SMR drives. Those changes >>>> aren't yet done. >>>> >>>> The patches linked above include: >>>> o A new 'camcontrol zone' command that allows displaying and managing >>>> drive zones via SCSI/ATA passthrough. >>>> o A new zonectl(8) utility that uses the new DIOCZONECMD ioctl to display >>>> and manage zones via the da(4) (and later ada(4)) driver. >>>> o Changes to diskinfo -v to display the zone mode of a drive. >>>> o A new disk zone API, sys/sys/disk_zone.h. >>>> o A new bio type, BIO_ZONE, and modifications to GEOM to support it. This >>>> new bio will allow filesystems to query zone support in a drive and >>>> manage zoned drives. >>>> o Extensive modifications to the da(4) driver to handle probing SCSI and >>>> SATA behind SAS SMR drives. >>>> o Additional CAM CDB building functions for zone commands. >>>> >>>> The current issues that need to be addressed are: >>>> o The da(4) driver now has 6 additional probe states, 5 of which are >>>> needed for probing ATA drives behind SAS controllers. I have not yet >>>> added support for BIO_ZONE bios to ada(4), but it will be very similar >>>> to the da(4) driver version. The ATA probe code needs to be pulled >>>> out of the da(4) driver and changed into a form that will allow it to >>>> work for either the ada(4) or da(4) driver. Otherwise we'll have a fair >>>> amount of code duplication between the two drivers. >>>> >>>> o There is a reasonable amount of code duplication between 'camcontrol zone' >>>> and zonectl(8). This was done for speed / expediency's sake, but it may >>>> be possible to make more of the code common there. >>>> >>>> o In order to add the new BIO_ZONE bio command, I had to change the bio_cmd >>>> field in struct bio from a uint8_t to a uint16_t. This will cause >>>> binary compatibility problems with any 3rd party loadable modules. >>>> Advice on how to handle this would be welcome. >>>> >>>> o In the process of developing these changes, I discovered that the >>>> dxfer_len paramter for scsi_ata_pass_16() was too small (uint16_t, and >>>> it needed to be uint32_t). I increased it, but that will potentially >>>> cause a binary incompatibility problem with any existing applications >>>> that use the current API via libcam. Advice on how to handle that >>>> would be welcome. >>>> >>>> If you look through the code, you'll notice that the disk_zone.h API is >>>> separate from the SCSI and ATA APIs. The intent is to allow filesystems >>>> and other consumers of the API to just talk to the disk zone API without >>>> dealing with the SCSI and ATA specifics. Another reason behind all of this >>>> is that even though the SCSI ZBC and ATA ZAC specs were developed in >>>> concert, and are intended to be functionally identical, they are still SCSI >>>> and ATA. As usual, SCSI is big endian and ATA is little endian. So to >>>> present a common API to the filesystem, we give all of the zone data back >>>> in native byte order, regardless of the underlying device protocol. >>>> >>>> Another thing to note is the extensive use of ATA passthrough in the da(4) >>>> driver. This is necessary because the SCSI SAT (SCSI to ATA Translation) >>>> specification has not yet caught up with translating SCSI zone commands >>>> (ZBC) to ATA zone commands (ZAC). So, until the spec is updated and LSI >>>> and other vendors update their SCSI to ATA translation layers, we'll have >>>> to use the ATA version of the commands when talking to ATA drives via SAS >>>> controllers. >>>> >>>> I have only tested the code so far with Seagate SATA Drive Managed and Host >>>> Aware drives. I would appreciate testing with any drives. (And testing to >>>> make sure that the patches don't cause problems with existing hardware.) >>>> Right now, all you can really do is manage the zones manually using >>>> camcontrol(8) or zonectl(8). Automatic management will come with the ZFS >>>> changes. (Or changes to other filesysems if people want to do it.) >>>> >>>> If you have a SATA Host Aware drive, in theory camcontrol(8) should allow >>>> you to manage the drive if you have it attached to a SATA controller. >>>> >>>> Here is an example of some of the commands. >>>> >>>> Get general zoning information: >>>> >>>> [root@sm4u-1 ~]# zonectl -c params -d /dev/da21 >>>> Zone Mode: Host Aware >>>> Command support: Report Zones, Open, Close, Finish, Reset Write Pointer >>>> Unrestricted Read in Sequential Write Required Zone (URSWRZ): No >>>> Optimal Number of Open Sequential Write Preferred Zones: 128 >>>> Optimal Number of Non-Sequentially Written Sequential Write Preferred Zones: 8 >>>> Maximum Number of Open Sequential Write Required Zones: Unlimited >>>> >>>> Look at information from the da(4) driver: >>>> >>>> [root@sm4u-1 ~]# sysctl kern.cam.da.21 >>>> kern.cam.da.21.delete_method: NONE >>>> kern.cam.da.21.delete_max: 1081344 >>>> kern.cam.da.21.minimum_cmd_size: 6 >>>> kern.cam.da.21.sort_io_queue: -1 >>>> kern.cam.da.21.zone_mode: Host Aware >>>> kern.cam.da.21.zone_support: Report Zones, Open, Close, Finish, Reset Write Pointer >>>> kern.cam.da.21.optimal_seq_zones: 128 >>>> kern.cam.da.21.optimal_nonseq_zones: 8 >>>> kern.cam.da.21.max_seq_zones: 4294967295 >>>> kern.cam.da.21.error_inject: 0 >>>> >>>> Display all of the zones with zonectl(8): >>>> >>>> [root@sm4u-1 ~]# zonectl -d /dev/da21 -c rz >>>> 29809 zones, Maximum LBA 0x3a3812aaf (15628053167) >>>> Zone lengths and types may vary >>>> Start LBA Length WP LBA Zone Type Condition Sequential Reset >>>> 0, 524288, 0x80000, Conventional, NWP, Sequential, No Reset Needed >>>> 0x80000, 524288, 0x100000, Conventional, NWP, Sequential, No Reset Needed >>>> 0x100000, 524288, 0x180000, Conventional, NWP, Sequential, No Reset Needed >>>> 0x180000, 524288, 0x200000, Conventional, NWP, Sequential, No Reset Needed >>>> 0x200000, 524288, 0x280000, Conventional, NWP, Sequential, No Reset Needed >>>> 0x280000, 524288, 0x300000, Conventional, NWP, Sequential, No Reset Needed >>>> 0x300000, 524288, 0x380000, Conventional, NWP, Sequential, No Reset Needed >>>> 0x380000, 524288, 0x400000, Conventional, NWP, Sequential, No Reset Needed >>>> 0x400000, 524288, 0x480000, Conventional, NWP, Sequential, No Reset Needed >>>> 0x480000, 524288, 0x500000, Conventional, NWP, Sequential, No Reset Needed >>>> 0x500000, 524288, 0x580000, Conventional, NWP, Sequential, No Reset Needed >>>> 0x580000, 524288, 0x600000, Conventional, NWP, Sequential, No Reset Needed >>>> 0x600000, 524288, 0x680000, Conventional, NWP, Sequential, No Reset Needed >>>> 0x680000, 524288, 0x700000, Conventional, NWP, Sequential, No Reset Needed >>>> 0x700000, 524288, 0x780000, Conventional, NWP, Sequential, No Reset Needed >>>> [ ... ] >>>> 0x1f00000, 524288, 0x1f80000, Conventional, NWP, Sequential, No Reset Needed >>>> 0x1f80000, 524288, 0x2000000, Conventional, NWP, Sequential, No Reset Needed >>>> 0x2000000, 524288, 0x2080000, Seq Preferred, Full, Sequential, No Reset Needed >>>> 0x2080000, 524288, 0x2100000, Seq Preferred, Full, Sequential, No Reset Needed >>>> 0x2100000, 524288, 0x2180000, Seq Preferred, Full, Sequential, No Reset Needed >>>> 0x2180000, 524288, 0x2200000, Seq Preferred, Full, Sequential, No Reset Needed >>>> 0x2200000, 524288, 0x2280000, Seq Preferred, Full, Sequential, No Reset Needed >>>> 0x2280000, 524288, 0x2300000, Seq Preferred, Full, Sequential, No Reset Needed >>>> [ ... ] >>>> >>>> Use camcontrol zone to reset the write pointer for the first shingled zone >>>> listed above: >>>> >>>> [root@sm4u-1 ~]# camcontrol zone da21 -v -c rwp -l 0x2000000 >>>> >>>> Use camcontrol zone to ask the drive to report empty zones: >>>> >>>> [root@sm4u-1 ~]# camcontrol zone da21 -v -c rz -o empty >>>> 1 zones, Maximum LBA 0x3a3812aaf (15628053167) >>>> Zone lengths and types may vary >>>> Start LBA Length WP LBA Zone Type Condition Sequential Reset >>>> 0x2000000, 524288, 0x2000000, Seq Preferred, Empty, Sequential, No Reset Needed >>>> >>>> Get information on a Host Aware drive: >>>> >>>> root@sm4u-1 ~]# diskinfo -v da21 >>>> da21 >>>> 512 # sectorsize >>>> 8001563222016 # mediasize in bytes (7.3T) >>>> 15628053168 # mediasize in sectors >>>> 4096 # stripesize >>>> 0 # stripeoffset >>>> 972801 # Cylinders according to firmware. >>>> 255 # Heads according to firmware. >>>> 63 # Sectors according to firmware. >>>> Z84008NY # Disk ident. >>>> enc@5003048001f311fd/elmtype@array_device/slot@22 # Physical path >>>> Host Aware # Zone Mode >>>> >>>> Get information on a drive managed drive: >>>> >>>> [root@sm4u-1 ~]# diskinfo -v da20 >>>> da20 >>>> 512 # sectorsize >>>> 8001563222016 # mediasize in bytes (7.3T) >>>> 15628053168 # mediasize in sectors >>>> 4096 # stripesize >>>> 0 # stripeoffset >>>> 972801 # Cylinders according to firmware. >>>> 255 # Heads according to firmware. >>>> 63 # Sectors according to firmware. >>>> Z8405938 # Disk ident. >>>> enc@5003048001f311fd/elmtype@array_device/slot@21 # Physical path >>>> Drive Managed # Zone Mode >>>> >>>> Get information on a non-zoned drive: >>>> >>>> [root@sm4u-1 ~]# diskinfo -v da4 >>>> da4 >>>> 512 # sectorsize >>>> 100030242816 # mediasize in bytes (93G) >>>> 195371568 # mediasize in sectors >>>> 0 # stripesize >>>> 0 # stripeoffset >>>> 12161 # Cylinders according to firmware. >>>> 255 # Heads according to firmware. >>>> 63 # Sectors according to firmware. >>>> 124903574F36 # Disk ident. >>>> enc@5003048001f311fd/elmtype@array_device/slot@5 # Physical path >>>> Not Zoned # Zone Mode >>>> >>>> Testing and comments are welcome. >>>> >>>> Thanks, >>>> >>>> Ken >>>> -- >>>> Kenneth Merry >>>> ken@FreeBSD.ORG >>> >>> -- >>> Kenneth Merry >>> ken@FreeBSD.ORG >> >> -- >> Kenneth Merry >> ken@FreeBSD.ORG >> _______________________________________________ >> freebsd-scsi@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-scsi >> To unsubscribe, send any mail to "freebsd-scsi-unsubscribe@freebsd.org" > > _______________________________________________ > freebsd-scsi@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-scsi > To unsubscribe, send any mail to "freebsd-scsi-unsubscribe@freebsd.org" > From owner-freebsd-scsi@freebsd.org Wed Mar 2 18:46:13 2016 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 65433AC2DE3 for ; Wed, 2 Mar 2016 18:46:13 +0000 (UTC) (envelope-from scott4long@yahoo.com) Received: from nm6-vm2.bullet.mail.gq1.yahoo.com (nm6-vm2.bullet.mail.gq1.yahoo.com [98.136.218.193]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 41A351E70 for ; Wed, 2 Mar 2016 18:46:13 +0000 (UTC) (envelope-from scott4long@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1456944204; bh=yuFWMEJ4l+2HQxeKHLeNOiZ4uYg2TyeWBgYUEwrSVaI=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject; b=X52tIAYdM2ern8bzsqjB3tStot9L2e9Goa8MtcCosmDpM7wX1oBJelK8lY5zytG8d3Ico8encgpUyar+pTCt22rx20Y8zrDNrtMsZu8K2FVfg/ENbZjikQc87XKKRsYjdXdKqSY+P9Bcf4eJOlnQLrE0dhLmW1rtOF2m/Ry5TVu3Ts2fkZiPH60zngxIYRPPIJBKBDDZXI77ac3PPNgIEnOii71HnFMLeQJo4EioexcJ7ZjsHbe6LYt+SIITj11Ay7YdTKGMN2Q2+ThzPgxvRPIvKC9WN7ELUwbp1jDqm5TzxfKyYCd6Yxhk8mnjt7zq9tPh21AgLR1rAFSMhdTQXw== Received: from [98.137.12.188] by nm6.bullet.mail.gq1.yahoo.com with NNFMP; 02 Mar 2016 18:43:24 -0000 Received: from [208.71.42.212] by tm9.bullet.mail.gq1.yahoo.com with NNFMP; 02 Mar 2016 18:43:24 -0000 Received: from [127.0.0.1] by smtp223.mail.gq1.yahoo.com with NNFMP; 02 Mar 2016 18:43:24 -0000 X-Yahoo-Newman-Id: 636728.20780.bm@smtp223.mail.gq1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: J8WZLZ0VM1n9EEL1jKxgHsjqHrToRO41.zj0dfXUUWq3rC5 E9Op.34zMjrONSIDO30NcV3wDniZVdHbFJ32qSqmMpl4.Ju0ghJMqKaoQaJK 59ggGDJMK4zkUmt8JTygLuYQ_0yMferdOCXSVQarBWCGxedB7LNgVFdskJg1 5aLQZ3P6NPADCiuPdvvhIh15VBL20NyE7eLjRVQ_mxNJ8Is4gLZt01PYjz18 j4IvWGXggy78HM8cGHuQoSCtGBdYQQSJR6_wvyR1dRm5rA0arwfvWQaSle9U cr8xZdXZwKHoRy4S8kXPy8wSGq9blZDzB0DdcoKzwj1PVw7eqRAX3eWCjuum Fi5HpviXQK92QGtpkypBGqZgX3zZvQkfmwmalQ2A_xQci0qWTRkK.17pi1IX bjjvhaxar6m8SvhkEJ8QOrE6oKhruONW9AnQjlg28L7plleQrmPkD9nKrF2z KfZ7q63Yw5v6QjZyb8EnHYH3aYmA.HFQ7QuI6ShTsEEUnR8RzLvOivL7uFKe j38iW82vuoDBmHbpw0liVNCK.Spzv2Vz8vNE4PhJ22UY.tg-- X-Yahoo-SMTP: clhABp.swBB7fs.LwIJpv3jkWgo2NU8- Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.2 \(3112\)) Subject: Re: mpr(4) SAS3008 Repeated Crashing From: Scott Long In-Reply-To: Date: Wed, 2 Mar 2016 11:43:23 -0700 Cc: Steven Hartland , freebsd-scsi@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <56D5FDB8.8040402@freebsd.org> <56D612FA.6090909@multiplay.co.uk> To: Borja Marcos X-Mailer: Apple Mail (2.3112) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Mar 2016 18:46:13 -0000 > On Mar 2, 2016, at 12:23 AM, Borja Marcos wrote: >=20 >=20 >> On 01 Mar 2016, at 23:08, Steven Hartland = wrote: >>=20 >> Initial ideas would be bad signalling. >>=20 >> If you have the option to drop the speeds down and that helps then = almost certainly the case. >>=20 >> The original mfi driver was very bad at recovering from issues like = this too, I spent over a month fixing and patching it to get it working = reliably when there where hardware related issues. In my case it turned = out the be a dodge CPU causing memory corruption but you'll get similar = behaviour from badly designed installs, particularly with expanders in = play for high speed devices (6-12Gbps) link speed. >=20 > I=E2=80=99ve suffered similar problems, although not as severe, on one = of my storage servers. It=E2=80=99s an IBM X Series with a LSI 3008 HBA=20= > connected to the backplane, using SATA SSDs. But mine are almost = certainly hardware problems. An identical system is working > without issues. >=20 > The symptom: with high I/O activity, for example, running Bonnie++, = some commands abort with the disks returning a > unit attention (power on/reset) asc 0,29. >=20 In your case, the UA is actually a secondary effect. What=E2=80=99s = happening is that a command is timing out so the driver is resetting the = disk. That causes the disk to report a UA with an ASC of 29/0 on the = next command it gets after it comes back up. It=E2=80=99s not fatal and = I=E2=80=99m not sure if it should actually cause a retry, but that=E2=80=99= s an investigation for a different time. It does produce a lot of noise = on the console/log, though. One thing I noticed in your log is that one of the commands was a = passthrough ATA command of 0x06 and feature of 0x01, which is DSM TRIM. = It=E2=80=99s not clear if this command was at fault, I need to add = better logging for this case, but it=E2=80=99s highly suspect. It was = only being asked to trim one sector, but given how unpredictable TRIM = responses are from the drive, I don=E2=80=99t know if this matters. = What it might point to, though, is that either the timeout for the = command was too short, the drive doesn=E2=80=99t support DSM TRIM that = well, or the LSI adapter doesn=E2=80=99t support it well (since it=E2=80=99= s not an NCQ command, the LSI firmware would have to remember to flush = out the pending NCQ reads and writes first before doing the DSM = command). The default timeout is 60 seconds, which should be enough = unless you changed it deliberately. If this is a reproducible case, = would you be willing to re-try with a different delete method, i.e. = fiddle with the kern.cam.da.X.delete_method sysctl? In any case, I doubt that the problem is with cabling. Active = backplanes have been known to cause problems with LSI controllers and = SATA disks, but the problem that reported in your log doesn=E2=80=99t = match the typical pattern for that. Scott From owner-freebsd-scsi@freebsd.org Wed Mar 2 20:23:17 2016 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 16DE7AC2B7A for ; Wed, 2 Mar 2016 20:23:17 +0000 (UTC) (envelope-from sbruno@freebsd.org) Received: from mail.ignoranthack.me (ignoranthack.me [199.102.79.106]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DA1E01A5F for ; Wed, 2 Mar 2016 20:23:16 +0000 (UTC) (envelope-from sbruno@freebsd.org) Received: from [192.168.200.206] (unknown [50.136.155.142]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: sbruno@ignoranthack.me) by mail.ignoranthack.me (Postfix) with ESMTPSA id 0A0E2192993 for ; Wed, 2 Mar 2016 20:23:14 +0000 (UTC) Subject: Re: mpr(4) SAS3008 Repeated Crashing To: freebsd-scsi@freebsd.org References: <56D5FDB8.8040402@freebsd.org> <56D612FA.6090909@multiplay.co.uk> From: Sean Bruno Message-ID: <56D74BB2.5020108@freebsd.org> Date: Wed, 2 Mar 2016 12:23:14 -0800 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.5.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Mar 2016 20:23:17 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 03/02/16 10:43, Scott Long via freebsd-scsi wrote: > >> On Mar 2, 2016, at 12:23 AM, Borja Marcos >> wrote: >> >> >>> On 01 Mar 2016, at 23:08, Steven Hartland >>> wrote: >>> >>> Initial ideas would be bad signalling. >>> >>> If you have the option to drop the speeds down and that helps >>> then almost certainly the case. >>> >>> The original mfi driver was very bad at recovering from issues >>> like this too, I spent over a month fixing and patching it to >>> get it working reliably when there where hardware related >>> issues. In my case it turned out the be a dodge CPU causing >>> memory corruption but you'll get similar behaviour from badly >>> designed installs, particularly with expanders in play for high >>> speed devices (6-12Gbps) link speed. >> >> I’ve suffered similar problems, although not as severe, on one of >> my storage servers. It’s an IBM X Series with a LSI 3008 HBA >> connected to the backplane, using SATA SSDs. But mine are almost >> certainly hardware problems. An identical system is working >> without issues. >> >> The symptom: with high I/O activity, for example, running >> Bonnie++, some commands abort with the disks returning a unit >> attention (power on/reset) asc 0,29. >> > > In your case, the UA is actually a secondary effect. What’s > happening is that a command is timing out so the driver is > resetting the disk. That causes the disk to report a UA with an > ASC of 29/0 on the next command it gets after it comes back up. > It’s not fatal and I’m not sure if it should actually cause a > retry, but that’s an investigation for a different time. It does > produce a lot of noise on the console/log, though. > > One thing I noticed in your log is that one of the commands was a > passthrough ATA command of 0x06 and feature of 0x01, which is DSM > TRIM. It’s not clear if this command was at fault, I need to add > better logging for this case, but it’s highly suspect. It was only > being asked to trim one sector, but given how unpredictable TRIM > responses are from the drive, I don’t know if this matters. What > it might point to, though, is that either the timeout for the > command was too short, the drive doesn’t support DSM TRIM that > well, or the LSI adapter doesn’t support it well (since it’s not an > NCQ command, the LSI firmware would have to remember to flush out > the pending NCQ reads and writes first before doing the DSM > command). The default timeout is 60 seconds, which should be > enough unless you changed it deliberately. If this is a > reproducible case, would you be willing to re-try with a different > delete method, i.e. fiddle with the kern.cam.da.X.delete_method > sysctl? > > In any case, I doubt that the problem is with cabling. Active > backplanes have been known to cause problems with LSI controllers > and SATA disks, but the problem that reported in your log doesn’t > match the typical pattern for that. > > Scott > The controller itself was running a quite "old" firmware revision, 6.00.00.00 We've updated it to 10.00.03.00 and are restarting our tests to see what, if anything is different. I've disabled the tool that was sending the ATA commands down with TRIM and we've rewired the cabling inside the host a bit to give us a better idea of what's going on. sean -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQF8BAEBCgBmBQJW10uuXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRCQUFENDYzMkU3MTIxREU4RDIwOTk3REQx MjAxRUZDQTFFNzI3RTY0AAoJEBIB78oecn5k8r4H/RFu+OT6sa0qaivncWPLzOqQ Q5Mzv8znFWHYyxX5es9EPwodEPnfOg/9QXLzC7TNha9ukDu+a723nka/1WUVl2Wq 9G93AImGy4AxjA6W/0bB0TXYGU26x8hVQ71E/xZB6XaVywGczuBbAtQIEESPi2n7 ScpBpOd6ctXexO4bCHPfu3Hz7Sq6Tbr3F1IHOqXGhbYTLekwDtBRzCDs+LTAr/qN tF9ou4vL2Hn3KjfFFSDIiTKT2vcod7aCHsNJMAkXnmHe9HdCbQBEElvN48Al0O6E 6iLhYsqeuIfQ2THsn4/T2/f8MuSxa9xTxGtG8WqWeEVUtXU1V9A2l84/EqhoCpI= =tMOi -----END PGP SIGNATURE----- From owner-freebsd-scsi@freebsd.org Wed Mar 2 21:25:34 2016 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 556F5AC0B22 for ; Wed, 2 Mar 2016 21:25:34 +0000 (UTC) (envelope-from ken@kdm.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 40D031E55 for ; Wed, 2 Mar 2016 21:25:34 +0000 (UTC) (envelope-from ken@kdm.org) Received: by mailman.ysv.freebsd.org (Postfix) id 3A556AC0B20; Wed, 2 Mar 2016 21:25:34 +0000 (UTC) Delivered-To: scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 399EEAC0B1D; Wed, 2 Mar 2016 21:25:34 +0000 (UTC) (envelope-from ken@kdm.org) Received: from mithlond.kdm.org (mithlond.kdm.org [96.89.93.250]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "A1-33714", Issuer "A1-33714" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 1638D1E52; Wed, 2 Mar 2016 21:25:33 +0000 (UTC) (envelope-from ken@kdm.org) Received: from mithlond.kdm.org (localhost [127.0.0.1]) by mithlond.kdm.org (8.15.2/8.14.9) with ESMTPS id u22LPPa5006990 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 2 Mar 2016 16:25:25 -0500 (EST) (envelope-from ken@mithlond.kdm.org) Received: (from ken@localhost) by mithlond.kdm.org (8.15.2/8.14.9/Submit) id u22LPPPD006989; Wed, 2 Mar 2016 16:25:25 -0500 (EST) (envelope-from ken) Date: Wed, 2 Mar 2016 16:25:25 -0500 From: "Kenneth D. Merry" To: Scott Long Cc: scsi@freebsd.org, current@freebsd.org Subject: Re: CAM Shingled Disk support patches available Message-ID: <20160302212525.GA5920@mithlond.kdm.org> References: <20151118171309.GA3564@mithlond.kdm.org> <20160118223704.GA87506@mithlond.kdm.org> <20160301224758.GA86834@mithlond.kdm.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (mithlond.kdm.org [127.0.0.1]); Wed, 02 Mar 2016 16:25:25 -0500 (EST) X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS autolearn=ham autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on mithlond.kdm.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Mar 2016 21:25:34 -0000 On Tue, Mar 01, 2016 at 20:07:19 -0700, Scott Long wrote: > Hi Ken, > > I???m against changing the function signature of scsi_ata_pass_16(). Even > if you manage to get things right with symbol versioning, it still leads to > problems of code compatibility. Maybe pre-existing binaries will work, but > source code will forever have to include an #if __FreeBSD_version < > xxxxxx bit of nonsense. Good point, that would be annoying. > I agree that it was incorrect for dxferlen to be declared as a uint16_t. > However, the function already contains a sector count argument pair. In > theory the sector count multiplied by the sector length, both of which the > application should know in order to arrive at a sensible dxferlen, can > substitute for the dxferlen argument. If so, then we can just ignore that > argument and declare that sector_count has logical priority. Okay. That will probably work for the most part. > Really though, I think that scsi_ata_pass_16() is a crummy function. If its > purpose is to implement SAT-3 12.2.2, it does an incredibly poor job at it: > > - By my count, it only covers 12 of the available 13 registers. > > - It has no 12 byte, opcode 0xa1 variant. > > - It doesn???t make any allowance for providing the response registers to the > caller on completion. Well, maybe it kinda does through a sense descriptor, > but???. it???s kinda open to vague interpretation. > > - Its use of the registers is clunky, assuming for example that you???ll only want > to fill the six LBA registers with a host-ordered 64-bit number. There are > plenty of commands that re-use sub-parts of the LBA, features, and/or sector > count registers for different things. > > I know you stated that you didn???t want to do this, but I think it???s better to start > over with a better function that has a better signature and a new name. In fact, > I think it???s better to use the existing ata_cmd and ata_res structures from > sys/cam/ata/ata_all.h, provide accessors for the multi-byte registers if needed, > provide a 12-byte compatibility, and simply the signature. Something like this: > > void scsi_ata_pass(struct ccb_scsiio *csio, u_int32_t retries, > void (*cbfcnp)(struct cam_periph *, union ccb *), > u_int32_t flags, u_int8_t tag_action, > struct ata_cmd *cmd, struct ata_res *res, > u_int8_t *data_ptr, u_int32_t dxfer_len, > u_int8_t *data_ptr, u_int16_t dxfer_len, I assume you only intended one line there, not two. :) > u_int8_t sense_len, u_int32_t timeout); > > To differentiate between the 12 and 16 byte variants, you???d look at the > AP_EXTEND flag in the protocol field. Btw, the handling of that flag is > inconsistent in the implementation of the existing scsi_ata_pass_16(). If > the caller providse an ata_res pointer then it gets filled on completion, > otherwise the caller does its best to look at 12.2.2.6 and extract what it > can from the sense descriptor. > > So my proposal is to create a new scsi_ata_pass and deprecate but not remove > scsi_ata_pass_16. Tell people that if they need to use it, dxfer_len is going to > have lower priority than sector_count/sector_count_exp if the latter multiply to > more than 65535. In general I think that's a reasonable idea, but we should probably go further. While we're at it, we should figure out what we need to do to add the Auxiliary register to struct ata_cmd. We'll need that to do the NCQ versions of the various SMR commands, as well as TRIM. The obvious challenge is that probably means changing the existing struct ccb_ataio CCB and bumping the CAM version. At least that will be source compatible, but will require ifdefs if people want to compile on older versions of FreeBSD. But in that case, they'll also be faced with no support for sending the NCQ versions of the commands, anyway. No way around that, though, since we have to follow the changing specs. Ken -- Kenneth Merry ken@FreeBSD.ORG From owner-freebsd-scsi@freebsd.org Thu Mar 3 04:40:58 2016 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7FA6BAC25FC for ; Thu, 3 Mar 2016 04:40:58 +0000 (UTC) (envelope-from scott4long@yahoo.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 6274E101A for ; Thu, 3 Mar 2016 04:40:58 +0000 (UTC) (envelope-from scott4long@yahoo.com) Received: by mailman.ysv.freebsd.org (Postfix) id 619FDAC25FB; Thu, 3 Mar 2016 04:40:58 +0000 (UTC) Delivered-To: scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 613BCAC25FA for ; Thu, 3 Mar 2016 04:40:58 +0000 (UTC) (envelope-from scott4long@yahoo.com) Received: from nm13-vm4.bullet.mail.gq1.yahoo.com (nm13-vm4.bullet.mail.gq1.yahoo.com [98.136.218.223]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3D91D1018 for ; Thu, 3 Mar 2016 04:40:58 +0000 (UTC) (envelope-from scott4long@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1456979648; bh=WrFp9V/33FP1bsmF71UyyvBcSlbjuUb+r1vGM1zpmGg=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject; b=cd+ODG2dIsA2LujYbbnOzsgVcdaFx9ZjkTeUBts+Ulhw+9lzosrlWTmVejlJdsZXvMreHZLMDh1vewyOmtWXHk3oh7EU8Zn+QMH4pT6qm3ES5/4IRAT51NaKJFa5W6OFIAEXUVN3u4mHOwWdV4w0uRAYXaY1H48rtIN0wRqBkmutADUaa5CEl14vudh8ASP7DXpMDtxmusZfFZcYgBj1gOhAJ+QE9TSgehyo9qLfU5sNDlYPmVg1xG45j/HjD3rF6u489ltGEtrSBoMPUPAgotO6YCS4EqZLRhQn6lCH/1FEyv//bYapoHrG542Ae/vZVny/JCqSqp/HVgNUw4ZzXA== Received: from [98.137.12.62] by nm13.bullet.mail.gq1.yahoo.com with NNFMP; 03 Mar 2016 04:34:08 -0000 Received: from [208.71.42.193] by tm7.bullet.mail.gq1.yahoo.com with NNFMP; 03 Mar 2016 04:34:08 -0000 Received: from [127.0.0.1] by smtp204.mail.gq1.yahoo.com with NNFMP; 03 Mar 2016 04:34:08 -0000 X-Yahoo-Newman-Id: 214424.84781.bm@smtp204.mail.gq1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: WOvY3psVM1mdxXpbsNSZCHMK3IcoNYxTKiQRIHglkua2SqL wlr_D694MaVtAl6ZSrinNnQmMm_w.QtgxThnea095M_Tx3unjMUh1ufPRGlh xtJCHG5czo_LweuXHT98B4gBiBVAD4U72VcVHpNOWlIP9B1QGtq7ta9EAppj .G0iIXjZx4mme00VNJ7GDkhvlwdm9u8qH5spv28_b_1F8qch.jgWPCeRQP.G BWf3DGJtaeRdRIEkmsYUc4iQawi5GMpZgACeWIB802a9XRONTipRkgPKLdZT TBlji5a80R0N2Vx.Lie.W3h9rO6eUWhKKPFx0jW9zACbUeAOYtgnXdsk5BV0 il14i0m8Y3wSfKygKvIs4TZT3uASeWRZMs33B2Y5VOXyyE3g47NWPXCyYVBJ 0nAqsn3fJjC54yATeQHC4viI7bJzVvuiKvZqoeA.PExJSHxtUnzY_gqu0ZKa 1s48yWzK_PvLDUokEIBqoqoy8TavkTspu8G58jw_2R6.u3tUc3mpxubIKWze G6JI3P2OZA9DzaS0zxwYKT5e36m7N.uokWRD3ayGR X-Yahoo-SMTP: clhABp.swBB7fs.LwIJpv3jkWgo2NU8- Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.2 \(3112\)) Subject: Re: CAM Shingled Disk support patches available From: Scott Long In-Reply-To: <20160302212525.GA5920@mithlond.kdm.org> Date: Wed, 2 Mar 2016 21:34:05 -0700 Cc: scsi@freebsd.org, current@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <20151118171309.GA3564@mithlond.kdm.org> <20160118223704.GA87506@mithlond.kdm.org> <20160301224758.GA86834@mithlond.kdm.org> <20160302212525.GA5920@mithlond.kdm.org> To: "Kenneth D. Merry" X-Mailer: Apple Mail (2.3112) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Mar 2016 04:40:58 -0000 > On Mar 2, 2016, at 2:25 PM, Kenneth D. Merry wrote: >=20 >>=20 >>=20 >> void scsi_ata_pass(struct ccb_scsiio *csio, u_int32_t retries, >> void (*cbfcnp)(struct cam_periph *, union ccb = *), >> u_int32_t flags, u_int8_t tag_action, >> struct ata_cmd *cmd, struct ata_res *res, >> u_int8_t *data_ptr, u_int32_t dxfer_len, >> u_int8_t *data_ptr, u_int16_t dxfer_len, >=20 > I assume you only intended one line there, not two. :) >=20 Indeed =3D-) >> u_int8_t sense_len, u_int32_t timeout); >>=20 >> To differentiate between the 12 and 16 byte variants, you???d look at = the >> AP_EXTEND flag in the protocol field. Btw, the handling of that flag = is >> inconsistent in the implementation of the existing = scsi_ata_pass_16(). If >> the caller providse an ata_res pointer then it gets filled on = completion, >> otherwise the caller does its best to look at 12.2.2.6 and extract = what it >> can from the sense descriptor. >>=20 >> So my proposal is to create a new scsi_ata_pass and deprecate but not = remove >> scsi_ata_pass_16. Tell people that if they need to use it, dxfer_len = is going to >> have lower priority than sector_count/sector_count_exp if the latter = multiply to >> more than 65535. >=20 > In general I think that's a reasonable idea, but we should probably go > further. >=20 > While we're at it, we should figure out what we need to do to add the > Auxiliary register to struct ata_cmd. We'll need that to do the NCQ > versions of the various SMR commands, as well as TRIM. >=20 Warner and I talked about this, and I thought that at one point we had a = patch that defined a =E2=80=98struct ata_cmd_aux=E2=80=99. As with function = signatures, I=E2=80=99m very much against redefining structures, and I think it=E2=80=99s reasonable = to create a new structure for what=E2=80=99s basically a late addition to the specs. = I need to read more of the draft ACS and SATA specs to see if there=E2=80=99s anything = else on the horizon that should also be included. However, and I talk a little bit = about this below, I think the situation is a bit more complicated than just = adding a field to the structure. The ata_cmd structure mostly models what=E2=80=99= s in an ATA taskfile, and the taskfile definition, whether from ACS or APT, has = never been expanded to include the aux (and aux_exp) registers. They exist = only in SATA as part of the Device-to-Host (D2H) FIS. However, I=E2=80=99m = kinda back and forth on this; maybe my interpretation of ata_cmd is too strict, and = we can stick in whatever is needed and let the SIM deal with it. At one point I had some patches that defined the various FIS packets and allowed the periphs to send in an XPT_SATA_FIS instead of an XPT_ATA_IO, but it seemed to not provide much value since most drivers (and hardware) still operate in terms of legacy ATA = taskfiles. It also wasn=E2=80=99t clear in the scheme of driving I/O via a FIS = where the responsibility of the periph stopped and the SIM started; If the periph sends a H2D FIS to initiate an I/O, does it need to then drive the PIO and/or DMA FIS=E2=80=99s as well? The FIS is really in the realm of the = transport, not the protocol, and it=E2=80=99s a huge shame that ATA is starting to = blur the lines by having the FIS Aux registers be part of the protocol. > The obvious challenge is that probably means changing the existing = struct > ccb_ataio CCB and bumping the CAM version. At least that will be = source > compatible, but will require ifdefs if people want to compile on older > versions of FreeBSD. But in that case, they'll also be faced with no > support for sending the NCQ versions of the commands, anyway. No way > around that, though, since we have to follow the changing specs. >=20 Again, not a fan of redefining the structures. A couple of other thoughts here. Steve Hartland was right about not = abusing the AP_EXTEND flag. What are your thoughts on having a new flag in the function to signal 12 vs 16 byte variants? Also do you have any = thoughts on the existing arguments? I=E2=80=99m not sure that tag_action has = ever made any sense for the ATA transport, there=E2=80=99s no such a thing as ordered = tags in ATA and we don=E2=80=99t expose the NCQ tag number outside of the SIM. MSG_SIMPLE_Q_TAG definitely has no meaning in ATA. I think the argument could go away. Second, I=E2=80=99m not sure that cam_ata_pass (or cam_ata_pass_16, for = that matter) is the right place to include the aux register. My reading is that = it=E2=80=99s implementing the 12.2.2.x chapter of SAT-3, and that doesn=E2=80=99t have any = allowance for the aux register. SAT-4 r.04a doesn=E2=80=99t seem to mention this register = either. The register seems to only be exposed when you have access to creating a H2D = FIS, and SAT is pretty much silent on that. IIRC, when Warner implemented NCQ TRIM, which required the Aux register, it could only be used on = devices attached via AHCI; LSI controllers had no way of expressing the = register. Still, we could add this ad-hoc to cam_ata_pass(). Maybe instead of it taking = an ata_cmd struct, it takes a union of ata_cmd and ata_cmd_aux, and we add = an argument to the function that tells it what is in the union. Maybe the = union could also include a H2D FIS? Anyways, here=E2=80=99s a possibility: union ata_cmds { struct ata_cmd cmd; struct ata_cmd_aux cmd_aux; /* Don=E2=80=99t = forget that this includes both aux registers! */ struct ata_fis_h2d fis; }; #define ATA_FORMAT_CMD 0x01 #define ATA_FORMAT_CMD_AUX 0x02 #define ATA_FORMAT_FIS_H2D 0x03 void scsi_ata_pass(struct ccb_scsiio *csio, u_int32_t retries, void (*cbfcnp)(struct cam_periph *, union ccb *), u_int32_t flags, u_int format, union ata_cmds *cmd, struct ata_res *res, u_int8_t *data_ptr, u_int32_t dxfer_len, u_int8_t *data_ptr, u_int16_t dxfer_len, u_int8_t sense_len, u_int32_t timeout); Scott > Ken > --=20 > Kenneth Merry > ken@FreeBSD.ORG From owner-freebsd-scsi@freebsd.org Thu Mar 3 07:42:31 2016 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E575CAC1282 for ; Thu, 3 Mar 2016 07:42:30 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from cu01176a.smtpx.saremail.com (cu01176a.smtpx.saremail.com [195.16.150.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A575E1228 for ; Thu, 3 Mar 2016 07:42:30 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from [172.16.8.96] (izaro.sarenet.es [192.148.167.11]) by proxypop03.sare.net (Postfix) with ESMTPSA id 28C9C9DE037; Thu, 3 Mar 2016 08:42:21 +0100 (CET) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.2 \(3112\)) Subject: Re: mpr(4) SAS3008 Repeated Crashing From: Borja Marcos In-Reply-To: Date: Thu, 3 Mar 2016 08:42:20 +0100 Cc: Steven Hartland , FreeBSD-scsi Content-Transfer-Encoding: quoted-printable Message-Id: References: <56D5FDB8.8040402@freebsd.org> <56D612FA.6090909@multiplay.co.uk> To: Scott Long X-Mailer: Apple Mail (2.3112) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Mar 2016 07:42:31 -0000 > On 02 Mar 2016, at 19:43, Scott Long wrote: >> I=E2=80=99ve suffered similar problems, although not as severe, on = one of my storage servers. It=E2=80=99s an IBM X Series with a LSI 3008 = HBA=20 >> connected to the backplane, using SATA SSDs. But mine are almost = certainly hardware problems. An identical system is working >> without issues. >>=20 >> The symptom: with high I/O activity, for example, running Bonnie++, = some commands abort with the disks returning a >> unit attention (power on/reset) asc 0,29. >>=20 >=20 > In your case, the UA is actually a secondary effect. What=E2=80=99s = happening is that a command is timing out so the driver is resetting the = disk. That causes the disk to report a UA with an ASC of 29/0 on the = next command it gets after it comes back up. It=E2=80=99s not fatal and = I=E2=80=99m not sure if it should actually cause a retry, but that=E2=80=99= s an investigation for a different time. It does produce a lot of noise = on the=20 > console/log, though. Hmm. Interesting. It does indeed cause problems, although nothing that a = ZFS scrub cannot fix.=20 So it=E2=80=99s the driver that is resetting the disks? I was assuming = that the disks were resetting themselves for some reason.=20 > One thing I noticed in your log is that one of the commands was a = passthrough ATA command of 0x06 and feature of 0x01, which is DSM TRIM. = It=E2=80=99s not clear if this command was at fault, I need to add = better logging for this case, but it=E2=80=99s highly suspect. It was = only being asked to trim one sector, but given how unpredictable TRIM = responses are from the drive, I don=E2=80=99t know if this matters. = What it might point to, though, is that either the timeout for the = command was too short, the drive doesn=E2=80=99t support DSM TRIM that = well, or the LSI adapter doesn=E2=80=99t support it well (since it=E2=80=99= s not an NCQ command, the LSI firmware would have to remember to flush = out the pending NCQ reads and writes first before doing the DSM = command). The default timeout is 60 seconds, which should be enough = unless you changed it deliberately. If this is a reproducible case, = would you be willing to re-try with a different delete method, i.e. = fiddle with the kern.cam.da.X.delete_method sysctl? The server is not in production for now, so I can run experiments on it. = I am trying with delete_method=3DDISABLE. Although using these disks = without trim would have a performance impact I guess.=20 What is puzzling is, the =E2=80=9Ctwin=E2=80=9D server is working like a = charm. Same hardware, same software. We only updated firmwares on the = ailing one when we noticed problems, just in case. Actually we=E2=80=99ve been poking the dealer and they are going to send = a new one to test. Given how the twin works, the problem should go away. Thanks! Borja.= From owner-freebsd-scsi@freebsd.org Thu Mar 3 10:26:32 2016 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 39593A93B9C for ; Thu, 3 Mar 2016 10:26:32 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from cu01176a.smtpx.saremail.com (cu01176a.smtpx.saremail.com [195.16.150.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id F01FB987 for ; Thu, 3 Mar 2016 10:26:31 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from [172.16.8.96] (izaro.sarenet.es [192.148.167.11]) by proxypop03.sare.net (Postfix) with ESMTPSA id BEAF29DE159; Thu, 3 Mar 2016 11:26:27 +0100 (CET) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.2 \(3112\)) Subject: Re: mpr(4) SAS3008 Repeated Crashing From: Borja Marcos In-Reply-To: <56D805FD.50500@multiplay.co.uk> Date: Thu, 3 Mar 2016 11:26:27 +0100 Cc: Scott Long , FreeBSD-scsi Content-Transfer-Encoding: quoted-printable Message-Id: References: <56D5FDB8.8040402@freebsd.org> <56D612FA.6090909@multiplay.co.uk> <56D805FD.50500@multiplay.co.uk> To: Steven Hartland X-Mailer: Apple Mail (2.3112) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Mar 2016 10:26:32 -0000 > On 03 Mar 2016, at 10:38, Steven Hartland = wrote: >=20 > We've seen HW issues before where the first thing to start triggering = the problem was TRIM requests, it seems like its an afterthought in most = FW's unfortunately, so one of the first things to go bad. I'm not saying = this is you issue, but its something to keep in mind. Thanks :) Not trim related, it seems. I=E2=80=99ve ran the tests with = kern.cam.X.delete_method set to DISABLE and I still see errors: In paranormal cases like this it would be awesome to have access to a = logic analyzer=E2=80=A6 I keep dreaming of course :) Mar 3 11:12:53 clientes-ssd8 kernel: (noperiph:mpr0:0:4294967295:0): = SMID 2 Aborting command 0xfffffe0000c7cab0 Mar 3 11:12:54 clientes-ssd8 kernel: (da2:mpr0:0:26:0): READ(10). CDB: = 28 00 26 d7 d0 98 00 00 20 00 length 16384 SMID 322 terminated ioc 804b = scsi 0 state c xfer 0 Mar 3 11:12:54 clientes-ssd8 kernel: (da2:mpr0:0:26:0): READ(10). CDB: = 28 00 26 d7 d0 b8 00 00 18 00 length 12288 SMID 217 terminated ioc 804b = scsi 0 state c xfer 0 Mar 3 11:12:54 clientes-ssd8 kernel: (da2:mpr0:0:26:0): SYNCHRONIZE = CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 length 0 SMID 205 = terminated ioc 804b scsi 0 sta(da2:mpr0:0:26:0): READ(10). CDB: 28 00 26 = d7 d0 18 00 00 78 00=20 Mar 3 11:12:54 clientes-ssd8 kernel: te c xfer 0 Mar 3 11:12:54 clientes-ssd8 kernel: (da2:mpr0:0:26:0): CAM status: = Command timeout Mar 3 11:12:54 clientes-ssd8 kernel: (da2:mpr0:0:26:0): Retrying = command Mar 3 11:12:54 clientes-ssd8 kernel: (da2:mpr0:0:26:0): SYNCHRONIZE = CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00=20 Mar 3 11:12:54 clientes-ssd8 kernel: (da2:mpr0:0:26:0): CAM status: = SCSI Status Error Mar 3 11:12:54 clientes-ssd8 kernel: (da2:mpr0:0:26:0): SCSI status: = Check Condition Mar 3 11:12:54 clientes-ssd8 kernel: (da2:mpr0:0:26:0): SCSI sense: = UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred) Mar 3 11:12:54 clientes-ssd8 kernel: (da2:mpr0:0:26:0): Retrying = command (per sense data) Borja. From owner-freebsd-scsi@freebsd.org Thu Mar 3 09:45:25 2016 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ABBF0A93FDE for ; Thu, 3 Mar 2016 09:45:25 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wm0-x22b.google.com (mail-wm0-x22b.google.com [IPv6:2a00:1450:400c:c09::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 39D56F92 for ; Thu, 3 Mar 2016 09:45:25 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wm0-x22b.google.com with SMTP id n186so123158428wmn.1 for ; Thu, 03 Mar 2016 01:45:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding; bh=DYBsKtTThCB30sFB3tNjMSWHQd98jzxqby51lq9X+NE=; b=hAW4ONnpYpN0srTVgkhjZdcxeTHR1TS+VZm+dBj8ccNLKY947mybBn8BfMlwxSDrb5 LNVi/iIeq+VRPDRKaKILwiFPlk4vSmfAKirPlEaw9hFjI7mgGi8tT5UUld6FO3N+kzFA P+pv7r0/jgxSqycJzTnZRx7FJHpmpbV1i50byse4GxX1Hqpf8ZvUyCY1ICbzO2niRThj xSZZ+M4Cd4L5knZ7qSqTrwFEiAlF3JUT/W03PbO8onbqtMlSQbGRBonh2oske4ddCdvs d0/O2Y4Z0rniH0MvOcUmaro3FcAXyoz2phqnOnLrtCq2yEsYKbRtcn8A/NKNuKBatNst MOdg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=DYBsKtTThCB30sFB3tNjMSWHQd98jzxqby51lq9X+NE=; b=V+pljqo5ZNx6tLUwt2EsjnulVO5zoK9KTHET5h7p7m5AGclF/ARqshk8vMkKivJpVa 4YSs1ErvM97+k6z4QKrbJvjymivpqIJLO6/jf+e2Gw2zoNdyUIZfDifAermRxKwFr6b3 icWUqFITUQRcNHRqWsd6aSu6bPflo9Zyt3IvRa3IlkNDiq2LHKTWPH7zLyEulDYcDakt labgKiIXUVLckEUSgl576QPbF/ACNJNTSu0a8owavkVQshIkV46mKO9lPJBGWoDX7LtX MemVOmGtj8eTFYVS8WGb2rVhtl/LXfX5YnFrhQdkv1s+IWt8PtVjNxvCDT57euWoaxIV olKA== X-Gm-Message-State: AD7BkJJSbVSw45ntminLG9H/47QsVqcON+bvBXdizG2iL3COj+nWS+DfI2j4cl3YRclsnQ3y X-Received: by 10.194.48.111 with SMTP id k15mr2076869wjn.103.1456998323436; Thu, 03 Mar 2016 01:45:23 -0800 (PST) Received: from [10.10.1.58] (liv3d.labs.multiplay.co.uk. [82.69.141.171]) by smtp.gmail.com with ESMTPSA id t7sm10187475wjf.39.2016.03.03.01.45.22 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 03 Mar 2016 01:45:22 -0800 (PST) Subject: Re: CAM Shingled Disk support patches available To: Scott Long References: <20151118171309.GA3564@mithlond.kdm.org> <20160118223704.GA87506@mithlond.kdm.org> <20160301224758.GA86834@mithlond.kdm.org> <56D6BAA7.1050004@multiplay.co.uk> Cc: freebsd-scsi@freebsd.org From: Steven Hartland Message-ID: <56D807C2.5090801@multiplay.co.uk> Date: Thu, 3 Mar 2016 09:45:38 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.6.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Mar 2016 09:45:25 -0000 On 02/03/2016 16:12, Scott Long wrote: >> On Mar 2, 2016, at 3:04 AM, Steven Hartland wrote: >> >> scsi_ata_pass_16 was added to support TRIM and implements 12.2.3 (not 12.2.2) providing access to all features as described by Table 104. I'll admit I didn't take into account callers which make non LBA use of the LBA fields, but I'm sure you'll agree that's an experience thing i.e. knowing about about the existence of features which abuse structure fields for other purposes ;-) >> > I’m going from SAT-3r5, which unfortunately is the only rev I have since T10 makes it hard to get SAT-3r7. Maybe the numbers changed? I meant 12.2.2.x, which covers 12.2.2.1, 12.2.2.2, and 12.2.2.3 in the draft that I have. There is no 12.2.3 in my draft. > >> Its use of u_int16_t dxfer_len is a bug, surprised the compiler didn't flag this, not sure this can be fixed without breaking backwards compatibility though? >> >> Not sure I understand the 12 vs 13 registers comment, I'm guessing you mean the 12 bytes of 12.2.2, however it actually exposes all 16 bytes of 12.2.3. >> > scsi_ata_pass_16 doesn’t accept a device argument and assumes it to be ATA_DEV_LBA, 0x40. There are two commands in ACS-3, READ_FPDMA_QUEUED and WRITE_FPDMA_QUEUED, that could use a different value for bit 7. Maybe that’s an unlikely case for userland passthrough, but not impossible. However bit 4 is also a wildcard that is transport-dependent for many commands. SATA is pretty clear about the bit always being zero. Technically in PATA you might want to control bit 4 to specify the master vs slave device, but I’m guessing that FreeBSD would be more likely to override that than support it. Still, I haven’t done a comprehensive review of all command sets and transports, and I think it would be prudent to leave the device register exposed to the API rather than hard-coded. > >> For reference it has two callers in the kernel scsi_ata_identify used to probe TRIM capability and scsi_ata_trim used to actually perform TRIM operations. >> >> In addition to this its also used by camcontrol to support passthrough from ata_do_cmd and ata_do_28bit_cmd again supporting TRIM, in this case the ATA security feature set as well as identify. >> >> I would be very surprised if its used elsewhere but in order to maintain compatibility I'd definitely agree with Scott on creating an scsi_ata_pass which can be used to implement and scsi_ata_pass_16 which can then be deprecated. >> > It’s exposed in libcam.so, and it sounds like Ken is also expanding its use, so it’s fair to say that it might have a wider audience than just those TRIM uses. > >> With regards to providing 12.2.2 and 12.2.3 by looking at AP_EXTEND I don't believe that's possible as you need this to tell the difference between 28bit and 48bit ATA within 12.2.3. >> > 12.2.2.3 states the following: > > If the EXTEND bit is set to zero, then the FEATURES (15:8) field, the SECTOR_COUNT (15:8) field, the LBA_LOW (15:8) field, then the LBA_MID (15:8) field, and the LBA_HIGH (15:8) field shall be ignored by the SATL, and the SATL shall process this command as specified in 12.2.2.2. > > My proposal isn’t completely correct, it would be abusing the EXTEND bit to force the use of opcode A1 vs 85, and really the spec says that 85 can be used with the bit being zero. One might want to send an 0x85 opcode with EXTEND set to zero to test that the receiver will do the right thing with ignoring the 5 extended fields, or you might have a receiver that only accepts 0x85 but you need to send a 28 bit command. So yeah, I concede it’s not great. Might need to either add an extra flag, or split the function into two. > > Scott > Sounds like the specs have changed a bit, in some subtle but important ways, since this I implemented this 3 years ago, that and I didn't consider PATA write or wrong. That's not to say I didn't miss things but it did achieve what it was intended to do at the time. It sounds like it could have had more flexibility designed in, but that was for features / use cases which weren't apparent to me at the time so sorry about that. Regards Steve From owner-freebsd-scsi@freebsd.org Thu Mar 3 09:37:53 2016 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9FF4FA93E0B for ; Thu, 3 Mar 2016 09:37:53 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wm0-x22d.google.com (mail-wm0-x22d.google.com [IPv6:2a00:1450:400c:c09::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 405E1CBD for ; Thu, 3 Mar 2016 09:37:52 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wm0-x22d.google.com with SMTP id l68so122454876wml.0 for ; Thu, 03 Mar 2016 01:37:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding; bh=gzutXeqdXCHZ2f9wwjx8fRPPEfWFY1ZrEj0Gmxri0w4=; b=W0Mw/TCT+SZGmW5D/JjedS00C23EmuBO1iTuL371tz+ucMUvaQME4Fzx9arYAJyAwE Fw0eaj6yQiStix48qqg2BFzWckDZ4YLs8sdHsbm9vFBSoV/CvvMOSJCPnneDONp2rQYM FVV0s8JTNhdZQ1R2sZer6gQtUtbEzt8oVxTWyXoxNmjvDE/GPfqz23zrybUs/y34ScVl NQsANOymfKC8bjJhzIvscJOHWr3m1w4TBN7Oxv0MAAWKsaXYmbbqb7125Rby4y9wTVb9 2Hxq642/wtKy/cHElYAwLXxjyAqGslvSHR8HG0FqmlJMbY/rEQ2kn2gs9v9WtIXB1XRm vw3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=gzutXeqdXCHZ2f9wwjx8fRPPEfWFY1ZrEj0Gmxri0w4=; b=KnHKaPmDy76X5P9s0GSASrIz4LPiW/aee+3juZusFTUMqrYUkL/YXEGIgZRhA1kmJ+ Wm3tXvqmB9LdT67PYT5DEeP/OyfsiZvaVk4tHWyaXjyb/hdwJUjEbfEndwwdV3bScG+p 0K0g4REP9DEWzslxnkZAX7aMUgdivnpvEfU+nD4fhMvnB0YjkIo+G8MIQHgQmKna9B3C Sj7/iigZxGzBMvdhQoFx6t+FaNqYoGteJ4sWBjuMId8WmMGut4gKNL2zFkCojrFK2Kk/ MqywoOLvV9aYN5lox4PSgbL03nTsZgW33iExHBrcjdweS3dcRaO5dKiI02D1YlthbxH4 uI7A== X-Gm-Message-State: AD7BkJLro6U3MOYHeusIiMLD+XymoeA+0nky9RhzbiA18OAfrxHH6Js6cM6GF7e+YoA+f16R X-Received: by 10.28.189.67 with SMTP id n64mr4947052wmf.24.1456997871171; Thu, 03 Mar 2016 01:37:51 -0800 (PST) Received: from [10.10.1.58] (liv3d.labs.multiplay.co.uk. [82.69.141.171]) by smtp.gmail.com with ESMTPSA id 192sm8075417wmw.0.2016.03.03.01.37.49 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 03 Mar 2016 01:37:50 -0800 (PST) Subject: Re: mpr(4) SAS3008 Repeated Crashing To: Borja Marcos , Scott Long References: <56D5FDB8.8040402@freebsd.org> <56D612FA.6090909@multiplay.co.uk> Cc: FreeBSD-scsi From: Steven Hartland Message-ID: <56D805FD.50500@multiplay.co.uk> Date: Thu, 3 Mar 2016 09:38:05 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.6.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Mar 2016 09:37:53 -0000 On 03/03/2016 07:42, Borja Marcos wrote: >> On 02 Mar 2016, at 19:43, Scott Long wrote: >>> I=E2=80=99ve suffered similar problems, although not as severe, on on= e of my storage servers. It=E2=80=99s an IBM X Series with a LSI 3008 HBA= >>> connected to the backplane, using SATA SSDs. But mine are almost cert= ainly hardware problems. An identical system is working >>> without issues. >>> >>> The symptom: with high I/O activity, for example, running Bonnie++, s= ome commands abort with the disks returning a >>> unit attention (power on/reset) asc 0,29. >>> >> In your case, the UA is actually a secondary effect. What=E2=80=99s h= appening is that a command is timing out so the driver is resetting the d= isk. That causes the disk to report a UA with an ASC of 29/0 on the next= command it gets after it comes back up. It=E2=80=99s not fatal and I=E2= =80=99m not sure if it should actually cause a retry, but that=E2=80=99s = an investigation for a different time. It does produce a lot of noise on= the >> console/log, though. This sounds similar to what we saw in mfi; while the cause was different = the real problem was the error paths in the driver where untested and=20 buggy causing more problems and resulting in panics. I was lucky, or unlucky depending on your point of view, that the HW=20 issue we had was very good at triggering pretty much every failure path=20 in the driver which allowed me to fix them, without that its really hard = to truly test these code paths which hardly ever get exercised. > Hmm. Interesting. It does indeed cause problems, although nothing that = a ZFS scrub cannot fix. > > So it=E2=80=99s the driver that is resetting the disks? I was assuming = that the disks were resetting themselves for some reason. > >> One thing I noticed in your log is that one of the commands was a pass= through ATA command of 0x06 and feature of 0x01, which is DSM TRIM. It=E2= =80=99s not clear if this command was at fault, I need to add better logg= ing for this case, but it=E2=80=99s highly suspect. It was only being as= ked to trim one sector, but given how unpredictable TRIM responses are fr= om the drive, I don=E2=80=99t know if this matters. What it might point = to, though, is that either the timeout for the command was too short, the= drive doesn=E2=80=99t support DSM TRIM that well, or the LSI adapter doe= sn=E2=80=99t support it well (since it=E2=80=99s not an NCQ command, the = LSI firmware would have to remember to flush out the pending NCQ reads an= d writes first before doing the DSM command). The default timeout is 60 = seconds, which should be enough unless you changed it deliberately. If t= his is a reproducible case, would you be willing to re-try with a differe= nt delete method, i.e. fiddle with the kern.cam.da.X.delete_method sysctl= ? > The server is not in production for now, so I can run experiments on it= =2E I am trying with delete_method=3DDISABLE. Although using these disks = without trim would have > a performance impact I guess. > > What is puzzling is, the =E2=80=9Ctwin=E2=80=9D server is working like = a charm. Same hardware, same software. We only updated firmwares on the a= iling one when we noticed problems, > just in case. > > Actually we=E2=80=99ve been poking the dealer and they are going to sen= d a new one to test. Given how the twin works, the problem should go away= =2E > We've seen HW issues before where the first thing to start triggering=20 the problem was TRIM requests, it seems like its an afterthought in most = FW's unfortunately, so one of the first things to go bad. I'm not saying = this is you issue, but its something to keep in mind. Regards Steve From owner-freebsd-scsi@freebsd.org Thu Mar 3 17:12:52 2016 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E4FE5A93F55 for ; Thu, 3 Mar 2016 17:12:52 +0000 (UTC) (envelope-from scott4long@yahoo.com) Received: from nm10-vm10.bullet.mail.gq1.yahoo.com (nm10-vm10.bullet.mail.gq1.yahoo.com [98.136.218.91]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4BB17818 for ; Thu, 3 Mar 2016 17:12:51 +0000 (UTC) (envelope-from scott4long@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1457024979; bh=wQhsy0bDQ6PB7pAWu7hAfHGKEJ9t+G2FO3T7W7unxfc=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject; b=TXCQ/BoplnKoZKnn/gktIBSX6dthuQD9coTueP4jOATb5XeiT9yX9cbN1PjP3ZcPnU9JX62s+iyUto3qARo7wMyhJ2NWD+viLz1N3aHEu/9fM4mN5kC3rSbEf+xQdGkAu5PG/vS83c8QQoPjK8E62r3z0niaZ+wMNBDFmatcWDpQH/0vnb4Q7lOCPVbIAhm9bZD/ErWaSHn3YDnE+gw0JEPRS20GbBivXQl7zCFrPcupATjnFNqRhZJFKVju3I0wuM3x8iCCZOiKRxOpCW0X8sUNoJHJAK37EI7nLAXaAn6RVRYOPzzkvKYsLSkt8XQjyB/MLDMM3lpyGdflcDhXMg== Received: from [216.39.60.180] by nm10.bullet.mail.gq1.yahoo.com with NNFMP; 03 Mar 2016 17:09:39 -0000 Received: from [98.136.164.75] by tm16.bullet.mail.gq1.yahoo.com with NNFMP; 03 Mar 2016 17:09:39 -0000 Received: from [127.0.0.1] by smtp237.mail.gq1.yahoo.com with NNFMP; 03 Mar 2016 17:09:39 -0000 X-Yahoo-Newman-Id: 198679.53640.bm@smtp237.mail.gq1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: g33At.AVM1lPUjEncDYhy_VugM2kMRGcQtyB9sLwHIWTIVF 65oLUMp3WqLwdvCzmdVZu89N84YvwREdvb7GE5ucHPTVJw2ET6vtHZH6P_VG SqcS4zzQ_XGZZRq156eIhL_6Ee0mOdAPEcwBKTNM6IlbQJ1GMlTa01dwZ48. EeJvxNK5eAsLkQLi9i1Kb7KKgjvltKNshfYLw.cKDI6GxD7o0WLrYZLmceoR CuN6DQfHYGlTJhiE6BSHNZi6L.LA1atRpwlJPQDtEcRTVSPFW98DM8UBCjlh V7DTSUc.NbrczOIAsihGu0DiDpgBX2d94S8njmWnKsAxLdW2t0L6px.qSwuj RYoP7zAJtfVNKzCWmGbcNYnMDHTX6sha3.tyA6LxopPvDpIL_FYaRihBWtD6 KYlznjG362QpVWefTjxBVvgqoKy5jlywFJsUCo3.iWz0jxAOT2KzLaRnq_BP ogn.2FENMaovODQEialHGlSOxrSVP5Lnc2NDnJz77AlaLkUA2auATahAOmDY xCfPxQJ51KLI7ZcuSwTJnU_MNIeDrPNcH5j71cdY2UTUKnA-- X-Yahoo-SMTP: clhABp.swBB7fs.LwIJpv3jkWgo2NU8- Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.2 \(3112\)) Subject: Re: mpr(4) SAS3008 Repeated Crashing From: Scott Long In-Reply-To: Date: Thu, 3 Mar 2016 10:09:37 -0700 Cc: Steven Hartland , FreeBSD-scsi Content-Transfer-Encoding: quoted-printable Message-Id: References: <56D5FDB8.8040402@freebsd.org> <56D612FA.6090909@multiplay.co.uk> <56D805FD.50500@multiplay.co.uk> To: Borja Marcos X-Mailer: Apple Mail (2.3112) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Mar 2016 17:12:53 -0000 > On Mar 3, 2016, at 3:26 AM, Borja Marcos wrote: >=20 >=20 >> On 03 Mar 2016, at 10:38, Steven Hartland = wrote: >>=20 >> We've seen HW issues before where the first thing to start triggering = the problem was TRIM requests, it seems like its an afterthought in most = FW's unfortunately, so one of the first things to go bad. I'm not saying = this is you issue, but its something to keep in mind. >=20 > Thanks :) >=20 > Not trim related, it seems. I=E2=80=99ve ran the tests with = kern.cam.X.delete_method set to DISABLE and I still see errors: >=20 > In paranormal cases like this it would be awesome to have access to a = logic analyzer=E2=80=A6 I keep dreaming of course :) >=20 >=20 >=20 > Mar 3 11:12:53 clientes-ssd8 kernel: (noperiph:mpr0:0:4294967295:0): = SMID 2 Aborting command 0xfffffe0000c7cab0 > Mar 3 11:12:54 clientes-ssd8 kernel: (da2:mpr0:0:26:0): READ(10). = CDB: 28 00 26 d7 d0 98 00 00 20 00 length 16384 SMID 322 terminated ioc = 804b scsi 0 state c xfer 0 > Mar 3 11:12:54 clientes-ssd8 kernel: (da2:mpr0:0:26:0): READ(10). = CDB: 28 00 26 d7 d0 b8 00 00 18 00 length 12288 SMID 217 terminated ioc = 804b scsi 0 state c xfer 0 > Mar 3 11:12:54 clientes-ssd8 kernel: (da2:mpr0:0:26:0): SYNCHRONIZE = CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 length 0 SMID 205 = terminated ioc 804b scsi 0 sta(da2:mpr0:0:26:0): READ(10). CDB: 28 00 26 = d7 d0 18 00 00 78 00=20 > Mar 3 11:12:54 clientes-ssd8 kernel: te c xfer 0 > Mar 3 11:12:54 clientes-ssd8 kernel: (da2:mpr0:0:26:0): CAM status: = Command timeout > Mar 3 11:12:54 clientes-ssd8 kernel: (da2:mpr0:0:26:0): Retrying = command > Mar 3 11:12:54 clientes-ssd8 kernel: (da2:mpr0:0:26:0): SYNCHRONIZE = CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00=20 > Mar 3 11:12:54 clientes-ssd8 kernel: (da2:mpr0:0:26:0): CAM status: = SCSI Status Error > Mar 3 11:12:54 clientes-ssd8 kernel: (da2:mpr0:0:26:0): SCSI status: = Check Condition > Mar 3 11:12:54 clientes-ssd8 kernel: (da2:mpr0:0:26:0): SCSI sense: = UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred) > Mar 3 11:12:54 clientes-ssd8 kernel: (da2:mpr0:0:26:0): Retrying = command (per sense data) >=20 SYNC CACHE seems to have been involved this time, and while it=E2=80=99s = sometimes a source of trouble with SATA disks, I=E2=80=99m very hesitant = to blame it. Given the seemingly random nature of your problems, I=E2=80=99= m not as certain anymore to rule out a fault of the disk enclosure. = This looks to be a different disk than your last report, and your = statement that a sibling system exhibits no problems is very = interesting. Maybe there=E2=80=99s an issue with the power supply, and = the disks are getting under-voltage conditions periodically. If you can = run smartctl against the disks, the output might be useful. Also, if = you=E2=80=99re able, could you make sure that both this system and the = one that is working well are being fed with sufficient and similar AC = power? And if the power supply modules in your enclosures are = swappable, maybe swap them between systems and see if the problem = follows the module? If that doesn=E2=80=99t fix it then I=E2=80=99ll = think of ways to provide more instrumentation. Scott From owner-freebsd-scsi@freebsd.org Thu Mar 3 13:03:02 2016 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8C99EA932CC for ; Thu, 3 Mar 2016 13:03:02 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wm0-x232.google.com (mail-wm0-x232.google.com [IPv6:2a00:1450:400c:c09::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 23DCDC26 for ; Thu, 3 Mar 2016 13:03:02 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wm0-x232.google.com with SMTP id l68so130237752wml.0 for ; Thu, 03 Mar 2016 05:03:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding; bh=2ZtvMVquqssoiwuaRZr/x7Qwgle9S30O1YjI4qhB3fE=; b=IV+aH+H1L4VcND8BtTWlLbgC3FoltTPl5G2O2EQy6M3Cf1/x+FXsMSQhOnFaH9MZuI shIS/OD44MWPtaxsgeK3m8mfLKC8o1orjoHVEdAue51Bs4LPwRSFNIZ3u0CfgZa2OU6V Xqe3cvIEaMDF4aRxkpvNpM1RztT0KMUeii+dVYU8uHwlwD8tsG7x5vzJL8Uz/cWkkj+l kWDjeCZ1egyPAeUtg3kCMLiZUCVipTs9ErsmbVyjLnOp38vnna/8OBNfo76PE44mHu7q p4jg44r06u4ciXKyRLte2IlMHGbrzG0NjPtlacLbvO412sm+7cl4ofll8aRkMHXDjHdG JGoQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=2ZtvMVquqssoiwuaRZr/x7Qwgle9S30O1YjI4qhB3fE=; b=IZQSRWD8NisGJOBXf129InyEQRDydQKQupYOmCudd82hfLHZvUo3JHEWXbaHptbbMK IJreRMWUVygU1YSR93azXQr1h5tiWs/Fl2G+b7wf0JVB8C+BLsvi/85cALK/e+374G9A AMqBtHLp/Q3ywNwzHX6ZWfm+YuxCw/wCmQuiLOj1xaH9ctS7qEDkiUAiYF7LMmISKNKg Lku64O3TuZ3zhz9HhR8fwrw01wJSAapM2e7maO6z3u28NeCK0hg7T+BaKhFqT4dlJ5KP 0Qyv38ZUotJyEau5aJ1H3CnH0+cu1/1GTHfnbRHXyACrQZWMqeLcLv7bf6bnEV9byfiR kq0w== X-Gm-Message-State: AD7BkJLvdlojSFPKRmEAdd32E60F78gAG53QvjKkhrJh/BdlT36BQZShkSfSBqm66MZY0k8E X-Received: by 10.28.86.10 with SMTP id k10mr5924705wmb.28.1457010179470; Thu, 03 Mar 2016 05:02:59 -0800 (PST) Received: from [10.10.1.58] (liv3d.labs.multiplay.co.uk. [82.69.141.171]) by smtp.gmail.com with ESMTPSA id v2sm8863676wmd.24.2016.03.03.05.02.57 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 03 Mar 2016 05:02:57 -0800 (PST) Subject: Re: mpr(4) SAS3008 Repeated Crashing To: Borja Marcos References: <56D5FDB8.8040402@freebsd.org> <56D612FA.6090909@multiplay.co.uk> <56D805FD.50500@multiplay.co.uk> Cc: Scott Long , FreeBSD-scsi From: Steven Hartland Message-ID: <56D83600.6090400@multiplay.co.uk> Date: Thu, 3 Mar 2016 13:02:56 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.6.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Mar 2016 13:03:02 -0000 On 03/03/2016 10:26, Borja Marcos wrote: >> On 03 Mar 2016, at 10:38, Steven Hartland wrote: >> >> We've seen HW issues before where the first thing to start triggering the problem was TRIM requests, it seems like its an afterthought in most FW's unfortunately, so one of the first things to go bad. I'm not saying this is you issue, but its something to keep in mind. > Thanks :) > > Not trim related, it seems. I’ve ran the tests with kern.cam.X.delete_method set to DISABLE and I still see errors: > > In paranormal cases like this it would be awesome to have access to a logic analyzer… I keep dreaming of course :) > > > > Mar 3 11:12:53 clientes-ssd8 kernel: (noperiph:mpr0:0:4294967295:0): SMID 2 Aborting command 0xfffffe0000c7cab0 > Mar 3 11:12:54 clientes-ssd8 kernel: (da2:mpr0:0:26:0): READ(10). CDB: 28 00 26 d7 d0 98 00 00 20 00 length 16384 SMID 322 terminated ioc 804b scsi 0 state c xfer 0 > Mar 3 11:12:54 clientes-ssd8 kernel: (da2:mpr0:0:26:0): READ(10). CDB: 28 00 26 d7 d0 b8 00 00 18 00 length 12288 SMID 217 terminated ioc 804b scsi 0 state c xfer 0 > Mar 3 11:12:54 clientes-ssd8 kernel: (da2:mpr0:0:26:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 length 0 SMID 205 terminated ioc 804b scsi 0 sta(da2:mpr0:0:26:0): READ(10). CDB: 28 00 26 d7 d0 18 00 00 78 00 > Mar 3 11:12:54 clientes-ssd8 kernel: te c xfer 0 > Mar 3 11:12:54 clientes-ssd8 kernel: (da2:mpr0:0:26:0): CAM status: Command timeout > Mar 3 11:12:54 clientes-ssd8 kernel: (da2:mpr0:0:26:0): Retrying command > Mar 3 11:12:54 clientes-ssd8 kernel: (da2:mpr0:0:26:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 > Mar 3 11:12:54 clientes-ssd8 kernel: (da2:mpr0:0:26:0): CAM status: SCSI Status Error > Mar 3 11:12:54 clientes-ssd8 kernel: (da2:mpr0:0:26:0): SCSI status: Check Condition > Mar 3 11:12:54 clientes-ssd8 kernel: (da2:mpr0:0:26:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred) > Mar 3 11:12:54 clientes-ssd8 kernel: (da2:mpr0:0:26:0): Retrying command (per sense data) > Have you tried mrsas instead of mpr? You can do so by adding the following to /boot/loader.conf: hw.mfi.mrsas_enable="1" Regards Steve From owner-freebsd-scsi@freebsd.org Thu Mar 3 20:30:47 2016 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8896EA944B0 for ; Thu, 3 Mar 2016 20:30:47 +0000 (UTC) (envelope-from gisiw@sexgirling.pw) Received: from mail.sexgirling.pw (mail.sexgirling.pw [121.42.144.60]) by mx1.freebsd.org (Postfix) with ESMTP id 197B0FF9 for ; Thu, 3 Mar 2016 20:30:46 +0000 (UTC) (envelope-from gisiw@sexgirling.pw) Received: from Y480-PC (unknown [119.136.170.5]) by mail.sexgirling.pw (Postfix) with ESMTPA id 3C8F0C3C9A for ; Fri, 4 Mar 2016 04:23:10 +0800 (CST) From: =?UTF-8?B?6ZuF576O6J225ZWG5Z+O?= Reply-To: customer@ymdlife.com To: freebsd-scsi@freebsd.org Message-ID: <10068866.5148251457036538720.JavaMail.Y480@Y480-PC> Subject: =?UTF-8?Q?=E6=88=91=E7=9A=84=E7=88=B1=E7=88=B1=E6=9C=80=E5=A4=A7=E5=BF=AB?= =?UTF-8?Q?=E6=84=9F=E7=A7=98=E6=96=B9_2016?= =?UTF-8?Q?-03-04_04:22:18?= MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.21 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Date: Thu, 03 Mar 2016 20:30:47 -0000 X-List-Received-Date: Thu, 03 Mar 2016 20:30:47 -0000 From owner-freebsd-scsi@freebsd.org Fri Mar 4 08:02:35 2016 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ACF0C9DA0CA for ; Fri, 4 Mar 2016 08:02:35 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from cu01176b.smtpx.saremail.com (cu01176b.smtpx.saremail.com [195.16.151.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6E863F21 for ; Fri, 4 Mar 2016 08:02:34 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from [172.16.8.96] (izaro.sarenet.es [192.148.167.11]) by proxypop01.sare.net (Postfix) with ESMTPSA id B5CB19DDF16; Fri, 4 Mar 2016 09:02:25 +0100 (CET) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.2 \(3112\)) Subject: Re: mpr(4) SAS3008 Repeated Crashing From: Borja Marcos In-Reply-To: Date: Fri, 4 Mar 2016 09:02:25 +0100 Cc: Steven Hartland , FreeBSD-scsi Content-Transfer-Encoding: quoted-printable Message-Id: References: <56D5FDB8.8040402@freebsd.org> <56D612FA.6090909@multiplay.co.uk> <56D805FD.50500@multiplay.co.uk> To: Scott Long X-Mailer: Apple Mail (2.3112) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Mar 2016 08:02:35 -0000 > On 03 Mar 2016, at 18:09, Scott Long wrote: >=20 >=20 > SYNC CACHE seems to have been involved this time, and while it=E2=80=99s= sometimes a source of trouble with SATA disks, I=E2=80=99m very = hesitant to blame it. Given the seemingly random nature of your = problems, I=E2=80=99m not as certain anymore to rule out a fault of the = disk enclosure. This looks to be a different disk than your last = report, and your statement that a sibling system exhibits no problems is = very interesting. Maybe there=E2=80=99s an issue with the power supply, = and the disks are getting under-voltage conditions periodically. If you = can run smartctl against the disks, the output might be useful. Also, = if you=E2=80=99re able, could you make sure that both this system and = the one that is working well are being fed with sufficient and similar = AC power? And if the power supply modules in your enclosures are = swappable, maybe swap them between systems and see if the problem = follows the module? If that doesn=E2=80=99t fix it then I=E2=80=99ll = think of ways to provide more instrumentation. The affected disks are completely random. I didn=E2=80=99t copy a lot of = instances to avoid too much litter, but each time it=E2=80=99s a = different disk. Both systems are in the same datacenter, and yes, the power = infrastructure is working. Swapping modules can be done if the dealer sends us another one because I prefer not to mess with a = working system. The fact that it=E2=80=99s a different disk each time, and that the = other system works perfectly is what makes me quite certain that it=E2=80=99= s a hardware problem. Either some trouble with the backplane or a power problem. I am tempted to go the oscilloscope route (monitoring the internal power = rails). But if the problem is in the power distribution of the backplane = itself I=E2=80=99ll need to destroy a broken disk to build a backplane power = probe :) Borja. From owner-freebsd-scsi@freebsd.org Fri Mar 4 09:16:25 2016 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 604859DA3E5 for ; Fri, 4 Mar 2016 09:16:25 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wm0-x22f.google.com (mail-wm0-x22f.google.com [IPv6:2a00:1450:400c:c09::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E8EDF929 for ; Fri, 4 Mar 2016 09:16:24 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wm0-x22f.google.com with SMTP id p65so11617523wmp.1 for ; Fri, 04 Mar 2016 01:16:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding; bh=TOo+addsXbNzHm2TzNT/hhUF72OQ0W5ywugPGdV9eY4=; b=S2KWhlozUg/ruTyOQgOYeIJ9GCh1NGaY3G8VY0UndawTcDWIjn+kd95JYPo64ZdFW8 7Of6ls353kTGTxdyRZFcdSL/QBuSIPAsbmkbQU+u4bFHLLL8yS6+nisG/qEUCIjtwcUe ywXJRfirD7YPasvkKU4zQs6VUk/CRtTHD79kgsMIVDVBN8Ypz0SjZbEPF3PdZ/mPp4ge Ev2zR6xBws9YblgBU9W6hlkYel4pGjWIaoFbxO/ZszjlY2q1oOxtdTLSSEjZHQNq31R4 M/MfYxjPsZ/Dic6GaXG8RDiiR1z8iQQkck/6okpoQbMKqtPaL2bo6x2wHr418AhVy3y2 K8Vg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=TOo+addsXbNzHm2TzNT/hhUF72OQ0W5ywugPGdV9eY4=; b=Yrd3Nv3f+H1VugI8zzBqOTfORXXZgW2/0Wz4sReJljST2/ebeDw+YPKR5vqg8XfYH8 ZrBI80KrYRZbX6Hd+D8Ue7vf+8XyXGrv2lyKPZze6963xt0QEnjlWnaycDg1r/iWJ+aT mRmY/386NAcmYcAXjC+SJBmICOA4RAYbZuI6znuVFNbUeM/Gnw4UqeZAmmcd97bxHsp9 s36ATFbVr/UjYjTvTC1TdvhfF9LU95LV0YNN4cAS+NY4dCD4ta6/ua6K1mSJM+ICp9Sz hDmhyIh8xM1W2nBqtBKeqae3Ah7mfwCxSI3fNK6eyCW+ycQjUnelaLLBMnpV31X3PfOK 7sNQ== X-Gm-Message-State: AD7BkJLJigcLhVQ8bTvt1u6p27hfzOZRiVA5WtEXJnAiwdwyiwF8C7+3A+4XZ7vs8/6M1Qvz X-Received: by 10.28.138.198 with SMTP id m189mr3831346wmd.19.1457082983459; Fri, 04 Mar 2016 01:16:23 -0800 (PST) Received: from [10.10.1.58] (liv3d.labs.multiplay.co.uk. [82.69.141.171]) by smtp.gmail.com with ESMTPSA id e4sm2155211wma.10.2016.03.04.01.16.22 (version=TLSv1/SSLv3 cipher=OTHER); Fri, 04 Mar 2016 01:16:22 -0800 (PST) Subject: Re: mpr(4) SAS3008 Repeated Crashing To: Borja Marcos , Scott Long References: <56D5FDB8.8040402@freebsd.org> <56D612FA.6090909@multiplay.co.uk> <56D805FD.50500@multiplay.co.uk> Cc: FreeBSD-scsi From: Steven Hartland Message-ID: <56D95266.301@multiplay.co.uk> Date: Fri, 4 Mar 2016 09:16:22 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.6.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Mar 2016 09:16:25 -0000 On 04/03/2016 08:02, Borja Marcos wrote: >> On 03 Mar 2016, at 18:09, Scott Long wrote: >> >> >> SYNC CACHE seems to have been involved this time, and while it=E2=80=99= s sometimes a source of trouble with SATA disks, I=E2=80=99m very hesitan= t to blame it. Given the seemingly random nature of your problems, I=E2=80= =99m not as certain anymore to rule out a fault of the disk enclosure. T= his looks to be a different disk than your last report, and your statemen= t that a sibling system exhibits no problems is very interesting. Maybe = there=E2=80=99s an issue with the power supply, and the disks are getting= under-voltage conditions periodically. If you can run smartctl against = the disks, the output might be useful. Also, if you=E2=80=99re able, cou= ld you make sure that both this system and the one that is working well a= re being fed with sufficient and similar AC power? And if the power supp= ly modules in your enclosures are swappable, maybe swap them between syst= ems and see if the problem follows the module? If that doesn=E2=80=99t f= ix it then I=E2=80=99ll think of ways to provide more instrumentation. > The affected disks are completely random. I didn=E2=80=99t copy a lot o= f instances to avoid too much litter, but each time it=E2=80=99s a differ= ent disk. > > Both systems are in the same datacenter, and yes, the power infrastruct= ure is working. Swapping modules can be done if > the dealer sends us another one because I prefer not to mess with a wor= king system. > > The fact that it=E2=80=99s a different disk each time, and that the oth= er system works perfectly is what makes me quite certain that it=E2=80=99= s a hardware problem. Either some trouble > with the backplane or a power problem. > > I am tempted to go the oscilloscope route (monitoring the internal powe= r rails). But if the problem is in the power distribution of the backplan= e itself > I=E2=80=99ll need to destroy a broken disk to build a backplane power p= robe :) > Its very rare but we've also seen this type of behaviour from a failing=20 Intel CPU. There was no other indication the CPU had an issue, which one = might expect, so just wanted to make you aware of the possibility. That said the most common cause of this we've seen, when its not a=20 common disk or disks, is a bad backplane or cabling to the backplane. Regards Steve From owner-freebsd-scsi@freebsd.org Fri Mar 4 10:59:03 2016 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 237E39DAA1E for ; Fri, 4 Mar 2016 10:59:03 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from cu01176b.smtpx.saremail.com (cu01176b.smtpx.saremail.com [195.16.151.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D46AD91E for ; Fri, 4 Mar 2016 10:59:02 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from [172.16.8.96] (izaro.sarenet.es [192.148.167.11]) by proxypop01.sare.net (Postfix) with ESMTPSA id B15CF9DD2D9; Fri, 4 Mar 2016 11:58:58 +0100 (CET) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.2 \(3112\)) Subject: Re: mpr(4) SAS3008 Repeated Crashing From: Borja Marcos In-Reply-To: <56D95266.301@multiplay.co.uk> Date: Fri, 4 Mar 2016 11:58:58 +0100 Cc: Scott Long , FreeBSD-scsi Content-Transfer-Encoding: quoted-printable Message-Id: References: <56D5FDB8.8040402@freebsd.org> <56D612FA.6090909@multiplay.co.uk> <56D805FD.50500@multiplay.co.uk> <56D95266.301@multiplay.co.uk> To: Steven Hartland X-Mailer: Apple Mail (2.3112) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Mar 2016 10:59:03 -0000 > On 04 Mar 2016, at 10:16, Steven Hartland = wrote: >=20 > Its very rare but we've also seen this type of behaviour from a = failing Intel CPU. There was no other indication the CPU had an issue, = which one might expect, so just wanted to make you aware of the = possibility. >=20 > That said the most common cause of this we've seen, when its not a = common disk or disks, is a bad backplane or cabling to the backplane. Now I=E2=80=99m really curious! How did you determine that it was the CPU? And what kind of issue was it = causing? Noise in the power rails? Interference? Borja. From owner-freebsd-scsi@freebsd.org Fri Mar 4 11:07:52 2016 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7ECAC9DAD4D for ; Fri, 4 Mar 2016 11:07:52 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wm0-x22e.google.com (mail-wm0-x22e.google.com [IPv6:2a00:1450:400c:c09::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1C759C64 for ; Fri, 4 Mar 2016 11:07:51 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wm0-x22e.google.com with SMTP id l68so29686062wml.0 for ; Fri, 04 Mar 2016 03:07:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding; bh=mtpYvUxUDWlxy8PPWW66mjbTWTSxQx4CJySEQiRv8rM=; b=YU/cnMZaEpcQOp2YN4973qJxcwyePML2w3LmbxmeSe9OhTKMpukdhPM4gYBAI5GN3K OYpoDM9sJAbl5JIwTGU8nfrqFxex8nAnU6MaWlzGB2DByppKbkJa+SOElwxW4Fs+rUQy zle7ZLxmeNiHjQeJzrLQKGKDsY/UQF+XrhpqIbtK80cV0icPq1Dpty75QtOQ5Z2ar0FL SXME2m5Cyppk0T9SBsK7x+Cez92q7XmFJvS29b2UCvG1ToUegycmQUL9juA43+RF5i8j k3jLTmjYO2nzSJuxpRasJkwBFGOfUlJAiJErrUQKH7KBpc86P4gnr//gY9dp6qLXB41C hSHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=mtpYvUxUDWlxy8PPWW66mjbTWTSxQx4CJySEQiRv8rM=; b=kDdifVn7WhM1SQPYmQw3JRzeFSyHzeBy2xa2sFc5GwCNJFHm3/t3COAMZrLuq3eNVP H/NEejUTQ0TQ8lMqzwP0H8dW84ED+X9sdw+FASyI6AoGuuGXRG8gGXQeRFSVUrgCxjrG chkGEBj0o2eK9Rv0NR1BawLJ2UOl0lmkpfCihP1T0H8P3jzksewNc3je4umgLpTPqe0c V7UsZIa9IDzDDt/gNbbj7dPYaJnvV915Q/U+BvyW+KApVcztXwI1MMgbD1cn0e602vWS vrHRQE7e+oCF56OGAA++pG0xXqfJKIbDK4EjZJaMaVBMEVb7M20Wva2xZyw+ytAHVkIo 4lcg== X-Gm-Message-State: AD7BkJLZjbWIlfgxQGjvRclNZwzkasUz3JKZCwpvrReYtDseB+/JIdq2m0CNRJHrnOQfuW4P X-Received: by 10.194.120.229 with SMTP id lf5mr9084654wjb.151.1457089670145; Fri, 04 Mar 2016 03:07:50 -0800 (PST) Received: from [10.10.1.58] (liv3d.labs.multiplay.co.uk. [82.69.141.171]) by smtp.gmail.com with ESMTPSA id e19sm2828407wmd.1.2016.03.04.03.07.49 (version=TLSv1/SSLv3 cipher=OTHER); Fri, 04 Mar 2016 03:07:49 -0800 (PST) Subject: Re: mpr(4) SAS3008 Repeated Crashing To: Borja Marcos References: <56D5FDB8.8040402@freebsd.org> <56D612FA.6090909@multiplay.co.uk> <56D805FD.50500@multiplay.co.uk> <56D95266.301@multiplay.co.uk> Cc: Scott Long , FreeBSD-scsi From: Steven Hartland Message-ID: <56D96C84.7070507@multiplay.co.uk> Date: Fri, 4 Mar 2016 11:07:48 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.6.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Mar 2016 11:07:52 -0000 On 04/03/2016 10:58, Borja Marcos wrote: >> On 04 Mar 2016, at 10:16, Steven Hartland wrote: >> >> Its very rare but we've also seen this type of behaviour from a failing Intel CPU. There was no other indication the CPU had an issue, which one might expect, so just wanted to make you aware of the possibility. >> >> That said the most common cause of this we've seen, when its not a common disk or disks, is a bad backplane or cabling to the backplane. > Now I’m really curious! > > How did you determine that it was the CPU? And what kind of issue was it causing? Noise in the power rails? Interference? After a month or so of fixing mfi so it recovered from all bad events and prevented all the various kernel panics, the machine stayed running long enough to log an MCA which pointed to a failing CPU cache. We we're lucky it was CPU #2 so we disabled all cores for said CPU in /boot/loader.conf and all the issues disappeared. We replaced the CPU and no more issues. We we're in the same situation as you, two machines identical configs, one which was constantly panicing in mfi the other was rock solid. Regards Steve