Date: Sat, 28 Feb 2015 11:48:01 -0500 From: Dan Langille <dan@langille.org> To: "Kenneth D. Merry" <ken@freebsd.org> Cc: current@freebsd.org, scsi@freebsd.org Subject: Re: sa(4) driver changes available for test Message-ID: <329C1EFF-FB33-491F-B179-8831039B4698@langille.org> In-Reply-To: <20150219001347.GA57416@mithlond.kdm.org> References: <20150214003232.GA63990@mithlond.kdm.org> <20150219001347.GA57416@mithlond.kdm.org>
next in thread | previous in thread | raw e-mail | index | archive | help
> On Feb 18, 2015, at 7:13 PM, Kenneth D. Merry <ken@freebsd.org> wrote: >=20 >=20 > I have updated the patches. >=20 > I have removed the XPT_DEV_ADVINFO changes from the patches to head, = since > I committed those separately. >=20 > I have (hopefully) fixed the build for the stable/10 patches by MFCing > dependencies. (One of them mav did for me, thanks!) >=20 > Rough draft commit message: >=20 > http://people.freebsd.org/~ken/sa_changes_commitmsg.20150218.1.txt >=20 > The patches against FreeBSD/head as of SVN revision 278975: >=20 > http://people.freebsd.org/~ken/sa_changes.20150218.1.txt >=20 > And (untested) patches against FreeBSD stable/10 as of SVN revision = 278974: >=20 > http://people.freebsd.org/~ken/sa_changes.stable_10.20150218.1.txt Not sure what I've done wrong here. I've applied your patches and run make buildworld against: [root@cuppy:/usr/src] # svn info Path: . Working Copy Root Path: /usr/src URL: https://svn0.us-west.freebsd.org/base/stable/10 Relative URL: ^/stable/10 Repository Root: https://svn0.us-west.freebsd.org/base Repository UUID: ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f Revision: 278721 Node Kind: directory Schedule: normal Last Changed Author: ngie Last Changed Rev: 278721 Last Changed Date: 2015-02-13 21:46:22 +0000 (Fri, 13 Feb 2015) But I get: rm -f .depend GPATH GRTAGS GSYMS GTAGS =3D=3D=3D> lib/libmp (cleandir) rm -f Version.map libmp.3.gz libmp.3.cat.gz rm -f a.out mpasbn.o mpasbn.o.tmp=20 rm -f mpasbn.po mpasbn.po.tmp rm -f mpasbn.So mpasbn.so mpasbn.So.tmp rm -f libmp.so rm -f libmp.a libmp_p.a libmp.so.7 rm -f Version.map rm -f .depend GPATH GRTAGS GSYMS GTAGS =3D=3D=3D> lib/libmt (cleandir) cd: /usr/src/lib/libmt: No such file or directory *** Error code 2 Stop. make[3]: stopped in /usr/src/lib *** Error code 1 Stop. make[2]: stopped in /usr/src *** Error code 1 Stop. make[1]: stopped in /usr/src *** Error code 1 Stop. make: stopped in /usr/src >=20 > Thanks, >=20 > Ken >=20 > On Fri, Feb 13, 2015 at 17:32:32 -0700, Kenneth D. Merry wrote: >>=20 >> I have a fairly large set of changes to the sa(4) driver and mt(1) = driver >> that I'm planning to commit in the near future. >>=20 >> A description of the changes is here and below in this message. >>=20 >> If you have tape hardware and the inclination, I'd appreciate testing = and >> feedback. >>=20 >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> Rough draft commit message: >>=20 >> http://people.freebsd.org/~ken/sa_changes_commitmsg.20150213.3.txt >>=20 >> The patches against FreeBSD/head as of SVN revision 278706: >>=20 >> http://people.freebsd.org/~ken/sa_changes.20150213.3.txt >>=20 >> And (untested) patches against FreeBSD stable/10 as of SVN revision = 278721. >>=20 >> http://people.freebsd.org/~ken/sa_changes.stable_10.20150213.3.txt >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >>=20 >> The intent is to get the tape infrastructure more up to date, so we = can >> support LTFS and more modern tape drives: >>=20 >> http://www.ibm.com/systems/storage/tape/ltfs/ >>=20 >> I have ported IBM's LTFS Single Drive Edition to FreeBSD. The port = depends >> on the patches linked above. It isn't fully cleaned up and ready for >> redistribution. If you're interested, though, let me know and I'll = tell >> you when it is ready to go out. You need an IBM LTO-5, LTO-6, TS1140 = or >> TS1150 tape drive. HP drives aren't supported by IBM's LTFS, and = older >> drives don't have the necessary features to support LTFS. >>=20 >> The commit message below outlines most of the changes. >>=20 >> A few comments: >>=20 >> 1. I'm planning to commit the XPT_DEV_ADVINFO changes separately. >>=20 >> 2. The XML output is similar to what GEOM and CTL do. It would be = nice to >> figure out how to put a standard schema on it so that standard = tools >> could read it. I don't know how feasible that is, since I haven't >> time to dig into it. If anyone has suggestions on whether that is >> feasible or advisable, I'd appreciate feedback. >>=20 >> 3. I have tested with a reasonable amount of tape hardware (see below = for a >> list), but more testing and feedback would be good. >>=20 >> 4. Standard 'mt status' output looks like this: >>=20 >> # mt -f /dev/nsa3 status -v >> Drive: sa3: <IBM ULTRIUM-HH6 E4J1> Serial Number: 101500520A >> --------------------------------- >> Mode Density Blocksize bpi Compression >> Current: 0x5a:LTO-6 variable 384607 enabled (0xff) >> --------------------------------- >> Current Driver State: at rest. >> --------------------------------- >> Partition: 0 Calc File Number: 0 Calc Record Number: 0 >> Residual: 0 Reported File Number: 0 Reported Record Number: 0 >> Flags: BOP >>=20 >> 5. 'mt status -v' looks like this: >>=20 >> # mt -f /dev/nsa3 status -v >> Drive: sa3: <IBM ULTRIUM-HH6 E4J1> Serial Number: 101500520A >> --------------------------------- >> Mode Density Blocksize bpi Compression >> Current: 0x5a:LTO-6 variable 384607 enabled (0xff) >> --------------------------------- >> Current Driver State: at rest. >> --------------------------------- >> Partition: 0 Calc File Number: 0 Calc Record Number: 0 >> Residual: 0 Reported File Number: 0 Reported Record Number: 0 >> Flags: BOP >> --------------------------------- >> Tape I/O parameters: >> Maximum I/O size allowed by driver and controller (maxio): 1081344 = bytes >> Maximum I/O size reported by controller (cpi_maxio): 5197824 bytes >> Maximum block size supported by tape drive and media (max_blk): = 8388608 bytes >> Minimum block size supported by tape drive and media (min_blk): 1 = bytes >> Block granularity supported by tape drive and media (blk_gran): 0 = bytes >> Maximum possible I/O size (max_effective_iosize): 1081344 bytes >>=20 >> 6. Existing applications should work without changes. If not, please = let >> me know. Hopefully they will move over time to the new interfaces. >>=20 >> 7. There are lots of additional features that could be added later. >> Append-only support, encryption, more log pages, etc. >>=20 >> 8. I have SCSI READ ATTRIBUTE changes for camcontrol(8) that will go = in >> separately. These changes allow displaying the contents of the MAM >> (Medium Auxiliary Memory) chips on LTO, TS and other modern tape = drives. >> These are good, and a future possible direction is adding = attributes=20 >> to the status XML from the sa(4) driver. >>=20 >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> Significant upgrades to sa(4) and mt(1). >>=20 >> The primary focus of these changes is to modernize FreeBSD's >> tape infrastructure so that we can take advantage of some of the >> features of modern tape drives and allow support for LTFS. >>=20 >> Significant changes and new features include: >>=20 >> o sa(4) driver status and parameter information is now exported via = an >> XML structure. This will allow for changes and improvements later >> on that will not break userland applications. The old MTIOCGET >> status ioctl remains, so applications using the existing interface >> will not break. >>=20 >> o 'mt status' now reports drive-reported tape position information >> as well as the previously available calculated tape position >> information. These numbers will be different at times, because >> the drive-reported block numbers are relative to BOP (Beginning >> of Partition), but the block numbers calculated previously via >> sa(4) (and still provided) are relative to the last filemark. >> Both numbers are now provided. 'mt status' now also shows the >> drive INQUIRY information, serial number and any position flags >> (BOP, EOT, etc.) provided with the tape position information. >> 'mt status -v' adds information on the maximum possible I/O size, >> and the underlying values used to calculate it. >>=20 >> o The extra sa(4) /dev entries (/dev/saN.[0-3]) have been removed. >>=20 >> The extra devices were originally added as place holders for >> density-specific device nodes. Some OSes (NetBSD, NetApp's OnTap >> and Solaris) have had device nodes that, when you write to them, >> will automatically select a given density for particular tape = drives. >>=20 >> This is a convenient way of switching densities, but it was never >> implemented in FreeBSD. Only the device nodes were there, and that >> sometimes confused users. >>=20 >> For modern tape devices, the density is generally not selectable >> (e.g. with LTO) or defaults to the highest availble density when >> the tape is rewritten from BOT (e.g. TS11X0). So, for most users, >> density selection won't be necessary. If they do need to select >> the density, it is easy enough to use 'mt density' to change it. >>=20 >> o Protection information is now supported. This is either a >> Reed-Solomon CRC or CRC32 that is included at the end of each block >> read and written. On write, the tape drive verifies the CRC, and >> on read, the tape drive provides a CRC for the userland application >> to verify. >>=20 >> o New, extensible tape driver parameter get/set interface. >>=20 >> o Density reporting information. For drives that support it, >> 'mt getdensity' will show detailed information on what formats the >> tape drive supports, and what formats the tape drive supports. >>=20 >> o Some mt(1) functionality moved into a new mt(3) library so that >> external applications can reuse the code. >>=20 >> o The new mt(3) library includes helper routines to aid in parsing >> the XML output of the sa(4) driver, and build a tree of driver >> metadata. >>=20 >> o Support for the MTLOAD (load a tape in the drive) and MTWEOFI >> (write filemark immediate) ioctls needed by IBM's LTFS >> implementation. >>=20 >> o Improve device departure behavior for the sa(4) driver. The = previous >> implementation led to hangs when the device was open. >>=20 >> o This has been tested on the following types of drives: >> IBM TS1150 >> IBM TS1140 >> IBM LTO-6 >> IBM LTO-5 >> HP LTO-2 >> Seagate DDS-4 >> Quantum DLT-4000 >> Exabyte 8505 >> Sony DDS-2 >>=20 >> contrib/groff/tmac/doc-syms, >> share/mk/bsd.libnames.mk, >> lib/Makefile, >> Add libmt. >>=20 >> lib/libmt/Makefile, >> lib/libmt/mt.3, >> lib/libmt/mtlib.c, >> lib/libmt/mtlib.h, >> New mt(3) library that contains functions moved from mt(1) and >> new functions needed to interact with the updated sa(4) driver. >>=20 >> This includes XML parser helper functions that application = writers >> can use when writing code to query tape parameters. >>=20 >> rescue/rescue/Makefile: >> Add -lmt to CRUNCH_LIBS. >>=20 >> sys/cam/cam_ccb.h >> Add a new flag value for the XPT_DEV_ADVINFO CCB, = CDAI_FLAG_NONE. >>=20 >> sbin/camcontrol/camcontrol.c, >> sys/cam/scsi/scsi_da.c, >> sys/cam/scsi/scsi_enc_ses.c, >> sys/dev/mps/mps_sas.c: >> Make sure the flags for the XPT_DEV_ADVINFO CCB are set = correctly. >> This prevents unintended attempts to set advanced information >> values when XPT_DEV_ADVINFO CCBs are not pre-zeroed. >>=20 >> src/share/man/man4/mtio.4 >> Clarify this man page a bit, and since it contains what is >> essentially the mtio.h header file, add new ioctls and structure >> definitions from mtio.h. >>=20 >> src/share/man/man4/sa.4 >> Update BUGS and maintainer section. >>=20 >> sys/cam/scsi/scsi_all.c, >> sys/cam/scsi/scsi_all.h: >> Add SCSI SECURITY PROTOCOL IN/OUT CDB definitions and CDB = building >> functions. >>=20 >> sys/cam/scsi/scsi_sa.c >> sys/cam/scsi/scsi_sa.h >> Many tape driver changes, largely outlined above. >>=20 >> Increase the sa(4) driver read/write timeout from 4 to 32 >> minutes. This is based on the recommended values for IBM LTO >> 5/6 drives. This may also avoid timeouts for other tape >> hardware that can take a long time to do retries and error >> recovery. Longer term, a better way to handle this is to ask >> the drive for recommended timeout values using the REPORT >> SUPPORTED OPCODES command. Modern IBM and Oracle tape drives >> at least support that command, and it would allow for more >> accurate timeout values. >>=20 >> Add XML status generation. This is done with a series of >> macros to eliminate as much duplicate code as possible. The >> new XML-based status values are reported through the new >> MTIOCEXTGET ioctl. >>=20 >> Add XML driver parameter reporting, using the new MTIOCPARAMGET >> ioctl. >>=20 >> Add a new driver parameter setting interface, using the new >> MTIOCPARAMSET and MTIOCSETLIST ioctls. >>=20 >> Add a new MTIOCRBLIM ioctl to get block limits information. >>=20 >> Add CCB/CDB building routines scsi_locate_16, scsi_locate_10, >> and scsi_read_position_10(). >>=20 >> scsi_locate_10 implements the LOCATE command, as does the >> existing scsi_set_position() command. It just supports >> additional arguments and features. If/when we figure out a >> good way to provide backward compatibility for older >> applications using the old function API, we can just revamp >> scsi_set_position(). The same goes for >> scsi_read_position_10() and the existing scsi_read_position() >> function. >>=20 >> Revamp sasetpos() to take the new mtlocate structure as an >> argument. It now will use either scsi_locate_10() or >> scsi_locate_16(), depending upon the arguments the user >> supplies. As before, once we change position we don't have a >> clear idea of what the current logical position of the tape >> drive is. >>=20 >> For tape drives that support long form position data, we >> read the current position and store that for later reporting >> after changing the position. This should help applications >> like Bacula speed tape access under FreeBSD once they are >> modified to support the new ioctls. >>=20 >> Add a new quirk, SA_QUIRK_NO_LONG_POS, that is set for all >> drives that report SCSI-2 or older, as well as drives that >> report an Illegal Request type error for READ POSITION with >> the long format. So we should automatically detect drives >> that don't support the long form and stop asking for it after >> an initial try. >>=20 >> Add a partition number to the sa(4) softc. >>=20 >> Improve device departure handling. The previous implementation >> led to hangs when the device was open. >>=20 >> If an application had the sa(4) driver open, and attempted to >> close it after it went away, the cam_periph_release() call in >> saclose() would cause the periph to get destroyed because that >> was the last reference to it. Because destroy_dev() was >> called from the sa(4) driver's cleanup routine (sacleanup()), >> and would block waiting for the close to happen, a deadlock >> would result. >>=20 >> So instead of calling destroy_dev() from the cleanup routine, >> call destroy_dev_sched_cb() from saoninvalidate() and wait for >> the callback. >>=20 >> Acquire a reference for devfs in saregister(), and release it >> in the new sadevgonecb() routine when all devfs devices for=09 >> the particular sa(4) driver instance are gone. >>=20 >> Add a new function, sasetupdev(), to centralize setting >> per-instance devfs device parameters instead of repeating the >> code in saregister(). >>=20 >> Add an open count to the softc, so we know how many >> peripheral driver references are a result of open >> sessions. >>=20 >> Add the D_TRACKCLOSE flag to the cdevsw flags so >> that we get a 1:1 mapping of open to close calls >> instead of a N:1 mapping. >>=20 >> This should be a no-op for everything except the >> control device, since we don't allow more than one >> open on non-control devices. >>=20 >> However, since we do allow multiple opens on the >> control device, the combination of the open count >> and the D_TRACKCLOSE flag should result in an >> accurate peripheral driver reference count, and an >> accurate open count. >>=20 >> The accurate open count allows us to release all >> peripheral driver references that are the result >> of open contexts once we get the callback from devfs. >>=20 >> sys/sys/mtio.h: >> Add a number of new mt(4) ioctls and the requisite data >> structures. None of the existing interfaces been removed >> or changed. >>=20 >> This includes definitions for the following new ioctls: >>=20 >> MTIOCRBLIM /* get block limits */ >> MTIOCEXTLOCATE /* seek to position */ >> MTIOCEXTGET /* get tape status */ >> MTIOCPARAMGET /* get tape params */ >> MTIOCPARAMSET /* set tape params */ >> MTIOCSETLIST /* set N params */ >>=20 >> usr.bin/mt/Makefile: >> mt(1) now depends on libmt, libsbuf and libbsdxml. >>=20 >> usr.bin/mt/mt.1: >> Document new mt(1) features and subcommands. >>=20 >> usr.bin/mt/mt.c: >> Implement support for mt(1) subcommands that need to >> use getopt(3) for their arguments. >>=20 >> Implement a new 'mt status' command to replace the old >> 'mt status' command. The old status command has been >> renamed 'ostatus'. >>=20 >> The new status function uses the MTIOCEXTGET ioctl, and >> therefore parses the XML data to determine drive status. >> The -x argument to 'mt status' allows the user to dump out >> the raw XML reported by the kernel. >>=20 >> The new status display is mostly the same as the old status >> display, except that it doesn't print the redundant density >> mode information, and it does print the current partition >> number and position flags. >>=20 >> Add a new command, 'mt locate', that will supersede the >> old 'mt setspos' and 'mt sethpos' commands. 'mt locate' >> implements all of the functionality of the MTIOCEXTLOCATE >> ioctl, and allows the user to change the logical position >> of the tape drive in a number of ways. (Partition, >> block number, file number, set mark number, end of data.) >> The immediate bit and the explicit address bits are >> implemented, but not documented in the man page. >>=20 >> Add a new 'mt weofi' command to use the new MTWEOFI ioctl. >> This allows the user to ask the drive to write a filemark >> without waiting around for the operation to complete. >>=20 >> Add a new 'mt getdensity' command that gets the XML-based >> tape drive density report from the sa(4) driver and displays >> it. This uses the SCSI REPORT DENSITY SUPPORT command >> to get comprehensive information from the tape drive about >> what formats it is able to read and write. >>=20 >> Add a new 'mt protect' command that allows getting and setting >> tape drive protection information. The protection information >> is a CRC tacked on to the end of every read/write from and to >> the tape drive. >>=20 >> Sponsored by: Spectra Logic >> MFC after: 1 month >>=20 >> Thanks, >>=20 >> Ken >> --=20 >> Kenneth Merry >> ken@FreeBSD.ORG >=20 > --=20 > Kenneth Merry > ken@FreeBSD.ORG > _______________________________________________ > freebsd-scsi@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-scsi > To unsubscribe, send any mail to = "freebsd-scsi-unsubscribe@freebsd.org" =E2=80=94=20 Dan Langille http://langille.org/
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?329C1EFF-FB33-491F-B179-8831039B4698>