From owner-freebsd-scsi@FreeBSD.ORG Tue Jul 17 16:46:17 2012 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id DA45B1065707 for ; Tue, 17 Jul 2012 16:46:17 +0000 (UTC) (envelope-from ken@kdm.org) Received: from nargothrond.kdm.org (nargothrond.kdm.org [70.56.43.81]) by mx1.freebsd.org (Postfix) with ESMTP id 9B4798FC08 for ; Tue, 17 Jul 2012 16:46:17 +0000 (UTC) Received: from nargothrond.kdm.org (localhost [127.0.0.1]) by nargothrond.kdm.org (8.14.2/8.14.2) with ESMTP id q6HGkHEZ017053; Tue, 17 Jul 2012 10:46:17 -0600 (MDT) (envelope-from ken@nargothrond.kdm.org) Received: (from ken@localhost) by nargothrond.kdm.org (8.14.2/8.14.2/Submit) id q6HGkG7K017052; Tue, 17 Jul 2012 10:46:16 -0600 (MDT) (envelope-from ken) Date: Tue, 17 Jul 2012 10:46:16 -0600 From: "Kenneth D. Merry" To: Trent Nelson Message-ID: <20120717164616.GA10673@nargothrond.kdm.org> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2i Cc: "freebsd-scsi@freebsd.org" Subject: Re: CAM Target Layer and dev/isp X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jul 2012 16:46:17 -0000 On Mon, Jul 16, 2012 at 20:14:31 -0700, Trent Nelson wrote: > Hi Ken, > > So, first off, nice work on CTL! I accidentally stumbled onto this little > gem yesterday and was shocked I hadn't heard of it before. (Although it > seems I'm not alone, even Google only seems to know about your original > mailing list posts -- `+freebsd +ctladm` returns surprisingly little.) Glad you like it! > (Somewhat related question before I get into my main issue: `ctladm` > should remove the need for share/examples/scsi_target, right? The CAM > Target Layer stuff encompasses all the original functionality exposed in > scsi_target?) Well, scsi_target is still useful as an example of how to create a userland target application. > I'm not sure how to expose LUNs via my isp devices. Here's what I've got > so far: > > [root@s24/ttypts/1(../misc/isp/bin)#] for i in {0..3}; do; ./isp_getrole > /dev/isp$i; done > /dev/isp0 chan 0: role target > /dev/isp1 chan 0: role target > /dev/isp2 chan 0: role target > /dev/isp3 chan 0: role target Looks good. What model Qlogic board is it? I know the driver works pretty well in target mode with 4Gb and 8Gb boards, but I don't know how well 2Gb and older boards work. > [root@s24/ttypts/1(../misc/isp/bin)#] ctladm devlist -v > LUN Backend Size (Blocks) BS Serial Number Device ID > 0 ramdisk 2097152 512 MYSERIAL 0 MYDEVID 0 > lun_type=0 That looks fine. Note that it is a fake ramdisk that is only backed by 1MB of memory no matter how large the reported size is. So you can make it as large as you want. > [root@s24/ttypts/1(../misc/isp/bin)#] ctladm port -l > Port Online Type Name pp vp WWNN WWPN > > 0 YES IOCTL CTL ioctl 0 0 0 0 > > 1 YES INTERNAL ctl2cam 0 0 0x5000000357375b00 > 0x5000000357375b02 > 2 YES INTERNAL CTL internal 0 0 0 0 > > 3 YES FC isp0 0 0 0x200000e08b146178 > 0x210000e08b146178 > 4 YES FC isp1 1 0 0x400000007f000009 > 0x400000007f000009 > 5 YES FC isp2 2 0 0x200000e08b102f5b > 0x210000e08b102f5b > 6 YES FC isp3 3 0 0x200000e08b302f5b > 0x210100e08b302f5b All of the ports are online, so that looks good. > It's not obvious from the docs how to export CTL LUNs through my isp > devices. I tried passing -D /dev/isp[n] to `ctladm create`, but that > returns: 'ctladm: cctl_create_lun: error issuing CTL_LUN_REQ ioctl: > Inappropriate ioctl for device'. The block backend only works on block devices or files. e.g.: ctladm create -b block -o file=/path/to/my/file ctladm create -b block -o file=/dev/da5 If you use a block device (like a zvol) as the backing store, you'll want to disable sending cache syncs to the disk, since that will trigger a GEOM assertion. ctladm realsync off (Do that before putting the ports online.) > Sample output from one of the switches on my fabric (s24's /dev/isp1 is > plugged into port 8 of sf1): > > sf1:admin> switchshow > switchName: sf1 > switchType: 17.2 > switchState: Online > switchMode: Native > switchRole: Principal > switchDomain: 1 > switchId: fffc01 > switchWwn: 10:00:00:60:69:5a:1a:40 > switchBeacon: OFF > Zoning: ON (cfg_2012_07_16) > port 0: id N2 Online F-Port 50:06:0b:00:00:13:18:72 > port 1: id N2 No_Light > port 2: id N2 Online E-Port 10:00:00:60:69:5a:09:c0 "sf2" > (downstream) > port 3: id N2 No_Light > port 4: id N2 No_Light > port 5: id N2 Online F-Port 21:01:00:e0:8b:ab:cf:be > port 6: id N2 Online F-Port 21:01:00:e0:8b:a6:98:ca > port 7: id N2 Online F-Port 21:01:00:e0:8b:30:2f:5b > port 8: id N1 Online L-Port 1 public Looks like it is in loop mode. Can your switch make a loop mode device visible on another port? What are you using for an initiator? Does it work if you connect the initiator directly to the FreeBSD target? > Ideally I'd like to be able to use CTL to export multiple ZFS zvols as > separate targets (i.e. all with unique WWNN/WWPNs), such that, from the > fabric's point of view, the port would look like just another FC-AL JBOD, > like, say, port 11: [ ... ] > That'll allow me to zone the zvols just as if they were actual JBOD disks. > Which will be awesome. > > Is that possible? CTL will just create multiple LUNs, not multiple targets. Each LUN will show up on all of the ports. If your switch has NP-IV support, you can also try creating multiple virtual ports with the isp(4) driver if you set the hint.isp.0.vports=N loader tunable, where N is the number of virtual ports. I haven't tried that in several years, though, and Matt Jacob has indicated it needs more testing. As for using zvols, the code that is in FreeBSD right now will lead to very slow performance with zvols. Justin Gibbs and Will Andrews gave a talk at BSDCan that explained their work to eliminate COW (Copy On Write) faults for files used as block devices and zvols. Their slides are here: http://www.bsdcan.org/2012/schedule/events/316.en.html And the talk itself is here: http://www.youtube.com/watch?v=LtY3vpX-cdM It's fine to use zvols now, but you may want to wait until their code does into FreeBSD/head at least to use zvols for anything that requires reasonable performance. Until their code goes in the tree, you'll probably get somewhat faster performance by using files on top of ZFS, or on top of UFS. (UFS will be much faster at the moment, but you don't get the software RAID functionality of ZFS.) The first write pass through on a zvol or a ZFS file will go very quickly, but subsequent writes will be pretty slow, especially if they are not on exact ZFS record boundaries. Also, Matt Jacob and I are chasing down a possible data corruption bug. We don't know exactly where it is, and it might not be in the code that is in FreeBSD right now. The point is, run some data integrity tests before using this in production. > Also, the HA stuff sounds bad-ass. What's the best way to stay up to date > with CTL development? Watch commits to sys/cam/ctl? (There's no separate > list or anything for this stuff, right?) That's pretty much the best way to keep up with it, there's not a separate mailing list. I don't think I'm going to have time to do anything with the HA hooks in the near future. Hopefully other folks will be interested and do some development in that area. It would be nice to have a fully HA block stack, but that will take a lot of effort. Ken -- Kenneth Merry ken@FreeBSD.ORG