Date: Tue, 17 Jul 2012 10:46:16 -0600 From: "Kenneth D. Merry" <ken@freebsd.org> To: Trent Nelson <trent@snakebite.org> Cc: "freebsd-scsi@freebsd.org" <freebsd-scsi@freebsd.org> Subject: Re: CAM Target Layer and dev/isp Message-ID: <20120717164616.GA10673@nargothrond.kdm.org> In-Reply-To: <CC2A50D7.3264B%trent@snakebite.org> References: <CC2A50D7.3264B%trent@snakebite.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Jul 16, 2012 at 20:14:31 -0700, Trent Nelson wrote:
> Hi Ken,
>
> So, first off, nice work on CTL! I accidentally stumbled onto this little
> gem yesterday and was shocked I hadn't heard of it before. (Although it
> seems I'm not alone, even Google only seems to know about your original
> mailing list posts -- `+freebsd +ctladm` returns surprisingly little.)
Glad you like it!
> (Somewhat related question before I get into my main issue: `ctladm`
> should remove the need for share/examples/scsi_target, right? The CAM
> Target Layer stuff encompasses all the original functionality exposed in
> scsi_target?)
Well, scsi_target is still useful as an example of how to create a userland
target application.
> I'm not sure how to expose LUNs via my isp devices. Here's what I've got
> so far:
>
> [root@s24/ttypts/1(../misc/isp/bin)#] for i in {0..3}; do; ./isp_getrole
> /dev/isp$i; done
> /dev/isp0 chan 0: role target
> /dev/isp1 chan 0: role target
> /dev/isp2 chan 0: role target
> /dev/isp3 chan 0: role target
Looks good. What model Qlogic board is it? I know the driver works pretty
well in target mode with 4Gb and 8Gb boards, but I don't know how well 2Gb
and older boards work.
> [root@s24/ttypts/1(../misc/isp/bin)#] ctladm devlist -v
> LUN Backend Size (Blocks) BS Serial Number Device ID
> 0 ramdisk 2097152 512 MYSERIAL 0 MYDEVID 0
> lun_type=0
That looks fine. Note that it is a fake ramdisk that is only backed by 1MB
of memory no matter how large the reported size is. So you can make it as
large as you want.
> [root@s24/ttypts/1(../misc/isp/bin)#] ctladm port -l
> Port Online Type Name pp vp WWNN WWPN
>
> 0 YES IOCTL CTL ioctl 0 0 0 0
>
> 1 YES INTERNAL ctl2cam 0 0 0x5000000357375b00
> 0x5000000357375b02
> 2 YES INTERNAL CTL internal 0 0 0 0
>
> 3 YES FC isp0 0 0 0x200000e08b146178
> 0x210000e08b146178
> 4 YES FC isp1 1 0 0x400000007f000009
> 0x400000007f000009
> 5 YES FC isp2 2 0 0x200000e08b102f5b
> 0x210000e08b102f5b
> 6 YES FC isp3 3 0 0x200000e08b302f5b
> 0x210100e08b302f5b
All of the ports are online, so that looks good.
> It's not obvious from the docs how to export CTL LUNs through my isp
> devices. I tried passing -D /dev/isp[n] to `ctladm create`, but that
> returns: 'ctladm: cctl_create_lun: error issuing CTL_LUN_REQ ioctl:
> Inappropriate ioctl for device'.
The block backend only works on block devices or files. e.g.:
ctladm create -b block -o file=/path/to/my/file
ctladm create -b block -o file=/dev/da5
If you use a block device (like a zvol) as the backing store, you'll want
to disable sending cache syncs to the disk, since that will trigger a GEOM
assertion.
ctladm realsync off
(Do that before putting the ports online.)
> Sample output from one of the switches on my fabric (s24's /dev/isp1 is
> plugged into port 8 of sf1):
>
> sf1:admin> switchshow
> switchName: sf1
> switchType: 17.2
> switchState: Online
> switchMode: Native
> switchRole: Principal
> switchDomain: 1
> switchId: fffc01
> switchWwn: 10:00:00:60:69:5a:1a:40
> switchBeacon: OFF
> Zoning: ON (cfg_2012_07_16)
> port 0: id N2 Online F-Port 50:06:0b:00:00:13:18:72
> port 1: id N2 No_Light
> port 2: id N2 Online E-Port 10:00:00:60:69:5a:09:c0 "sf2"
> (downstream)
> port 3: id N2 No_Light
> port 4: id N2 No_Light
> port 5: id N2 Online F-Port 21:01:00:e0:8b:ab:cf:be
> port 6: id N2 Online F-Port 21:01:00:e0:8b:a6:98:ca
> port 7: id N2 Online F-Port 21:01:00:e0:8b:30:2f:5b
> port 8: id N1 Online L-Port 1 public
Looks like it is in loop mode. Can your switch make a loop mode device
visible on another port? What are you using for an initiator? Does it
work if you connect the initiator directly to the FreeBSD target?
> Ideally I'd like to be able to use CTL to export multiple ZFS zvols as
> separate targets (i.e. all with unique WWNN/WWPNs), such that, from the
> fabric's point of view, the port would look like just another FC-AL JBOD,
> like, say, port 11:
[ ... ]
> That'll allow me to zone the zvols just as if they were actual JBOD disks.
> Which will be awesome.
>
> Is that possible?
CTL will just create multiple LUNs, not multiple targets. Each LUN will
show up on all of the ports.
If your switch has NP-IV support, you can also try creating multiple
virtual ports with the isp(4) driver if you set the hint.isp.0.vports=N
loader tunable, where N is the number of virtual ports. I haven't tried
that in several years, though, and Matt Jacob has indicated it needs more
testing.
As for using zvols, the code that is in FreeBSD right now will lead to very
slow performance with zvols. Justin Gibbs and Will Andrews gave a talk at
BSDCan that explained their work to eliminate COW (Copy On Write) faults
for files used as block devices and zvols. Their slides are here:
http://www.bsdcan.org/2012/schedule/events/316.en.html
And the talk itself is here:
http://www.youtube.com/watch?v=LtY3vpX-cdM
It's fine to use zvols now, but you may want to wait until their code does
into FreeBSD/head at least to use zvols for anything that requires
reasonable performance.
Until their code goes in the tree, you'll probably get somewhat faster
performance by using files on top of ZFS, or on top of UFS. (UFS will be
much faster at the moment, but you don't get the software RAID
functionality of ZFS.)
The first write pass through on a zvol or a ZFS file will go very quickly,
but subsequent writes will be pretty slow, especially if they are not on
exact ZFS record boundaries.
Also, Matt Jacob and I are chasing down a possible data corruption bug.
We don't know exactly where it is, and it might not be in the code that
is in FreeBSD right now. The point is, run some data integrity tests
before using this in production.
> Also, the HA stuff sounds bad-ass. What's the best way to stay up to date
> with CTL development? Watch commits to sys/cam/ctl? (There's no separate
> list or anything for this stuff, right?)
That's pretty much the best way to keep up with it, there's not a separate
mailing list.
I don't think I'm going to have time to do anything with the HA hooks in
the near future. Hopefully other folks will be interested and do some
development in that area. It would be nice to have a fully HA block stack,
but that will take a lot of effort.
Ken
--
Kenneth Merry
ken@FreeBSD.ORG
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20120717164616.GA10673>