From owner-freebsd-scsi@FreeBSD.ORG Wed Jul 18 13:13:44 2012 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id EDBC1106564A; Wed, 18 Jul 2012 13:13:43 +0000 (UTC) (envelope-from trent@snakebite.org) Received: from EXHUB04.exchhosting.com (exchla3.liveoffice.com [64.70.67.188]) by mx1.freebsd.org (Postfix) with ESMTP id C4EB98FC12; Wed, 18 Jul 2012 13:13:43 +0000 (UTC) Received: from EXMBX10.exchhosting.com ([fe80::9c37:32f6:a508:a44f]) by EXHUB04.exchhosting.com ([fe80::e08b:16b6:14a0:73b0%12]) with mapi; Wed, 18 Jul 2012 06:13:36 -0700 From: Trent Nelson To: "Kenneth D. Merry" Date: Wed, 18 Jul 2012 06:13:33 -0700 Thread-Topic: CAM Target Layer and dev/isp Thread-Index: Ac1k5yMmTcYIavedSEaB63mhmcrTaA== Message-ID: In-Reply-To: <20120717164616.GA10673@nargothrond.kdm.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/14.2.3.120616 acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "freebsd-scsi@freebsd.org" Subject: Re: CAM Target Layer and dev/isp X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jul 2012 13:13:44 -0000 On 7/17/12 12:46 PM, "Kenneth D. Merry" wrote: >On Mon, Jul 16, 2012 at 20:14:31 -0700, Trent Nelson wrote: >> I'm not sure how to expose LUNs via my isp devices. Here's what I've >>got >> so far: >>=20 >> [root@s24/ttypts/1(../misc/isp/bin)#] for i in {0..3}; do; ./isp_getrole >> /dev/isp$i; done >> /dev/isp0 chan 0: role target >> /dev/isp1 chan 0: role target >> /dev/isp2 chan 0: role target >> /dev/isp3 chan 0: role target > >Looks good. What model Qlogic board is it? I know the driver works >pretty >well in target mode with 4Gb and 8Gb boards, but I don't know how well 2Gb >and older boards work. I'm still frolicking in 2Gb land, SAN-wise. All my HBAs are Qlogic 2Gb (single and dual port) models. Sample dmesg from one of the dual port ones: sp0: Board Type 2312, Chip Revision 0x2, loaded F/W Revision 3.3.26 isp0: Attributes: TargetMode SCC-Lun Fabric 2K-Login isp0: 876 max I/O command limit set > >> [root@s24/ttypts/1(../misc/isp/bin)#] ctladm devlist -v >> LUN Backend Size (Blocks) BS Serial Number Device ID >> 0 ramdisk 2097152 512 MYSERIAL 0 MYDEVID 0 >> lun_type=3D0 > >That looks fine. Note that it is a fake ramdisk that is only backed by >1MB >of memory no matter how large the reported size is. So you can make it as >large as you want. Haha, yeah I was wondering about the "10485760000000000" you used in the examples. You should definitely make mention of that 1MB fact in the docs somewhere if you're going to use 9536TB as the size in the example ;-) (My interaction went like this: created the crazy huge one used in the examples, was thoroughly confused when I realized it was 9536TB, deleted it, then created a 1GB one instead.) >All of the ports are online, so that looks good. > >> It's not obvious from the docs how to export CTL LUNs through my isp >> devices. I tried passing -D /dev/isp[n] to `ctladm create`, but that >> returns: 'ctladm: cctl_create_lun: error issuing CTL_LUN_REQ ioctl: >> Inappropriate ioctl for device'. > >The block backend only works on block devices or files. e.g.: > >ctladm create -b block -o file=3D/path/to/my/file >ctladm create -b block -o file=3D/dev/da5 > >If you use a block device (like a zvol) as the backing store, you'll want >to disable sending cache syncs to the disk, since that will trigger a GEOM >assertion. > >ctladm realsync off > >(Do that before putting the ports online.) Ah, ok, I played a bit more with block devices instead of ramdisk, and it cleared up some stuff. > >> Sample output from one of the switches on my fabric (s24's /dev/isp1 is >> plugged into port 8 of sf1): >>=20 >> sf1:admin> switchshow >> switchName: sf1 >> switchType: 17.2 >> switchState: Online >> switchMode: Native >> switchRole: Principal >> switchDomain: 1 >> switchId: fffc01 >> switchWwn: 10:00:00:60:69:5a:1a:40 >> switchBeacon: OFF >> Zoning: ON (cfg_2012_07_16) >> port 0: id N2 Online F-Port 50:06:0b:00:00:13:18:72 >> port 1: id N2 No_Light >> port 2: id N2 Online E-Port 10:00:00:60:69:5a:09:c0 "sf2" >> (downstream)=20 >> port 3: id N2 No_Light >> port 4: id N2 No_Light >> port 5: id N2 Online F-Port 21:01:00:e0:8b:ab:cf:be >> port 6: id N2 Online F-Port 21:01:00:e0:8b:a6:98:ca >> port 7: id N2 Online F-Port 21:01:00:e0:8b:30:2f:5b >> port 8: id N1 Online L-Port 1 public > >Looks like it is in loop mode. Can your switch make a loop mode device >visible on another port? You know what, I have no idea what was going on there. I can't replicate that behavior (getting the port to present itself to the fabric as an L-port) on another (almost identical) box. And I managed to panic that box whilst composing this e-mail, to the point where it doesn't even get past the PCI BIOS routines. (It happened when I unplugged the HBA, I'll paste a backtrace and CC Matt in a separate e-mail.) > What are you using for an initiator? Does it >work if you connect the initiator directly to the FreeBSD target? Ah, ok, those questions (and various other comments you made) have cleared up a couple of things. What I was expecting to see from the switch when I did `portshow [n]` was a list of all the LUNs (via their WWNs) being made available by camctl/isp/target-mode. This is what I'm alluding to when I say this: > > Ideally I'd like to be able to use CTL to export multiple ZFS zvols as > > separate targets (i.e. all with unique WWNN/WWPNs), such that, from the > > fabric's point of view, the port would look like just another FC-AL >JBOD, > > like, say, port 11: > >=20 > > sf1:admin> portshow 11 [snip] > > portWwn: 20:0b:00:60:69:5a:1a:40 > > portWwn of device(s) connected: 21:00:00:14:c3:ca:23:ca > > 21:00:00:14:c3:c1:9c:90 > > 21:00:00:14:c3:c4:41:9a > > 21:00:00:14:c3:c1:23:6b > > 21:00:00:14:c3:ca:23:df > > 21:00:00:14:c3:c4:47:97 > > 21:00:00:14:c3:ca:20:0e > > 21:00:00:14:c3:c4:40:ca However, it just occurred to me that in order for that to happen, the block LUN I'm exporting would essentially have to mimic/implement an FC-AL-ported disk (that is public-loop aware) -- I.e. it would have to support fabric login and all the other fancy cruft my physical dual-ported FC SCSI drives in the FC-AL JBODs implement. I reckon it's safe to assume none of that is in place, no? ;-) (Would that even be possible? Seems like it would need a lot of underlying driver support in dev/isp, at the very least. As well as a bucket-load of ctl support. Perhaps better suited to a new backend type? I.e. `ctladm create -b fc-al ...`.) Now, with that being said, back to your questions... > What are you using for an initiator? > Does it work if you connect the initiator directly to the FreeBSD target? Right, so, I haven't played around with any initiators yet as I wasn't seeing the expected output from the SAN switch (I.e. the WWNs of "devices connected to this port"). Now that I know this isn't going to happen, I'll try see if I can forcibly connect initiators. I was happy to see that when I created a block device via ctladm, it was assigned WWPN/WWNNs automatically. I'm not sure if this means I'll still be able to use a fabric (with zoning enabled) or not. If I have no luck with that approach, I'll try plug the initiator's HBA port directly into the target HBA and see if that helps. >If your switch has NP-IV support, you can also try creating multiple >virtual ports with the isp(4) driver if you set the hint.isp.0.vports=3DN >loader tunable, where N is the number of virtual ports. I haven't tried >that in several years, though, and Matt Jacob has indicated it needs more >testing. Yeah unfortunately neither my switches nor my HBAs have support for VSAN stuff. >As for using zvols, the code that is in FreeBSD right now will lead to >very >slow performance with zvols. Justin Gibbs and Will Andrews gave a talk at >BSDCan that explained their work to eliminate COW (Copy On Write) faults >for files used as block devices and zvols. Their slides are here: > >http://www.bsdcan.org/2012/schedule/events/316.en.html > >And the talk itself is here: > >http://www.youtube.com/watch?v=3DLtY3vpX-cdM > >It's fine to use zvols now, but you may want to wait until their code does >into FreeBSD/head at least to use zvols for anything that requires >reasonable performance. > >Until their code goes in the tree, you'll probably get somewhat faster >performance by using files on top of ZFS, or on top of UFS. (UFS will be >much faster at the moment, but you don't get the software RAID >functionality of ZFS.) > >The first write pass through on a zvol or a ZFS file will go very quickly, >but subsequent writes will be pretty slow, especially if they are not on >exact ZFS record boundaries. Gotcha'. Super info, thanks. The "perfect" solution down the track would be zvol, but there's an enormous amount of other stuff I'd need to flush about before that. (What I'm aiming to do is present zvols to either virtual or physical boxes for the OS install disk. That would allow me to snapshot/clone entire OS instances (I.e. AIX/HP-UX/IRIX/Tru64-UNIX et al) and export them as a new disk that could be picked up by another box, which would be incredibly cool for provisioning, tinkering, dev stuff, etc.) >Also, Matt Jacob and I are chasing down a possible data corruption bug. >We don't know exactly where it is, and it might not be in the code that >is in FreeBSD right now. The point is, run some data integrity tests >before using this in production. Heh, roger. (FWIW, this is all pie-in-the-sky level at the moment -- it'll be a while before I can put it into production.) > >> Also, the HA stuff sounds bad-ass. What's the best way to stay up to >>date >> with CTL development? Watch commits to sys/cam/ctl? (There's no >>separate >> list or anything for this stuff, right?) > >That's pretty much the best way to keep up with it, there's not a separate >mailing list. > >I don't think I'm going to have time to do anything with the HA hooks in >the near future. Hopefully other folks will be interested and do some >development in that area. It would be nice to have a fully HA block >stack, >but that will take a lot of effort. Nod. It's certainly not a deal breaker for me, and I can get around it by other (clunky) means, but it certainly would be awesome. I'd essentially be able to build a ZFS-backed storage controller indistinguishable from the (proprietary) HA RAID controllers that I was originally intended to use with my disk arrays instead of just straight JBOD.) Thanks for such thorough information. Very useful. I've got a few things to play around with now. Regards, Trent.