From owner-freebsd-scsi@FreeBSD.ORG Thu May 3 13:34:52 2012 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id EC8491065675 for ; Thu, 3 May 2012 13:34:52 +0000 (UTC) (envelope-from mj@feral.com) Received: from ns1.feral.com (ns1.feral.com [192.67.166.1]) by mx1.freebsd.org (Postfix) with ESMTP id BB0A08FC1D for ; Thu, 3 May 2012 13:34:52 +0000 (UTC) Received: from [192.168.135.100] (c-76-126-166-136.hsd1.ca.comcast.net [76.126.166.136]) (authenticated bits=0) by ns1.feral.com (8.14.4/8.14.4) with ESMTP id q43DYpIU029504 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Thu, 3 May 2012 06:34:51 -0700 (PDT) (envelope-from mj@feral.com) Message-ID: <4FA28976.2070607@feral.com> Date: Thu, 03 May 2012 06:34:46 -0700 From: Matthew Jacob Organization: Feral Software User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:12.0) Gecko/20120428 Thunderbird/12.0.1 MIME-Version: 1.0 To: Trent Nelson References: <20120502203948.GA1141@snakebite.org> <4FA1DF1C.2030706@feral.com> <20120503050956.GA18983@snakebite.org> In-Reply-To: <20120503050956.GA18983@snakebite.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (ns1.feral.com [192.67.166.1]); Thu, 03 May 2012 06:34:52 -0700 (PDT) Cc: freebsd-scsi@freebsd.org Subject: Re: Missing sysctl options for isp driver X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: mj@feral.com List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 May 2012 13:34:53 -0000 > I, uh, I don't think I'm using target mode. I've got four Xyratex RS- > 1600-FC2 JBODs (16x146GB + 48x36GB) connected to two-but-soon-to-be-four > HP StorageWorks SAN Switch 2/16s. Each FreeBSD box has a minimum of two > FC HBA ports; each HBA port goes to a different switch, and zoning config > on the switches controls which disks each hosts sees (although more on > this later). > > So, uh, I think that constitutes fabric mode right? The switch reports > the HBA ports as F-Ports and, when zoning's enabled, all the FLOGI/PLOGI > stuff seems to work properly. > > (My SAN/FC-fu has been cobbled together from trial-and-error, so I'm not > entirely sure what target mode is or whether or not target mode bugs are > likely to affect me.) Target Mode is where the FreeBSD box can pretend to be a disk. Yes, that's fabric. > > >> Did you have specific bugs you were concerned with? > Well... as I'm still playing around with the setup, I'm not sure if some > of the oddities I'm seeing are bugs or not. Things seem to work when I've > got zoning enabled on the SAN switches. > > When zoning's disabled and the HBAs are plugged into the fabric, things > sort of go a bit ballistic... The isp driver doesn't seem to handle getting > 128 RSCNs (2 for each dual-ported FC drive) in one go very well. Mailbox > timeouts, hung `camcontrol rescan all`'s, lots of isp[n] dmesg errors and > warnings -- even the SAN switch console starts barfing errors. > > This sort of stuff (queue random dmesg snippets from earlier): > > (da37:isp1:0:88:0): got CAM status 0x4a > (da37:isp1:0:88:0): fatal error, failed to attach to device > (da37:isp1:0:88:0): removing device entry > (da2:isp0:0:9:0): Synchronize cache failed, status == 0x1b, scsi status == 0x0 > (da38:isp1:0:102:0): got CAM status 0x4a > (da38:isp1:0:102:0): fatal error, failed to attach to device > (da38:isp1:0:102:0): removing device entry > isp0: Chan 0 PLOGI 0x020500 failed > isp0: Chan 0 PLOGI 0x020600 failed > isp0: Chan 0 PLOGI 0x020800 failed > isp0: Mailbox Command 'BUS RESET' failed (COMMAND ERROR) > isp1: Mailbox Command 'BUS RESET' failed (COMMAND ERROR) > (da119:isp1:0:3:0): lost device - 0 outstanding, 0 refs > (da119:isp1:0:3:0): removing device entry > isp1: Chan 0 PLOGI 0x0201ca failed > isp1: Chan 0 PLOGI 0x0201cb failed > isp0: Chan 0 PLOGI 0x0101e0 failed > isp0: Chan 0 PLOGI 0x0101e1 failed > isp0: Chan 0 PortID 0x0102ad already at 512 handle 35 state 5 > isp0: Chan 0 PortID 0x0102ae already at 512 handle 34 state 5 > isp0: Chan 0 PortID 0x0102b1 already at 512 handle 33 state 5 > isp0: Chan 0 PortID 0x0102b2 already at 512 handle 32 state 5 > isp0: Chan 0 PortID 0x0102b3 already at 512 handle 31 state 5 > > If I plug the HBA directly into a port on the JBOD, I get no such issues; > all /dev/da* device nodes are created instantly and everything just seems > to work. (I presume HBA->JBOD would result in an FC-AL topology rather > than an F-port topology, which I figure translates into a different code > path in the isp driver.) > > When I enabled zoning and configured a HBA to only see a handful of disks, > everything behaved properly. I'm going to update to head and get a debug > build going with all the ISP debug bells and whistles; I'll report back on > HBA->fabric behaviour when zoning is off after that. > > (FWIW, I've got two HP-UX (PA-RISC and IA64) boxes with two HBAs also > plugged into the fabric; they don't go into fits like FreeBSD/isp seems > to when zoning is turned off.) > What kind of switches are you using? This is definitely wrong, and probably means that I need to do some work again here.