From owner-freebsd-scsi@freebsd.org Wed Sep 9 23:42:37 2015 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 65BD0A0091E for ; Wed, 9 Sep 2015 23:42:37 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-oi0-x242.google.com (mail-oi0-x242.google.com [IPv6:2607:f8b0:4003:c06::242]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2CCC619C2; Wed, 9 Sep 2015 23:42:37 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by oibi136 with SMTP id i136so1411789oib.3; Wed, 09 Sep 2015 16:42:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=vDZNKT+1SrvCs9XWEmqmyxD2pbq4marWbrNxrkUn9vI=; b=RVdu23ZZJD6EKY7CMkENnCPDtmRwPr2QlnEgCqMUfes7x+U2yRFccrxCLjB5ejI+tr MvoCanQsdecYAwjD1hWgDrm5Bs8Nm7cl2sN/izxOr/Xsy7Z+AvwJzbThLa7dMHAgX+zo fZU9iLsjRnIolMObqgTGwbpT5ne/0WxeN/FUYIDUPeDEZU9VyDqTTOp37ygvdeXtdENl yLfnYN8UKZPKg7LaNNhzx1pyBiB8ECDUXSVu20NtUnM7cYcRijCxR080wlMrHztIuHca EWvH381wmn2CwQyzPS4942jbiEi3oP0qMquyziab8p1mRDvSDPxx3VNRvOVNLGrYokxX 64Pg== MIME-Version: 1.0 X-Received: by 10.202.215.68 with SMTP id o65mr5543028oig.100.1441842156278; Wed, 09 Sep 2015 16:42:36 -0700 (PDT) Sender: asomers@gmail.com Received: by 10.202.79.140 with HTTP; Wed, 9 Sep 2015 16:42:36 -0700 (PDT) In-Reply-To: <20150909231756.GA88399@FreeBSD.org> References: <20150909033510.GA34413@FreeBSD.org> <20150909231756.GA88399@FreeBSD.org> Date: Wed, 9 Sep 2015 17:42:36 -0600 X-Google-Sender-Auth: YEYg87EG5f5cA6q-Cf532OxcIR0 Message-ID: Subject: Re: ses/pass devices (enclosure/processor devices) not all showing up? From: Alan Somers To: John De Boskey Cc: Alan Somers , FreeBSD-SCSI Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Sep 2015 23:42:37 -0000 On Wed, Sep 9, 2015 at 5:17 PM, John De Boskey wrote: > ----- Alan Somers's Original Message ----- >> On Tue, Sep 8, 2015 at 9:35 PM, John De Boskey wrote: >> > Hi Folks - >> > >> > I have a shelf with 84 sata drives. All drives show up >> > correctly and are accessible. The shelf appears to have >> > multiple processor devices and one enclosure device internally. >> > For instance: >> > >> > # camcontrol devlist | grep XYRATEX >> > at scbus7 target 159 lun 0 (pass18) >> > at scbus7 target 188 lun 0 (pass47) >> > at scbus7 target 217 lun 0 (pass76) >> > at scbus7 target 232 lun 0 (pass91) >> > >> > # camcontrol devlist | grep ses >> > at scbus7 target 144 lun 0 (ses0,pass3) >> > >> > # camcontrol smprg pass18 | grep 'Number of Phys:' >> > Number of Phys: 25 >> > # camcontrol smprg pass47 | grep 'Number of Phys:' >> > Number of Phys: 37 >> > # camcontrol smprg pass76 | grep 'Number of Phys:' >> > Number of Phys: 37 >> > # camcontrol smprg pass91 | grep 'Number of Phys:' >> > Number of Phys: 25 >> > # camcontrol smprg ses0 | grep 'Number of Phys:' >> > Number of Phys: 37 >> > >> > # camcontrol smpphylist pass18 >> > 25 PHYs: >> > PHY Attached SAS Address >> > 0 0x5000c500585f4b52 (pass4,da0) > > ... > >> > I have linked dmesg, camcontrol devlist, and sas2ircu output below: >> > >> > http://people.freebsd.org/~jwd/sespass/dmesg.txt mps messaging enabled. >> > >> > http://people.freebsd.org/~jwd/sespass/devlist.txt >> > >> > http://people.freebsd.org/~jwd/sespass/sas2ircu.txt >> > >> > Thanks, >> > John > > Hi Alan, > >> I'm not sure exactly what you're asking. Are you just wondering why > > In a simple form, I'm trying to figure out why 'camcontrol smpphyslist ses0' > produces no output, while 'camcontrol smpphyslist pass18' does. > >> your pass devices don't also have ses nodes? I think I know why. >> Your JBOD problem has five SAS expander chips, though ses0 might >> actually be some other kind of SAS target chip. pass18, pass47, >> pass76, and pass91 are configured to report ses0's SAS Address as the >> address of their SEP. The LSI HBA's interpretation is that there is > > Could you expand on the concept of the SAS Address being reported? > When I look at the address of the devices in the sas2ircu output they > are unique. For example: Each expander IC actually contains several logically independent devices. First there is the expander itself. FreeBSD doesn't have any visibility into that. But you can see the address if you do "sg_ses -p 10 ses0" and look for "attached SAS address". In your case, you should see four unique attached SAS addresses. Second, there is the SEP (SCSI Enclosure Processor). This is the SCSI target that ends up getting a device node in FreeBSD. Third, the expander will automatically generate a SAS address for any attached SATA devices. Now, I don't understand all the details, but somehow your four slave expanders are telling your HBA that their disks are managed by the SEP whose SAS address is the address of ses0. As you've seen, that information is not contained with the SMP Report General response. It's probably some other SMP command, but I don't know which. > > Device is a Processor device > Enclosure # : 2 > Slot # : 1 > SAS Address : 50050cc-1-0d2f-fabe > State : Standby (SBY) > Manufacturer : XYRATEX > Model Number : DEFAULT-SD-R24 > Firmware Revision : 3034 > Serial No : PMCSIERA > > Device is a Processor device > Enclosure # : 2 > Slot # : 2 > SAS Address : 50050cc-1-0d2f-fafe > State : Standby (SBY) > Manufacturer : XYRATEX > Model Number : DEFAULT-SD-R36 > Firmware Revision : 3034 > Serial No : PMCSIERA > > ie: they show as unique. However, they don't appear to be unique > as reported by camcontrol smprg. I haven't been able to determine > where in the code this linkage is being done yet. > > >> only one SES processor. So FreeBSD reports one ses device, and the >> other expanders just show up as pass devices. Unless the manufacturer >> royally screwed up (I doubt it), ses0 will report info for all 84 >> disks in its SES status pages. > > camcontrol smpphylist ses0 - returns no information. > camcontrol smpphylist pass18 - returns a drive list > camcontrol smpphylist pass47 - returns a drive list > camcontrol smpphylist pass76 - returns a drive list > camcontrol smpphylist pass91 - returns a drive list > >> Bill is right, sg3_utils is your friend. However, I doubt you'll see >> any secondary subenclosures. That feature isn't much used, and it's >> not necessary in order for ses0 to report all 84 drives. Running >> these two commands will probably tell you most of what you need to >> know: >> >> sg_ses -p 1 ses0 >> sg_ses -p 2 ses0 > > I have run the above commands and linked the output below. > > http://people.freebsd.org/~jwd/sespass/sgsesp1.txt > > http://people.freebsd.org/~jwd/sespass/sgsesp2.txt descriptor for each disk > > Thoughts? It's as I expected. ses0 is reporting all 84 disk slots. You can use sg_ses on ses0 for pretty much all drive management tasks. The only reason to use smpphylist and smppc, IMHO, is to disable a drive's phy. That's mostly useful for simulating a drive pull, or for disabling a device that is broken and doesn't probe cleanly. -Alan > > Thanks, > John > >> -Alan