From owner-freebsd-scsi@freebsd.org Wed Sep 9 15:38:21 2015 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 77BADA018DF for ; Wed, 9 Sep 2015 15:38:21 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-oi0-x230.google.com (mail-oi0-x230.google.com [IPv6:2607:f8b0:4003:c06::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 410EE1B24; Wed, 9 Sep 2015 15:38:21 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by oiev17 with SMTP id v17so7819890oie.1; Wed, 09 Sep 2015 08:38:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=EJMDWzIVIY3rrtBNiP0GS/xsDn9F8jMTbAM+vtwbgNU=; b=qUts5cjazpVv//HPLMb2MTH+yanUsqS7e8fwIJWkeaIz7Da0J6gT1FQR3t52ltzxNO 4P0Wz4ksYuazWotrLUuBEDtC9X/rpxelKFQ/5bbvb8gxHic18GfVwS1sZ7ko+y+TEuwy BVtaROfI9Y+NwpQtlpy2M3yncvfi89HZtntb3xC6cTWVtSAx4KhbCr8eS+n+20r4bFqi uQVVcbpP+f9kPo/Z9pRff1ZD+BX4kr+by6oNZEfYkPzc+d7h9Y5mNkuyAtXVtHnhRJha g8cprSqgCKk8JBQxL42hBCFed+D5BVVcmAxgcAeaG6T4Z/6ugApo5oF0ELmzHPY+jc1p zVWQ== MIME-Version: 1.0 X-Received: by 10.202.212.73 with SMTP id l70mr25711589oig.54.1441813100601; Wed, 09 Sep 2015 08:38:20 -0700 (PDT) Sender: asomers@gmail.com Received: by 10.202.79.140 with HTTP; Wed, 9 Sep 2015 08:38:20 -0700 (PDT) In-Reply-To: <20150909033510.GA34413@FreeBSD.org> References: <20150909033510.GA34413@FreeBSD.org> Date: Wed, 9 Sep 2015 09:38:20 -0600 X-Google-Sender-Auth: byoRRni8VxAaD5HSXwScT5KLjbw Message-ID: Subject: Re: ses/pass devices (enclosure/processor devices) not all showing up? From: Alan Somers To: John De Boskey Cc: FreeBSD-SCSI Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Sep 2015 15:38:21 -0000 On Tue, Sep 8, 2015 at 9:35 PM, John De Boskey wrote: > Hi Folks - > > I have a shelf with 84 sata drives. All drives show up > correctly and are accessible. The shelf appears to have > multiple processor devices and one enclosure device internally. > For instance: > > # camcontrol devlist | grep XYRATEX > at scbus7 target 159 lun 0 (pass18) > at scbus7 target 188 lun 0 (pass47) > at scbus7 target 217 lun 0 (pass76) > at scbus7 target 232 lun 0 (pass91) > > # camcontrol devlist | grep ses > at scbus7 target 144 lun 0 (ses0,pass3) > > # camcontrol smprg pass18 | grep 'Number of Phys:' > Number of Phys: 25 > # camcontrol smprg pass47 | grep 'Number of Phys:' > Number of Phys: 37 > # camcontrol smprg pass76 | grep 'Number of Phys:' > Number of Phys: 37 > # camcontrol smprg pass91 | grep 'Number of Phys:' > Number of Phys: 25 > # camcontrol smprg ses0 | grep 'Number of Phys:' > Number of Phys: 37 > > # camcontrol smpphylist pass18 > 25 PHYs: > PHY Attached SAS Address > 0 0x5000c500585f4b52 (pass4,da0) > 1 0x5000c500585f4476 (pass5,da1) > 2 0x5000c500585f6efe (pass6,da2) > 3 0x5000c500585f14d6 (pass7,da3) > 4 0x5000c500585f683e (pass8,da4) > 5 0x5000c500585f15e6 (pass9,da5) > 6 0x5000c500585f49ba (pass10,da6) > 7 0x5000c500585f14ea (pass11,da7) > 8 0x5000c500585f5ec6 (pass12,da8) > 9 0x5000c500585f0d12 (pass13,da9) > 10 0x5000c500585f620a (pass14,da10) > 11 0x5000c500585f0d1e (pass15,da11) > 12 0x5000c500585f64ae (pass16,da12) > 13 0x5000c500585f27aa (pass17,da13) > 14 0x0000000000000000 > 15 0x0000000000000000 > 16 0x0000000000000000 > 17 0x0000000000000000 > 18 0x0000000000000000 > 19 0x0000000000000000 > 20 0x50050cc1177513bf > 21 0x50050cc1177513bf > 22 0x50050cc1177513bf > 23 0x50050cc1177513bf > 24 0x50050cc10d2fe7be > > > (sas2ircu processor devices) > > # camcontrol smpphylist pass18 -q | wc -l > 14 > # camcontrol smpphylist pass47 -q | wc -l > 28 > # camcontrol smpphylist pass76 -q | wc -l > 28 > # camcontrol smpphylist pass91 -q | wc -l > 14 > > Which adds up to the 84 expected drives. No devices show up under ses0: > > (sas2ircu enclosure services device) > > # camcontrol smpphylist ses0 -q | wc -l > 0 > > I don't know that I understand the hierarchy of the backplane attachments. > Should the 4 pass only processor devices actually be ses devices? Or should > they be linked up under ses0 and all the devices show up under ses0? > > Any thoughts or ideas on how to fix this up(debug) would be appreciated. > > I have linked dmesg, camcontrol devlist, and sas2ircu output below: > > http://people.freebsd.org/~jwd/sespass/dmesg.txt mps messaging enabled. > > http://people.freebsd.org/~jwd/sespass/devlist.txt > > http://people.freebsd.org/~jwd/sespass/sas2ircu.txt > > Thanks, > John I'm not sure exactly what you're asking. Are you just wondering why your pass devices don't also have ses nodes? I think I know why. Your JBOD problem has five SAS expander chips, though ses0 might actually be some other kind of SAS target chip. pass18, pass47, pass76, and pass91 are configured to report ses0's SAS Address as the address of their SEP. The LSI HBA's interpretation is that there is only one SES processor. So FreeBSD reports one ses device, and the other expanders just show up as pass devices. Unless the manufacturer royally screwed up (I doubt it), ses0 will report info for all 84 disks in its SES status pages. Bill is right, sg3_utils is your friend. However, I doubt you'll see any secondary subenclosures. That feature isn't much used, and it's not necessary in order for ses0 to report all 84 drives. Running these two commands will probably tell you most of what you need to know: sg_ses -p 1 ses0 sg_ses -p 2 ses0 -Alan