From owner-freebsd-scsi@freebsd.org Tue Sep 8 18:47:08 2015 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 028EBA00993 for ; Tue, 8 Sep 2015 18:47:08 +0000 (UTC) (envelope-from sbruno@freebsd.org) Received: from mail.ignoranthack.me (ignoranthack.me [199.102.79.106]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DB7B81E71 for ; Tue, 8 Sep 2015 18:47:07 +0000 (UTC) (envelope-from sbruno@freebsd.org) Received: from [192.168.100.102] (unknown [50.136.155.142]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: sbruno@ignoranthack.me) by mail.ignoranthack.me (Postfix) with ESMTPSA id C10AB193E68 for ; Tue, 8 Sep 2015 18:47:05 +0000 (UTC) Subject: Re: da2:ciss1:0:0:0): Periph destroyed To: freebsd-scsi@freebsd.org References: <55E72440.8070507@intersonic.se> <55E7309C.8010406@freebsd.org> <55E73900.5080302@intersonic.se> <55E742B9.1060002@freebsd.org> <55E747AC.6020302@intersonic.se> <55E7488C.3070602@freebsd.org> From: Sean Bruno Message-ID: <55EF2D29.9090105@freebsd.org> Date: Tue, 8 Sep 2015 11:47:05 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: <55E7488C.3070602@freebsd.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Sep 2015 18:47:08 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 09/02/15 12:05, Sean Bruno wrote: > > > On 09/02/15 12:02, Per olof Ljungmark wrote: > > > Definitely. I'll yank it out today and setup a 3 day MFC > > sean ____________________________ This was MFC'd today. sean -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQF8BAEBCgBmBQJV7y0mXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRCQUFENDYzMkU3MTIxREU4RDIwOTk3REQx MjAxRUZDQTFFNzI3RTY0AAoJEBIB78oecn5kTs8H/0z3Di+I0rejVzsrqgHBfBhA Q0ir8lPvJ4XFYT3dil623s9aKw6+IGDs8qrQ++ud5tInaSpGTn/nsyerjUjokvCb 0j/yAU7/CGMwoq43voIcFagV/ZCX/7reN1bCzibsVVno+mAeeEqR7wF4j7WLnJMn Vc0QZheLy6KzNKIyQWSpl8b+MD8zmv2BdT7kjx1bLzwKXsOVLgGgdQ7Jut5D9ekd HFGlgwx5S7fdQT9T7nLCafnwLWzgalqMGwCpX5pfp3a+DH4jsmt13/Xr4lcUfM73 6UaMg2T+EKLNpwkaYFXM8bd1XQjTYEq5c4KCs1MvJj5wEvbYxa5oB3gMsNncSDk= =baFB -----END PGP SIGNATURE----- From owner-freebsd-scsi@freebsd.org Wed Sep 9 03:35:11 2015 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1FD169CD257 for ; Wed, 9 Sep 2015 03:35:11 +0000 (UTC) (envelope-from jwd@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id 15EBC153B for ; Wed, 9 Sep 2015 03:35:11 +0000 (UTC) (envelope-from jwd@freebsd.org) Received: by freefall.freebsd.org (Postfix, from userid 821) id 1462D1B9B; Wed, 9 Sep 2015 03:35:11 +0000 (UTC) Date: Wed, 9 Sep 2015 03:35:11 +0000 From: John De Boskey Cc: FreeBSD-SCSI Subject: ses/pass devices (enclosure/processor devices) not all showing up? Message-ID: <20150909033510.GA34413@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Sep 2015 03:35:11 -0000 Hi Folks - I have a shelf with 84 sata drives. All drives show up correctly and are accessible. The shelf appears to have multiple processor devices and one enclosure device internally. For instance: # camcontrol devlist | grep XYRATEX at scbus7 target 159 lun 0 (pass18) at scbus7 target 188 lun 0 (pass47) at scbus7 target 217 lun 0 (pass76) at scbus7 target 232 lun 0 (pass91) # camcontrol devlist | grep ses at scbus7 target 144 lun 0 (ses0,pass3) # camcontrol smprg pass18 | grep 'Number of Phys:' Number of Phys: 25 # camcontrol smprg pass47 | grep 'Number of Phys:' Number of Phys: 37 # camcontrol smprg pass76 | grep 'Number of Phys:' Number of Phys: 37 # camcontrol smprg pass91 | grep 'Number of Phys:' Number of Phys: 25 # camcontrol smprg ses0 | grep 'Number of Phys:' Number of Phys: 37 # camcontrol smpphylist pass18 25 PHYs: PHY Attached SAS Address 0 0x5000c500585f4b52 (pass4,da0) 1 0x5000c500585f4476 (pass5,da1) 2 0x5000c500585f6efe (pass6,da2) 3 0x5000c500585f14d6 (pass7,da3) 4 0x5000c500585f683e (pass8,da4) 5 0x5000c500585f15e6 (pass9,da5) 6 0x5000c500585f49ba (pass10,da6) 7 0x5000c500585f14ea (pass11,da7) 8 0x5000c500585f5ec6 (pass12,da8) 9 0x5000c500585f0d12 (pass13,da9) 10 0x5000c500585f620a (pass14,da10) 11 0x5000c500585f0d1e (pass15,da11) 12 0x5000c500585f64ae (pass16,da12) 13 0x5000c500585f27aa (pass17,da13) 14 0x0000000000000000 15 0x0000000000000000 16 0x0000000000000000 17 0x0000000000000000 18 0x0000000000000000 19 0x0000000000000000 20 0x50050cc1177513bf 21 0x50050cc1177513bf 22 0x50050cc1177513bf 23 0x50050cc1177513bf 24 0x50050cc10d2fe7be (sas2ircu processor devices) # camcontrol smpphylist pass18 -q | wc -l 14 # camcontrol smpphylist pass47 -q | wc -l 28 # camcontrol smpphylist pass76 -q | wc -l 28 # camcontrol smpphylist pass91 -q | wc -l 14 Which adds up to the 84 expected drives. No devices show up under ses0: (sas2ircu enclosure services device) # camcontrol smpphylist ses0 -q | wc -l 0 I don't know that I understand the hierarchy of the backplane attachments. Should the 4 pass only processor devices actually be ses devices? Or should they be linked up under ses0 and all the devices show up under ses0? Any thoughts or ideas on how to fix this up(debug) would be appreciated. I have linked dmesg, camcontrol devlist, and sas2ircu output below: http://people.freebsd.org/~jwd/sespass/dmesg.txt mps messaging enabled. http://people.freebsd.org/~jwd/sespass/devlist.txt http://people.freebsd.org/~jwd/sespass/sas2ircu.txt Thanks, John From owner-freebsd-scsi@freebsd.org Wed Sep 9 05:42:54 2015 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A7FEF9CD598 for ; Wed, 9 Sep 2015 05:42:54 +0000 (UTC) (envelope-from ethernext@gmail.com) Received: from mail-io0-x243.google.com (mail-io0-x243.google.com [IPv6:2607:f8b0:4001:c06::243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7A1931EB2; Wed, 9 Sep 2015 05:42:54 +0000 (UTC) (envelope-from ethernext@gmail.com) Received: by ioii196 with SMTP id i196so983814ioi.2; Tue, 08 Sep 2015 22:42:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=cmhl8M66R46XglF8FAPwfrtjGz6AoBcOxqwCn9vXia4=; b=sPh+K0DaAv5x76yLZovqM8cGqAnVoY2WGhZfk2yDopcJ/KPlgmpWNQgjH0tUg/dRSo lHzZhq6x9PFv5YRysLcdsNdT2JZieMRyHr1MR9o8IS/LB7KzhtUlg3QKQSv7xocoVtNU 3/0ofqqnVdBJiiC018nGnk//CnrAzT2tT70eNYGrVrGU8WeOF9iI7HxU1v4WgXN0J5qC OcORcDHTcFEoopLV0cch92OwsdLpxv6cWzwgpAIH9Fw90FgAd8aiIrLzlKropMGMgkJP uVmjvuICpVtTL9L+HlW5KaQJtNFbphzaGvzXUYocP0Q/Lru6y96S7AZyljZL1HD/YyCk zsgg== MIME-Version: 1.0 X-Received: by 10.107.3.94 with SMTP id 91mr44962829iod.178.1441777373962; Tue, 08 Sep 2015 22:42:53 -0700 (PDT) Received: by 10.107.140.12 with HTTP; Tue, 8 Sep 2015 22:42:53 -0700 (PDT) In-Reply-To: <20150909033510.GA34413@FreeBSD.org> References: <20150909033510.GA34413@FreeBSD.org> Date: Wed, 9 Sep 2015 01:42:53 -0400 Message-ID: Subject: Re: ses/pass devices (enclosure/processor devices) not all showing up? From: "Bill (EtherneXt)" To: John De Boskey Cc: FreeBSD-SCSI Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Sep 2015 05:42:54 -0000 Hi John, On Tue, Sep 8, 2015 at 11:35 PM, John De Boskey wrote: > Hi Folks - > > I have a shelf with 84 sata drives. All drives show up > correctly and are accessible. The shelf appears to have > multiple processor devices and one enclosure device internally. > For instance: > > # camcontrol devlist | grep XYRATEX > at scbus7 target 159 lun 0 (pass18) > at scbus7 target 188 lun 0 (pass47) > at scbus7 target 217 lun 0 (pass76) > at scbus7 target 232 lun 0 (pass91) > > # camcontrol devlist | grep ses > at scbus7 target 144 lun 0 (ses0,pass3) > > # camcontrol smprg pass18 | grep 'Number of Phys:' > Number of Phys: 25 > # camcontrol smprg pass47 | grep 'Number of Phys:' > Number of Phys: 37 > # camcontrol smprg pass76 | grep 'Number of Phys:' > Number of Phys: 37 > # camcontrol smprg pass91 | grep 'Number of Phys:' > Number of Phys: 25 > # camcontrol smprg ses0 | grep 'Number of Phys:' > Number of Phys: 37 > ...... you definitely want to install and use the sg3_utils package to manage the enclosure. the sg_ses utility in sg3_utils will provide most all the info you will ever need. you can use sg_ses to manage drive/slot LED's, fans, power supplies and more. sg3_utils installs a lot of useful tools so take a look at them all. there are also some basic ses tools in /usr/share/examples/ses. sdparm is also a handy tool for managing drives. for the 4 pass devices, I am not sure. there is a scsi processor device driver named pt but I have no idea if it is related. do a man pt for more info. you may also want to build the kernel with the sg device which is another pass-thru device for compatibility with the scsi generic linux driver. with sg_ses, take a look at the line for subenclosures: #>sg_ses ses1 -p 1 SUN CSM200_E 98C4 Configuration diagnostic page: number of secondary subenclosures: 0 ^^^^^^^^^^^^^^^^^^^^^^^^ if you see 4 subenclosures, those scsi processor devices may be used for internal enclosure management and communication with the ses0 device, but thats only a guess on my part. you can also try using sg_ses on the pass devices to see if they report enclosure services. also see the sg3_utils website, lots of detail on using the utilities. Best, -- Bill From owner-freebsd-scsi@freebsd.org Wed Sep 9 09:07:13 2015 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3B1F39CC11E for ; Wed, 9 Sep 2015 09:07:13 +0000 (UTC) (envelope-from k.kulikov2@gmail.com) Received: from mail-wi0-x231.google.com (mail-wi0-x231.google.com [IPv6:2a00:1450:400c:c05::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DBD6811E2 for ; Wed, 9 Sep 2015 09:07:12 +0000 (UTC) (envelope-from k.kulikov2@gmail.com) Received: by wiclk2 with SMTP id lk2so148928811wic.0 for ; Wed, 09 Sep 2015 02:07:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :content-type; bh=3+XZae+lbZfaZSPp7npKEBgiEWH14NbyzBa6Ux8w7LY=; b=sIOjB/FEjNG15fTlACACZS42kECDpHcakUAzIvY8a+oEI7uZqR4P1I7XlOjplIiqlb sW23qcjIIrd/PlEjWXfJ9c5y9BnTNSpqySU8JPi5N/8dINht3Y6rcTu7al4U6bJt0k47 f2/kt7iUtcqymd4/ZwSYSkLWEE4IY8DJoF8sXjiUFyXWpBPoCq8SnUGRuFYNm17xrI4K ZVn0Kf3I7TN/h7Yx6CWZFOBVSZIw2jPD3e3DRI7z2AEnkgTwwsBk2PLkE2Ul1XR1MVWs 9/fG0atq7cLMRCM5BnzDALKQRRnQ22r8wYjFQ2zMJ9mXAGV0JQc7Yf3UkGTqTos37Z02 zEtg== X-Received: by 10.180.82.233 with SMTP id l9mr52189462wiy.71.1441789631045; Wed, 09 Sep 2015 02:07:11 -0700 (PDT) MIME-Version: 1.0 References: <55DE5365.6090608@physics.umn.edu> <55DF1F78.2080204@physics.umn.edu> <55DF6F1F.9070909@physics.umn.edu> In-Reply-To: <55DF6F1F.9070909@physics.umn.edu> From: Konstantin Kulikov Date: Wed, 09 Sep 2015 09:07:01 +0000 Message-ID: Subject: Re: mps driver on 10.2-BETA To: Graham Allan , freebsd-scsi@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Sep 2015 09:07:13 -0000 Just tested lsi 9201-16i 20.00.04.00 with 10.2-RELEASE. Looks good here as well. On Thu, Aug 27, 2015 at 11:12 PM Graham Allan wrote: > On 8/27/2015 9:32 AM, Graham Allan wrote: > > > > I have another system which isn't in production yet - I can do the same > > test on that and see if the newer 20.00.00.04 is stable. This is also > > running 9.3 but the test should still be valid. I will be using this > > system for 10.2 testing, but would like to keep it on 9.3 for a short > > time to try and resolve any issues with that. > > I updated the firmware on this HBA to 20.00.04.00 and ran some stress > tests - all looks fine after 4+ hours. I'm pretty certain that issues > showed up long before that with 20.00.00.00 (ideally I'd like to retest > that, but I don't have a stashed copy of that firmware). > > To reiterate, this is with FreeBSD 9.3, and in this particular case the > HBA is a 9205-8e. > > I expect I'll get to re-test within the next few weeks with FreeBSD > 10.2. But for now it seems like firmware 20.00.04.00 has the previous > problems fixed. > > Graham > _______________________________________________ > freebsd-scsi@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-scsi > To unsubscribe, send any mail to "freebsd-scsi-unsubscribe@freebsd.org" > From owner-freebsd-scsi@freebsd.org Wed Sep 9 12:22:32 2015 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 688B9A00BD2; Wed, 9 Sep 2015 12:22:32 +0000 (UTC) (envelope-from mikej@mikej.com) Received: from mx2.paymentallianceintl.com (mx2.paymentallianceintl.com [216.26.158.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx2.paymentallianceintl.com", Issuer "Go Daddy Secure Certification Authority" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 40D9117F7; Wed, 9 Sep 2015 12:22:31 +0000 (UTC) (envelope-from mikej@mikej.com) Received: from firewall.mikej.com (162-230-214-65.lightspeed.lsvlky.sbcglobal.net [162.230.214.65]) by mx2.paymentallianceintl.com (8.15.1/8.15.1) with ESMTPS id t89CBP7I041331 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 9 Sep 2015 08:11:26 -0400 (EDT) (envelope-from mikej@mikej.com) X-Authentication-Warning: mx2.paymentallianceintl.com: Host 162-230-214-65.lightspeed.lsvlky.sbcglobal.net [162.230.214.65] claimed to be firewall.mikej.com Received: from mail.mikej.com (firewall.mikej.com [192.168.6.63]) by firewall.mikej.com (8.15.2/8.14.9) with ESMTP id t89CBPbk053222; Wed, 9 Sep 2015 08:11:25 -0400 (EDT) (envelope-from mikej@mikej.com) X-Authentication-Warning: firewall.mikej.com: Host firewall.mikej.com [192.168.6.63] claimed to be mail.mikej.com MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Wed, 09 Sep 2015 08:11:24 -0400 From: Michael Jung To: "Bill (EtherneXt)" Cc: John De Boskey , FreeBSD-SCSI , owner-freebsd-scsi@freebsd.org Subject: Re: ses/pass devices (enclosure/processor devices) not all showing up? In-Reply-To: References: <20150909033510.GA34413@FreeBSD.org> Message-ID: <269a4cc51fb8e2200fe92d592a9eabd2@mail.mikej.com> X-Sender: mikej@mikej.com User-Agent: Roundcube Webmail/1.1.2 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Sep 2015 12:22:32 -0000 On 2015-09-09 01:42, Bill (EtherneXt) wrote: > Hi John, > > On Tue, Sep 8, 2015 at 11:35 PM, John De Boskey > wrote: >> Hi Folks - >> >> I have a shelf with 84 sata drives. All drives show up >> correctly and are accessible. The shelf appears to have >> multiple processor devices and one enclosure device internally. >> For instance: >> >> # camcontrol devlist | grep XYRATEX >> at scbus7 target 159 lun 0 (pass18) >> at scbus7 target 188 lun 0 (pass47) >> at scbus7 target 217 lun 0 (pass76) >> at scbus7 target 232 lun 0 (pass91) >> >> # camcontrol devlist | grep ses >> at scbus7 target 144 lun 0 >> (ses0,pass3) >> >> # camcontrol smprg pass18 | grep 'Number of Phys:' >> Number of Phys: 25 >> # camcontrol smprg pass47 | grep 'Number of Phys:' >> Number of Phys: 37 >> # camcontrol smprg pass76 | grep 'Number of Phys:' >> Number of Phys: 37 >> # camcontrol smprg pass91 | grep 'Number of Phys:' >> Number of Phys: 25 >> # camcontrol smprg ses0 | grep 'Number of Phys:' >> Number of Phys: 37 >> ...... > > you definitely want to install and use the sg3_utils package to manage > the enclosure. the sg_ses utility in sg3_utils will provide most all > the info you will ever need. you can use sg_ses to manage drive/slot > LED's, fans, power supplies and more. sg3_utils installs a lot of > useful tools so take a look at them all. there are also some basic ses > tools in /usr/share/examples/ses. sdparm is also a handy tool for > managing drives. > > for the 4 pass devices, I am not sure. there is a scsi processor > device driver named pt but I have no idea if it is related. do a man > pt for more info. you may also want to build the kernel with the sg > device which is another pass-thru device for compatibility with the > scsi generic linux driver. with sg_ses, take a look at the line for > subenclosures: > > #>sg_ses ses1 -p 1 > SUN CSM200_E 98C4 > Configuration diagnostic page: > number of secondary subenclosures: 0 > ^^^^^^^^^^^^^^^^^^^^^^^^ > > if you see 4 subenclosures, those scsi processor devices may be used > for internal enclosure management and communication with the ses0 > device, but thats only a guess on my part. you can also try using > sg_ses on the pass devices to see if they report enclosure services. > > also see the sg3_utils website, lots of detail on using the utilities. > > > Best, I had a similar problem with no /dev/sesX devices. As explained to me by Alan Somers my backplane in my enclosure did not have a SEP. "A SEP is a microcontroller, normally builtin to an expander SOC, that shows up as a SCSI target device. It's the device that talks the SES protocol, and it's what gets the /dev/sesX device node." Once I changed my backplane I then had /dev/ses0 and in my case could simply use sas2icru (LSI) with my supermicro backplane. Even sg3_utils did not help for me until I changed my backplane. http://lists.freebsd.org/pipermail/freebsd-scsi/2015-May/006713.html Hope this helps. --mikej From owner-freebsd-scsi@freebsd.org Wed Sep 9 15:38:21 2015 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 77BADA018DF for ; Wed, 9 Sep 2015 15:38:21 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-oi0-x230.google.com (mail-oi0-x230.google.com [IPv6:2607:f8b0:4003:c06::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 410EE1B24; Wed, 9 Sep 2015 15:38:21 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by oiev17 with SMTP id v17so7819890oie.1; Wed, 09 Sep 2015 08:38:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=EJMDWzIVIY3rrtBNiP0GS/xsDn9F8jMTbAM+vtwbgNU=; b=qUts5cjazpVv//HPLMb2MTH+yanUsqS7e8fwIJWkeaIz7Da0J6gT1FQR3t52ltzxNO 4P0Wz4ksYuazWotrLUuBEDtC9X/rpxelKFQ/5bbvb8gxHic18GfVwS1sZ7ko+y+TEuwy BVtaROfI9Y+NwpQtlpy2M3yncvfi89HZtntb3xC6cTWVtSAx4KhbCr8eS+n+20r4bFqi uQVVcbpP+f9kPo/Z9pRff1ZD+BX4kr+by6oNZEfYkPzc+d7h9Y5mNkuyAtXVtHnhRJha g8cprSqgCKk8JBQxL42hBCFed+D5BVVcmAxgcAeaG6T4Z/6ugApo5oF0ELmzHPY+jc1p zVWQ== MIME-Version: 1.0 X-Received: by 10.202.212.73 with SMTP id l70mr25711589oig.54.1441813100601; Wed, 09 Sep 2015 08:38:20 -0700 (PDT) Sender: asomers@gmail.com Received: by 10.202.79.140 with HTTP; Wed, 9 Sep 2015 08:38:20 -0700 (PDT) In-Reply-To: <20150909033510.GA34413@FreeBSD.org> References: <20150909033510.GA34413@FreeBSD.org> Date: Wed, 9 Sep 2015 09:38:20 -0600 X-Google-Sender-Auth: byoRRni8VxAaD5HSXwScT5KLjbw Message-ID: Subject: Re: ses/pass devices (enclosure/processor devices) not all showing up? From: Alan Somers To: John De Boskey Cc: FreeBSD-SCSI Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Sep 2015 15:38:21 -0000 On Tue, Sep 8, 2015 at 9:35 PM, John De Boskey wrote: > Hi Folks - > > I have a shelf with 84 sata drives. All drives show up > correctly and are accessible. The shelf appears to have > multiple processor devices and one enclosure device internally. > For instance: > > # camcontrol devlist | grep XYRATEX > at scbus7 target 159 lun 0 (pass18) > at scbus7 target 188 lun 0 (pass47) > at scbus7 target 217 lun 0 (pass76) > at scbus7 target 232 lun 0 (pass91) > > # camcontrol devlist | grep ses > at scbus7 target 144 lun 0 (ses0,pass3) > > # camcontrol smprg pass18 | grep 'Number of Phys:' > Number of Phys: 25 > # camcontrol smprg pass47 | grep 'Number of Phys:' > Number of Phys: 37 > # camcontrol smprg pass76 | grep 'Number of Phys:' > Number of Phys: 37 > # camcontrol smprg pass91 | grep 'Number of Phys:' > Number of Phys: 25 > # camcontrol smprg ses0 | grep 'Number of Phys:' > Number of Phys: 37 > > # camcontrol smpphylist pass18 > 25 PHYs: > PHY Attached SAS Address > 0 0x5000c500585f4b52 (pass4,da0) > 1 0x5000c500585f4476 (pass5,da1) > 2 0x5000c500585f6efe (pass6,da2) > 3 0x5000c500585f14d6 (pass7,da3) > 4 0x5000c500585f683e (pass8,da4) > 5 0x5000c500585f15e6 (pass9,da5) > 6 0x5000c500585f49ba (pass10,da6) > 7 0x5000c500585f14ea (pass11,da7) > 8 0x5000c500585f5ec6 (pass12,da8) > 9 0x5000c500585f0d12 (pass13,da9) > 10 0x5000c500585f620a (pass14,da10) > 11 0x5000c500585f0d1e (pass15,da11) > 12 0x5000c500585f64ae (pass16,da12) > 13 0x5000c500585f27aa (pass17,da13) > 14 0x0000000000000000 > 15 0x0000000000000000 > 16 0x0000000000000000 > 17 0x0000000000000000 > 18 0x0000000000000000 > 19 0x0000000000000000 > 20 0x50050cc1177513bf > 21 0x50050cc1177513bf > 22 0x50050cc1177513bf > 23 0x50050cc1177513bf > 24 0x50050cc10d2fe7be > > > (sas2ircu processor devices) > > # camcontrol smpphylist pass18 -q | wc -l > 14 > # camcontrol smpphylist pass47 -q | wc -l > 28 > # camcontrol smpphylist pass76 -q | wc -l > 28 > # camcontrol smpphylist pass91 -q | wc -l > 14 > > Which adds up to the 84 expected drives. No devices show up under ses0: > > (sas2ircu enclosure services device) > > # camcontrol smpphylist ses0 -q | wc -l > 0 > > I don't know that I understand the hierarchy of the backplane attachments. > Should the 4 pass only processor devices actually be ses devices? Or should > they be linked up under ses0 and all the devices show up under ses0? > > Any thoughts or ideas on how to fix this up(debug) would be appreciated. > > I have linked dmesg, camcontrol devlist, and sas2ircu output below: > > http://people.freebsd.org/~jwd/sespass/dmesg.txt mps messaging enabled. > > http://people.freebsd.org/~jwd/sespass/devlist.txt > > http://people.freebsd.org/~jwd/sespass/sas2ircu.txt > > Thanks, > John I'm not sure exactly what you're asking. Are you just wondering why your pass devices don't also have ses nodes? I think I know why. Your JBOD problem has five SAS expander chips, though ses0 might actually be some other kind of SAS target chip. pass18, pass47, pass76, and pass91 are configured to report ses0's SAS Address as the address of their SEP. The LSI HBA's interpretation is that there is only one SES processor. So FreeBSD reports one ses device, and the other expanders just show up as pass devices. Unless the manufacturer royally screwed up (I doubt it), ses0 will report info for all 84 disks in its SES status pages. Bill is right, sg3_utils is your friend. However, I doubt you'll see any secondary subenclosures. That feature isn't much used, and it's not necessary in order for ses0 to report all 84 drives. Running these two commands will probably tell you most of what you need to know: sg_ses -p 1 ses0 sg_ses -p 2 ses0 -Alan From owner-freebsd-scsi@freebsd.org Wed Sep 9 23:17:56 2015 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EAE02A01BB9 for ; Wed, 9 Sep 2015 23:17:56 +0000 (UTC) (envelope-from jwd@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id CEA1E1BC0; Wed, 9 Sep 2015 23:17:56 +0000 (UTC) (envelope-from jwd@freebsd.org) Received: by freefall.freebsd.org (Postfix, from userid 821) id CD70A1B8E; Wed, 9 Sep 2015 23:17:56 +0000 (UTC) Date: Wed, 9 Sep 2015 23:17:56 +0000 From: John De Boskey To: Alan Somers Cc: FreeBSD-SCSI Subject: Re: ses/pass devices (enclosure/processor devices) not all showing up? Message-ID: <20150909231756.GA88399@FreeBSD.org> References: <20150909033510.GA34413@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Sep 2015 23:17:57 -0000 ----- Alan Somers's Original Message ----- > On Tue, Sep 8, 2015 at 9:35 PM, John De Boskey wrote: > > Hi Folks - > > > > I have a shelf with 84 sata drives. All drives show up > > correctly and are accessible. The shelf appears to have > > multiple processor devices and one enclosure device internally. > > For instance: > > > > # camcontrol devlist | grep XYRATEX > > at scbus7 target 159 lun 0 (pass18) > > at scbus7 target 188 lun 0 (pass47) > > at scbus7 target 217 lun 0 (pass76) > > at scbus7 target 232 lun 0 (pass91) > > > > # camcontrol devlist | grep ses > > at scbus7 target 144 lun 0 (ses0,pass3) > > > > # camcontrol smprg pass18 | grep 'Number of Phys:' > > Number of Phys: 25 > > # camcontrol smprg pass47 | grep 'Number of Phys:' > > Number of Phys: 37 > > # camcontrol smprg pass76 | grep 'Number of Phys:' > > Number of Phys: 37 > > # camcontrol smprg pass91 | grep 'Number of Phys:' > > Number of Phys: 25 > > # camcontrol smprg ses0 | grep 'Number of Phys:' > > Number of Phys: 37 > > > > # camcontrol smpphylist pass18 > > 25 PHYs: > > PHY Attached SAS Address > > 0 0x5000c500585f4b52 (pass4,da0) ... > > I have linked dmesg, camcontrol devlist, and sas2ircu output below: > > > > http://people.freebsd.org/~jwd/sespass/dmesg.txt mps messaging enabled. > > > > http://people.freebsd.org/~jwd/sespass/devlist.txt > > > > http://people.freebsd.org/~jwd/sespass/sas2ircu.txt > > > > Thanks, > > John Hi Alan, > I'm not sure exactly what you're asking. Are you just wondering why In a simple form, I'm trying to figure out why 'camcontrol smpphyslist ses0' produces no output, while 'camcontrol smpphyslist pass18' does. > your pass devices don't also have ses nodes? I think I know why. > Your JBOD problem has five SAS expander chips, though ses0 might > actually be some other kind of SAS target chip. pass18, pass47, > pass76, and pass91 are configured to report ses0's SAS Address as the > address of their SEP. The LSI HBA's interpretation is that there is Could you expand on the concept of the SAS Address being reported? When I look at the address of the devices in the sas2ircu output they are unique. For example: Device is a Processor device Enclosure # : 2 Slot # : 1 SAS Address : 50050cc-1-0d2f-fabe State : Standby (SBY) Manufacturer : XYRATEX Model Number : DEFAULT-SD-R24 Firmware Revision : 3034 Serial No : PMCSIERA Device is a Processor device Enclosure # : 2 Slot # : 2 SAS Address : 50050cc-1-0d2f-fafe State : Standby (SBY) Manufacturer : XYRATEX Model Number : DEFAULT-SD-R36 Firmware Revision : 3034 Serial No : PMCSIERA ie: they show as unique. However, they don't appear to be unique as reported by camcontrol smprg. I haven't been able to determine where in the code this linkage is being done yet. > only one SES processor. So FreeBSD reports one ses device, and the > other expanders just show up as pass devices. Unless the manufacturer > royally screwed up (I doubt it), ses0 will report info for all 84 > disks in its SES status pages. camcontrol smpphylist ses0 - returns no information. camcontrol smpphylist pass18 - returns a drive list camcontrol smpphylist pass47 - returns a drive list camcontrol smpphylist pass76 - returns a drive list camcontrol smpphylist pass91 - returns a drive list > Bill is right, sg3_utils is your friend. However, I doubt you'll see > any secondary subenclosures. That feature isn't much used, and it's > not necessary in order for ses0 to report all 84 drives. Running > these two commands will probably tell you most of what you need to > know: > > sg_ses -p 1 ses0 > sg_ses -p 2 ses0 I have run the above commands and linked the output below. http://people.freebsd.org/~jwd/sespass/sgsesp1.txt http://people.freebsd.org/~jwd/sespass/sgsesp2.txt descriptor for each disk Thoughts? Thanks, John > -Alan From owner-freebsd-scsi@freebsd.org Wed Sep 9 23:42:37 2015 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 65BD0A0091E for ; Wed, 9 Sep 2015 23:42:37 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-oi0-x242.google.com (mail-oi0-x242.google.com [IPv6:2607:f8b0:4003:c06::242]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2CCC619C2; Wed, 9 Sep 2015 23:42:37 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by oibi136 with SMTP id i136so1411789oib.3; Wed, 09 Sep 2015 16:42:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=vDZNKT+1SrvCs9XWEmqmyxD2pbq4marWbrNxrkUn9vI=; b=RVdu23ZZJD6EKY7CMkENnCPDtmRwPr2QlnEgCqMUfes7x+U2yRFccrxCLjB5ejI+tr MvoCanQsdecYAwjD1hWgDrm5Bs8Nm7cl2sN/izxOr/Xsy7Z+AvwJzbThLa7dMHAgX+zo fZU9iLsjRnIolMObqgTGwbpT5ne/0WxeN/FUYIDUPeDEZU9VyDqTTOp37ygvdeXtdENl yLfnYN8UKZPKg7LaNNhzx1pyBiB8ECDUXSVu20NtUnM7cYcRijCxR080wlMrHztIuHca EWvH381wmn2CwQyzPS4942jbiEi3oP0qMquyziab8p1mRDvSDPxx3VNRvOVNLGrYokxX 64Pg== MIME-Version: 1.0 X-Received: by 10.202.215.68 with SMTP id o65mr5543028oig.100.1441842156278; Wed, 09 Sep 2015 16:42:36 -0700 (PDT) Sender: asomers@gmail.com Received: by 10.202.79.140 with HTTP; Wed, 9 Sep 2015 16:42:36 -0700 (PDT) In-Reply-To: <20150909231756.GA88399@FreeBSD.org> References: <20150909033510.GA34413@FreeBSD.org> <20150909231756.GA88399@FreeBSD.org> Date: Wed, 9 Sep 2015 17:42:36 -0600 X-Google-Sender-Auth: YEYg87EG5f5cA6q-Cf532OxcIR0 Message-ID: Subject: Re: ses/pass devices (enclosure/processor devices) not all showing up? From: Alan Somers To: John De Boskey Cc: Alan Somers , FreeBSD-SCSI Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Sep 2015 23:42:37 -0000 On Wed, Sep 9, 2015 at 5:17 PM, John De Boskey wrote: > ----- Alan Somers's Original Message ----- >> On Tue, Sep 8, 2015 at 9:35 PM, John De Boskey wrote: >> > Hi Folks - >> > >> > I have a shelf with 84 sata drives. All drives show up >> > correctly and are accessible. The shelf appears to have >> > multiple processor devices and one enclosure device internally. >> > For instance: >> > >> > # camcontrol devlist | grep XYRATEX >> > at scbus7 target 159 lun 0 (pass18) >> > at scbus7 target 188 lun 0 (pass47) >> > at scbus7 target 217 lun 0 (pass76) >> > at scbus7 target 232 lun 0 (pass91) >> > >> > # camcontrol devlist | grep ses >> > at scbus7 target 144 lun 0 (ses0,pass3) >> > >> > # camcontrol smprg pass18 | grep 'Number of Phys:' >> > Number of Phys: 25 >> > # camcontrol smprg pass47 | grep 'Number of Phys:' >> > Number of Phys: 37 >> > # camcontrol smprg pass76 | grep 'Number of Phys:' >> > Number of Phys: 37 >> > # camcontrol smprg pass91 | grep 'Number of Phys:' >> > Number of Phys: 25 >> > # camcontrol smprg ses0 | grep 'Number of Phys:' >> > Number of Phys: 37 >> > >> > # camcontrol smpphylist pass18 >> > 25 PHYs: >> > PHY Attached SAS Address >> > 0 0x5000c500585f4b52 (pass4,da0) > > ... > >> > I have linked dmesg, camcontrol devlist, and sas2ircu output below: >> > >> > http://people.freebsd.org/~jwd/sespass/dmesg.txt mps messaging enabled. >> > >> > http://people.freebsd.org/~jwd/sespass/devlist.txt >> > >> > http://people.freebsd.org/~jwd/sespass/sas2ircu.txt >> > >> > Thanks, >> > John > > Hi Alan, > >> I'm not sure exactly what you're asking. Are you just wondering why > > In a simple form, I'm trying to figure out why 'camcontrol smpphyslist ses0' > produces no output, while 'camcontrol smpphyslist pass18' does. > >> your pass devices don't also have ses nodes? I think I know why. >> Your JBOD problem has five SAS expander chips, though ses0 might >> actually be some other kind of SAS target chip. pass18, pass47, >> pass76, and pass91 are configured to report ses0's SAS Address as the >> address of their SEP. The LSI HBA's interpretation is that there is > > Could you expand on the concept of the SAS Address being reported? > When I look at the address of the devices in the sas2ircu output they > are unique. For example: Each expander IC actually contains several logically independent devices. First there is the expander itself. FreeBSD doesn't have any visibility into that. But you can see the address if you do "sg_ses -p 10 ses0" and look for "attached SAS address". In your case, you should see four unique attached SAS addresses. Second, there is the SEP (SCSI Enclosure Processor). This is the SCSI target that ends up getting a device node in FreeBSD. Third, the expander will automatically generate a SAS address for any attached SATA devices. Now, I don't understand all the details, but somehow your four slave expanders are telling your HBA that their disks are managed by the SEP whose SAS address is the address of ses0. As you've seen, that information is not contained with the SMP Report General response. It's probably some other SMP command, but I don't know which. > > Device is a Processor device > Enclosure # : 2 > Slot # : 1 > SAS Address : 50050cc-1-0d2f-fabe > State : Standby (SBY) > Manufacturer : XYRATEX > Model Number : DEFAULT-SD-R24 > Firmware Revision : 3034 > Serial No : PMCSIERA > > Device is a Processor device > Enclosure # : 2 > Slot # : 2 > SAS Address : 50050cc-1-0d2f-fafe > State : Standby (SBY) > Manufacturer : XYRATEX > Model Number : DEFAULT-SD-R36 > Firmware Revision : 3034 > Serial No : PMCSIERA > > ie: they show as unique. However, they don't appear to be unique > as reported by camcontrol smprg. I haven't been able to determine > where in the code this linkage is being done yet. > > >> only one SES processor. So FreeBSD reports one ses device, and the >> other expanders just show up as pass devices. Unless the manufacturer >> royally screwed up (I doubt it), ses0 will report info for all 84 >> disks in its SES status pages. > > camcontrol smpphylist ses0 - returns no information. > camcontrol smpphylist pass18 - returns a drive list > camcontrol smpphylist pass47 - returns a drive list > camcontrol smpphylist pass76 - returns a drive list > camcontrol smpphylist pass91 - returns a drive list > >> Bill is right, sg3_utils is your friend. However, I doubt you'll see >> any secondary subenclosures. That feature isn't much used, and it's >> not necessary in order for ses0 to report all 84 drives. Running >> these two commands will probably tell you most of what you need to >> know: >> >> sg_ses -p 1 ses0 >> sg_ses -p 2 ses0 > > I have run the above commands and linked the output below. > > http://people.freebsd.org/~jwd/sespass/sgsesp1.txt > > http://people.freebsd.org/~jwd/sespass/sgsesp2.txt descriptor for each disk > > Thoughts? It's as I expected. ses0 is reporting all 84 disk slots. You can use sg_ses on ses0 for pretty much all drive management tasks. The only reason to use smpphylist and smppc, IMHO, is to disable a drive's phy. That's mostly useful for simulating a drive pull, or for disabling a device that is broken and doesn't probe cleanly. -Alan > > Thanks, > John > >> -Alan