From owner-freebsd-scsi@freebsd.org Tue May 30 20:09:19 2017 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B8F27BEB3CB for ; Tue, 30 May 2017 20:09:19 +0000 (UTC) (envelope-from dustinwenz@ebureau.com) Received: from internet06.ebureau.com (internet06.ebureau.com [65.127.24.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "internet06.ebureau.com", Issuer "internet06.ebureau.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 8E79577692; Tue, 30 May 2017 20:09:19 +0000 (UTC) (envelope-from dustinwenz@ebureau.com) Received: from localhost (localhost [127.0.0.1]) by internet06.ebureau.com (Postfix) with ESMTP id 847B179188F8; Tue, 30 May 2017 15:09:17 -0500 (CDT) X-Virus-Scanned: amavisd-new at mydomain = ebureau.com Received: from internet06.ebureau.com ([127.0.0.1]) by localhost (internet06.ebureau.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id IVVt68QwbqZU; Tue, 30 May 2017 15:09:16 -0500 (CDT) Received: from square.office.ebureau.com (unknown [10.10.20.22]) by internet06.ebureau.com (Postfix) with ESMTPSA id ABF5779188EE; Tue, 30 May 2017 15:09:16 -0500 (CDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: Re: Inferring SAS expander topology From: Dustin Wenz In-Reply-To: Date: Tue, 30 May 2017 15:09:15 -0500 Cc: FreeBSD-scsi Content-Transfer-Encoding: quoted-printable Message-Id: <5124123C-B996-4DC2-A338-64ACC33A5C09@ebureau.com> References: <18E31C87-AB53-491F-9E40-F496AE31E305@ebureau.com> To: Alan Somers X-Mailer: Apple Mail (2.3273) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 30 May 2017 20:09:19 -0000 Thanks! It looks like smp_discover will do what I need, though is not = particularly user-friendly for this specific application.=20 I needed to use both "smp_discover sesX" in addition to "smp_discover = --phy=3DY --brief sesX" in order to correlate SAS interfaces with the = addresses of devices attached to them. - .Dustin > On May 30, 2017, at 12:31 PM, Alan Somers wrote: >=20 > On Tue, May 30, 2017 at 11:00 AM, Dustin Wenz = wrote: >> I'm working on a server that has 6 LSI SAS expanders connected to it. = Two of them are throwing intermittent command errors and need some = attention. Because the host only has four external 8088 ports, I know = some of the expanders are chained through each other. >>=20 >> Before I can send parts for replacement, I need to determine if the = two problematic expanders are either daisy-chained or directly connected = to the host. On Linux, this is trivial; I would just browse the device = topology in /sys/. However, this machine is running FreeBSD 10.3, and I = am unable to find a way to do this. I have sg3_utils available, as well = as any build-in FreeBSD tools. Using this software, is it possible to = determine if a SAS expander is chained off of another? >>=20 >> - .Dustin >=20 > I don't think sg3_utils will help you. You want sysutils/smp_utils > instead. If you install that and then run "smp_discover /dev/ses0", > it will show you what each phy is connected to. Usually, there will > be four phys connected to the upstream port. You can tell by their > EUI64s whether they're connected directly to the HBA or to another > expander. And you can tell which expander by comparing the exact > EUI64 to each other expander's SEP phy. Note that with LSI expanders, > the SEP's address usually differs from the expander's address by a few > bits in the last byte. For example, if the expander's address is > 0x50000000000000ff, then the SEP's address might by > 0x50000000000000ffd. Here's some example output from one of my > systems: >=20 > # smp_discover /dev/ses3 > phy 0:U:attached:[500093d23000a000:00 t(SATA)] 6 Gbps > phy 1:U:attached:[500093d23000a001:00 t(SATA)] 6 Gbps > phy 2:U:attached:[500093d23000a002:00 t(SATA)] 6 Gbps > phy 3:U:attached:[500093d23000a003:00 t(SATA)] 6 Gbps > phy 4:U:attached:[500093d23000a17f:11 exp t(SMP)] 6 Gbps > <- Connected to the other expander > phy 5:U:attached:[500093d23000a17f:10 exp t(SMP)] 6 Gbps > <- Connected to the other expander > phy 6:U:attached:[500093d23000a17f:09 exp t(SMP)] 6 Gbps > <- Connected to the other expander > phy 7:U:attached:[500093d23000a17f:08 exp t(SMP)] 6 Gbps > <- Connected to the other expander > phy 8:U:attached:[500093d23000a1bf:11 exp t(SMP)] 6 Gbps > <- Connected to a third expander > phy 9:U:attached:[500093d23000a1bf:10 exp t(SMP)] 6 Gbps > <- Connected to a third expander > phy 10:U:attached:[500093d23000a1bf:09 exp t(SMP)] 6 Gbps > <- Connected to a third expander > phy 11:U:attached:[500093d23000a1bf:08 exp t(SMP)] 6 Gbps > <- Connected to a third expander > phy 16:U:attached:[500093d23000a010:00 t(SATA)] 6 Gbps > phy 17:U:attached:[500093d23000a011:00 t(SATA)] 6 Gbps > phy 18:U:attached:[500093d23000a012:00 t(SATA)] 6 Gbps > phy 19:U:attached:[500093d23000a013:00 t(SATA)] 6 Gbps > phy 20:U:attached:[500093d23000a014:00 t(SATA)] 6 Gbps > phy 21:U:attached:[500093d23000a015:00 t(SATA)] 6 Gbps > phy 22:U:attached:[500093d23000a016:00 t(SATA)] 6 Gbps > phy 23:U:attached:[500093d23000a017:00 t(SATA)] 6 Gbps > phy 25:U:attached:[500093d23000a019:00 t(SATA)] 6 Gbps > phy 26:U:attached:[500093d23000a01a:00 t(SATA)] 6 Gbps > phy 29:U:attached:[500093d23000a01d:00 t(SATA)] 6 Gbps > phy 30:U:attached:[500093d23000a01e:00 t(SATA)] 6 Gbps > phy 31:U:attached:[500093d23000a01f:00 t(SATA)] 6 Gbps > phy 33:U:attached:[500093d23000a021:00 t(SATA)] 6 Gbps > phy 34:U:attached:[500093d23000a022:00 t(SATA)] 6 Gbps > phy 35:U:attached:[500093d23000a023:00 t(SATA)] 6 Gbps > phy 36:U:attached:[500093d23000a024:00 t(SATA)] 6 Gbps > phy 37:U:attached:[500093d23000a025:00 t(SATA)] 6 Gbps > phy 38:U:attached:[500093d23000a026:00 t(SATA)] 6 Gbps > phy 39:U:attached:[500093d23000a027:00 t(SATA)] 6 Gbps > phy 40:U:attached:[500605b008a93990:04 i(SSP+STP+SMP)] 6 Gbps > <- Connected to an LSI HBA > phy 41:U:attached:[500605b008a93990:07 i(SSP+STP+SMP)] 6 Gbps > <- Connected to an LSI HBA > phy 42:U:attached:[500605b008a93990:05 i(SSP+STP+SMP)] 6 Gbps > <- Connected to an LSI HBA > phy 43:U:attached:[500605b008a93990:06 i(SSP+STP+SMP)] 6 Gbps > <- Connected to an LSI HBA > phy 48:D:attached:[500093d23000a03d:00 V i(SMP) t(SSP)] 12 Gbps > <- This expander's SEP >=20 > # smp_discover /dev/pass121 > phy 0:U:attached:[500093d23000a140:00 t(SATA)] 6 Gbps > phy 1:U:attached:[500093d23000a141:00 t(SATA)] 6 Gbps > phy 2:U:attached:[500093d23000a142:00 t(SATA)] 6 Gbps > phy 3:U:attached:[500093d23000a143:00 t(SATA)] 6 Gbps > phy 4:U:attached:[500093d23000a144:00 t(SATA)] 6 Gbps > phy 5:U:attached:[500093d23000a145:00 t(SATA)] 6 Gbps > phy 6:U:attached:[500093d23000a146:00 t(SATA)] 6 Gbps > phy 7:U:attached:[500093d23000a147:00 t(SATA)] 6 Gbps > phy 8:U:attached:[500093d23000a03f:07 exp t(SMP)] 6 Gbps > <- Connected to the other expander > phy 9:U:attached:[500093d23000a03f:06 exp t(SMP)] 6 Gbps > <- Connected to the other expander > phy 10:U:attached:[500093d23000a03f:05 exp t(SMP)] 6 Gbps > <- Connected to the other expander > phy 11:U:attached:[500093d23000a03f:04 exp t(SMP)] 6 Gbps > <- Connected to the other expander > phy 12:U:attached:[500093d23000a14c:00 t(SATA)] 6 Gbps > phy 14:U:attached:[500093d23000a14e:00 t(SATA)] 6 Gbps > phy 15:U:attached:[500093d23000a14f:00 t(SATA)] 6 Gbps > phy 16:U:attached:[500093d23000a150:00 t(SATA)] 6 Gbps > phy 17:U:attached:[500093d23000a151:00 t(SATA)] 6 Gbps > phy 18:U:attached:[500093d23000a152:00 t(SATA)] 6 Gbps > phy 19:U:attached:[500093d23000a153:00 t(SATA)] 6 Gbps > phy 20:U:attached:[500093d23000a154:00 t(SATA)] 6 Gbps > phy 21:U:attached:[500093d23000a155:00 t(SATA)] 6 Gbps > phy 22:U:attached:[500093d23000a156:00 t(SATA)] 6 Gbps > phy 23:U:attached:[500093d23000a157:00 t(SATA)] 6 Gbps > phy 24:U:attached:[500093d23000a158:00 t(SATA)] 6 Gbps > phy 25:U:attached:[500093d23000a159:00 t(SATA)] 6 Gbps > phy 26:U:attached:[500093d23000a15a:00 t(SATA)] 6 Gbps > phy 27:U:attached:[500093d23000a15b:00 t(SATA)] 6 Gbps > phy 28:U:attached:[500093d23000a15c:00 t(SATA)] 6 Gbps > phy 29:U:attached:[500093d23000a15d:00 t(SATA)] 6 Gbps > phy 30:U:attached:[500093d23000a15e:00 t(SATA)] 6 Gbps > phy 32:U:attached:[500093d23000a160:00 t(SATA)] 6 Gbps > phy 33:U:attached:[500093d23000a161:00 t(SATA)] 6 Gbps > phy 34:U:attached:[500093d23000a162:00 t(SATA)] 6 Gbps > phy 35:U:attached:[500093d23000a163:00 t(SATA)] 6 Gbps > phy 36:U:attached:[500093d23000a164:00 t(SATA)] 6 Gbps > phy 37:U:attached:[500093d23000a165:00 t(SATA)] 6 Gbps > phy 38:U:attached:[500093d23000a166:00 t(SATA)] 6 Gbps > phy 39:U:attached:[500093d23000a167:00 t(SATA)] 6 Gbps > phy 40:D:attached:[500093d23000a17d:00 V i(SMP) t(SSP)] 12 Gbps > <- This expander's SEP >=20 > -Alan