From owner-freebsd-scsi@freebsd.org Tue May 30 17:31:35 2017 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F3940B94682 for ; Tue, 30 May 2017 17:31:34 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-yw0-x22f.google.com (mail-yw0-x22f.google.com [IPv6:2607:f8b0:4002:c05::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A3EA07121E for ; Tue, 30 May 2017 17:31:34 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-yw0-x22f.google.com with SMTP id l14so43268439ywk.1 for ; Tue, 30 May 2017 10:31:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc:content-transfer-encoding; bh=46QHQby/uSLGS83XzIsVuH50jd0pXVRCrF/kbrbOSsM=; b=nNCaN89pSJdw7Q684cMQMcPyFJ54ARkFESBUNiUVvuFRYnjP0+MMTpZXLsB6usRoms he6BbYd/7KsIp5wIz/vUd/Nkyr8FUME7SAGPTZpp6JwmPwd1xjd+bi/4lJi1rUkxagB2 Zlft3k8SHZ4b6kHI6AuJ50VicvKALhxKrOC9IqQ3AZaZSEC6ylNiDc8gkjMsyEawqQkG cbm2xHIBVOM+XZXyNdSd9JkW2JxrQep6c5+013CH+y9GAghBwK6VqARarT9/DRTW88a4 gm+HRJnJnTnZZ8K3DuOvqzZ3gUyLJk3RDr7Rg9dQTasJ8hD4OTBDyQGOzXKemMAqyqBl 7dFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc:content-transfer-encoding; bh=46QHQby/uSLGS83XzIsVuH50jd0pXVRCrF/kbrbOSsM=; b=U6JrvtoYYLzTNv5atZ2KKb3TN8sOkmQ/N8Vn8ysn55gYd3kjcZmRvZJN8oOsdmIvc5 7X/aKFB/+KGNJMhDiQw2Dycoj1LZVrodQObhC6sUb/FFkTZViUvHz3oRNrMjqrZULcpO pEw4dclaKn4Ci0/sjXUL2kVdNhZMZBjjoM9PjAmdtEi+RcAltEpBlScvMmdHVEE2aLYh kJWEUJjGatu9GJDRkm69uMMzwhHXY6PF6pAP1XM1+LxYrvpobxrRptrGqZGiUmEz2LBS UyiialCrveqzikwZpj+gin/VpJmaJOni0kBg+2wt6ovIiRmLVCvMRoUvqa9D2xUGR37c Kzmg== X-Gm-Message-State: AODbwcC5wsD6uu9I+d/ckmidFa6WZw7CizmRhtWAitrYIwJ762WN1nGR LbN6t0lTGbi3dcbp6dL0vCC26EHdVA== X-Received: by 10.129.89.85 with SMTP id n82mr16616828ywb.94.1496165493615; Tue, 30 May 2017 10:31:33 -0700 (PDT) MIME-Version: 1.0 Sender: asomers@gmail.com Received: by 10.13.206.199 with HTTP; Tue, 30 May 2017 10:31:33 -0700 (PDT) In-Reply-To: <18E31C87-AB53-491F-9E40-F496AE31E305@ebureau.com> References: <18E31C87-AB53-491F-9E40-F496AE31E305@ebureau.com> From: Alan Somers Date: Tue, 30 May 2017 11:31:33 -0600 X-Google-Sender-Auth: 4a_mFTvjPUPL_n9wzrKRm6k4eZw Message-ID: Subject: Re: Inferring SAS expander topology To: Dustin Wenz Cc: FreeBSD-scsi Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 30 May 2017 17:31:35 -0000 On Tue, May 30, 2017 at 11:00 AM, Dustin Wenz wrot= e: > I'm working on a server that has 6 LSI SAS expanders connected to it. Two= of them are throwing intermittent command errors and need some attention. = Because the host only has four external 8088 ports, I know some of the expa= nders are chained through each other. > > Before I can send parts for replacement, I need to determine if the two p= roblematic expanders are either daisy-chained or directly connected to the = host. On Linux, this is trivial; I would just browse the device topology in= /sys/. However, this machine is running FreeBSD 10.3, and I am unable to f= ind a way to do this. I have sg3_utils available, as well as any build-in F= reeBSD tools. Using this software, is it possible to determine if a SAS exp= ander is chained off of another? > > - .Dustin I don't think sg3_utils will help you. You want sysutils/smp_utils instead. If you install that and then run "smp_discover /dev/ses0", it will show you what each phy is connected to. Usually, there will be four phys connected to the upstream port. You can tell by their EUI64s whether they're connected directly to the HBA or to another expander. And you can tell which expander by comparing the exact EUI64 to each other expander's SEP phy. Note that with LSI expanders, the SEP's address usually differs from the expander's address by a few bits in the last byte. For example, if the expander's address is 0x50000000000000ff, then the SEP's address might by 0x50000000000000ffd. Here's some example output from one of my systems: # smp_discover /dev/ses3 phy 0:U:attached:[500093d23000a000:00 t(SATA)] 6 Gbps phy 1:U:attached:[500093d23000a001:00 t(SATA)] 6 Gbps phy 2:U:attached:[500093d23000a002:00 t(SATA)] 6 Gbps phy 3:U:attached:[500093d23000a003:00 t(SATA)] 6 Gbps phy 4:U:attached:[500093d23000a17f:11 exp t(SMP)] 6 Gbps <- Connected to the other expander phy 5:U:attached:[500093d23000a17f:10 exp t(SMP)] 6 Gbps <- Connected to the other expander phy 6:U:attached:[500093d23000a17f:09 exp t(SMP)] 6 Gbps <- Connected to the other expander phy 7:U:attached:[500093d23000a17f:08 exp t(SMP)] 6 Gbps <- Connected to the other expander phy 8:U:attached:[500093d23000a1bf:11 exp t(SMP)] 6 Gbps <- Connected to a third expander phy 9:U:attached:[500093d23000a1bf:10 exp t(SMP)] 6 Gbps <- Connected to a third expander phy 10:U:attached:[500093d23000a1bf:09 exp t(SMP)] 6 Gbps <- Connected to a third expander phy 11:U:attached:[500093d23000a1bf:08 exp t(SMP)] 6 Gbps <- Connected to a third expander phy 16:U:attached:[500093d23000a010:00 t(SATA)] 6 Gbps phy 17:U:attached:[500093d23000a011:00 t(SATA)] 6 Gbps phy 18:U:attached:[500093d23000a012:00 t(SATA)] 6 Gbps phy 19:U:attached:[500093d23000a013:00 t(SATA)] 6 Gbps phy 20:U:attached:[500093d23000a014:00 t(SATA)] 6 Gbps phy 21:U:attached:[500093d23000a015:00 t(SATA)] 6 Gbps phy 22:U:attached:[500093d23000a016:00 t(SATA)] 6 Gbps phy 23:U:attached:[500093d23000a017:00 t(SATA)] 6 Gbps phy 25:U:attached:[500093d23000a019:00 t(SATA)] 6 Gbps phy 26:U:attached:[500093d23000a01a:00 t(SATA)] 6 Gbps phy 29:U:attached:[500093d23000a01d:00 t(SATA)] 6 Gbps phy 30:U:attached:[500093d23000a01e:00 t(SATA)] 6 Gbps phy 31:U:attached:[500093d23000a01f:00 t(SATA)] 6 Gbps phy 33:U:attached:[500093d23000a021:00 t(SATA)] 6 Gbps phy 34:U:attached:[500093d23000a022:00 t(SATA)] 6 Gbps phy 35:U:attached:[500093d23000a023:00 t(SATA)] 6 Gbps phy 36:U:attached:[500093d23000a024:00 t(SATA)] 6 Gbps phy 37:U:attached:[500093d23000a025:00 t(SATA)] 6 Gbps phy 38:U:attached:[500093d23000a026:00 t(SATA)] 6 Gbps phy 39:U:attached:[500093d23000a027:00 t(SATA)] 6 Gbps phy 40:U:attached:[500605b008a93990:04 i(SSP+STP+SMP)] 6 Gbps <- Connected to an LSI HBA phy 41:U:attached:[500605b008a93990:07 i(SSP+STP+SMP)] 6 Gbps <- Connected to an LSI HBA phy 42:U:attached:[500605b008a93990:05 i(SSP+STP+SMP)] 6 Gbps <- Connected to an LSI HBA phy 43:U:attached:[500605b008a93990:06 i(SSP+STP+SMP)] 6 Gbps <- Connected to an LSI HBA phy 48:D:attached:[500093d23000a03d:00 V i(SMP) t(SSP)] 12 Gbps <- This expander's SEP # smp_discover /dev/pass121 phy 0:U:attached:[500093d23000a140:00 t(SATA)] 6 Gbps phy 1:U:attached:[500093d23000a141:00 t(SATA)] 6 Gbps phy 2:U:attached:[500093d23000a142:00 t(SATA)] 6 Gbps phy 3:U:attached:[500093d23000a143:00 t(SATA)] 6 Gbps phy 4:U:attached:[500093d23000a144:00 t(SATA)] 6 Gbps phy 5:U:attached:[500093d23000a145:00 t(SATA)] 6 Gbps phy 6:U:attached:[500093d23000a146:00 t(SATA)] 6 Gbps phy 7:U:attached:[500093d23000a147:00 t(SATA)] 6 Gbps phy 8:U:attached:[500093d23000a03f:07 exp t(SMP)] 6 Gbps <- Connected to the other expander phy 9:U:attached:[500093d23000a03f:06 exp t(SMP)] 6 Gbps <- Connected to the other expander phy 10:U:attached:[500093d23000a03f:05 exp t(SMP)] 6 Gbps <- Connected to the other expander phy 11:U:attached:[500093d23000a03f:04 exp t(SMP)] 6 Gbps <- Connected to the other expander phy 12:U:attached:[500093d23000a14c:00 t(SATA)] 6 Gbps phy 14:U:attached:[500093d23000a14e:00 t(SATA)] 6 Gbps phy 15:U:attached:[500093d23000a14f:00 t(SATA)] 6 Gbps phy 16:U:attached:[500093d23000a150:00 t(SATA)] 6 Gbps phy 17:U:attached:[500093d23000a151:00 t(SATA)] 6 Gbps phy 18:U:attached:[500093d23000a152:00 t(SATA)] 6 Gbps phy 19:U:attached:[500093d23000a153:00 t(SATA)] 6 Gbps phy 20:U:attached:[500093d23000a154:00 t(SATA)] 6 Gbps phy 21:U:attached:[500093d23000a155:00 t(SATA)] 6 Gbps phy 22:U:attached:[500093d23000a156:00 t(SATA)] 6 Gbps phy 23:U:attached:[500093d23000a157:00 t(SATA)] 6 Gbps phy 24:U:attached:[500093d23000a158:00 t(SATA)] 6 Gbps phy 25:U:attached:[500093d23000a159:00 t(SATA)] 6 Gbps phy 26:U:attached:[500093d23000a15a:00 t(SATA)] 6 Gbps phy 27:U:attached:[500093d23000a15b:00 t(SATA)] 6 Gbps phy 28:U:attached:[500093d23000a15c:00 t(SATA)] 6 Gbps phy 29:U:attached:[500093d23000a15d:00 t(SATA)] 6 Gbps phy 30:U:attached:[500093d23000a15e:00 t(SATA)] 6 Gbps phy 32:U:attached:[500093d23000a160:00 t(SATA)] 6 Gbps phy 33:U:attached:[500093d23000a161:00 t(SATA)] 6 Gbps phy 34:U:attached:[500093d23000a162:00 t(SATA)] 6 Gbps phy 35:U:attached:[500093d23000a163:00 t(SATA)] 6 Gbps phy 36:U:attached:[500093d23000a164:00 t(SATA)] 6 Gbps phy 37:U:attached:[500093d23000a165:00 t(SATA)] 6 Gbps phy 38:U:attached:[500093d23000a166:00 t(SATA)] 6 Gbps phy 39:U:attached:[500093d23000a167:00 t(SATA)] 6 Gbps phy 40:D:attached:[500093d23000a17d:00 V i(SMP) t(SSP)] 12 Gbps <- This expander's SEP -Alan