From owner-freebsd-scsi@FreeBSD.ORG Thu May 28 14:04:32 2015 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 53A2B18E for ; Thu, 28 May 2015 14:04:32 +0000 (UTC) (envelope-from mikej@mikej.com) Received: from mx2.paymentallianceintl.com (mx2.paymentallianceintl.com [216.26.158.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx2.paymentallianceintl.com", Issuer "Go Daddy Secure Certification Authority" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1658BCCB for ; Thu, 28 May 2015 14:04:31 +0000 (UTC) (envelope-from mikej@mikej.com) Received: from firewall.mikej.com (162-230-214-65.lightspeed.lsvlky.sbcglobal.net [162.230.214.65]) by mx2.paymentallianceintl.com (8.15.1/8.15.1) with ESMTPS id t4SDjxSM088856 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 28 May 2015 09:46:00 -0400 (EDT) (envelope-from mikej@mikej.com) X-Authentication-Warning: mx2.paymentallianceintl.com: Host 162-230-214-65.lightspeed.lsvlky.sbcglobal.net [162.230.214.65] claimed to be firewall.mikej.com Received: from mail.mikej.com (firewall [192.168.6.63]) by firewall.mikej.com (8.15.1/8.14.9) with ESMTP id t4SDjwmW032897 for ; Thu, 28 May 2015 09:45:59 -0400 (EDT) (envelope-from mikej@mikej.com) X-Authentication-Warning: firewall.mikej.com: Host firewall [192.168.6.63] claimed to be mail.mikej.com MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Thu, 28 May 2015 09:45:58 -0400 From: Michael Jung To: freebsd-scsi@freebsd.org Subject: /dev/ses not created Message-ID: <76e9e3f41990026290382e5fa9c9b8d3@mail.mikej.com> X-Sender: mikej@mikej.com User-Agent: Roundcube Webmail/1.0.4 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 May 2015 14:04:32 -0000 I hoping to get some direction as I simply am unsure what to do next. I have three LSI 9200-8e adapters connected to a Super Micro CSE-846A-R1200B yet no SES devices get created. The Backplane in the chassis BPN-SAS_846A does not have a expander. So I have six SFF-8088 on the backplane that connect directly to the ports on the 9200-8e albeit externally.The backplane does have a Marvel MG9072 supporting SES-2 and is jumpered for SGPIO. The server in question is running 9.3, but I have tested with 10.1-STABLE and 11-Current live CD's and I do not see /dev/ses created. I have also booted for testing openIndiana and do not see /dev/es/ses created. I have an open ticket with SM but that so far has them asking a lot of questions and giving no direction. On the LSI cards I have tested P16, P17 and P19 all in IT mode on 9.3. sas2icru on each of the adpapters does return output such as Enclosure information ------------------------------------------------------------------------ Enclosure# : 1 Logical ID : 500605b0:065e26b0 Numslots : 8 StartSlot : 0 ------------------------------------------------------------------------ What stands out is is not identefied for 'mps0' in dmesg. However, drives attached to all controllers do work fine and are identified. Full dmesg and output of sas2icru for each controller https://charon.gopai.com/dmesg Booting 9.3 verbosely with hw.mps.debug_level=0xd in /boot/loader.conf yeilds: mps1: port 0xcc00-0xccff mem 0xde8bc000-0xde8bffff,0xde8c0000-0xde8fffff irq 35 at device 0.0 on pci19 mps1: Firmware: 16.00.00.00, Driver: 16.00.00.00-fbsd mps1: IOCCapabilities: 1285c mps1: EventReply : mps1: EventReply : mps1: attempting to allocate 1 MSI-X vectors (15 supported) mps1: using IRQ 272 for MSI-X mps2: port 0xdc00-0xdcff mem 0xdeabc000-0xdeabffff,0xdeac0000-0xdeafffff irq 33 at device 0.0 on pci18 mps2: Firmware: 16.00.00.00, Driver: 16.00.00.00-fbsd mps2: IOCCapabilities: 1285c mps2: EventReply : mps2: EventReply : mps2: attempting to allocate 1 MSI-X vectors (15 supported) mps2: using IRQ 273 for MSI-X mps0: EventReply : mps0: EventReply : mps0: EventReply : mps0: EventReply : mps0: EventReply : mps0: EventReply : mps0: EventReply : mps0: EventReply : mps0: EventReply : mps0: EventReply : mps0: EventReply : mps0: EventReply : mps0: EventReply : mps0: EventReply : mps1: EventReply : mps1: EventReply : mps1: EventReply : mps1: EventReply : mps1: EventReply : mps1: EventReply : mps1: EventReply : mps1: EventReply : mps1: EventReply : mps1: EventReply : mps1: EventReply : mps1: EventReply : mps2: EventReply : mps2: EventReply : mps2: EventReply : mps2: EventReply : mps2: EventReply : mps2: EventReply : mps2: EventReply : mps2: EventReply : mps2: EventReply : mps2: EventReply : mps2: EventReply : mps2: EventReply : mps2: EventReply : mps2: EventReply : mps0: (0)->(mpssas_fw_work) Working on Event: [16] mps0: (1)->(mpssas_fw_work) Event Free: [16] mps0: (1)->(mpssas_fw_work) Working on Event: [1d] mps0: (2)->(mpssas_fw_work) Event Free: [1d] mps0: (2)->(mpssas_fw_work) Working on Event: [1c] mps1: (3)->(mpssas_fw_work) Working on Event: [16] mps1: (4)->(mpssas_fw_work) Event Free: [16] mps1: (4)->(mpssas_fw_work) Working on Event: [1d] mps1: (5)->(mpssas_fw_work) Event Free: [1d] mps1: (5)->(mpssas_fw_work) Working on Event: [1c] mps2: (6)->(mpssas_fw_work) Working on Event: [16] mps2: (7)->(mpssas_fw_work) Event Free: [16] mps2: (7)->(mpssas_fw_work) Working on Event: [1d] mps2: (8)->(mpssas_fw_work) Event Free: [1d] mps2: (8)->(mpssas_fw_work) Working on Event: [1c] (probe0:mps1:0:0:0): Down reving Protocol Version from 4 to 0? (probe1:mps1:0:1:0): Down reving Protocol Version from 4 to 0? (probe2:mps1:0:2:0): Down reving Protocol Version from 4 to 0? (probe3:mps1:0:4:0): Down reving Protocol Version from 4 to 0? (probe4:mps2:0:0:0): Down reving Protocol Version from 4 to 0? (probe5:mps2:0:1:0): Down reving Protocol Version from 4 to 0? mps1: (9)->(mpssas_fw_work) Event Free: [1c] mps1: (9)->(mpssas_fw_work) Working on Event: [16] mps1: (10)->(mpssas_fw_work) Event Free: [16] mps1: (10)->(mpssas_fw_work) Working on Event: [16] mps1: (11)->(mpssas_fw_work) Event Free: [16] mps1: (11)->(mpssas_fw_work) Working on Event: [16] mps1: (12)->(mpssas_fw_work) Event Free: [16] mps1: (12)->(mpssas_fw_work) Working on Event: [16] mps1: (13)->(mpssas_fw_work) Event Free: [16] mps1: (13)->(mpssas_fw_work) Working on Event: [16] mps1: (14)->(mpssas_fw_work) Event Free: [16] mps1: (14)->(mpssas_fw_work) Working on Event: [16] mps1: (15)->(mpssas_fw_work) Event Free: [16] mps1: (15)->(mpssas_fw_work) Working on Event: [16] mps1: (16)->(mpssas_fw_work) Event Free: [16] mps1: (16)->(mpssas_fw_work) Working on Event: [16] mps1: (17)->(mpssas_fw_work) Event Free: [16] mps1: (17)->(mpssas_fw_work) Working on Event: [16] mps1: (18)->(mpssas_fw_work) Event Free: [16] (probe6:mps1:0:5:0): Down reving Protocol Version from 4 to 0? (probe7:mps2:0:2:0): Down reving Protocol Version from 4 to 0? (probe8:mps2:0:6:0): Down reving Protocol Version from 4 to 0? (probe9:mps2:0:4:0): Down reving Protocol Version from 4 to 0? mps2: (18)->(mpssas_fw_work) Event Free: [1c] mps2: (18)->(mpssas_fw_work) Working on Event: [16] mps2: (19)->(mpssas_fw_work) Event Free: [16] mps2: (19)->(mpssas_fw_work) Working on Event: [16] mps2: (20)->(mpssas_fw_work) Event Free: [16] mps2: (20)->(mpssas_fw_work) Working on Event: [16] mps2: (21)->(mpssas_fw_work) Event Free: [16] mps2: (21)->(mpssas_fw_work) Working on Event: [16] mps2: (22)->(mpssas_fw_work) Event Free: [16] mps2: (22)->(mpssas_fw_work) Working on Event: [16] mps2: (23)->(mpssas_fw_work) Event Free: [16] mps2: (23)->(mpssas_fw_work) Working on Event: [16] mps2: (24)->(mpssas_fw_work) Event Free: [16] mps2: (24)->(mpssas_fw_work) Working on Event: [16] mps2: (25)->(mpssas_fw_work) Event Free: [16] mps2: (25)->(mpssas_fw_work) Working on Event: [16] mps2: (26)->(mpssas_fw_work) Event Free: [16] mps2: (26)->(mpssas_fw_work) Working on Event: [16] mps2: (27)->(mpssas_fw_work) Event Free: [16] mps2: (27)->(mpssas_fw_work) Working on Event: [16] mps2: (28)->(mpssas_fw_work) Event Free: [16] mps2: (28)->(mpssas_fw_work) Working on Event: [16] mps2: (29)->(mpssas_fw_work) Event Free: [16] (probe0:mps2:0:5:0): Down reving Protocol Version from 4 to 0? (probe1:mps0:0:0:0): Down reving Protocol Version from 4 to 0? (probe2:mps0:0:1:0): Down reving Protocol Version from 4 to 0? (probe3:mps0:0:2:0): Down reving Protocol Version from 4 to 0? (probe5:mps0:0:3:0): Down reving Protocol Version from 4 to 0? (probe6:mps0:0:4:0): Down reving Protocol Version from 4 to 0? mps0: (29)->(mpssas_fw_work) Event Free: [1c] mps0: (29)->(mpssas_fw_work) Working on Event: [16] mps0: (30)->(mpssas_fw_work) Event Free: [16] mps0: (30)->(mpssas_fw_work) Working on Event: [16] mps0: (31)->(mpssas_fw_work) Event Free: [16] mps0: (31)->(mpssas_fw_work) Working on Event: [16] mps0: (32)->(mpssas_fw_work) Event Free: [16] mps0: (32)->(mpssas_fw_work) Working on Event: [16] mps0: (33)->(mpssas_fw_work) Event Free: [16] mps0: (33)->(mpssas_fw_work) Working on Event: [16] mps0: (34)->(mpssas_fw_work) Event Free: [16] mps0: (34)->(mpssas_fw_work) Working on Event: [16] mps0: (35)->(mpssas_fw_work) Event Free: [16] mps0: (35)->(mpssas_fw_work) Working on Event: [16] mps0: (36)->(mpssas_fw_work) Event Free: [16] mps0: (36)->(mpssas_fw_work) Working on Event: [16] mps0: (37)->(mpssas_fw_work) Event Free: [16] mps0: (37)->(mpssas_fw_work) Working on Event: [16] mps0: (38)->(mpssas_fw_work) Event Free: [16] mps0: (38)->(mpssas_fw_work) Working on Event: [16] mps0: (39)->(mpssas_fw_work) Event Free: [16] mps0: (39)->(mpssas_fw_work) Working on Event: [16] mps0: (40)->(mpssas_fw_work) Event Free: [16] (probe0:mps0:0:6:0): Down reving Protocol Version from 4 to 0? da10 at mps0 bus 0 scbus0 target 0 lun 0 da0 at mps0 bus 0 scbus0 target 2 lun 0 da1 at mps0 bus 0 scbus0 target 3 lun 0 da11 at mps0 bus 0 scbus0 target 1 lun 0 da2 at mps0 bus 0 scbus0 target 4 lun 0 da18 at mps1 bus 0 scbus1 target 0 lun 0 da19 at mps1 bus 0 scbus1 target 1 lun 0 da20 at mps1 bus 0 scbus1 target 2 lun 0 da23 at mps1 bus 0 scbus1 target 5 lun 0 da14 at mps2 bus 0 scbus2 target 0 lun 0 da15 at mps2 bus 0 scbus2 target 1 lun 0 da16 at mps2 bus 0 scbus2 target 2 lun 0 da22 at mps1 bus 0 scbus1 target 4 lun 0 da6 at mps2 bus 0 scbus2 target 4 lun 0 pass0 at mps0 bus 0 scbus0 target 0 lun 0 da8 at mps2 bus 0 scbus2 target 6 lun 0 pass1 at mps0 bus 0 scbus0 target 1 lun 0 pass2 at mps0 bus 0 scbus0 target 2 lun 0 pass3 at mps0 bus 0 scbus0 target 3 lun 0 pass4 at mps0 bus 0 scbus0 target 4 lun 0 pass5 at mps0 bus 0 scbus0 target 6 lun 0 pass6 at mps1 bus 0 scbus1 target 0 lun 0 pass7 at mps1 bus 0 scbus1 target 1 lun 0 da7 at mps2 bus 0 scbus2 target 5 lun 0 pass8 at mps1 bus 0 scbus1 target 2 lun 0 pass9 at mps1 bus 0 scbus1 target 4 lun 0 pass10 at mps1 bus 0 scbus1 target 5 lun 0 pass11 at mps2 bus 0 scbus2 target 0 lun 0 pass12 at mps2 bus 0 scbus2 target 1 lun 0 pass13 at mps2 bus 0 scbus2 target 2 lun 0 pass14 at mps2 bus 0 scbus2 target 4 lun 0 pass15 at mps2 bus 0 scbus2 target 5 lun 0 pass16 at mps2 bus 0 scbus2 target 6 lun 0 da4 at mps0 bus 0 scbus0 target 6 lun 0 I admit I am new to enclosure management - What am I missing? Thank you. --mikej From owner-freebsd-scsi@FreeBSD.ORG Thu May 28 14:05:29 2015 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5E4051C8 for ; Thu, 28 May 2015 14:05:29 +0000 (UTC) (envelope-from mikej@mikej.com) Received: from mx2.paymentallianceintl.com (mx2.paymentallianceintl.com [216.26.158.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx2.paymentallianceintl.com", Issuer "Go Daddy Secure Certification Authority" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 36085CDC for ; Thu, 28 May 2015 14:05:28 +0000 (UTC) (envelope-from mikej@mikej.com) Received: from firewall.mikej.com (162-230-214-65.lightspeed.lsvlky.sbcglobal.net [162.230.214.65]) by mx2.paymentallianceintl.com (8.15.1/8.15.1) with ESMTPS id t4SE5QHF091169 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 28 May 2015 10:05:27 -0400 (EDT) (envelope-from mikej@mikej.com) X-Authentication-Warning: mx2.paymentallianceintl.com: Host 162-230-214-65.lightspeed.lsvlky.sbcglobal.net [162.230.214.65] claimed to be firewall.mikej.com Received: from mail.mikej.com (firewall [192.168.6.63]) by firewall.mikej.com (8.15.1/8.14.9) with ESMTP id t4SE5Qrj036874 for ; Thu, 28 May 2015 10:05:26 -0400 (EDT) (envelope-from mikej@mikej.com) X-Authentication-Warning: firewall.mikej.com: Host firewall [192.168.6.63] claimed to be mail.mikej.com MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Thu, 28 May 2015 10:05:26 -0400 From: Michael Jung To: freebsd-scsi@freebsd.org Subject: Re: /dev/ses not created In-Reply-To: <76e9e3f41990026290382e5fa9c9b8d3@mail.mikej.com> References: <76e9e3f41990026290382e5fa9c9b8d3@mail.mikej.com> Message-ID: <0aa49c9f2b27059fb1e8d56e344f0f53@mail.mikej.com> X-Sender: mikej@mikej.com User-Agent: Roundcube Webmail/1.0.4 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 May 2015 14:05:29 -0000 Excuse my errant statement about 'mpt0' not identified in dmesg. I did not release the kern.msgbufsize was not large enough. full dmesg updated https://charon.gopai.com/dmesg. Thank you. On 2015-05-28 09:45, Michael Jung wrote: > I hoping to get some direction as I simply am unsure what to do next. > > I have three LSI 9200-8e adapters connected to a Super Micro > CSE-846A-R1200B > yet no SES devices get created. The Backplane in the chassis > BPN-SAS_846A > does not have a expander. So I have six SFF-8088 on the backplane that > connect > directly to the ports on the 9200-8e albeit externally.The backplane > does have > a Marvel MG9072 supporting SES-2 and is jumpered for SGPIO. > > The server in question is running 9.3, but I have tested with > 10.1-STABLE and > 11-Current live CD's and I do not see /dev/ses created. I have also > booted > for testing openIndiana and do not see /dev/es/ses created. I > have an open ticket with SM but that so far has them asking a lot of > questions > and giving no direction. > > On the LSI cards I have tested P16, P17 and P19 all in IT mode on 9.3. > sas2icru on each of the adpapters does return output such as > > Enclosure information > ------------------------------------------------------------------------ > Enclosure# : 1 > Logical ID : 500605b0:065e26b0 > Numslots : 8 > StartSlot : 0 > ------------------------------------------------------------------------ > > What stands out is is not identefied for 'mps0' in dmesg. > However, > drives attached to all controllers do work fine and are identified. > > Full dmesg and output of sas2icru for each controller > https://charon.gopai.com/dmesg > > Booting 9.3 verbosely with hw.mps.debug_level=0xd in /boot/loader.conf > yeilds: > > mps1: port 0xcc00-0xccff mem > 0xde8bc000-0xde8bffff,0xde8c0000-0xde8fffff irq 35 at device 0.0 on > pci19 > mps1: Firmware: 16.00.00.00, Driver: 16.00.00.00-fbsd > mps1: IOCCapabilities: > 1285c > mps1: EventReply : > mps1: EventReply : > mps1: attempting to allocate 1 MSI-X vectors (15 supported) > mps1: using IRQ 272 for MSI-X > mps2: port 0xdc00-0xdcff mem > 0xdeabc000-0xdeabffff,0xdeac0000-0xdeafffff irq 33 at device 0.0 on > pci18 > mps2: Firmware: 16.00.00.00, Driver: 16.00.00.00-fbsd > mps2: IOCCapabilities: > 1285c > mps2: EventReply : > mps2: EventReply : > mps2: attempting to allocate 1 MSI-X vectors (15 supported) > mps2: using IRQ 273 for MSI-X > mps0: EventReply : > mps0: EventReply : > mps0: EventReply : > mps0: EventReply : > mps0: EventReply : > mps0: EventReply : > mps0: EventReply : > mps0: EventReply : > mps0: EventReply : > mps0: EventReply : > mps0: EventReply : > mps0: EventReply : > mps0: EventReply : > mps0: EventReply : > mps1: EventReply : > mps1: EventReply : > mps1: EventReply : > mps1: EventReply : > mps1: EventReply : > mps1: EventReply : > mps1: EventReply : > mps1: EventReply : > mps1: EventReply : > mps1: EventReply : > mps1: EventReply : > mps1: EventReply : > mps2: EventReply : > mps2: EventReply : > mps2: EventReply : > mps2: EventReply : > mps2: EventReply : > mps2: EventReply : > mps2: EventReply : > mps2: EventReply : > mps2: EventReply : > mps2: EventReply : > mps2: EventReply : > mps2: EventReply : > mps2: EventReply : > mps2: EventReply : > mps0: (0)->(mpssas_fw_work) Working on Event: [16] > mps0: (1)->(mpssas_fw_work) Event Free: [16] > mps0: (1)->(mpssas_fw_work) Working on Event: [1d] > mps0: (2)->(mpssas_fw_work) Event Free: [1d] > mps0: (2)->(mpssas_fw_work) Working on Event: [1c] > mps1: (3)->(mpssas_fw_work) Working on Event: [16] > mps1: (4)->(mpssas_fw_work) Event Free: [16] > mps1: (4)->(mpssas_fw_work) Working on Event: [1d] > mps1: (5)->(mpssas_fw_work) Event Free: [1d] > mps1: (5)->(mpssas_fw_work) Working on Event: [1c] > mps2: (6)->(mpssas_fw_work) Working on Event: [16] > mps2: (7)->(mpssas_fw_work) Event Free: [16] > mps2: (7)->(mpssas_fw_work) Working on Event: [1d] > mps2: (8)->(mpssas_fw_work) Event Free: [1d] > mps2: (8)->(mpssas_fw_work) Working on Event: [1c] > (probe0:mps1:0:0:0): Down reving Protocol Version from 4 to 0? > (probe1:mps1:0:1:0): Down reving Protocol Version from 4 to 0? > (probe2:mps1:0:2:0): Down reving Protocol Version from 4 to 0? > (probe3:mps1:0:4:0): Down reving Protocol Version from 4 to 0? > (probe4:mps2:0:0:0): Down reving Protocol Version from 4 to 0? > (probe5:mps2:0:1:0): Down reving Protocol Version from 4 to 0? > mps1: (9)->(mpssas_fw_work) Event Free: [1c] > mps1: (9)->(mpssas_fw_work) Working on Event: [16] > mps1: (10)->(mpssas_fw_work) Event Free: [16] > mps1: (10)->(mpssas_fw_work) Working on Event: [16] > mps1: (11)->(mpssas_fw_work) Event Free: [16] > mps1: (11)->(mpssas_fw_work) Working on Event: [16] > mps1: (12)->(mpssas_fw_work) Event Free: [16] > mps1: (12)->(mpssas_fw_work) Working on Event: [16] > mps1: (13)->(mpssas_fw_work) Event Free: [16] > mps1: (13)->(mpssas_fw_work) Working on Event: [16] > mps1: (14)->(mpssas_fw_work) Event Free: [16] > mps1: (14)->(mpssas_fw_work) Working on Event: [16] > mps1: (15)->(mpssas_fw_work) Event Free: [16] > mps1: (15)->(mpssas_fw_work) Working on Event: [16] > mps1: (16)->(mpssas_fw_work) Event Free: [16] > mps1: (16)->(mpssas_fw_work) Working on Event: [16] > mps1: (17)->(mpssas_fw_work) Event Free: [16] > mps1: (17)->(mpssas_fw_work) Working on Event: [16] > mps1: (18)->(mpssas_fw_work) Event Free: [16] > (probe6:mps1:0:5:0): Down reving Protocol Version from 4 to 0? > (probe7:mps2:0:2:0): Down reving Protocol Version from 4 to 0? > (probe8:mps2:0:6:0): Down reving Protocol Version from 4 to 0? > (probe9:mps2:0:4:0): Down reving Protocol Version from 4 to 0? > mps2: (18)->(mpssas_fw_work) Event Free: [1c] > mps2: (18)->(mpssas_fw_work) Working on Event: [16] > mps2: (19)->(mpssas_fw_work) Event Free: [16] > mps2: (19)->(mpssas_fw_work) Working on Event: [16] > mps2: (20)->(mpssas_fw_work) Event Free: [16] > mps2: (20)->(mpssas_fw_work) Working on Event: [16] > mps2: (21)->(mpssas_fw_work) Event Free: [16] > mps2: (21)->(mpssas_fw_work) Working on Event: [16] > mps2: (22)->(mpssas_fw_work) Event Free: [16] > mps2: (22)->(mpssas_fw_work) Working on Event: [16] > mps2: (23)->(mpssas_fw_work) Event Free: [16] > mps2: (23)->(mpssas_fw_work) Working on Event: [16] > mps2: (24)->(mpssas_fw_work) Event Free: [16] > mps2: (24)->(mpssas_fw_work) Working on Event: [16] > mps2: (25)->(mpssas_fw_work) Event Free: [16] > mps2: (25)->(mpssas_fw_work) Working on Event: [16] > mps2: (26)->(mpssas_fw_work) Event Free: [16] > mps2: (26)->(mpssas_fw_work) Working on Event: [16] > mps2: (27)->(mpssas_fw_work) Event Free: [16] > mps2: (27)->(mpssas_fw_work) Working on Event: [16] > mps2: (28)->(mpssas_fw_work) Event Free: [16] > mps2: (28)->(mpssas_fw_work) Working on Event: [16] > mps2: (29)->(mpssas_fw_work) Event Free: [16] > (probe0:mps2:0:5:0): Down reving Protocol Version from 4 to 0? > (probe1:mps0:0:0:0): Down reving Protocol Version from 4 to 0? > (probe2:mps0:0:1:0): Down reving Protocol Version from 4 to 0? > (probe3:mps0:0:2:0): Down reving Protocol Version from 4 to 0? > (probe5:mps0:0:3:0): Down reving Protocol Version from 4 to 0? > (probe6:mps0:0:4:0): Down reving Protocol Version from 4 to 0? > mps0: (29)->(mpssas_fw_work) Event Free: [1c] > mps0: (29)->(mpssas_fw_work) Working on Event: [16] > mps0: (30)->(mpssas_fw_work) Event Free: [16] > mps0: (30)->(mpssas_fw_work) Working on Event: [16] > mps0: (31)->(mpssas_fw_work) Event Free: [16] > mps0: (31)->(mpssas_fw_work) Working on Event: [16] > mps0: (32)->(mpssas_fw_work) Event Free: [16] > mps0: (32)->(mpssas_fw_work) Working on Event: [16] > mps0: (33)->(mpssas_fw_work) Event Free: [16] > mps0: (33)->(mpssas_fw_work) Working on Event: [16] > mps0: (34)->(mpssas_fw_work) Event Free: [16] > mps0: (34)->(mpssas_fw_work) Working on Event: [16] > mps0: (35)->(mpssas_fw_work) Event Free: [16] > mps0: (35)->(mpssas_fw_work) Working on Event: [16] > mps0: (36)->(mpssas_fw_work) Event Free: [16] > mps0: (36)->(mpssas_fw_work) Working on Event: [16] > mps0: (37)->(mpssas_fw_work) Event Free: [16] > mps0: (37)->(mpssas_fw_work) Working on Event: [16] > mps0: (38)->(mpssas_fw_work) Event Free: [16] > mps0: (38)->(mpssas_fw_work) Working on Event: [16] > mps0: (39)->(mpssas_fw_work) Event Free: [16] > mps0: (39)->(mpssas_fw_work) Working on Event: [16] > mps0: (40)->(mpssas_fw_work) Event Free: [16] > (probe0:mps0:0:6:0): Down reving Protocol Version from 4 to 0? > da10 at mps0 bus 0 scbus0 target 0 lun 0 > da0 at mps0 bus 0 scbus0 target 2 lun 0 > da1 at mps0 bus 0 scbus0 target 3 lun 0 > da11 at mps0 bus 0 scbus0 target 1 lun 0 > da2 at mps0 bus 0 scbus0 target 4 lun 0 > da18 at mps1 bus 0 scbus1 target 0 lun 0 > da19 at mps1 bus 0 scbus1 target 1 lun 0 > da20 at mps1 bus 0 scbus1 target 2 lun 0 > da23 at mps1 bus 0 scbus1 target 5 lun 0 > da14 at mps2 bus 0 scbus2 target 0 lun 0 > da15 at mps2 bus 0 scbus2 target 1 lun 0 > da16 at mps2 bus 0 scbus2 target 2 lun 0 > da22 at mps1 bus 0 scbus1 target 4 lun 0 > da6 at mps2 bus 0 scbus2 target 4 lun 0 > pass0 at mps0 bus 0 scbus0 target 0 lun 0 > da8 at mps2 bus 0 scbus2 target 6 lun 0 > pass1 at mps0 bus 0 scbus0 target 1 lun 0 > pass2 at mps0 bus 0 scbus0 target 2 lun 0 > pass3 at mps0 bus 0 scbus0 target 3 lun 0 > pass4 at mps0 bus 0 scbus0 target 4 lun 0 > pass5 at mps0 bus 0 scbus0 target 6 lun 0 > pass6 at mps1 bus 0 scbus1 target 0 lun 0 > pass7 at mps1 bus 0 scbus1 target 1 lun 0 > da7 at mps2 bus 0 scbus2 target 5 lun 0 > pass8 at mps1 bus 0 scbus1 target 2 lun 0 > pass9 at mps1 bus 0 scbus1 target 4 lun 0 > pass10 at mps1 bus 0 scbus1 target 5 lun 0 > pass11 at mps2 bus 0 scbus2 target 0 lun 0 > pass12 at mps2 bus 0 scbus2 target 1 lun 0 > pass13 at mps2 bus 0 scbus2 target 2 lun 0 > pass14 at mps2 bus 0 scbus2 target 4 lun 0 > pass15 at mps2 bus 0 scbus2 target 5 lun 0 > pass16 at mps2 bus 0 scbus2 target 6 lun 0 > da4 at mps0 bus 0 scbus0 target 6 lun 0 > > I admit I am new to enclosure management - What am I missing? > > Thank you. > > --mikej From owner-freebsd-scsi@FreeBSD.ORG Thu May 28 14:55:50 2015 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 10D4FE7E for ; Thu, 28 May 2015 14:55:50 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-oi0-x22c.google.com (mail-oi0-x22c.google.com [IPv6:2607:f8b0:4003:c06::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CCEFCBD6 for ; Thu, 28 May 2015 14:55:49 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by oihb142 with SMTP id b142so34144324oih.3 for ; Thu, 28 May 2015 07:55:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=/osY+Z38iZYy31MIJaXiPnBGkYZvXS8PDj65nlyY8Ik=; b=w/kH3VzBWXO51yqW1kUtAemuKMT0Nj0lm60+wNxFrCl7WeOscEcUKxLZhs8WP16tG3 sgLaP/vY1lpYF/AwDVdjPpHe/G0yFgcgt2szxITy3p2uGyBzdWAAkkSuVH6tBettanoo uM1ZwE0h5rsOWSodHawUj4h3DIW7Ma79Fx1T/OmaNQo8pqa/Zz+4xbevFgdXHn6AU+EM Oj3vBQSfqookbclK5072XvKi4II5qz7GQriLGgpGP+1DToimuK/jBWt5VUx8Ov3RYPxN oJCexK9RFBTa8mVwcneWXs7cwjExyt4X1T4+7Fq3aNj78CxiE+TKCOLFp2lduhDIE2ap syJQ== MIME-Version: 1.0 X-Received: by 10.182.112.138 with SMTP id iq10mr2857080obb.38.1432824949020; Thu, 28 May 2015 07:55:49 -0700 (PDT) Sender: asomers@gmail.com Received: by 10.202.107.3 with HTTP; Thu, 28 May 2015 07:55:48 -0700 (PDT) In-Reply-To: <76e9e3f41990026290382e5fa9c9b8d3@mail.mikej.com> References: <76e9e3f41990026290382e5fa9c9b8d3@mail.mikej.com> Date: Thu, 28 May 2015 08:55:48 -0600 X-Google-Sender-Auth: qBlnJLIqSeQUUblGLh2vQbB80ZY Message-ID: Subject: Re: /dev/ses not created From: Alan Somers To: Michael Jung Cc: FreeBSD-scsi Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 May 2015 14:55:50 -0000 SGPIO is an out-of-band protocol. It doesn't use the same wires as the SAS connections. It's odd that the manual says the enclosure management chip supports SES-2 and yet there is no SAS port for it. The chassis manual also shows that headers 7-9 are for SGPIO. Are they connected to anything? Some Supermicro motherboards have SGPIO headers. But I would think that if yours did, then the ses device would show up even if the cable were unplugged. LSI HBAs are also supposed to have builtin SGPIO controllers. But I don't know how to enable them or if FreeBSD has driver support for them. You might learn more by asking LSI support than SuperMicro support. If you do have an SGPIO controller on the motherboard, you will need FreeBSD 10.0 or later to use it. Finally, I know you probably don't want to spend more money, but if you want enclosure management then your best option is to replace the backplane with a SAS-846EL. It should be mechanically compatible. That will give you SES, which is way better than SGPIO. It also has an expander, so you'll only need one HBA. -Alan On Thu, May 28, 2015 at 7:45 AM, Michael Jung wrote: > I hoping to get some direction as I simply am unsure what to do next. > > I have three LSI 9200-8e adapters connected to a Super Micro CSE-846A-R1200B > yet no SES devices get created. The Backplane in the chassis BPN-SAS_846A > does not have a expander. So I have six SFF-8088 on the backplane that > connect > directly to the ports on the 9200-8e albeit externally.The backplane does > have > a Marvel MG9072 supporting SES-2 and is jumpered for SGPIO. > > The server in question is running 9.3, but I have tested with 10.1-STABLE > and > 11-Current live CD's and I do not see /dev/ses created. I have also > booted > for testing openIndiana and do not see /dev/es/ses created. I > have an open ticket with SM but that so far has them asking a lot of > questions > and giving no direction. > > On the LSI cards I have tested P16, P17 and P19 all in IT mode on 9.3. > sas2icru on each of the adpapters does return output such as > > Enclosure information > ------------------------------------------------------------------------ > Enclosure# : 1 > Logical ID : 500605b0:065e26b0 > Numslots : 8 > StartSlot : 0 > ------------------------------------------------------------------------ > > What stands out is is not identefied for 'mps0' in dmesg. > However, > drives attached to all controllers do work fine and are identified. > > Full dmesg and output of sas2icru for each controller > https://charon.gopai.com/dmesg > > Booting 9.3 verbosely with hw.mps.debug_level=0xd in /boot/loader.conf > yeilds: > > mps1: port 0xcc00-0xccff mem > 0xde8bc000-0xde8bffff,0xde8c0000-0xde8fffff irq 35 at device 0.0 on pci19 > mps1: Firmware: 16.00.00.00, Driver: 16.00.00.00-fbsd > mps1: IOCCapabilities: > 1285c > mps1: EventReply : > mps1: EventReply : > mps1: attempting to allocate 1 MSI-X vectors (15 supported) > mps1: using IRQ 272 for MSI-X > mps2: port 0xdc00-0xdcff mem > 0xdeabc000-0xdeabffff,0xdeac0000-0xdeafffff irq 33 at device 0.0 on pci18 > mps2: Firmware: 16.00.00.00, Driver: 16.00.00.00-fbsd > mps2: IOCCapabilities: > 1285c > mps2: EventReply : > mps2: EventReply : > mps2: attempting to allocate 1 MSI-X vectors (15 supported) > mps2: using IRQ 273 for MSI-X > mps0: EventReply : > mps0: EventReply : > mps0: EventReply : > mps0: EventReply : > mps0: EventReply : > mps0: EventReply : > mps0: EventReply : > mps0: EventReply : > mps0: EventReply : > mps0: EventReply : > mps0: EventReply : > mps0: EventReply : > mps0: EventReply : > mps0: EventReply : > mps1: EventReply : > mps1: EventReply : > mps1: EventReply : > mps1: EventReply : > mps1: EventReply : > mps1: EventReply : > mps1: EventReply : > mps1: EventReply : > mps1: EventReply : > mps1: EventReply : > mps1: EventReply : > mps1: EventReply : > mps2: EventReply : > mps2: EventReply : > mps2: EventReply : > mps2: EventReply : > mps2: EventReply : > mps2: EventReply : > mps2: EventReply : > mps2: EventReply : > mps2: EventReply : > mps2: EventReply : > mps2: EventReply : > mps2: EventReply : > mps2: EventReply : > mps2: EventReply : > mps0: (0)->(mpssas_fw_work) Working on Event: [16] > mps0: (1)->(mpssas_fw_work) Event Free: [16] > mps0: (1)->(mpssas_fw_work) Working on Event: [1d] > mps0: (2)->(mpssas_fw_work) Event Free: [1d] > mps0: (2)->(mpssas_fw_work) Working on Event: [1c] > mps1: (3)->(mpssas_fw_work) Working on Event: [16] > mps1: (4)->(mpssas_fw_work) Event Free: [16] > mps1: (4)->(mpssas_fw_work) Working on Event: [1d] > mps1: (5)->(mpssas_fw_work) Event Free: [1d] > mps1: (5)->(mpssas_fw_work) Working on Event: [1c] > mps2: (6)->(mpssas_fw_work) Working on Event: [16] > mps2: (7)->(mpssas_fw_work) Event Free: [16] > mps2: (7)->(mpssas_fw_work) Working on Event: [1d] > mps2: (8)->(mpssas_fw_work) Event Free: [1d] > mps2: (8)->(mpssas_fw_work) Working on Event: [1c] > (probe0:mps1:0:0:0): Down reving Protocol Version from 4 to 0? > (probe1:mps1:0:1:0): Down reving Protocol Version from 4 to 0? > (probe2:mps1:0:2:0): Down reving Protocol Version from 4 to 0? > (probe3:mps1:0:4:0): Down reving Protocol Version from 4 to 0? > (probe4:mps2:0:0:0): Down reving Protocol Version from 4 to 0? > (probe5:mps2:0:1:0): Down reving Protocol Version from 4 to 0? > mps1: (9)->(mpssas_fw_work) Event Free: [1c] > mps1: (9)->(mpssas_fw_work) Working on Event: [16] > mps1: (10)->(mpssas_fw_work) Event Free: [16] > mps1: (10)->(mpssas_fw_work) Working on Event: [16] > mps1: (11)->(mpssas_fw_work) Event Free: [16] > mps1: (11)->(mpssas_fw_work) Working on Event: [16] > mps1: (12)->(mpssas_fw_work) Event Free: [16] > mps1: (12)->(mpssas_fw_work) Working on Event: [16] > mps1: (13)->(mpssas_fw_work) Event Free: [16] > mps1: (13)->(mpssas_fw_work) Working on Event: [16] > mps1: (14)->(mpssas_fw_work) Event Free: [16] > mps1: (14)->(mpssas_fw_work) Working on Event: [16] > mps1: (15)->(mpssas_fw_work) Event Free: [16] > mps1: (15)->(mpssas_fw_work) Working on Event: [16] > mps1: (16)->(mpssas_fw_work) Event Free: [16] > mps1: (16)->(mpssas_fw_work) Working on Event: [16] > mps1: (17)->(mpssas_fw_work) Event Free: [16] > mps1: (17)->(mpssas_fw_work) Working on Event: [16] > mps1: (18)->(mpssas_fw_work) Event Free: [16] > (probe6:mps1:0:5:0): Down reving Protocol Version from 4 to 0? > (probe7:mps2:0:2:0): Down reving Protocol Version from 4 to 0? > (probe8:mps2:0:6:0): Down reving Protocol Version from 4 to 0? > (probe9:mps2:0:4:0): Down reving Protocol Version from 4 to 0? > mps2: (18)->(mpssas_fw_work) Event Free: [1c] > mps2: (18)->(mpssas_fw_work) Working on Event: [16] > mps2: (19)->(mpssas_fw_work) Event Free: [16] > mps2: (19)->(mpssas_fw_work) Working on Event: [16] > mps2: (20)->(mpssas_fw_work) Event Free: [16] > mps2: (20)->(mpssas_fw_work) Working on Event: [16] > mps2: (21)->(mpssas_fw_work) Event Free: [16] > mps2: (21)->(mpssas_fw_work) Working on Event: [16] > mps2: (22)->(mpssas_fw_work) Event Free: [16] > mps2: (22)->(mpssas_fw_work) Working on Event: [16] > mps2: (23)->(mpssas_fw_work) Event Free: [16] > mps2: (23)->(mpssas_fw_work) Working on Event: [16] > mps2: (24)->(mpssas_fw_work) Event Free: [16] > mps2: (24)->(mpssas_fw_work) Working on Event: [16] > mps2: (25)->(mpssas_fw_work) Event Free: [16] > mps2: (25)->(mpssas_fw_work) Working on Event: [16] > mps2: (26)->(mpssas_fw_work) Event Free: [16] > mps2: (26)->(mpssas_fw_work) Working on Event: [16] > mps2: (27)->(mpssas_fw_work) Event Free: [16] > mps2: (27)->(mpssas_fw_work) Working on Event: [16] > mps2: (28)->(mpssas_fw_work) Event Free: [16] > mps2: (28)->(mpssas_fw_work) Working on Event: [16] > mps2: (29)->(mpssas_fw_work) Event Free: [16] > (probe0:mps2:0:5:0): Down reving Protocol Version from 4 to 0? > (probe1:mps0:0:0:0): Down reving Protocol Version from 4 to 0? > (probe2:mps0:0:1:0): Down reving Protocol Version from 4 to 0? > (probe3:mps0:0:2:0): Down reving Protocol Version from 4 to 0? > (probe5:mps0:0:3:0): Down reving Protocol Version from 4 to 0? > (probe6:mps0:0:4:0): Down reving Protocol Version from 4 to 0? > mps0: (29)->(mpssas_fw_work) Event Free: [1c] > mps0: (29)->(mpssas_fw_work) Working on Event: [16] > mps0: (30)->(mpssas_fw_work) Event Free: [16] > mps0: (30)->(mpssas_fw_work) Working on Event: [16] > mps0: (31)->(mpssas_fw_work) Event Free: [16] > mps0: (31)->(mpssas_fw_work) Working on Event: [16] > mps0: (32)->(mpssas_fw_work) Event Free: [16] > mps0: (32)->(mpssas_fw_work) Working on Event: [16] > mps0: (33)->(mpssas_fw_work) Event Free: [16] > mps0: (33)->(mpssas_fw_work) Working on Event: [16] > mps0: (34)->(mpssas_fw_work) Event Free: [16] > mps0: (34)->(mpssas_fw_work) Working on Event: [16] > mps0: (35)->(mpssas_fw_work) Event Free: [16] > mps0: (35)->(mpssas_fw_work) Working on Event: [16] > mps0: (36)->(mpssas_fw_work) Event Free: [16] > mps0: (36)->(mpssas_fw_work) Working on Event: [16] > mps0: (37)->(mpssas_fw_work) Event Free: [16] > mps0: (37)->(mpssas_fw_work) Working on Event: [16] > mps0: (38)->(mpssas_fw_work) Event Free: [16] > mps0: (38)->(mpssas_fw_work) Working on Event: [16] > mps0: (39)->(mpssas_fw_work) Event Free: [16] > mps0: (39)->(mpssas_fw_work) Working on Event: [16] > mps0: (40)->(mpssas_fw_work) Event Free: [16] > (probe0:mps0:0:6:0): Down reving Protocol Version from 4 to 0? > da10 at mps0 bus 0 scbus0 target 0 lun 0 > da0 at mps0 bus 0 scbus0 target 2 lun 0 > da1 at mps0 bus 0 scbus0 target 3 lun 0 > da11 at mps0 bus 0 scbus0 target 1 lun 0 > da2 at mps0 bus 0 scbus0 target 4 lun 0 > da18 at mps1 bus 0 scbus1 target 0 lun 0 > da19 at mps1 bus 0 scbus1 target 1 lun 0 > da20 at mps1 bus 0 scbus1 target 2 lun 0 > da23 at mps1 bus 0 scbus1 target 5 lun 0 > da14 at mps2 bus 0 scbus2 target 0 lun 0 > da15 at mps2 bus 0 scbus2 target 1 lun 0 > da16 at mps2 bus 0 scbus2 target 2 lun 0 > da22 at mps1 bus 0 scbus1 target 4 lun 0 > da6 at mps2 bus 0 scbus2 target 4 lun 0 > pass0 at mps0 bus 0 scbus0 target 0 lun 0 > da8 at mps2 bus 0 scbus2 target 6 lun 0 > pass1 at mps0 bus 0 scbus0 target 1 lun 0 > pass2 at mps0 bus 0 scbus0 target 2 lun 0 > pass3 at mps0 bus 0 scbus0 target 3 lun 0 > pass4 at mps0 bus 0 scbus0 target 4 lun 0 > pass5 at mps0 bus 0 scbus0 target 6 lun 0 > pass6 at mps1 bus 0 scbus1 target 0 lun 0 > pass7 at mps1 bus 0 scbus1 target 1 lun 0 > da7 at mps2 bus 0 scbus2 target 5 lun 0 > pass8 at mps1 bus 0 scbus1 target 2 lun 0 > pass9 at mps1 bus 0 scbus1 target 4 lun 0 > pass10 at mps1 bus 0 scbus1 target 5 lun 0 > pass11 at mps2 bus 0 scbus2 target 0 lun 0 > pass12 at mps2 bus 0 scbus2 target 1 lun 0 > pass13 at mps2 bus 0 scbus2 target 2 lun 0 > pass14 at mps2 bus 0 scbus2 target 4 lun 0 > pass15 at mps2 bus 0 scbus2 target 5 lun 0 > pass16 at mps2 bus 0 scbus2 target 6 lun 0 > da4 at mps0 bus 0 scbus0 target 6 lun 0 > > I admit I am new to enclosure management - What am I missing? > > Thank you. > > --mikej > _______________________________________________ > freebsd-scsi@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-scsi > To unsubscribe, send any mail to "freebsd-scsi-unsubscribe@freebsd.org" From owner-freebsd-scsi@FreeBSD.ORG Thu May 28 15:22:33 2015 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6300236C; Thu, 28 May 2015 15:22:33 +0000 (UTC) (envelope-from mikej@mikej.com) Received: from mx2.paymentallianceintl.com (mx2.paymentallianceintl.com [216.26.158.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx2.paymentallianceintl.com", Issuer "Go Daddy Secure Certification Authority" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 264CC322; Thu, 28 May 2015 15:22:32 +0000 (UTC) (envelope-from mikej@mikej.com) Received: from firewall.mikej.com (162-230-214-65.lightspeed.lsvlky.sbcglobal.net [162.230.214.65]) by mx2.paymentallianceintl.com (8.15.1/8.15.1) with ESMTPS id t4SFMUjq002425 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 28 May 2015 11:22:31 -0400 (EDT) (envelope-from mikej@mikej.com) X-Authentication-Warning: mx2.paymentallianceintl.com: Host 162-230-214-65.lightspeed.lsvlky.sbcglobal.net [162.230.214.65] claimed to be firewall.mikej.com Received: from mail.mikej.com (firewall [192.168.6.63]) by firewall.mikej.com (8.15.1/8.14.9) with ESMTP id t4SFMTMh051460; Thu, 28 May 2015 11:22:30 -0400 (EDT) (envelope-from mikej@mikej.com) X-Authentication-Warning: firewall.mikej.com: Host firewall [192.168.6.63] claimed to be mail.mikej.com MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Thu, 28 May 2015 11:22:29 -0400 From: Michael Jung To: Alan Somers Cc: FreeBSD-scsi , asomers@gmail.com Subject: Re: /dev/ses not created In-Reply-To: References: <76e9e3f41990026290382e5fa9c9b8d3@mail.mikej.com> Message-ID: <026292acf4288dbde00ce9bcb8686490@mail.mikej.com> X-Sender: mikej@mikej.com User-Agent: Roundcube Webmail/1.0.4 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 May 2015 15:22:33 -0000 Alan, Thank you for the information. So SGPIO is out of band. So the intent of Marvel chip is not for in-band SES-2? I see these events, one for each LSI card in dmesg: mps1: EventReply : EventDataLength: 5 AckRequired: 0 Event: SasEnclDeviceStatusChange (0x1d) <---- EventContext: 0x0 EnclosureHandle: 0x1 ReasonCode: Added PhysicalPort: 0 NumSlots: 8 <---- StartSlot: 0 PhyBits: 0x0 The motherboard is not in the same cabinet as the backplane, and and I would rather not just replace the backplane with one that had an expander on it $$$. Maybe I'm just seeking functionality that is not present but the fact that the backplane manual cleary states "The MG9072 is an enclosure management chip that supports the SES-2 controller and SES-2 protocols." make me -want- to believe something else is amiss. Thank you. --mikej On 2015-05-28 10:55, Alan Somers wrote: > SGPIO is an out-of-band protocol. It doesn't use the same wires as > the SAS connections. It's odd that the manual says the enclosure > management chip supports SES-2 and yet there is no SAS port for it. > The chassis manual also shows that headers 7-9 are for SGPIO. Are > they connected to anything? Some Supermicro motherboards have SGPIO > headers. But I would think that if yours did, then the ses device > would show up even if the cable were unplugged. LSI HBAs are also > supposed to have builtin SGPIO controllers. But I don't know how to > enable them or if FreeBSD has driver support for them. You might > learn more by asking LSI support than SuperMicro support. > > If you do have an SGPIO controller on the motherboard, you will need > FreeBSD 10.0 or later to use it. > > Finally, I know you probably don't want to spend more money, but if > you want enclosure management then your best option is to replace the > backplane with a SAS-846EL. It should be mechanically compatible. > That will give you SES, which is way better than SGPIO. It also has > an expander, so you'll only need one HBA. > > -Alan > > On Thu, May 28, 2015 at 7:45 AM, Michael Jung wrote: >> I hoping to get some direction as I simply am unsure what to do next. >> >> I have three LSI 9200-8e adapters connected to a Super Micro >> CSE-846A-R1200B >> yet no SES devices get created. The Backplane in the chassis >> BPN-SAS_846A >> does not have a expander. So I have six SFF-8088 on the backplane >> that >> connect >> directly to the ports on the 9200-8e albeit externally.The backplane >> does >> have >> a Marvel MG9072 supporting SES-2 and is jumpered for SGPIO. >> >> The server in question is running 9.3, but I have tested with >> 10.1-STABLE >> and >> 11-Current live CD's and I do not see /dev/ses created. I have also >> booted >> for testing openIndiana and do not see /dev/es/ses created. I >> have an open ticket with SM but that so far has them asking a lot of >> questions >> and giving no direction. >> >> On the LSI cards I have tested P16, P17 and P19 all in IT mode on 9.3. >> sas2icru on each of the adpapters does return output such as >> >> Enclosure information >> ------------------------------------------------------------------------ >> Enclosure# : 1 >> Logical ID : 500605b0:065e26b0 >> Numslots : 8 >> StartSlot : 0 >> ------------------------------------------------------------------------ >> >> What stands out is is not identefied for 'mps0' in >> dmesg. >> However, >> drives attached to all controllers do work fine and are identified. >> >> Full dmesg and output of sas2icru for each controller >> https://charon.gopai.com/dmesg >> >> Booting 9.3 verbosely with hw.mps.debug_level=0xd in /boot/loader.conf >> yeilds: >> >> mps1: port 0xcc00-0xccff mem >> 0xde8bc000-0xde8bffff,0xde8c0000-0xde8fffff irq 35 at device 0.0 on >> pci19 >> mps1: Firmware: 16.00.00.00, Driver: 16.00.00.00-fbsd >> mps1: IOCCapabilities: >> 1285c >> mps1: EventReply : >> mps1: EventReply : >> mps1: attempting to allocate 1 MSI-X vectors (15 supported) >> mps1: using IRQ 272 for MSI-X >> mps2: port 0xdc00-0xdcff mem >> 0xdeabc000-0xdeabffff,0xdeac0000-0xdeafffff irq 33 at device 0.0 on >> pci18 >> mps2: Firmware: 16.00.00.00, Driver: 16.00.00.00-fbsd >> mps2: IOCCapabilities: >> 1285c >> mps2: EventReply : >> mps2: EventReply : >> mps2: attempting to allocate 1 MSI-X vectors (15 supported) >> mps2: using IRQ 273 for MSI-X >> mps0: EventReply : >> mps0: EventReply : >> mps0: EventReply : >> mps0: EventReply : >> mps0: EventReply : >> mps0: EventReply : >> mps0: EventReply : >> mps0: EventReply : >> mps0: EventReply : >> mps0: EventReply : >> mps0: EventReply : >> mps0: EventReply : >> mps0: EventReply : >> mps0: EventReply : >> mps1: EventReply : >> mps1: EventReply : >> mps1: EventReply : >> mps1: EventReply : >> mps1: EventReply : >> mps1: EventReply : >> mps1: EventReply : >> mps1: EventReply : >> mps1: EventReply : >> mps1: EventReply : >> mps1: EventReply : >> mps1: EventReply : >> mps2: EventReply : >> mps2: EventReply : >> mps2: EventReply : >> mps2: EventReply : >> mps2: EventReply : >> mps2: EventReply : >> mps2: EventReply : >> mps2: EventReply : >> mps2: EventReply : >> mps2: EventReply : >> mps2: EventReply : >> mps2: EventReply : >> mps2: EventReply : >> mps2: EventReply : >> mps0: (0)->(mpssas_fw_work) Working on Event: [16] >> mps0: (1)->(mpssas_fw_work) Event Free: [16] >> mps0: (1)->(mpssas_fw_work) Working on Event: [1d] >> mps0: (2)->(mpssas_fw_work) Event Free: [1d] >> mps0: (2)->(mpssas_fw_work) Working on Event: [1c] >> mps1: (3)->(mpssas_fw_work) Working on Event: [16] >> mps1: (4)->(mpssas_fw_work) Event Free: [16] >> mps1: (4)->(mpssas_fw_work) Working on Event: [1d] >> mps1: (5)->(mpssas_fw_work) Event Free: [1d] >> mps1: (5)->(mpssas_fw_work) Working on Event: [1c] >> mps2: (6)->(mpssas_fw_work) Working on Event: [16] >> mps2: (7)->(mpssas_fw_work) Event Free: [16] >> mps2: (7)->(mpssas_fw_work) Working on Event: [1d] >> mps2: (8)->(mpssas_fw_work) Event Free: [1d] >> mps2: (8)->(mpssas_fw_work) Working on Event: [1c] >> (probe0:mps1:0:0:0): Down reving Protocol Version from 4 to 0? >> (probe1:mps1:0:1:0): Down reving Protocol Version from 4 to 0? >> (probe2:mps1:0:2:0): Down reving Protocol Version from 4 to 0? >> (probe3:mps1:0:4:0): Down reving Protocol Version from 4 to 0? >> (probe4:mps2:0:0:0): Down reving Protocol Version from 4 to 0? >> (probe5:mps2:0:1:0): Down reving Protocol Version from 4 to 0? >> mps1: (9)->(mpssas_fw_work) Event Free: [1c] >> mps1: (9)->(mpssas_fw_work) Working on Event: [16] >> mps1: (10)->(mpssas_fw_work) Event Free: [16] >> mps1: (10)->(mpssas_fw_work) Working on Event: [16] >> mps1: (11)->(mpssas_fw_work) Event Free: [16] >> mps1: (11)->(mpssas_fw_work) Working on Event: [16] >> mps1: (12)->(mpssas_fw_work) Event Free: [16] >> mps1: (12)->(mpssas_fw_work) Working on Event: [16] >> mps1: (13)->(mpssas_fw_work) Event Free: [16] >> mps1: (13)->(mpssas_fw_work) Working on Event: [16] >> mps1: (14)->(mpssas_fw_work) Event Free: [16] >> mps1: (14)->(mpssas_fw_work) Working on Event: [16] >> mps1: (15)->(mpssas_fw_work) Event Free: [16] >> mps1: (15)->(mpssas_fw_work) Working on Event: [16] >> mps1: (16)->(mpssas_fw_work) Event Free: [16] >> mps1: (16)->(mpssas_fw_work) Working on Event: [16] >> mps1: (17)->(mpssas_fw_work) Event Free: [16] >> mps1: (17)->(mpssas_fw_work) Working on Event: [16] >> mps1: (18)->(mpssas_fw_work) Event Free: [16] >> (probe6:mps1:0:5:0): Down reving Protocol Version from 4 to 0? >> (probe7:mps2:0:2:0): Down reving Protocol Version from 4 to 0? >> (probe8:mps2:0:6:0): Down reving Protocol Version from 4 to 0? >> (probe9:mps2:0:4:0): Down reving Protocol Version from 4 to 0? >> mps2: (18)->(mpssas_fw_work) Event Free: [1c] >> mps2: (18)->(mpssas_fw_work) Working on Event: [16] >> mps2: (19)->(mpssas_fw_work) Event Free: [16] >> mps2: (19)->(mpssas_fw_work) Working on Event: [16] >> mps2: (20)->(mpssas_fw_work) Event Free: [16] >> mps2: (20)->(mpssas_fw_work) Working on Event: [16] >> mps2: (21)->(mpssas_fw_work) Event Free: [16] >> mps2: (21)->(mpssas_fw_work) Working on Event: [16] >> mps2: (22)->(mpssas_fw_work) Event Free: [16] >> mps2: (22)->(mpssas_fw_work) Working on Event: [16] >> mps2: (23)->(mpssas_fw_work) Event Free: [16] >> mps2: (23)->(mpssas_fw_work) Working on Event: [16] >> mps2: (24)->(mpssas_fw_work) Event Free: [16] >> mps2: (24)->(mpssas_fw_work) Working on Event: [16] >> mps2: (25)->(mpssas_fw_work) Event Free: [16] >> mps2: (25)->(mpssas_fw_work) Working on Event: [16] >> mps2: (26)->(mpssas_fw_work) Event Free: [16] >> mps2: (26)->(mpssas_fw_work) Working on Event: [16] >> mps2: (27)->(mpssas_fw_work) Event Free: [16] >> mps2: (27)->(mpssas_fw_work) Working on Event: [16] >> mps2: (28)->(mpssas_fw_work) Event Free: [16] >> mps2: (28)->(mpssas_fw_work) Working on Event: [16] >> mps2: (29)->(mpssas_fw_work) Event Free: [16] >> (probe0:mps2:0:5:0): Down reving Protocol Version from 4 to 0? >> (probe1:mps0:0:0:0): Down reving Protocol Version from 4 to 0? >> (probe2:mps0:0:1:0): Down reving Protocol Version from 4 to 0? >> (probe3:mps0:0:2:0): Down reving Protocol Version from 4 to 0? >> (probe5:mps0:0:3:0): Down reving Protocol Version from 4 to 0? >> (probe6:mps0:0:4:0): Down reving Protocol Version from 4 to 0? >> mps0: (29)->(mpssas_fw_work) Event Free: [1c] >> mps0: (29)->(mpssas_fw_work) Working on Event: [16] >> mps0: (30)->(mpssas_fw_work) Event Free: [16] >> mps0: (30)->(mpssas_fw_work) Working on Event: [16] >> mps0: (31)->(mpssas_fw_work) Event Free: [16] >> mps0: (31)->(mpssas_fw_work) Working on Event: [16] >> mps0: (32)->(mpssas_fw_work) Event Free: [16] >> mps0: (32)->(mpssas_fw_work) Working on Event: [16] >> mps0: (33)->(mpssas_fw_work) Event Free: [16] >> mps0: (33)->(mpssas_fw_work) Working on Event: [16] >> mps0: (34)->(mpssas_fw_work) Event Free: [16] >> mps0: (34)->(mpssas_fw_work) Working on Event: [16] >> mps0: (35)->(mpssas_fw_work) Event Free: [16] >> mps0: (35)->(mpssas_fw_work) Working on Event: [16] >> mps0: (36)->(mpssas_fw_work) Event Free: [16] >> mps0: (36)->(mpssas_fw_work) Working on Event: [16] >> mps0: (37)->(mpssas_fw_work) Event Free: [16] >> mps0: (37)->(mpssas_fw_work) Working on Event: [16] >> mps0: (38)->(mpssas_fw_work) Event Free: [16] >> mps0: (38)->(mpssas_fw_work) Working on Event: [16] >> mps0: (39)->(mpssas_fw_work) Event Free: [16] >> mps0: (39)->(mpssas_fw_work) Working on Event: [16] >> mps0: (40)->(mpssas_fw_work) Event Free: [16] >> (probe0:mps0:0:6:0): Down reving Protocol Version from 4 to 0? >> da10 at mps0 bus 0 scbus0 target 0 lun 0 >> da0 at mps0 bus 0 scbus0 target 2 lun 0 >> da1 at mps0 bus 0 scbus0 target 3 lun 0 >> da11 at mps0 bus 0 scbus0 target 1 lun 0 >> da2 at mps0 bus 0 scbus0 target 4 lun 0 >> da18 at mps1 bus 0 scbus1 target 0 lun 0 >> da19 at mps1 bus 0 scbus1 target 1 lun 0 >> da20 at mps1 bus 0 scbus1 target 2 lun 0 >> da23 at mps1 bus 0 scbus1 target 5 lun 0 >> da14 at mps2 bus 0 scbus2 target 0 lun 0 >> da15 at mps2 bus 0 scbus2 target 1 lun 0 >> da16 at mps2 bus 0 scbus2 target 2 lun 0 >> da22 at mps1 bus 0 scbus1 target 4 lun 0 >> da6 at mps2 bus 0 scbus2 target 4 lun 0 >> pass0 at mps0 bus 0 scbus0 target 0 lun 0 >> da8 at mps2 bus 0 scbus2 target 6 lun 0 >> pass1 at mps0 bus 0 scbus0 target 1 lun 0 >> pass2 at mps0 bus 0 scbus0 target 2 lun 0 >> pass3 at mps0 bus 0 scbus0 target 3 lun 0 >> pass4 at mps0 bus 0 scbus0 target 4 lun 0 >> pass5 at mps0 bus 0 scbus0 target 6 lun 0 >> pass6 at mps1 bus 0 scbus1 target 0 lun 0 >> pass7 at mps1 bus 0 scbus1 target 1 lun 0 >> da7 at mps2 bus 0 scbus2 target 5 lun 0 >> pass8 at mps1 bus 0 scbus1 target 2 lun 0 >> pass9 at mps1 bus 0 scbus1 target 4 lun 0 >> pass10 at mps1 bus 0 scbus1 target 5 lun 0 >> pass11 at mps2 bus 0 scbus2 target 0 lun 0 >> pass12 at mps2 bus 0 scbus2 target 1 lun 0 >> pass13 at mps2 bus 0 scbus2 target 2 lun 0 >> pass14 at mps2 bus 0 scbus2 target 4 lun 0 >> pass15 at mps2 bus 0 scbus2 target 5 lun 0 >> pass16 at mps2 bus 0 scbus2 target 6 lun 0 >> da4 at mps0 bus 0 scbus0 target 6 lun 0 >> >> I admit I am new to enclosure management - What am I missing? >> >> Thank you. >> >> --mikej >> _______________________________________________ >> freebsd-scsi@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-scsi >> To unsubscribe, send any mail to >> "freebsd-scsi-unsubscribe@freebsd.org" From owner-freebsd-scsi@FreeBSD.ORG Thu May 28 16:13:26 2015 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C63181EB for ; Thu, 28 May 2015 16:13:26 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-ob0-x230.google.com (mail-ob0-x230.google.com [IPv6:2607:f8b0:4003:c01::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8AEAB1AF for ; Thu, 28 May 2015 16:13:26 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by obew15 with SMTP id w15so36833931obe.1 for ; Thu, 28 May 2015 09:13:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=pT6/ddh8HQry2Gs8Q2JZSdyJRL2ery9lIWTFWOvgdGw=; b=dzIHRoaSNPRX/ffSAIqSTDhE95uprW7hgzTS67ZCv5ejSnZHNquw1JBpW1yiAD49PH F3Odip7eUYpYSSwUrj+7/JTno+o6m3emjAeXSmRGEaU20177TpVFNm8B3a61qPo4KKAd 9A+Rimvt+NKhpdbIiYTRdSHCR3pKWdvwuFYMyA3ziZdu8i6pTXkjCcLNOm3oNhSym1o+ 2lMuXJ1S2g6c9CiyjHrJsNPzQq/ov+OuLcE2y11qohyRoeWbrUpgH5tJuYLMciDYqyjU 0OzX/3mbkeSoxPtSDDtA8kRAJLwcEHJSeCYk6bWb/963yy07oUiZ2VLhxJqUoEA5rNDC pNHA== MIME-Version: 1.0 X-Received: by 10.202.54.3 with SMTP id d3mr2952389oia.103.1432829605698; Thu, 28 May 2015 09:13:25 -0700 (PDT) Sender: asomers@gmail.com Received: by 10.202.107.3 with HTTP; Thu, 28 May 2015 09:13:25 -0700 (PDT) In-Reply-To: <026292acf4288dbde00ce9bcb8686490@mail.mikej.com> References: <76e9e3f41990026290382e5fa9c9b8d3@mail.mikej.com> <026292acf4288dbde00ce9bcb8686490@mail.mikej.com> Date: Thu, 28 May 2015 10:13:25 -0600 X-Google-Sender-Auth: sCwKiX7G8_tqVIvnuPkeBReZMvY Message-ID: Subject: Re: /dev/ses not created From: Alan Somers To: Michael Jung Cc: FreeBSD-scsi Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 May 2015 16:13:26 -0000 On Thu, May 28, 2015 at 9:22 AM, Michael Jung wrote: > Alan, > > Thank you for the information. > > So SGPIO is out of band. > > So the intent of Marvel chip is not for in-band SES-2? > > I see these events, one for each LSI card in dmesg: > > mps1: EventReply : > EventDataLength: 5 > AckRequired: 0 > Event: SasEnclDeviceStatusChange (0x1d) <---- > EventContext: 0x0 > EnclosureHandle: 0x1 > ReasonCode: Added > PhysicalPort: 0 > NumSlots: 8 <---- > StartSlot: 0 > PhyBits: 0x0 Assuming you have IT firmware, those HBAs each have a builtin expander with the same number of slots as the card has ports. It's not a separate chip; it's just part of the HBA controller. That expander does not have SES functionality, however. > > The motherboard is not in the same cabinet as the backplane, > and and I would rather not just replace the backplane with > one that had an expander on it $$$. Understood. But HBAs are expensive, too. If you find yourself needing to order an HBA, you ought to consider whether it would be cheaper to order a new backplane and free up two of the HBAs that you have. > > Maybe I'm just seeking functionality that is not present but > the fact that the backplane manual cleary states "The MG9072 > is an enclosure management chip that supports the SES-2 controller > and SES-2 protocols." make me -want- to believe something else > is amiss. If there's no expander and no dedicated SAS port just for SES, then you definitely can't use SES. SES is in-band. Perhaps Supermicro meant "The MG9072 supports SES-2, but we didn't bother to route a SAS port to it, so this backplane is stuck with SGPIO". You might be able to figure out how to make those HBAs expose their SGPIO controllers. But even if you do, it won't gain you much. SGPIO is pretty lame. There is no way to automatically correlate an SGPIO element with a da device. You'll have to do it manually, and it will be dependent on your cabling arrangement. And if you add or remove drives, that will change your da numbers and you'll have to determine the mapping again. The mps driver supports "persistent mapping" mode, where each slot always gets mapped to the same bus-target-lun, but the da driver does not map the same bus-target-lun to a consistent da device. So your mapping process will have to involve "camcontrol devlist" to map da devices to bus-target-luns. There's also AFAIK no software that will automatically manage the SGPIO LEDs for you. You'll have to write your own based on the output of "zpool status" or something. Sorry to rain on the parade, but SGPIO just basically sucks. -Alan > > Thank you. > > --mikej > > > > > On 2015-05-28 10:55, Alan Somers wrote: >> >> SGPIO is an out-of-band protocol. It doesn't use the same wires as >> the SAS connections. It's odd that the manual says the enclosure >> management chip supports SES-2 and yet there is no SAS port for it. >> The chassis manual also shows that headers 7-9 are for SGPIO. Are >> they connected to anything? Some Supermicro motherboards have SGPIO >> headers. But I would think that if yours did, then the ses device >> would show up even if the cable were unplugged. LSI HBAs are also >> supposed to have builtin SGPIO controllers. But I don't know how to >> enable them or if FreeBSD has driver support for them. You might >> learn more by asking LSI support than SuperMicro support. >> >> If you do have an SGPIO controller on the motherboard, you will need >> FreeBSD 10.0 or later to use it. >> >> Finally, I know you probably don't want to spend more money, but if >> you want enclosure management then your best option is to replace the >> backplane with a SAS-846EL. It should be mechanically compatible. >> That will give you SES, which is way better than SGPIO. It also has >> an expander, so you'll only need one HBA. >> >> -Alan >> >> On Thu, May 28, 2015 at 7:45 AM, Michael Jung wrote: >>> >>> I hoping to get some direction as I simply am unsure what to do next. >>> >>> I have three LSI 9200-8e adapters connected to a Super Micro >>> CSE-846A-R1200B >>> yet no SES devices get created. The Backplane in the chassis >>> BPN-SAS_846A >>> does not have a expander. So I have six SFF-8088 on the backplane that >>> connect >>> directly to the ports on the 9200-8e albeit externally.The backplane does >>> have >>> a Marvel MG9072 supporting SES-2 and is jumpered for SGPIO. >>> >>> The server in question is running 9.3, but I have tested with 10.1-STABLE >>> and >>> 11-Current live CD's and I do not see /dev/ses created. I have also >>> booted >>> for testing openIndiana and do not see /dev/es/ses created. I >>> have an open ticket with SM but that so far has them asking a lot of >>> questions >>> and giving no direction. >>> >>> On the LSI cards I have tested P16, P17 and P19 all in IT mode on 9.3. >>> sas2icru on each of the adpapters does return output such as >>> >>> Enclosure information >>> ------------------------------------------------------------------------ >>> Enclosure# : 1 >>> Logical ID : 500605b0:065e26b0 >>> Numslots : 8 >>> StartSlot : 0 >>> ------------------------------------------------------------------------ >>> >>> What stands out is is not identefied for 'mps0' in dmesg. >>> However, >>> drives attached to all controllers do work fine and are identified. >>> >>> Full dmesg and output of sas2icru for each controller >>> https://charon.gopai.com/dmesg >>> >>> Booting 9.3 verbosely with hw.mps.debug_level=0xd in /boot/loader.conf >>> yeilds: >>> >>> mps1: port 0xcc00-0xccff mem >>> 0xde8bc000-0xde8bffff,0xde8c0000-0xde8fffff irq 35 at device 0.0 on pci19 >>> mps1: Firmware: 16.00.00.00, Driver: 16.00.00.00-fbsd >>> mps1: IOCCapabilities: >>> >>> 1285c >>> mps1: EventReply : >>> mps1: EventReply : >>> mps1: attempting to allocate 1 MSI-X vectors (15 supported) >>> mps1: using IRQ 272 for MSI-X >>> mps2: port 0xdc00-0xdcff mem >>> 0xdeabc000-0xdeabffff,0xdeac0000-0xdeafffff irq 33 at device 0.0 on pci18 >>> mps2: Firmware: 16.00.00.00, Driver: 16.00.00.00-fbsd >>> mps2: IOCCapabilities: >>> >>> 1285c >>> mps2: EventReply : >>> mps2: EventReply : >>> mps2: attempting to allocate 1 MSI-X vectors (15 supported) >>> mps2: using IRQ 273 for MSI-X >>> mps0: EventReply : >>> mps0: EventReply : >>> mps0: EventReply : >>> mps0: EventReply : >>> mps0: EventReply : >>> mps0: EventReply : >>> mps0: EventReply : >>> mps0: EventReply : >>> mps0: EventReply : >>> mps0: EventReply : >>> mps0: EventReply : >>> mps0: EventReply : >>> mps0: EventReply : >>> mps0: EventReply : >>> mps1: EventReply : >>> mps1: EventReply : >>> mps1: EventReply : >>> mps1: EventReply : >>> mps1: EventReply : >>> mps1: EventReply : >>> mps1: EventReply : >>> mps1: EventReply : >>> mps1: EventReply : >>> mps1: EventReply : >>> mps1: EventReply : >>> mps1: EventReply : >>> mps2: EventReply : >>> mps2: EventReply : >>> mps2: EventReply : >>> mps2: EventReply : >>> mps2: EventReply : >>> mps2: EventReply : >>> mps2: EventReply : >>> mps2: EventReply : >>> mps2: EventReply : >>> mps2: EventReply : >>> mps2: EventReply : >>> mps2: EventReply : >>> mps2: EventReply : >>> mps2: EventReply : >>> mps0: (0)->(mpssas_fw_work) Working on Event: [16] >>> mps0: (1)->(mpssas_fw_work) Event Free: [16] >>> mps0: (1)->(mpssas_fw_work) Working on Event: [1d] >>> mps0: (2)->(mpssas_fw_work) Event Free: [1d] >>> mps0: (2)->(mpssas_fw_work) Working on Event: [1c] >>> mps1: (3)->(mpssas_fw_work) Working on Event: [16] >>> mps1: (4)->(mpssas_fw_work) Event Free: [16] >>> mps1: (4)->(mpssas_fw_work) Working on Event: [1d] >>> mps1: (5)->(mpssas_fw_work) Event Free: [1d] >>> mps1: (5)->(mpssas_fw_work) Working on Event: [1c] >>> mps2: (6)->(mpssas_fw_work) Working on Event: [16] >>> mps2: (7)->(mpssas_fw_work) Event Free: [16] >>> mps2: (7)->(mpssas_fw_work) Working on Event: [1d] >>> mps2: (8)->(mpssas_fw_work) Event Free: [1d] >>> mps2: (8)->(mpssas_fw_work) Working on Event: [1c] >>> (probe0:mps1:0:0:0): Down reving Protocol Version from 4 to 0? >>> (probe1:mps1:0:1:0): Down reving Protocol Version from 4 to 0? >>> (probe2:mps1:0:2:0): Down reving Protocol Version from 4 to 0? >>> (probe3:mps1:0:4:0): Down reving Protocol Version from 4 to 0? >>> (probe4:mps2:0:0:0): Down reving Protocol Version from 4 to 0? >>> (probe5:mps2:0:1:0): Down reving Protocol Version from 4 to 0? >>> mps1: (9)->(mpssas_fw_work) Event Free: [1c] >>> mps1: (9)->(mpssas_fw_work) Working on Event: [16] >>> mps1: (10)->(mpssas_fw_work) Event Free: [16] >>> mps1: (10)->(mpssas_fw_work) Working on Event: [16] >>> mps1: (11)->(mpssas_fw_work) Event Free: [16] >>> mps1: (11)->(mpssas_fw_work) Working on Event: [16] >>> mps1: (12)->(mpssas_fw_work) Event Free: [16] >>> mps1: (12)->(mpssas_fw_work) Working on Event: [16] >>> mps1: (13)->(mpssas_fw_work) Event Free: [16] >>> mps1: (13)->(mpssas_fw_work) Working on Event: [16] >>> mps1: (14)->(mpssas_fw_work) Event Free: [16] >>> mps1: (14)->(mpssas_fw_work) Working on Event: [16] >>> mps1: (15)->(mpssas_fw_work) Event Free: [16] >>> mps1: (15)->(mpssas_fw_work) Working on Event: [16] >>> mps1: (16)->(mpssas_fw_work) Event Free: [16] >>> mps1: (16)->(mpssas_fw_work) Working on Event: [16] >>> mps1: (17)->(mpssas_fw_work) Event Free: [16] >>> mps1: (17)->(mpssas_fw_work) Working on Event: [16] >>> mps1: (18)->(mpssas_fw_work) Event Free: [16] >>> (probe6:mps1:0:5:0): Down reving Protocol Version from 4 to 0? >>> (probe7:mps2:0:2:0): Down reving Protocol Version from 4 to 0? >>> (probe8:mps2:0:6:0): Down reving Protocol Version from 4 to 0? >>> (probe9:mps2:0:4:0): Down reving Protocol Version from 4 to 0? >>> mps2: (18)->(mpssas_fw_work) Event Free: [1c] >>> mps2: (18)->(mpssas_fw_work) Working on Event: [16] >>> mps2: (19)->(mpssas_fw_work) Event Free: [16] >>> mps2: (19)->(mpssas_fw_work) Working on Event: [16] >>> mps2: (20)->(mpssas_fw_work) Event Free: [16] >>> mps2: (20)->(mpssas_fw_work) Working on Event: [16] >>> mps2: (21)->(mpssas_fw_work) Event Free: [16] >>> mps2: (21)->(mpssas_fw_work) Working on Event: [16] >>> mps2: (22)->(mpssas_fw_work) Event Free: [16] >>> mps2: (22)->(mpssas_fw_work) Working on Event: [16] >>> mps2: (23)->(mpssas_fw_work) Event Free: [16] >>> mps2: (23)->(mpssas_fw_work) Working on Event: [16] >>> mps2: (24)->(mpssas_fw_work) Event Free: [16] >>> mps2: (24)->(mpssas_fw_work) Working on Event: [16] >>> mps2: (25)->(mpssas_fw_work) Event Free: [16] >>> mps2: (25)->(mpssas_fw_work) Working on Event: [16] >>> mps2: (26)->(mpssas_fw_work) Event Free: [16] >>> mps2: (26)->(mpssas_fw_work) Working on Event: [16] >>> mps2: (27)->(mpssas_fw_work) Event Free: [16] >>> mps2: (27)->(mpssas_fw_work) Working on Event: [16] >>> mps2: (28)->(mpssas_fw_work) Event Free: [16] >>> mps2: (28)->(mpssas_fw_work) Working on Event: [16] >>> mps2: (29)->(mpssas_fw_work) Event Free: [16] >>> (probe0:mps2:0:5:0): Down reving Protocol Version from 4 to 0? >>> (probe1:mps0:0:0:0): Down reving Protocol Version from 4 to 0? >>> (probe2:mps0:0:1:0): Down reving Protocol Version from 4 to 0? >>> (probe3:mps0:0:2:0): Down reving Protocol Version from 4 to 0? >>> (probe5:mps0:0:3:0): Down reving Protocol Version from 4 to 0? >>> (probe6:mps0:0:4:0): Down reving Protocol Version from 4 to 0? >>> mps0: (29)->(mpssas_fw_work) Event Free: [1c] >>> mps0: (29)->(mpssas_fw_work) Working on Event: [16] >>> mps0: (30)->(mpssas_fw_work) Event Free: [16] >>> mps0: (30)->(mpssas_fw_work) Working on Event: [16] >>> mps0: (31)->(mpssas_fw_work) Event Free: [16] >>> mps0: (31)->(mpssas_fw_work) Working on Event: [16] >>> mps0: (32)->(mpssas_fw_work) Event Free: [16] >>> mps0: (32)->(mpssas_fw_work) Working on Event: [16] >>> mps0: (33)->(mpssas_fw_work) Event Free: [16] >>> mps0: (33)->(mpssas_fw_work) Working on Event: [16] >>> mps0: (34)->(mpssas_fw_work) Event Free: [16] >>> mps0: (34)->(mpssas_fw_work) Working on Event: [16] >>> mps0: (35)->(mpssas_fw_work) Event Free: [16] >>> mps0: (35)->(mpssas_fw_work) Working on Event: [16] >>> mps0: (36)->(mpssas_fw_work) Event Free: [16] >>> mps0: (36)->(mpssas_fw_work) Working on Event: [16] >>> mps0: (37)->(mpssas_fw_work) Event Free: [16] >>> mps0: (37)->(mpssas_fw_work) Working on Event: [16] >>> mps0: (38)->(mpssas_fw_work) Event Free: [16] >>> mps0: (38)->(mpssas_fw_work) Working on Event: [16] >>> mps0: (39)->(mpssas_fw_work) Event Free: [16] >>> mps0: (39)->(mpssas_fw_work) Working on Event: [16] >>> mps0: (40)->(mpssas_fw_work) Event Free: [16] >>> (probe0:mps0:0:6:0): Down reving Protocol Version from 4 to 0? >>> da10 at mps0 bus 0 scbus0 target 0 lun 0 >>> da0 at mps0 bus 0 scbus0 target 2 lun 0 >>> da1 at mps0 bus 0 scbus0 target 3 lun 0 >>> da11 at mps0 bus 0 scbus0 target 1 lun 0 >>> da2 at mps0 bus 0 scbus0 target 4 lun 0 >>> da18 at mps1 bus 0 scbus1 target 0 lun 0 >>> da19 at mps1 bus 0 scbus1 target 1 lun 0 >>> da20 at mps1 bus 0 scbus1 target 2 lun 0 >>> da23 at mps1 bus 0 scbus1 target 5 lun 0 >>> da14 at mps2 bus 0 scbus2 target 0 lun 0 >>> da15 at mps2 bus 0 scbus2 target 1 lun 0 >>> da16 at mps2 bus 0 scbus2 target 2 lun 0 >>> da22 at mps1 bus 0 scbus1 target 4 lun 0 >>> da6 at mps2 bus 0 scbus2 target 4 lun 0 >>> pass0 at mps0 bus 0 scbus0 target 0 lun 0 >>> da8 at mps2 bus 0 scbus2 target 6 lun 0 >>> pass1 at mps0 bus 0 scbus0 target 1 lun 0 >>> pass2 at mps0 bus 0 scbus0 target 2 lun 0 >>> pass3 at mps0 bus 0 scbus0 target 3 lun 0 >>> pass4 at mps0 bus 0 scbus0 target 4 lun 0 >>> pass5 at mps0 bus 0 scbus0 target 6 lun 0 >>> pass6 at mps1 bus 0 scbus1 target 0 lun 0 >>> pass7 at mps1 bus 0 scbus1 target 1 lun 0 >>> da7 at mps2 bus 0 scbus2 target 5 lun 0 >>> pass8 at mps1 bus 0 scbus1 target 2 lun 0 >>> pass9 at mps1 bus 0 scbus1 target 4 lun 0 >>> pass10 at mps1 bus 0 scbus1 target 5 lun 0 >>> pass11 at mps2 bus 0 scbus2 target 0 lun 0 >>> pass12 at mps2 bus 0 scbus2 target 1 lun 0 >>> pass13 at mps2 bus 0 scbus2 target 2 lun 0 >>> pass14 at mps2 bus 0 scbus2 target 4 lun 0 >>> pass15 at mps2 bus 0 scbus2 target 5 lun 0 >>> pass16 at mps2 bus 0 scbus2 target 6 lun 0 >>> da4 at mps0 bus 0 scbus0 target 6 lun 0 >>> >>> I admit I am new to enclosure management - What am I missing? >>> >>> Thank you. >>> >>> --mikej >>> _______________________________________________ >>> freebsd-scsi@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-scsi >>> To unsubscribe, send any mail to "freebsd-scsi-unsubscribe@freebsd.org" > > From owner-freebsd-scsi@FreeBSD.ORG Thu May 28 18:11:26 2015 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 27370551; Thu, 28 May 2015 18:11:26 +0000 (UTC) (envelope-from mikej@mikej.com) Received: from mx2.paymentallianceintl.com (mx2.paymentallianceintl.com [216.26.158.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx2.paymentallianceintl.com", Issuer "Go Daddy Secure Certification Authority" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C9FED346; Thu, 28 May 2015 18:11:25 +0000 (UTC) (envelope-from mikej@mikej.com) Received: from firewall.mikej.com (162-230-214-65.lightspeed.lsvlky.sbcglobal.net [162.230.214.65]) by mx2.paymentallianceintl.com (8.15.1/8.15.1) with ESMTPS id t4SIBNVg027572 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 28 May 2015 14:11:23 -0400 (EDT) (envelope-from mikej@mikej.com) X-Authentication-Warning: mx2.paymentallianceintl.com: Host 162-230-214-65.lightspeed.lsvlky.sbcglobal.net [162.230.214.65] claimed to be firewall.mikej.com Received: from mail.mikej.com (firewall [192.168.6.63]) by firewall.mikej.com (8.15.1/8.14.9) with ESMTP id t4SIBMYp084846; Thu, 28 May 2015 14:11:22 -0400 (EDT) (envelope-from mikej@mikej.com) X-Authentication-Warning: firewall.mikej.com: Host firewall [192.168.6.63] claimed to be mail.mikej.com MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Thu, 28 May 2015 14:11:22 -0400 From: Michael Jung To: Alan Somers Cc: FreeBSD-scsi , asomers@gmail.com Subject: Re: /dev/ses not created In-Reply-To: References: <76e9e3f41990026290382e5fa9c9b8d3@mail.mikej.com> <026292acf4288dbde00ce9bcb8686490@mail.mikej.com> Message-ID: <112e22fa5a66ac54f7a7f56efa62a7d9@mail.mikej.com> X-Sender: mikej@mikej.com User-Agent: Roundcube Webmail/1.0.4 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 May 2015 18:11:26 -0000 On 2015-05-28 12:13, Alan Somers wrote: > On Thu, May 28, 2015 at 9:22 AM, Michael Jung wrote: >> Alan, >> >> Thank you for the information. >> >> So SGPIO is out of band. >> >> So the intent of Marvel chip is not for in-band SES-2? >> >> I see these events, one for each LSI card in dmesg: >> >> mps1: EventReply : >> EventDataLength: 5 >> AckRequired: 0 >> Event: SasEnclDeviceStatusChange (0x1d) <---- >> EventContext: 0x0 >> EnclosureHandle: 0x1 >> ReasonCode: Added >> PhysicalPort: 0 >> NumSlots: 8 <---- >> StartSlot: 0 >> PhyBits: 0x0 > > > Assuming you have IT firmware, those HBAs each have a builtin expander > with the same number of slots as the card has ports. It's not a > separate chip; it's just part of the HBA controller. That expander > does not have SES functionality, however. > Alan, Ok, I understand this event is for the expander on the HBA. Per your suggestion I opened a ticket with LSI this AM, and, when I get something definitive will post back. If there is a sound FAQ on I2c, SGPIO, SES/SES-2, cabling along with common problems and use cases I have not found it - I can dream though. A lot of reading internet posts right or wrong and reading between the lines in manuals. Or I may just be getting older. Frustrating. Thank you again. From owner-freebsd-scsi@FreeBSD.ORG Thu May 28 19:39:37 2015 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DFA348F9 for ; Thu, 28 May 2015 19:39:36 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-oi0-x22d.google.com (mail-oi0-x22d.google.com [IPv6:2607:f8b0:4003:c06::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A51B2C3A for ; Thu, 28 May 2015 19:39:36 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by oihd6 with SMTP id d6so40603271oih.2 for ; Thu, 28 May 2015 12:39:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=8B84j884zyMNOUAZr8KccndQQSQvO7yxfYgmy6RoQP8=; b=ft5e0CG2QLy9BQxX2qFGvgwTV0Dmff+pwy3CgvKyTbTQMO2WPF0haXwOhCW32VPx+X 4d0FG0yOAJTZs1n0UEMmRTEWUxkhABVxSXX8MNm112xiQuT4gmU55o5QCCv3RKUez5zW tUk/wUIJpqWtKs0velxGMr1dVLCiJzcSwPUXOSiU/I3pjukW6ZPahVbjBOlrDJfanHBn hoFX+jivdAPqLbVM4Vw4flPbPHkCGwxxRZYdeVIz2Oy5FEfSyErFrBj9hIXxy6cN6QAw +O27d3DyKrQzz/uGocDhG4MSGsQf+pfdQ0RRbTgBtyn7gWxo7upWauXA/Ybl20+0AfbQ R3xg== MIME-Version: 1.0 X-Received: by 10.202.191.6 with SMTP id p6mr3652778oif.109.1432841975838; Thu, 28 May 2015 12:39:35 -0700 (PDT) Sender: asomers@gmail.com Received: by 10.202.107.3 with HTTP; Thu, 28 May 2015 12:39:35 -0700 (PDT) In-Reply-To: <112e22fa5a66ac54f7a7f56efa62a7d9@mail.mikej.com> References: <76e9e3f41990026290382e5fa9c9b8d3@mail.mikej.com> <026292acf4288dbde00ce9bcb8686490@mail.mikej.com> <112e22fa5a66ac54f7a7f56efa62a7d9@mail.mikej.com> Date: Thu, 28 May 2015 13:39:35 -0600 X-Google-Sender-Auth: AnjnPhOx7wDbFBhElLdkReDcg4A Message-ID: Subject: Re: /dev/ses not created From: Alan Somers To: Michael Jung Cc: FreeBSD-scsi Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 May 2015 19:39:37 -0000 On Thu, May 28, 2015 at 12:11 PM, Michael Jung wrote: > On 2015-05-28 12:13, Alan Somers wrote: >> >> On Thu, May 28, 2015 at 9:22 AM, Michael Jung wrote: >>> >>> Alan, >>> >>> Thank you for the information. >>> >>> So SGPIO is out of band. >>> >>> So the intent of Marvel chip is not for in-band SES-2? >>> >>> I see these events, one for each LSI card in dmesg: >>> >>> mps1: EventReply : >>> EventDataLength: 5 >>> AckRequired: 0 >>> Event: SasEnclDeviceStatusChange (0x1d) <---- >>> EventContext: 0x0 >>> EnclosureHandle: 0x1 >>> ReasonCode: Added >>> PhysicalPort: 0 >>> NumSlots: 8 <---- >>> StartSlot: 0 >>> PhyBits: 0x0 >> >> >> >> Assuming you have IT firmware, those HBAs each have a builtin expander >> with the same number of slots as the card has ports. It's not a >> separate chip; it's just part of the HBA controller. That expander >> does not have SES functionality, however. >> > > Alan, > > Ok, I understand this event is for the expander on the HBA. Per your > suggestion > I opened a ticket with LSI this AM, and, when I get something definitive > will post > back. > > If there is a sound FAQ on I2c, SGPIO, SES/SES-2, cabling along with common > problems and use cases I have not found it - I can dream though. The wikipedia article on SGPIO is pretty good. The SES article is less good, but the SES3 standard isn't too hard to understand by itself. I2C is a very general purpose bus that has nothing to do with storage. I didn't think there was any kind of standard for reporting SES-like information over I2C, but Google suggests that there is. I can't find the standard though. The SES2 and SES3 specs don't mention I2C at all. You might ask Alexander Motin (mav@). He wrote ses(4) and if anybody knows anything about SES over I2C, it would be him. https://en.wikipedia.org/wiki/SGPIO -Alan > > A lot of reading internet posts right or wrong and reading between the lines > in manuals. > > Or I may just be getting older. > > Frustrating. > > Thank you again.