Date: Thu, 28 May 2015 10:13:25 -0600 From: Alan Somers <asomers@freebsd.org> To: Michael Jung <mikej@mikej.com> Cc: FreeBSD-scsi <freebsd-scsi@freebsd.org> Subject: Re: /dev/ses<x> not created Message-ID: <CAOtMX2hEj%2BpUooEspM3_76nvuwNkDVSnKk-AqhMbZoU3NQHKaQ@mail.gmail.com> In-Reply-To: <026292acf4288dbde00ce9bcb8686490@mail.mikej.com> References: <76e9e3f41990026290382e5fa9c9b8d3@mail.mikej.com> <CAOtMX2jKOuUAFF2FFJXJzWDX-mMac=TGUTzwTjjVbjT-NtAmYw@mail.gmail.com> <026292acf4288dbde00ce9bcb8686490@mail.mikej.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, May 28, 2015 at 9:22 AM, Michael Jung <mikej@mikej.com> wrote: > Alan, > > Thank you for the information. > > So SGPIO is out of band. > > So the intent of Marvel chip is not for in-band SES-2? > > I see these events, one for each LSI card in dmesg: > > mps1: EventReply : > EventDataLength: 5 > AckRequired: 0 > Event: SasEnclDeviceStatusChange (0x1d) <---- > EventContext: 0x0 > EnclosureHandle: 0x1 > ReasonCode: Added > PhysicalPort: 0 > NumSlots: 8 <---- > StartSlot: 0 > PhyBits: 0x0 Assuming you have IT firmware, those HBAs each have a builtin expander with the same number of slots as the card has ports. It's not a separate chip; it's just part of the HBA controller. That expander does not have SES functionality, however. > > The motherboard is not in the same cabinet as the backplane, > and and I would rather not just replace the backplane with > one that had an expander on it $$$. Understood. But HBAs are expensive, too. If you find yourself needing to order an HBA, you ought to consider whether it would be cheaper to order a new backplane and free up two of the HBAs that you have. > > Maybe I'm just seeking functionality that is not present but > the fact that the backplane manual cleary states "The MG9072 > is an enclosure management chip that supports the SES-2 controller > and SES-2 protocols." make me -want- to believe something else > is amiss. If there's no expander and no dedicated SAS port just for SES, then you definitely can't use SES. SES is in-band. Perhaps Supermicro meant "The MG9072 supports SES-2, but we didn't bother to route a SAS port to it, so this backplane is stuck with SGPIO". You might be able to figure out how to make those HBAs expose their SGPIO controllers. But even if you do, it won't gain you much. SGPIO is pretty lame. There is no way to automatically correlate an SGPIO element with a da device. You'll have to do it manually, and it will be dependent on your cabling arrangement. And if you add or remove drives, that will change your da numbers and you'll have to determine the mapping again. The mps driver supports "persistent mapping" mode, where each slot always gets mapped to the same bus-target-lun, but the da driver does not map the same bus-target-lun to a consistent da device. So your mapping process will have to involve "camcontrol devlist" to map da devices to bus-target-luns. There's also AFAIK no software that will automatically manage the SGPIO LEDs for you. You'll have to write your own based on the output of "zpool status" or something. Sorry to rain on the parade, but SGPIO just basically sucks. -Alan > > Thank you. > > --mikej > > > > > On 2015-05-28 10:55, Alan Somers wrote: >> >> SGPIO is an out-of-band protocol. It doesn't use the same wires as >> the SAS connections. It's odd that the manual says the enclosure >> management chip supports SES-2 and yet there is no SAS port for it. >> The chassis manual also shows that headers 7-9 are for SGPIO. Are >> they connected to anything? Some Supermicro motherboards have SGPIO >> headers. But I would think that if yours did, then the ses device >> would show up even if the cable were unplugged. LSI HBAs are also >> supposed to have builtin SGPIO controllers. But I don't know how to >> enable them or if FreeBSD has driver support for them. You might >> learn more by asking LSI support than SuperMicro support. >> >> If you do have an SGPIO controller on the motherboard, you will need >> FreeBSD 10.0 or later to use it. >> >> Finally, I know you probably don't want to spend more money, but if >> you want enclosure management then your best option is to replace the >> backplane with a SAS-846EL. It should be mechanically compatible. >> That will give you SES, which is way better than SGPIO. It also has >> an expander, so you'll only need one HBA. >> >> -Alan >> >> On Thu, May 28, 2015 at 7:45 AM, Michael Jung <mikej@mikej.com> wrote: >>> >>> I hoping to get some direction as I simply am unsure what to do next. >>> >>> I have three LSI 9200-8e adapters connected to a Super Micro >>> CSE-846A-R1200B >>> yet no SES devices get created. The Backplane in the chassis >>> BPN-SAS_846A >>> does not have a expander. So I have six SFF-8088 on the backplane that >>> connect >>> directly to the ports on the 9200-8e albeit externally.The backplane does >>> have >>> a Marvel MG9072 supporting SES-2 and is jumpered for SGPIO. >>> >>> The server in question is running 9.3, but I have tested with 10.1-STABLE >>> and >>> 11-Current live CD's and I do not see /dev/ses<x> created. I have also >>> booted >>> for testing openIndiana and do not see /dev/es/ses<x> created. I >>> have an open ticket with SM but that so far has them asking a lot of >>> questions >>> and giving no direction. >>> >>> On the LSI cards I have tested P16, P17 and P19 all in IT mode on 9.3. >>> sas2icru on each of the adpapters does return output such as >>> >>> Enclosure information >>> ------------------------------------------------------------------------ >>> Enclosure# : 1 >>> Logical ID : 500605b0:065e26b0 >>> Numslots : 8 >>> StartSlot : 0 >>> ------------------------------------------------------------------------ >>> >>> What stands out is <LSI SAS2008> is not identefied for 'mps0' in dmesg. >>> However, >>> drives attached to all controllers do work fine and are identified. >>> >>> Full dmesg and output of sas2icru for each controller >>> https://charon.gopai.com/dmesg >>> >>> Booting 9.3 verbosely with hw.mps.debug_level=0xd in /boot/loader.conf >>> yeilds: >>> >>> mps1: <LSI SAS2008> port 0xcc00-0xccff mem >>> 0xde8bc000-0xde8bffff,0xde8c0000-0xde8fffff irq 35 at device 0.0 on pci19 >>> mps1: Firmware: 16.00.00.00, Driver: 16.00.00.00-fbsd >>> mps1: IOCCapabilities: >>> >>> 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc> >>> mps1: EventReply : >>> mps1: EventReply : >>> mps1: attempting to allocate 1 MSI-X vectors (15 supported) >>> mps1: using IRQ 272 for MSI-X >>> mps2: <LSI SAS2008> port 0xdc00-0xdcff mem >>> 0xdeabc000-0xdeabffff,0xdeac0000-0xdeafffff irq 33 at device 0.0 on pci18 >>> mps2: Firmware: 16.00.00.00, Driver: 16.00.00.00-fbsd >>> mps2: IOCCapabilities: >>> >>> 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc> >>> mps2: EventReply : >>> mps2: EventReply : >>> mps2: attempting to allocate 1 MSI-X vectors (15 supported) >>> mps2: using IRQ 273 for MSI-X >>> mps0: EventReply : >>> mps0: EventReply : >>> mps0: EventReply : >>> mps0: EventReply : >>> mps0: EventReply : >>> mps0: EventReply : >>> mps0: EventReply : >>> mps0: EventReply : >>> mps0: EventReply : >>> mps0: EventReply : >>> mps0: EventReply : >>> mps0: EventReply : >>> mps0: EventReply : >>> mps0: EventReply : >>> mps1: EventReply : >>> mps1: EventReply : >>> mps1: EventReply : >>> mps1: EventReply : >>> mps1: EventReply : >>> mps1: EventReply : >>> mps1: EventReply : >>> mps1: EventReply : >>> mps1: EventReply : >>> mps1: EventReply : >>> mps1: EventReply : >>> mps1: EventReply : >>> mps2: EventReply : >>> mps2: EventReply : >>> mps2: EventReply : >>> mps2: EventReply : >>> mps2: EventReply : >>> mps2: EventReply : >>> mps2: EventReply : >>> mps2: EventReply : >>> mps2: EventReply : >>> mps2: EventReply : >>> mps2: EventReply : >>> mps2: EventReply : >>> mps2: EventReply : >>> mps2: EventReply : >>> mps0: (0)->(mpssas_fw_work) Working on Event: [16] >>> mps0: (1)->(mpssas_fw_work) Event Free: [16] >>> mps0: (1)->(mpssas_fw_work) Working on Event: [1d] >>> mps0: (2)->(mpssas_fw_work) Event Free: [1d] >>> mps0: (2)->(mpssas_fw_work) Working on Event: [1c] >>> mps1: (3)->(mpssas_fw_work) Working on Event: [16] >>> mps1: (4)->(mpssas_fw_work) Event Free: [16] >>> mps1: (4)->(mpssas_fw_work) Working on Event: [1d] >>> mps1: (5)->(mpssas_fw_work) Event Free: [1d] >>> mps1: (5)->(mpssas_fw_work) Working on Event: [1c] >>> mps2: (6)->(mpssas_fw_work) Working on Event: [16] >>> mps2: (7)->(mpssas_fw_work) Event Free: [16] >>> mps2: (7)->(mpssas_fw_work) Working on Event: [1d] >>> mps2: (8)->(mpssas_fw_work) Event Free: [1d] >>> mps2: (8)->(mpssas_fw_work) Working on Event: [1c] >>> (probe0:mps1:0:0:0): Down reving Protocol Version from 4 to 0? >>> (probe1:mps1:0:1:0): Down reving Protocol Version from 4 to 0? >>> (probe2:mps1:0:2:0): Down reving Protocol Version from 4 to 0? >>> (probe3:mps1:0:4:0): Down reving Protocol Version from 4 to 0? >>> (probe4:mps2:0:0:0): Down reving Protocol Version from 4 to 0? >>> (probe5:mps2:0:1:0): Down reving Protocol Version from 4 to 0? >>> mps1: (9)->(mpssas_fw_work) Event Free: [1c] >>> mps1: (9)->(mpssas_fw_work) Working on Event: [16] >>> mps1: (10)->(mpssas_fw_work) Event Free: [16] >>> mps1: (10)->(mpssas_fw_work) Working on Event: [16] >>> mps1: (11)->(mpssas_fw_work) Event Free: [16] >>> mps1: (11)->(mpssas_fw_work) Working on Event: [16] >>> mps1: (12)->(mpssas_fw_work) Event Free: [16] >>> mps1: (12)->(mpssas_fw_work) Working on Event: [16] >>> mps1: (13)->(mpssas_fw_work) Event Free: [16] >>> mps1: (13)->(mpssas_fw_work) Working on Event: [16] >>> mps1: (14)->(mpssas_fw_work) Event Free: [16] >>> mps1: (14)->(mpssas_fw_work) Working on Event: [16] >>> mps1: (15)->(mpssas_fw_work) Event Free: [16] >>> mps1: (15)->(mpssas_fw_work) Working on Event: [16] >>> mps1: (16)->(mpssas_fw_work) Event Free: [16] >>> mps1: (16)->(mpssas_fw_work) Working on Event: [16] >>> mps1: (17)->(mpssas_fw_work) Event Free: [16] >>> mps1: (17)->(mpssas_fw_work) Working on Event: [16] >>> mps1: (18)->(mpssas_fw_work) Event Free: [16] >>> (probe6:mps1:0:5:0): Down reving Protocol Version from 4 to 0? >>> (probe7:mps2:0:2:0): Down reving Protocol Version from 4 to 0? >>> (probe8:mps2:0:6:0): Down reving Protocol Version from 4 to 0? >>> (probe9:mps2:0:4:0): Down reving Protocol Version from 4 to 0? >>> mps2: (18)->(mpssas_fw_work) Event Free: [1c] >>> mps2: (18)->(mpssas_fw_work) Working on Event: [16] >>> mps2: (19)->(mpssas_fw_work) Event Free: [16] >>> mps2: (19)->(mpssas_fw_work) Working on Event: [16] >>> mps2: (20)->(mpssas_fw_work) Event Free: [16] >>> mps2: (20)->(mpssas_fw_work) Working on Event: [16] >>> mps2: (21)->(mpssas_fw_work) Event Free: [16] >>> mps2: (21)->(mpssas_fw_work) Working on Event: [16] >>> mps2: (22)->(mpssas_fw_work) Event Free: [16] >>> mps2: (22)->(mpssas_fw_work) Working on Event: [16] >>> mps2: (23)->(mpssas_fw_work) Event Free: [16] >>> mps2: (23)->(mpssas_fw_work) Working on Event: [16] >>> mps2: (24)->(mpssas_fw_work) Event Free: [16] >>> mps2: (24)->(mpssas_fw_work) Working on Event: [16] >>> mps2: (25)->(mpssas_fw_work) Event Free: [16] >>> mps2: (25)->(mpssas_fw_work) Working on Event: [16] >>> mps2: (26)->(mpssas_fw_work) Event Free: [16] >>> mps2: (26)->(mpssas_fw_work) Working on Event: [16] >>> mps2: (27)->(mpssas_fw_work) Event Free: [16] >>> mps2: (27)->(mpssas_fw_work) Working on Event: [16] >>> mps2: (28)->(mpssas_fw_work) Event Free: [16] >>> mps2: (28)->(mpssas_fw_work) Working on Event: [16] >>> mps2: (29)->(mpssas_fw_work) Event Free: [16] >>> (probe0:mps2:0:5:0): Down reving Protocol Version from 4 to 0? >>> (probe1:mps0:0:0:0): Down reving Protocol Version from 4 to 0? >>> (probe2:mps0:0:1:0): Down reving Protocol Version from 4 to 0? >>> (probe3:mps0:0:2:0): Down reving Protocol Version from 4 to 0? >>> (probe5:mps0:0:3:0): Down reving Protocol Version from 4 to 0? >>> (probe6:mps0:0:4:0): Down reving Protocol Version from 4 to 0? >>> mps0: (29)->(mpssas_fw_work) Event Free: [1c] >>> mps0: (29)->(mpssas_fw_work) Working on Event: [16] >>> mps0: (30)->(mpssas_fw_work) Event Free: [16] >>> mps0: (30)->(mpssas_fw_work) Working on Event: [16] >>> mps0: (31)->(mpssas_fw_work) Event Free: [16] >>> mps0: (31)->(mpssas_fw_work) Working on Event: [16] >>> mps0: (32)->(mpssas_fw_work) Event Free: [16] >>> mps0: (32)->(mpssas_fw_work) Working on Event: [16] >>> mps0: (33)->(mpssas_fw_work) Event Free: [16] >>> mps0: (33)->(mpssas_fw_work) Working on Event: [16] >>> mps0: (34)->(mpssas_fw_work) Event Free: [16] >>> mps0: (34)->(mpssas_fw_work) Working on Event: [16] >>> mps0: (35)->(mpssas_fw_work) Event Free: [16] >>> mps0: (35)->(mpssas_fw_work) Working on Event: [16] >>> mps0: (36)->(mpssas_fw_work) Event Free: [16] >>> mps0: (36)->(mpssas_fw_work) Working on Event: [16] >>> mps0: (37)->(mpssas_fw_work) Event Free: [16] >>> mps0: (37)->(mpssas_fw_work) Working on Event: [16] >>> mps0: (38)->(mpssas_fw_work) Event Free: [16] >>> mps0: (38)->(mpssas_fw_work) Working on Event: [16] >>> mps0: (39)->(mpssas_fw_work) Event Free: [16] >>> mps0: (39)->(mpssas_fw_work) Working on Event: [16] >>> mps0: (40)->(mpssas_fw_work) Event Free: [16] >>> (probe0:mps0:0:6:0): Down reving Protocol Version from 4 to 0? >>> da10 at mps0 bus 0 scbus0 target 0 lun 0 >>> da0 at mps0 bus 0 scbus0 target 2 lun 0 >>> da1 at mps0 bus 0 scbus0 target 3 lun 0 >>> da11 at mps0 bus 0 scbus0 target 1 lun 0 >>> da2 at mps0 bus 0 scbus0 target 4 lun 0 >>> da18 at mps1 bus 0 scbus1 target 0 lun 0 >>> da19 at mps1 bus 0 scbus1 target 1 lun 0 >>> da20 at mps1 bus 0 scbus1 target 2 lun 0 >>> da23 at mps1 bus 0 scbus1 target 5 lun 0 >>> da14 at mps2 bus 0 scbus2 target 0 lun 0 >>> da15 at mps2 bus 0 scbus2 target 1 lun 0 >>> da16 at mps2 bus 0 scbus2 target 2 lun 0 >>> da22 at mps1 bus 0 scbus1 target 4 lun 0 >>> da6 at mps2 bus 0 scbus2 target 4 lun 0 >>> pass0 at mps0 bus 0 scbus0 target 0 lun 0 >>> da8 at mps2 bus 0 scbus2 target 6 lun 0 >>> pass1 at mps0 bus 0 scbus0 target 1 lun 0 >>> pass2 at mps0 bus 0 scbus0 target 2 lun 0 >>> pass3 at mps0 bus 0 scbus0 target 3 lun 0 >>> pass4 at mps0 bus 0 scbus0 target 4 lun 0 >>> pass5 at mps0 bus 0 scbus0 target 6 lun 0 >>> pass6 at mps1 bus 0 scbus1 target 0 lun 0 >>> pass7 at mps1 bus 0 scbus1 target 1 lun 0 >>> da7 at mps2 bus 0 scbus2 target 5 lun 0 >>> pass8 at mps1 bus 0 scbus1 target 2 lun 0 >>> pass9 at mps1 bus 0 scbus1 target 4 lun 0 >>> pass10 at mps1 bus 0 scbus1 target 5 lun 0 >>> pass11 at mps2 bus 0 scbus2 target 0 lun 0 >>> pass12 at mps2 bus 0 scbus2 target 1 lun 0 >>> pass13 at mps2 bus 0 scbus2 target 2 lun 0 >>> pass14 at mps2 bus 0 scbus2 target 4 lun 0 >>> pass15 at mps2 bus 0 scbus2 target 5 lun 0 >>> pass16 at mps2 bus 0 scbus2 target 6 lun 0 >>> da4 at mps0 bus 0 scbus0 target 6 lun 0 >>> >>> I admit I am new to enclosure management - What am I missing? >>> >>> Thank you. >>> >>> --mikej >>> _______________________________________________ >>> freebsd-scsi@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-scsi >>> To unsubscribe, send any mail to "freebsd-scsi-unsubscribe@freebsd.org" > >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOtMX2hEj%2BpUooEspM3_76nvuwNkDVSnKk-AqhMbZoU3NQHKaQ>