Date: Fri, 10 Mar 2017 08:39:20 -0700 From: Alan Somers <asomers@freebsd.org> To: John <jwd@freebsd.org> Cc: Alexander Motin <mav@freebsd.org>, FreeBSD-SCSI <freebsd-scsi@freebsd.org> Subject: Re: sesutil not seeing disk in first bay Message-ID: <CAOtMX2jT80XSPk7ZPkrCQCMJ0eWkF5jL6FmXha_wyC_CW7yssQ@mail.gmail.com> In-Reply-To: <20170310143610.GA15302@FreeBSD.org> References: <20170224102748.GA34792@FreeBSD.org> <0fec6ff8-ede3-d74d-0b81-6ed7e80dfc6d@FreeBSD.org> <20170224140002.GA77466@FreeBSD.org> <CAOtMX2i=6CeqGyCZR9e5zwj036-RCPxboBz1nUE7ALoOPHCUDg@mail.gmail.com> <20170310143610.GA15302@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
I doubt the change that mav mentioned is relevant, because EIIOE=0 in your case. But there is something very strange about your enclosure. Your SES page 2 show multiple sections for the same element type. It looks like you have three sections of SAS Connectors, two sections of Power Supplies, three sections of Enclosure Services Controller Electronics, two sections of Display, and three of Temperature Sensors. This will be more obvious if you do "sg_ses -p 1 /dev/ses0". I didn't think the SES spec allowed multiple sections of the same element type within the same subenclosure. Definitely weird. But that's not what's causing your problem. The first element in your SES page 10 has an Element Index of 1, and eiioe=0. That means it refers to Array Device element 1. There is no additional element status descriptor with Element Index 0. That means that sesutil is faithfully reporting the junk that your expander reports. You have a buggy SES controller. It ought to be setting EIIOE=0x1b, as described in Table 33 of SES 3 r13. You should contact HP and ask for a firmware upgrade. -Alan On Fri, Mar 10, 2017 at 7:36 AM, John <jwd@freebsd.org> wrote: > Sorry for the slow response, system was being moved. > > ----- Alan Somers's Original Message ----- >> Hard to say whether the problem is in sesutil or in your expander's >> firmware. Could you please install sysutils/sg3_utils and run "sg_ses >> -p2 /dev/ses0" and "sg_ses -p10 /dev/ses0" ? > > Requested outputs: > > sg_ses -p2 /dev/ses0 - https://people.freebsd.org/~jwd/ses0.sg_ses-p2.txt > > sg_ses -p10 /dev/ses0 - https://people.freebsd.org/~jwd/ses0.sg_ses-p10.txt > > For completeness: > > camcontrol smpphylist ses0 - https://people.freebsd.org/~jwd/ses0.camcontrol-smpphylist.txt > > sesutil -u /dev/ses0 map - https://people.freebsd.org/~jwd/ses0.sesutil-map.txt > > Using a combination of smpphylist to get the phy & disk seems > to always work in conjunction with sg_ses -p 10 to map the phy/disk > to a slot number by matching the phy. > > sesutil seems like it's off by 1. > > Thanks, > John > >> -Alan >> >> On Fri, Feb 24, 2017 at 7:00 AM, John <jwd@freebsd.org> wrote: >> > ----- Alexander Motin's Original Message ----- >> >> Hi. >> >> >> >> I am not sure I completely understand from your description how exactly >> >> detected disks match their enclosure slots, but make sure to take a look >> >> on this change: >> >> https://svnweb.freebsd.org/base?view=revision&revision=308425 >> > >> > Yes - this patch is on my system. >> > >> > To phrase it differently - the shelf contains 25 drives. Disks da0 >> > through da23 show up via sesutil: >> > >> > # sesutil map -u /dev/ses0 | grep da >> > Device Names: da0,pass0 >> > Device Names: da1,pass1 >> > Device Names: da2,pass2 >> > Device Names: da3,pass3 >> > Device Names: da4,pass4 >> > Device Names: da5,pass5 >> > Device Names: da6,pass6 >> > Device Names: da7,pass7 >> > Device Names: da8,pass8 >> > Device Names: da9,pass9 >> > Device Names: da10,pass10 >> > Device Names: da11,pass11 >> > Device Names: da12,pass12 >> > Device Names: da13,pass13 >> > Device Names: da14,pass14 >> > Device Names: da15,pass15 >> > Device Names: da16,pass16 >> > Device Names: da17,pass17 >> > Device Names: da18,pass18 >> > Device Names: da19,pass19 >> > Device Names: da20,pass20 >> > Device Names: da21,pass21 >> > Device Names: da22,pass22 >> > Device Names: da23,pass23 >> > >> > Disk da24 does not show up but is present in the first bay. da0 >> > is in bay 2, etc. >> > >> > I hope this helps. >> > >> > Thanks, >> > John >> > >> >> On 24.02.2017 12:27, John wrote: >> >> > Hi Folks - >> >> > >> >> > Running fresh fbsd 11.0-STABLE #1 r314197: Fri Feb 24 05:05:34 EST 2017. >> >> > >> >> > I'm wondering if anyone using sesutil has seen the following issue >> >> > with an HP D2700 shelf (25 x 600GB SAS drives): >> >> > >> >> > sesutil map -u /dev/ses0 >> >> > ses0: >> >> > Enclosure Name: HP D2700 SAS AJ941A 0149 >> >> > Enclosure ID: 5001438023432c00 >> >> > Element 0, Type: Array Device Slot >> >> > Status: Unsupported (0x00 0x80 0x00 0x00) >> >> > Element 1, Type: Array Device Slot >> >> > Status: OK (0x01 0x80 0x00 0x00) >> >> > Element 2, Type: Array Device Slot >> >> > Status: OK (0x01 0x80 0x00 0x00) >> >> > Device Names: da0,pass0 >> >> > Element 3, Type: Array Device Slot >> >> > Status: OK (0x01 0x80 0x00 0x00) >> >> > Device Names: da1,pass1 >> >> > Element 4, Type: Array Device Slot >> >> > Status: OK (0x01 0x80 0x00 0x00) >> >> > Device Names: da2,pass2 >> >> > ... >> >> > ... full list below >> >> > ... >> >> > Element 25, Type: Array Device Slot >> >> > Status: OK (0x01 0x80 0x00 0x00) >> >> > Device Names: da23,pass23 >> >> > >> >> > >> >> > Note Element 1 is an does not contain any device names. >> >> > >> >> > Sesutil is getting an ENODEV back from the ioctl requesting the names. >> >> > >> >> > In scsi_enc.c the error seems be coming from here: >> >> > >> >> > cam_periph_lock(periph); >> >> > error = (*enc->enc_vec.get_elm_devnames)(enc, &elmdn); >> >> > cam_periph_unlock(periph); >> >> > >> >> > which eventually calls cam_periph_list() which "looks" ok to me. >> >> > >> >> > I'm wondering if there is some sort of off by 1 error somewhere in >> >> > the code with respect to how this shelf reports devices and are >> >> > assigned since da0 is not allocated to the first bay, but instead bay 2. >> >> > >> >> > Device da24 (the 25th device) is in the first bay. If I run a dd or >> >> > iozone to da24 the drive works correctly and the activity light >> >> > comes on as expected. >> >> > >> >> > This process works correctly on a set of supermicro shelves. >> >> > >> >> > Any thoughts or pointers? >> >> > >> >> > Thanks, >> >> > John >> >> > >> >> > dmesg for da0/da1 >> >> > >> >> > (da0:mps0:0:8:0): UNMAPPED >> >> > da0 at mps0 bus 0 scbus0 target 8 lun 0 >> >> > da0: <HP EG0600FBDSR HPD6> Fixed Direct Access SPC-3 SCSI device >> >> > da0: Serial Number EA01PC91LLBB1239 >> >> > da0: 600.000MB/s transfers >> >> > da0: Command Queueing enabled >> >> > da0: 572325MB (1172123568 512 byte sectors) >> >> > (da1:mps0:0:9:0): UNMAPPED >> >> > da1 at mps0 bus 0 scbus0 target 9 lun 0 >> >> > da1: <HP EG0600FBDSR HPD6> Fixed Direct Access SPC-3 SCSI device >> >> > da1: Serial Number EA01PC91LPG91239 >> >> > da1: 600.000MB/s transfers >> >> > da1: Command Queueing enabled >> >> > da1: 572325MB (1172123568 512 byte sectors) >> >> > >> >> > >> >> > >> >> > # sesutil map -u /dev/ses0 >> >> > ses0: >> >> > Enclosure Name: HP D2700 SAS AJ941A 0149 >> >> > Enclosure ID: 5001438023432c00 >> >> > Element 0, Type: Array Device Slot >> >> > Status: Unsupported (0x00 0x80 0x00 0x00) >> >> > Element 1, Type: Array Device Slot >> >> > Status: OK (0x01 0x80 0x00 0x00) >> >> > Element 2, Type: Array Device Slot >> >> > Status: OK (0x01 0x80 0x00 0x00) >> >> > Device Names: da0,pass0 >> >> > Element 3, Type: Array Device Slot >> >> > Status: OK (0x01 0x80 0x00 0x00) >> >> > Device Names: da1,pass1 >> >> > Element 4, Type: Array Device Slot >> >> > Status: OK (0x01 0x80 0x00 0x00) >> >> > Device Names: da2,pass2 >> >> > Element 5, Type: Array Device Slot >> >> > Status: OK (0x01 0x80 0x00 0x00) >> >> > Device Names: da3,pass3 >> >> > Element 6, Type: Array Device Slot >> >> > Status: OK (0x01 0x80 0x00 0x00) >> >> > Device Names: da4,pass4 >> >> > Element 7, Type: Array Device Slot >> >> > Status: OK (0x01 0x80 0x00 0x00) >> >> > Device Names: da5,pass5 >> >> > Element 8, Type: Array Device Slot >> >> > Status: OK (0x01 0x80 0x00 0x00) >> >> > Device Names: da6,pass6 >> >> > Element 9, Type: Array Device Slot >> >> > Status: OK (0x01 0x80 0x00 0x00) >> >> > Device Names: da7,pass7 >> >> > Element 10, Type: Array Device Slot >> >> > Status: OK (0x01 0x80 0x00 0x00) >> >> > Device Names: da8,pass8 >> >> > Element 11, Type: Array Device Slot >> >> > Status: OK (0x01 0x80 0x00 0x00) >> >> > Device Names: da9,pass9 >> >> > Element 12, Type: Array Device Slot >> >> > Status: OK (0x01 0x80 0x00 0x00) >> >> > Device Names: da10,pass10 >> >> > Element 13, Type: Array Device Slot >> >> > Status: OK (0x01 0x80 0x00 0x00) >> >> > Device Names: da11,pass11 >> >> > Element 14, Type: Array Device Slot >> >> > Status: OK (0x01 0x80 0x00 0x00) >> >> > Device Names: da12,pass12 >> >> > Element 15, Type: Array Device Slot >> >> > Status: OK (0x01 0x80 0x00 0x00) >> >> > Device Names: da13,pass13 >> >> > Element 16, Type: Array Device Slot >> >> > Status: OK (0x01 0x80 0x00 0x00) >> >> > Device Names: da14,pass14 >> >> > Element 17, Type: Array Device Slot >> >> > Status: OK (0x01 0x80 0x00 0x00) >> >> > Device Names: da15,pass15 >> >> > Element 18, Type: Array Device Slot >> >> > Status: OK (0x01 0x80 0x00 0x00) >> >> > Device Names: da16,pass16 >> >> > Element 19, Type: Array Device Slot >> >> > Status: OK (0x01 0x80 0x00 0x00) >> >> > Device Names: da17,pass17 >> >> > Element 20, Type: Array Device Slot >> >> > Status: OK (0x01 0x80 0x00 0x00) >> >> > Device Names: da18,pass18 >> >> > Element 21, Type: Array Device Slot >> >> > Status: OK (0x01 0x80 0x00 0x00) >> >> > Device Names: da19,pass19 >> >> > Element 22, Type: Array Device Slot >> >> > Status: OK (0x01 0x80 0x00 0x00) >> >> > Device Names: da20,pass20 >> >> > Element 23, Type: Array Device Slot >> >> > Status: OK (0x01 0x80 0x00 0x00) >> >> > Device Names: da21,pass21 >> >> > Element 24, Type: Array Device Slot >> >> > Status: OK (0x01 0x80 0x00 0x00) >> >> > Device Names: da22,pass22 >> >> > Element 25, Type: Array Device Slot >> >> > Status: OK (0x01 0x80 0x00 0x00) >> >> > Device Names: da23,pass23 >> >> > Element 26, Type: SAS Connector >> >> > Status: Unsupported (0x00 0x00 0x00 0x00) >> >> > Element 27, Type: SAS Connector >> >> > Status: OK (0x01 0x20 0x00 0x00) >> >> > Element 28, Type: SAS Connector >> >> > Status: OK (0x01 0x20 0x00 0x00) >> >> > Element 29, Type: SAS Connector >> >> > Status: OK (0x01 0x20 0x00 0x00) >> >> > Element 30, Type: SAS Connector >> >> > Status: OK (0x01 0x20 0x00 0x00) >> >> > Element 31, Type: SAS Connector >> >> > Status: OK (0x01 0x20 0x00 0x00) >> >> > Element 32, Type: SAS Connector >> >> > Status: OK (0x01 0x20 0x00 0x00) >> >> > Element 33, Type: SAS Connector >> >> > Status: OK (0x01 0x20 0x00 0x00) >> >> > Element 34, Type: SAS Connector >> >> > Status: OK (0x01 0x20 0x00 0x00) >> >> > Element 35, Type: SAS Connector >> >> > Status: OK (0x01 0x20 0x00 0x00) >> >> > Element 36, Type: SAS Connector >> >> > Status: OK (0x01 0x20 0x00 0x00) >> >> > Element 37, Type: SAS Connector >> >> > Status: OK (0x01 0x20 0x00 0x00) >> >> > Element 38, Type: SAS Connector >> >> > Status: OK (0x01 0x20 0x00 0x00) >> >> > Element 39, Type: SAS Connector >> >> > Status: OK (0x01 0x20 0x00 0x00) >> >> > Element 40, Type: SAS Connector >> >> > Status: OK (0x01 0x20 0x00 0x00) >> >> > Element 41, Type: SAS Connector >> >> > Status: OK (0x01 0x20 0x00 0x00) >> >> > Element 42, Type: SAS Connector >> >> > Status: OK (0x01 0x20 0x00 0x00) >> >> > Element 43, Type: SAS Connector >> >> > Status: OK (0x01 0x20 0x00 0x00) >> >> > Element 44, Type: SAS Connector >> >> > Status: OK (0x01 0x20 0x00 0x00) >> >> > Element 45, Type: SAS Connector >> >> > Status: OK (0x01 0x20 0x00 0x00) >> >> > Element 46, Type: SAS Connector >> >> > Status: OK (0x01 0x20 0x00 0x00) >> >> > Element 47, Type: SAS Connector >> >> > Status: OK (0x01 0x20 0x00 0x00) >> >> > Element 48, Type: SAS Connector >> >> > Status: OK (0x01 0x20 0x00 0x00) >> >> > Element 49, Type: SAS Connector >> >> > Status: OK (0x01 0x20 0x00 0x00) >> >> > Element 50, Type: SAS Connector >> >> > Status: OK (0x01 0x20 0x00 0x00) >> >> > Element 51, Type: SAS Connector >> >> > Status: OK (0x01 0x20 0x00 0x00) >> >> > Element 52, Type: <Type 0xff> >> >> > Status: Unsupported (0x00 0x00 0x00 0x00) >> >> > Element 53, Type: <Type 0xff> >> >> > Status: OK (0x01 0x00 0x00 0x00) >> >> > Description: 5C7236P2L5 >> >> > Element 54, Type: Power Supply >> >> > Status: Unsupported (0x00 0x00 0x00 0x80) >> >> > Element 55, Type: Power Supply >> >> > Status: OK (0x01 0x00 0x00 0x80) >> >> > Description: 20Delta 5ANLD0C4D3A0XN >> >> > Element 56, Type: Power Supply >> >> > Status: OK (0x01 0x00 0x00 0x80) >> >> > Description: 20Delta 5ANLD0C4D3A0XS >> >> > Element 57, Type: Cooling >> >> > Status: OK (0x01 0x00 0x00 0x80) >> >> > Description: B PAVCA0B9V3B0GH >> >> > Extra status: >> >> > - Speed: 0 rpm >> >> > Element 58, Type: Cooling >> >> > Status: Unsupported (0x00 0x02 0x8f 0x02) >> >> > Extra status: >> >> > - Speed: 6550 rpm >> >> > Element 59, Type: Cooling >> >> > Status: Unsupported (0x00 0x02 0x6b 0x02) >> >> > Extra status: >> >> > - Speed: 6190 rpm >> >> > Element 60, Type: Cooling >> >> > Status: OK (0x01 0x00 0x00 0x80) >> >> > Description: B PAVCA0B9V3B0DL >> >> > Extra status: >> >> > - Speed: 0 rpm >> >> > Element 61, Type: Cooling >> >> > Status: Unsupported (0x00 0x02 0xa3 0x02) >> >> > Extra status: >> >> > - Speed: 6750 rpm >> >> > Element 62, Type: Cooling >> >> > Status: Unsupported (0x00 0x02 0x6b 0x02) >> >> > Extra status: >> >> > - Speed: 6190 rpm >> >> > Element 63, Type: Enclosure Services Controller Electronics >> >> > Status: Unsupported (0x00 0x00 0x00 0x00) >> >> > Element 64, Type: Enclosure Services Controller Electronics >> >> > Status: OK (0x01 0x00 0x00 0x00) >> >> > Description: 014901B0PAZAV0BTM3D3AX 0026 >> >> > Element 65, Type: SAS Connector >> >> > Status: Unsupported (0x00 0x02 0x00 0x00) >> >> > Element 66, Type: SAS Connector >> >> > Status: OK (0x01 0x02 0x00 0x00) >> >> > Element 67, Type: SAS Connector >> >> > Status: OK (0x01 0x02 0x01 0x00) >> >> > Element 68, Type: SAS Connector >> >> > Status: OK (0x01 0x02 0x02 0x00) >> >> > Element 69, Type: SAS Connector >> >> > Status: OK (0x01 0x02 0x03 0x00) >> >> > Element 70, Type: SAS Connector >> >> > Status: Unsupported (0x00 0x02 0x00 0x00) >> >> > Element 71, Type: SAS Connector >> >> > Status: OK (0x01 0x02 0x00 0x00) >> >> > Element 72, Type: SAS Connector >> >> > Status: OK (0x01 0x02 0x01 0x00) >> >> > Element 73, Type: SAS Connector >> >> > Status: OK (0x01 0x02 0x02 0x00) >> >> > Element 74, Type: SAS Connector >> >> > Status: OK (0x01 0x02 0x03 0x00) >> >> > Element 75, Type: SAS Expander >> >> > Status: Unsupported (0x00 0x00 0x00 0x00) >> >> > Element 76, Type: SAS Expander >> >> > Status: OK (0x01 0x00 0x00 0x00) >> >> > Description: 900080050002 >> >> > Element 77, Type: Display >> >> > Status: OK (0x01 0x00 0x00 0x00) >> >> > Description: 0.107SEGDISP C:03 >> >> > Element 78, Type: Display >> >> > Status: Unsupported (0x00 0x02 0x30 0x00) >> >> > Element 79, Type: Display >> >> > Status: Unsupported (0x00 0x02 0x31 0x00) >> >> > Element 80, Type: Temperature Sensors >> >> > Status: Unsupported (0x00 0x00 0x43 0x00) >> >> > Extra status: >> >> > - Temperature: 47 C >> >> > Element 81, Type: Temperature Sensors >> >> > Status: OK (0x01 0x00 0x43 0x00) >> >> > Extra status: >> >> > - Temperature: 47 C >> >> > Element 82, Type: Enclosure Services Controller Electronics >> >> > Status: Unsupported (0x00 0x00 0x00 0x00) >> >> > Element 83, Type: Enclosure Services Controller Electronics >> >> > Status: Unknown (0x06 0x00 0x00 0x00) >> >> > Description: 0149 0026 >> >> > Element 84, Type: SAS Connector >> >> > Status: Unsupported (0x00 0x02 0x00 0x00) >> >> > Element 85, Type: SAS Connector >> >> > Status: Unknown (0x06 0x02 0x00 0x00) >> >> > Element 86, Type: SAS Connector >> >> > Status: Unknown (0x06 0x02 0x01 0x00) >> >> > Element 87, Type: SAS Connector >> >> > Status: Unknown (0x06 0x02 0x02 0x00) >> >> > Element 88, Type: SAS Connector >> >> > Status: Unknown (0x06 0x02 0x03 0x00) >> >> > Element 89, Type: SAS Connector >> >> > Status: Unsupported (0x00 0x02 0x00 0x00) >> >> > Element 90, Type: SAS Connector >> >> > Status: Unknown (0x06 0x02 0x00 0x00) >> >> > Element 91, Type: SAS Connector >> >> > Status: Unknown (0x06 0x02 0x01 0x00) >> >> > Element 92, Type: SAS Connector >> >> > Status: Unknown (0x06 0x02 0x02 0x00) >> >> > Element 93, Type: SAS Connector >> >> > Status: Unknown (0x06 0x02 0x03 0x00) >> >> > Element 94, Type: SAS Expander >> >> > Status: Unsupported (0x00 0x00 0x00 0x00) >> >> > Element 95, Type: SAS Expander >> >> > Status: Unknown (0x06 0x00 0x00 0x00) >> >> > Description: >> >> > Element 96, Type: Display >> >> > Status: Unknown (0x06 0x00 0x00 0x00) >> >> > Description: >> >> > Element 97, Type: Display >> >> > Status: Unsupported (0x00 0x00 0x00 0x00) >> >> > Element 98, Type: Display >> >> > Status: Unsupported (0x00 0x00 0x00 0x00) >> >> > Element 99, Type: Temperature Sensors >> >> > Status: Unsupported (0x00 0x00 0x43 0x00) >> >> > Extra status: >> >> > - Temperature: 47 C >> >> > Element 100, Type: Temperature Sensors >> >> > Status: OK (0x01 0x00 0x43 0x00) >> >> > Extra status: >> >> > - Temperature: 47 C >> >> > Element 101, Type: Enclosure Services Controller Electronics >> >> > Status: Unsupported (0x00 0x00 0x00 0x00) >> >> > Element 102, Type: Enclosure Services Controller Electronics >> >> > Status: OK (0x01 0x00 0x00 0x00) >> >> > Description: 002803D0PARHV0DTM3B0FF >> >> > Element 103, Type: Enclosure Services Controller Electronics >> >> > Status: Unsupported (0x00 0x00 0x00 0x00) >> >> > Element 104, Type: Enclosure Services Controller Electronics >> >> > Status: OK (0x01 0x00 0x00 0x00) >> >> > Description: 1.02 >> >> > Element 105, Type: Enclosure Services Controller Electronics >> >> > Status: OK (0x01 0x00 0x00 0x00) >> >> > Description: 1.02 >> >> > Element 106, Type: Temperature Sensors >> >> > Status: Unsupported (0x00 0x00 0x30 0x00) >> >> > Extra status: >> >> > - Temperature: 28 C >> >> > Element 107, Type: Temperature Sensors >> >> > Status: OK (0x01 0x00 0x31 0x00) >> >> > Extra status: >> >> > - Temperature: 29 C >> >> > Element 108, Type: Temperature Sensors >> >> > Status: OK (0x01 0x00 0x30 0x00) >> >> > Extra status: >> >> > - Temperature: 28 C >> >> > Element 109, Type: Power Supply >> >> > Status: Unsupported (0x00 0x00 0x00 0x00) >> >> > Element 110, Type: Power Supply >> >> > Status: OK (0x01 0x00 0x00 0x00) >> >> > Element 111, Type: Power Supply >> >> > Status: OK (0x01 0x00 0x00 0x00) >> >> > >> >> >> >> -- >> >> Alexander Motin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOtMX2jT80XSPk7ZPkrCQCMJ0eWkF5jL6FmXha_wyC_CW7yssQ>