Date: Fri, 11 Dec 2015 17:34:16 -0500 From: Mykel@mWare.ca To: freebsd-scsi@freebsd.org Subject: Informal(?) sesX messages Message-ID: <566B4F68.2040807@mWare.ca>
next in thread | raw e-mail | index | archive | help
Hi all, please CC me on reply as I'm not subscribed to this list. I've got one of those Supermicro 72-drive monster machines, all ZFS'd up. https://www.supermicro.com/products/system/4u/6048/SSG-6048R-E1CR72L.cfm And before & after replacing a faulty SAS Expander and a pair of cables (gobs of WRITE/ABORT errors), I'm still occasionally seeing these kernel messages (in groups), and I'm not sure if they're benign, or pointing to a SAS expander event... or what. I admit, this is my first time dealing with a machine with SAS expanders, so I'm a bit out of my depth in diagnosis thereof. Dec 11 16:06:54 ZFS-AF kernel: ses5: da7,pass7: Element descriptor: 'Slot00' Dec 11 16:06:54 ZFS-AF kernel: ses5: da7,pass7: SAS Device Slot Element: 1 Phys at Slot 0 Dec 11 16:06:54 ZFS-AF kernel: ses5: phy 0: SAS device type 1 id 0 Dec 11 16:06:54 ZFS-AF kernel: ses5: phy 0: protocols: Initiator( None ) Target( SSP ) Dec 11 16:06:54 ZFS-AF kernel: ses5: phy 0: parent 500304801ea2df3f addr 5000c500844bd449 Every now and then (couple times per week) I'll get messages like that for a number of drives, but not all, and not particularly specific to any particular drive - but always spinning rust, not SSDs. It seems to happen under high load, but I'm not completely certain of that. - Smartctl is happy with all the drives. - We've got redundant power supplies and have rotated them around, run with spares - ZFS has logged zero checksum errors, many scrubs, massive writes/copies/send|recvs & bonnies later. - The messages come when we're using either istgt, ctld, local bonnie++s, dd'ing 0s, or just scrubbing. - Seem to only occur while there's regular activity on the pools, NOT when importing/exporting/snapshotting. - Tried without SSDs connected, and with TRIM off, no change (saw a post from a few years ago with the older version of the LSI SAS card having some issues with that.) - It's a new machine, we're still commissioning it, but plan to press it into service as a Vmware storage machine later next week - I've compiled and fired up sesd, but haven't had any messages go by recently - OS is on a pair of Intel SSDs using the motherboard's controller, unrelated/unaffected. Root on ZFS tho. Do I have anything to worry about? Are these normal things to see popping up sporadically? Could this be a firmware bug in the expander(s) or with the driver? I don't have any important data on there at this time, so data-destructive testing is possible right now. Including bunch of info that might be helpful for dx. Thanks for any advice/suggestion! Myke PS: Please CC me in your reply. PPS: That's the kind of hostname you get when you don't give me something better, or tell me that you don't care ;) FreeBSD ZFS-AF.$CLIENT.fqdn 10.2-RELEASE-p7 FreeBSD 10.2-RELEASE-p7 #0: Mon Nov 2 14:19:39 UTC 2015 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 (The device IDs are a bit skewed as I hot-rearranged a number of them (camcontrol stopping and reimporting the zpools), but it'll do it after a reboot as well.) Dec 11 16:06:54 ZFS-AF kernel: ses5: da7,pass7: Element descriptor: 'Slot00' Dec 11 16:06:54 ZFS-AF kernel: ses5: da7,pass7: SAS Device Slot Element: 1 Phys at Slot 0 Dec 11 16:06:54 ZFS-AF kernel: ses5: phy 0: SAS device type 1 id 0 Dec 11 16:06:54 ZFS-AF kernel: ses5: phy 0: protocols: Initiator( None ) Target( SSP ) Dec 11 16:06:54 ZFS-AF kernel: ses5: phy 0: parent 500304801ea2df3f addr 5000c500844bd449 Dec 11 16:06:54 ZFS-AF kernel: ses5: da4,pass4: Element descriptor: 'Slot01' Dec 11 16:06:54 ZFS-AF kernel: ses5: da4,pass4: SAS Device Slot Element: 1 Phys at Slot 1 Dec 11 16:06:54 ZFS-AF kernel: ses5: phy 0: SAS device type 1 id 0 Dec 11 16:06:54 ZFS-AF kernel: ses5: phy 0: protocols: Initiator( None ) Target( SSP ) Dec 11 16:06:54 ZFS-AF kernel: ses5: phy 0: parent 500304801ea2df3f addr 5000c500844bc9b1 Dec 11 16:06:54 ZFS-AF kernel: ses5: da2,pass2: Element descriptor: 'Slot02' Dec 11 16:06:54 ZFS-AF kernel: ses5: da2,pass2: SAS Device Slot Element: 1 Phys at Slot 2 Dec 11 16:06:54 ZFS-AF kernel: ses5: phy 0: SAS device type 1 id 0 Dec 11 16:06:54 ZFS-AF kernel: ses5: phy 0: protocols: Initiator( None ) Target( SSP ) Dec 11 16:06:54 ZFS-AF kernel: ses5: phy 0: parent 500304801ea2df3f addr 5000c500844d6ea1 Dec 11 16:06:54 ZFS-AF kernel: ses5: da1,pass1: Element descriptor: 'Slot03' Dec 11 16:06:54 ZFS-AF kernel: ses5: da1,pass1: SAS Device Slot Element: 1 Phys at Slot 3 Dec 11 16:06:54 ZFS-AF kernel: ses5: phy 0: SAS device type 1 id 0 Dec 11 16:06:54 ZFS-AF kernel: ses5: phy 0: protocols: Initiator( None ) Target( SSP ) Dec 11 16:06:54 ZFS-AF kernel: ses5: phy 0: parent 500304801ea2df3f addr 5000c500844b9785 Dec 11 16:06:54 ZFS-AF kernel: ses5: da32,pass39: Element descriptor: 'Slot04' Dec 11 16:06:54 ZFS-AF kernel: ses5: da32,pass39: SAS Device Slot Element: 1 Phys at Slot 4 Dec 11 16:06:54 ZFS-AF kernel: ses5: phy 0: SAS device type 1 id 0 Dec 11 16:06:54 ZFS-AF kernel: ses5: phy 0: protocols: Initiator( None ) Target( SSP ) Dec 11 16:06:54 ZFS-AF kernel: ses5: phy 0: parent 500304801ea2df3f addr 5000c500844bda21 Dec 11 16:06:54 ZFS-AF kernel: ses5: da33,pass40: Element descriptor: 'Slot05' Dec 11 16:06:54 ZFS-AF kernel: ses5: da33,pass40: SAS Device Slot Element: 1 Phys at Slot 5 Dec 11 16:06:54 ZFS-AF kernel: ses5: phy 0: SAS device type 1 id 0 Dec 11 16:06:54 ZFS-AF kernel: ses5: phy 0: protocols: Initiator( None ) Target( SSP ) Dec 11 16:06:54 ZFS-AF kernel: ses5: phy 0: parent 500304801ea2df3f addr 5000c500844d050d Dec 11 16:06:54 ZFS-AF kernel: ses5: da34,pass41: Element descriptor: 'Slot06' Dec 11 16:06:54 ZFS-AF kernel: ses5: da34,pass41: SAS Device Slot Element: 1 Phys at Slot 6 Dec 11 16:06:54 ZFS-AF kernel: ses5: phy 0: SAS device type 1 id 0 Dec 11 16:06:54 ZFS-AF kernel: ses5: phy 0: protocols: Initiator( None ) Target( SSP ) Dec 11 16:06:54 ZFS-AF kernel: ses5: phy 0: parent 500304801ea2df3f addr 5000c500844c81cd Dec 11 16:06:54 ZFS-AF kernel: ses5: da35,pass42: Element descriptor: 'Slot07' Dec 11 16:06:54 ZFS-AF kernel: ses5: da35,pass42: SAS Device Slot Element: 1 Phys at Slot 7 Dec 11 16:06:54 ZFS-AF kernel: ses5: phy 0: SAS device type 1 id 0 Dec 11 16:06:54 ZFS-AF kernel: ses5: phy 0: protocols: Initiator( None ) Target( SSP ) Dec 11 16:06:54 ZFS-AF kernel: ses5: phy 0: parent 500304801ea2df3f addr 5000c500844bf42d Dec 11 16:06:54 ZFS-AF kernel: ses5: da36,pass43: Element descriptor: 'Slot08' Dec 11 16:06:54 ZFS-AF kernel: ses5: da36,pass43: SAS Device Slot Element: 1 Phys at Slot 8 Dec 11 16:06:54 ZFS-AF kernel: ses5: phy 0: SAS device type 1 id 0 Dec 11 16:06:54 ZFS-AF kernel: ses5: phy 0: protocols: Initiator( None ) Target( SSP ) Dec 11 16:06:54 ZFS-AF kernel: ses5: phy 0: parent 500304801ea2df3f addr 5000c500844cf71d Dec 11 16:06:54 ZFS-AF kernel: ses5: da37,pass44: Element descriptor: 'Slot09' Dec 11 16:06:54 ZFS-AF kernel: ses5: da37,pass44: SAS Device Slot Element: 1 Phys at Slot 9 Dec 11 16:06:54 ZFS-AF kernel: ses5: phy 0: SAS device type 1 id 0 Dec 11 16:06:54 ZFS-AF kernel: ses5: phy 0: protocols: Initiator( None ) Target( SSP ) Dec 11 16:06:54 ZFS-AF kernel: ses5: phy 0: parent 500304801ea2df3f addr 5000c500844bde8d Dec 11 16:06:54 ZFS-AF kernel: ses5: da38,pass45: Element descriptor: 'Slot10' Dec 11 16:06:54 ZFS-AF kernel: ses5: da38,pass45: SAS Device Slot Element: 1 Phys at Slot 10 Dec 11 16:06:54 ZFS-AF kernel: ses5: phy 0: SAS device type 1 id 0 Dec 11 16:06:54 ZFS-AF kernel: ses5: phy 0: protocols: Initiator( None ) Target( SSP ) Dec 11 16:06:54 ZFS-AF kernel: ses5: phy 0: parent 500304801ea2df3f addr 5000c500844d66c9 Dec 11 16:06:54 ZFS-AF kernel: ses5: da39,pass46: Element descriptor: 'Slot11' Dec 11 16:06:54 ZFS-AF kernel: ses5: da39,pass46: SAS Device Slot Element: 1 Phys at Slot 11 Dec 11 16:06:54 ZFS-AF kernel: ses5: phy 0: SAS device type 1 id 0 Dec 11 16:06:54 ZFS-AF kernel: ses5: phy 0: protocols: Initiator( None ) Target( SSP ) Dec 11 16:06:54 ZFS-AF kernel: ses5: phy 0: parent 500304801ea2df3f addr 5000c500844cf01d [root@ZFS-AF ~]# camcontrol devlist <SEAGATE ST4000NM0034 E001> at scbus0 target 13 lun 0 (da12,pass13) <SEAGATE ST4000NM0034 E001> at scbus0 target 14 lun 0 (da14,pass15) <SEAGATE ST4000NM0034 E001> at scbus0 target 15 lun 0 (da16,pass17) <SEAGATE ST4000NM0034 E001> at scbus0 target 23 lun 0 (da19,pass20) <LSI SAS3x48 0701> at scbus0 target 32 lun 0 (ses0,pass8) <SEAGATE ST600MP0005 N003> at scbus1 target 8 lun 0 (pass9,da8) <SEAGATE ST600MP0005 N003> at scbus1 target 9 lun 0 (da10,pass11) <SEAGATE ST600MP0005 N003> at scbus1 target 10 lun 0 (da9,pass10) <SEAGATE ST600MP0005 N003> at scbus1 target 11 lun 0 (pass12,da11) <ATA SAMSUNG MZ7KM960 003Q> at scbus1 target 12 lun 0 (da22,pass28) <SEAGATE ST4000NM0034 E001> at scbus1 target 13 lun 0 (da13,pass14) <SEAGATE ST4000NM0034 E001> at scbus1 target 14 lun 0 (da15,pass16) <SEAGATE ST4000NM0034 E001> at scbus1 target 15 lun 0 (da17,pass18) <ATA SAMSUNG MZ7KM960 003Q> at scbus1 target 20 lun 0 (da23,pass29) <ATA SAMSUNG MZ7KM960 003Q> at scbus1 target 21 lun 0 (da21,pass27) <ATA SAMSUNG MZ7KM960 003Q> at scbus1 target 22 lun 0 (da18,pass19) <SEAGATE ST4000NM0034 E001> at scbus1 target 23 lun 0 (da20,pass26) <LSI SAS3x36 0701> at scbus1 target 32 lun 0 (ses1,pass21) <AHCI SGPIO Enclosure 1.00 0001> at scbus6 target 0 lun 0 (ses2,pass22) <INTEL SSDSC2BB080G6 G2010130> at scbus11 target 0 lun 0 (ada0,pass23) <INTEL SSDSC2BB080G6 G2010130> at scbus12 target 0 lun 0 (ada1,pass24) <AHCI SGPIO Enclosure 1.00 0001> at scbus13 target 0 lun 0 (ses3,pass25) <SEAGATE ST4000NM0034 E001> at scbus14 target 8 lun 0 (da6,pass6) <SEAGATE ST4000NM0034 E001> at scbus14 target 9 lun 0 (da5,pass5) <SEAGATE ST4000NM0034 E001> at scbus14 target 10 lun 0 (da3,pass3) <SEAGATE ST4000NM0034 E001> at scbus14 target 11 lun 0 (da0,pass0) <SEAGATE ST4000NM0034 E001> at scbus14 target 12 lun 0 (pass30,da24) <SEAGATE ST4000NM0034 E001> at scbus14 target 13 lun 0 (pass31,da25) <SEAGATE ST4000NM0034 E001> at scbus14 target 14 lun 0 (pass32,da26) <SEAGATE ST4000NM0034 E001> at scbus14 target 15 lun 0 (pass33,da27) <SEAGATE ST4000NM0034 E001> at scbus14 target 16 lun 0 (pass34,da28) <SEAGATE ST4000NM0034 E001> at scbus14 target 17 lun 0 (pass35,da29) <SEAGATE ST4000NM0034 E001> at scbus14 target 18 lun 0 (pass36,da30) <SEAGATE ST4000NM0034 E001> at scbus14 target 19 lun 0 (pass37,da31) <LSI SAS3x28 0701> at scbus14 target 20 lun 0 (ses4,pass38) <SEAGATE ST4000NM0034 E001> at scbus14 target 39 lun 0 (da7,pass7) <SEAGATE ST4000NM0034 E001> at scbus14 target 40 lun 0 (da4,pass4) <SEAGATE ST4000NM0034 E001> at scbus14 target 41 lun 0 (da2,pass2) <SEAGATE ST4000NM0034 E001> at scbus14 target 42 lun 0 (da1,pass1) <SEAGATE ST4000NM0034 E001> at scbus14 target 43 lun 0 (pass39,da32) <SEAGATE ST4000NM0034 E001> at scbus14 target 44 lun 0 (pass40,da33) <SEAGATE ST4000NM0034 E001> at scbus14 target 45 lun 0 (pass41,da34) <SEAGATE ST4000NM0034 E001> at scbus14 target 46 lun 0 (pass42,da35) <SEAGATE ST4000NM0034 E001> at scbus14 target 47 lun 0 (pass43,da36) <SEAGATE ST4000NM0034 E001> at scbus14 target 48 lun 0 (pass44,da37) <SEAGATE ST4000NM0034 E001> at scbus14 target 49 lun 0 (pass45,da38) <SEAGATE ST4000NM0034 E001> at scbus14 target 50 lun 0 (pass46,da39) <LSI SAS3x48 0701> at scbus14 target 63 lun 0 (ses5,pass47) [root@ZFS-AF ~]# [root@ZFS-AF ~]# smp_discover ses0 phy 21:D:attached:[5000c500844c64e5:00 t(SSP)] 12 Gbps phy 22:D:attached:[5000c500844d8de5:00 t(SSP)] 12 Gbps phy 23:D:attached:[5000c500844d6865:00 t(SSP)] 12 Gbps phy 31:D:attached:[5000c500844c9a3d:00 t(SSP)] 12 Gbps phy 40:U:attached:[5003048018c77101:03 i(SSP+STP+SMP)] 12 Gbps phy 41:U:attached:[5003048018c77101:02 i(SSP+STP+SMP)] 12 Gbps phy 42:U:attached:[5003048018c77101:01 i(SSP+STP+SMP)] 12 Gbps phy 43:U:attached:[5003048018c77101:00 i(SSP+STP+SMP)] 12 Gbps phy 44:U:attached:[5003048018c77101:07 i(SSP+STP+SMP)] 12 Gbps phy 45:U:attached:[5003048018c77101:06 i(SSP+STP+SMP)] 12 Gbps phy 46:U:attached:[5003048018c77101:05 i(SSP+STP+SMP)] 12 Gbps phy 47:U:attached:[5003048018c77101:04 i(SSP+STP+SMP)] 12 Gbps phy 48:D:attached:[50030480090b8d3d:00 V i(SMP) t(SSP)] 12 Gbps [root@ZFS-AF ~]# smp_discover ses1 phy 0:U:attached:[500304801970b401:00 i(SSP+STP+SMP)] 12 Gbps phy 1:U:attached:[500304801970b401:01 i(SSP+STP+SMP)] 12 Gbps phy 2:U:attached:[500304801970b401:03 i(SSP+STP+SMP)] 12 Gbps phy 3:U:attached:[500304801970b401:02 i(SSP+STP+SMP)] 12 Gbps phy 4:D:attached:[5000c5007788ad75:00 t(SSP)] 12 Gbps phy 5:D:attached:[5000c5007799d161:00 t(SSP)] 12 Gbps phy 6:D:attached:[5000c5007788fd71:00 t(SSP)] 12 Gbps phy 7:D:attached:[5000c5007788d7f9:00 t(SSP)] 12 Gbps phy 8:D:attached:[50030480090b8d88:00 t(SATA)] 12 Gbps phy 9:D:attached:[5000c500844c2185:00 t(SSP)] 12 Gbps phy 10:D:attached:[5000c500844d7ca1:00 t(SSP)] 12 Gbps phy 11:D:attached:[5000c500844c5db5:00 t(SSP)] 12 Gbps phy 16:D:attached:[50030480090b8d90:00 t(SATA)] 12 Gbps phy 17:D:attached:[50030480090b8d91:00 t(SATA)] 12 Gbps phy 18:D:attached:[50030480090b8d92:00 t(SATA)] 12 Gbps phy 19:D:attached:[5000c500844d028d:00 t(SSP)] 12 Gbps phy 22:U:disabled phy 23:U:disabled phy 24:U:attached:[500304801970b401:06 i(SSP+STP+SMP)] 12 Gbps phy 25:U:attached:[500304801970b401:07 i(SSP+STP+SMP)] 12 Gbps phy 26:U:attached:[500304801970b401:05 i(SSP+STP+SMP)] 12 Gbps phy 27:U:attached:[500304801970b401:04 i(SSP+STP+SMP)] 12 Gbps phy 31:U:disabled phy 32:U:disabled phy 36:D:attached:[50030480090b8dbd:00 V i(SMP) t(SSP)] 12 Gbps [root@ZFS-AF ~]# smp_discover ses4 phy 4:D:attached:[5000c500844bb641:00 t(SSP)] 12 Gbps phy 5:D:attached:[5000c500844ba865:00 t(SSP)] 12 Gbps phy 6:D:disabled phy 7:D:disabled phy 8:D:disabled phy 9:D:disabled phy 10:D:disabled phy 11:D:disabled phy 12:D:attached:[5000c500844c6699:00 t(SSP)] 12 Gbps phy 13:D:attached:[5000c500844bd9d5:00 t(SSP)] 12 Gbps phy 14:D:attached:[5000c500844d8c29:00 t(SSP)] 12 Gbps phy 15:D:attached:[5000c500844ba3c1:00 t(SSP)] 12 Gbps phy 16:D:attached:[5000c500844ca251:00 t(SSP)] 12 Gbps phy 17:D:attached:[5000c500844bcc31:00 t(SSP)] 12 Gbps phy 18:D:attached:[5000c500844bda45:00 t(SSP)] 12 Gbps phy 19:D:attached:[5000c500844c8579:00 t(SSP)] 12 Gbps phy 20:D:attached:[5000c500844c7ee1:00 t(SSP)] 12 Gbps phy 21:D:attached:[5000c500844d01fd:00 t(SSP)] 12 Gbps phy 22:U:disabled phy 23:U:disabled phy 24:U:attached:[500304801970b101:02 i(SSP+STP+SMP)] 12 Gbps phy 25:U:attached:[500304801970b101:03 i(SSP+STP+SMP)] 12 Gbps phy 26:U:attached:[500304801970b101:01 i(SSP+STP+SMP)] 12 Gbps phy 27:U:attached:[500304801970b101:00 i(SSP+STP+SMP)] 12 Gbps phy 28:D:attached:[500304801ea2dfbd:00 V i(SMP) t(SSP)] 12 Gbps [root@ZFS-AF ~]# smp_discover ses5 phy 0:D:attached:[5000c500844bd449:00 t(SSP)] 12 Gbps phy 1:D:attached:[5000c500844bc9b1:00 t(SSP)] 12 Gbps phy 2:D:attached:[5000c500844d6ea1:00 t(SSP)] 12 Gbps phy 3:D:attached:[5000c500844b9785:00 t(SSP)] 12 Gbps phy 20:D:attached:[5000c500844bda21:00 t(SSP)] 12 Gbps phy 21:D:attached:[5000c500844d050d:00 t(SSP)] 12 Gbps phy 22:D:attached:[5000c500844c81cd:00 t(SSP)] 12 Gbps phy 23:D:attached:[5000c500844bf42d:00 t(SSP)] 12 Gbps phy 24:D:attached:[5000c500844cf71d:00 t(SSP)] 12 Gbps phy 25:D:attached:[5000c500844bde8d:00 t(SSP)] 12 Gbps phy 26:D:attached:[5000c500844d66c9:00 t(SSP)] 12 Gbps phy 27:D:attached:[5000c500844cf01d:00 t(SSP)] 12 Gbps phy 40:U:attached:[500304801970b102:07 i(SSP+STP+SMP)] 12 Gbps phy 41:U:attached:[500304801970b102:06 i(SSP+STP+SMP)] 12 Gbps phy 42:U:attached:[500304801970b102:05 i(SSP+STP+SMP)] 12 Gbps phy 43:U:attached:[500304801970b102:04 i(SSP+STP+SMP)] 12 Gbps phy 48:D:attached:[500304801ea2df3d:00 V i(SMP) t(SSP)] 12 Gbps [root@ZFS-AF ~]# mpr0@pci0:2:0:0: class=0x010700 card=0x080815d9 chip=0x00971000 rev=0x02 hdr=0x00 vendor = 'LSI Logic / Symbios Logic' device = 'SAS3008 PCI-Express Fusion-MPT SAS-3' class = mass storage subclass = SAS mpr1@pci0:3:0:0: class=0x010700 card=0x080815d9 chip=0x00971000 rev=0x02 hdr=0x00 vendor = 'LSI Logic / Symbios Logic' device = 'SAS3008 PCI-Express Fusion-MPT SAS-3' class = mass storage subclass = SAS mpr2@pci0:133:0:0: class=0x010700 card=0x080815d9 chip=0x00971000 rev=0x02 hdr=0x00 vendor = 'LSI Logic / Symbios Logic' device = 'SAS3008 PCI-Express Fusion-MPT SAS-3' class = mass storage subclass = SAS
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?566B4F68.2040807>