From owner-freebsd-scsi@freebsd.org Sun Dec 13 02:33:09 2015 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D9C54A14C0B for ; Sun, 13 Dec 2015 02:33:09 +0000 (UTC) (envelope-from kevin.bowling@kev009.com) Received: from mail-qk0-x22f.google.com (mail-qk0-x22f.google.com [IPv6:2607:f8b0:400d:c09::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 955951788 for ; Sun, 13 Dec 2015 02:33:09 +0000 (UTC) (envelope-from kevin.bowling@kev009.com) Received: by qkht125 with SMTP id t125so92303403qkh.3 for ; Sat, 12 Dec 2015 18:33:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kev009.com; s=google; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=T+RQlhMuijt1v9jI5dOPnf4AD2UnK34CBCu1TmJAuYc=; b=q+ThG4a7lnWXzvBxr35JjuixRhHMbOwOymk0RNhha03Mp6632F/j5Iu5eEHAmu7pAC gbJ7OnsWrqMLm2i9LaxwidhL/G9n5MZkQ4IqEvCPZSFOs/TjaIn9Kx+pB+O5ejv/m//o nIkK37VSNClvrUlDuVPA8hpry48NuCraB0eGM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=T+RQlhMuijt1v9jI5dOPnf4AD2UnK34CBCu1TmJAuYc=; b=hkxxKUGFt4W6Y2ppEmSZX93bmYmvyYVsKXUsIBfJC52jb/zcM6w60gdWMfrJTt+4sI +q3CP0eLKyAJvMEeB49oJD3mqhuZLpdFmwsomCfuZ6Q9wYhPrgHbgfN4/WaTX7qcVkgG ovdxFsQ8HWksV3jaxQGA0tC5AtEmXRQjB2I213ZhsAqg6Z+xn35Ij26zeBOXpinuiXlO QsAdeoJAgaBXxA2ymjuTiCeyVOqtCnf6h+rL4e7LuWeLtwt8LVkyYkKx5fRno4BispwC TmYNZKH+llgWRiAKi9XNJGl5p97vkBc1ZOe8YOww2jpvw1s8eu9KqsNh36DKDU4Mqceu xmjQ== X-Gm-Message-State: ALoCoQldT9lU8gvdW0F0p7VZKvdsMntQmyuM1mpBYWHSGKqINlMKZNbzK/qdXdlZVAWXRBQjkUW2jYAXpYVz3eGN65G426XG0A== MIME-Version: 1.0 X-Received: by 10.129.81.147 with SMTP id f141mr14631115ywb.176.1449973988525; Sat, 12 Dec 2015 18:33:08 -0800 (PST) Received: by 10.37.13.147 with HTTP; Sat, 12 Dec 2015 18:33:08 -0800 (PST) In-Reply-To: References: <566B4F68.2040807@mWare.ca> <566B8E2A.8070404@mWare.ca> <566BC34D.2020404@mware.ca> Date: Sat, 12 Dec 2015 19:33:08 -0700 Message-ID: Subject: Re: Informal(?) sesX messages From: Kevin Bowling To: Alan Somers Cc: Mike Geiger , FreeBSD-scsi , Sean Bruno , jason@llnw.com Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Dec 2015 02:33:09 -0000 JFWIW we see this across a lot of our machines as well -- just arrival messages, usually some time well after the machine is fully booted. Dr. Google shows other instances. It appears to be harmless but noisy. root@cds124.lab:~# sysctl dev.mps.0.firmware_version dev.mps.0.firmware_version: 16.00.01.00 root@cds124.lab:~# sg_inq --hex --len=64 ses0 00 0d 00 05 02 34 00 40 02 4c 53 49 20 20 20 20 20 ....4.@.LSI 10 53 41 53 32 58 32 38 20 20 20 20 20 20 20 20 20 SAS2X28 20 30 65 31 32 78 33 36 2d 35 35 2e 31 34 2e 31 38 0e12x36-55.14.18 30 2e 30 00 20 20 20 20 20 .0. On Sat, Dec 12, 2015 at 8:41 AM, Alan Somers wrote: > On Fri, Dec 11, 2015 at 11:48 PM, wrote: > > On 2015/12/11 22:55, Alan Somers wrote: > >> > >> On Fri, Dec 11, 2015 at 8:02 PM, wrote: > >>> > >>> On 15-12-11 17:44, Alan Somers wrote: > >>>> > >>>> On Fri, Dec 11, 2015 at 3:34 PM, wrote: > >>>>> > >>>>> Hi all, please CC me on reply as I'm not subscribed to this list. > >>>>> > >>>>> I've got one of those Supermicro 72-drive monster machines, all ZFS'd > >>>>> up. > >>>>> > >>>>> > https://www.supermicro.com/products/system/4u/6048/SSG-6048R-E1CR72L.cfm > >>>>> > >>>>> And before & after replacing a faulty SAS Expander and a pair of > cables > >>>>> (gobs of WRITE/ABORT errors), I'm still occasionally seeing these > >>>>> kernel > >>>>> messages (in groups), and I'm not sure if they're benign, or pointing > >>>>> to > >>>>> a > >>>>> SAS expander event... or what. I admit, this is my first time dealing > >>>>> with a > >>>>> machine with SAS expanders, so I'm a bit out of my depth in diagnosis > >>>>> thereof. > >>>>> > >>>>> Dec 11 16:06:54 ZFS-AF kernel: ses5: da7,pass7: Element descriptor: > >>>>> 'Slot00' > >>>>> Dec 11 16:06:54 ZFS-AF kernel: ses5: da7,pass7: SAS Device Slot > >>>>> Element: > >>>>> 1 > >>>>> Phys at Slot 0 > >>>>> Dec 11 16:06:54 ZFS-AF kernel: ses5: phy 0: SAS device type 1 id 0 > >>>>> Dec 11 16:06:54 ZFS-AF kernel: ses5: phy 0: protocols: Initiator( > None > >>>>> ) > >>>>> Target( SSP ) > >>>>> Dec 11 16:06:54 ZFS-AF kernel: ses5: phy 0: parent 500304801ea2df3f > >>>>> addr > >>>>> 5000c500844bd449 > >>>>> > >>>> These look like device arrival notifications. If you scroll up, do > >>>> you see any departure notifications? They should look like this: > >>>> > >>>> mps0: mpssas_prepare_remove: Sending reset for target ID 10 > >>>> da0 at mps0 bus 0 scbus0 target 10 lun 0 > >>>> da0: s/n JPW930HQ15H26H detached > >>>> mps0: Unfreezing devq for target ID 10 > >>>> xpt_release_devq(): requested 1 > present 0 > >>>> (da0:mps0:0:10:0): Periph destroyed > >>>> > >>>> Also, could you post your HBA and expander firmware versions? > >>>> > >>>> -Alan > >>> > >>> > >>> I can say, without doubt, that I do NOT have any preceding > detachments... > >>> which is why I'm so baffled by the messages. If the devices aren't > >>> de/reattaching, what's the point of these informal/benign ones? I am > >>> familiar with them from other hot-swap and disk failure scenarios in > >>> other > >>> machines. > >>> > >>> Could this be a driver bug not logging the disconnection? But when I > >>> hot-unplugged them, I do see that in dmesg. > >>> Or does SAS do something where it might renegotiate or reconfigure the > >>> lanes, and I'm just seeing it do that? > >>> > >>> Thanks, > >>> > >>> Myke > >>> > >>> > >>> dev.mpr.0.driver_version: 09.255.01.00-fbsd > >>> dev.mpr.0.firmware_version: 06.00.00.00 > >>> dev.mpr.1.driver_version: 09.255.01.00-fbsd > >>> dev.mpr.1.firmware_version: 08.00.00.00 > >>> dev.mpr.2.driver_version: 09.255.01.00-fbsd > >>> dev.mpr.2.firmware_version: 08.00.00.00 > >>> > >>> [root@ZFS-AF ~]# sg_inq --hex --len=64 ses0 > >>> 00 0d 00 05 02 33 00 40 02 4c 53 49 20 20 20 20 20 ....3.@.LSI > >>> 10 53 41 53 33 78 34 38 20 20 20 20 20 20 20 20 20 SAS3x48 > >>> 20 30 37 30 31 78 34 38 2d 36 36 2e 37 2e 31 2e 31 > >>> 0701x48-66.7.1.1 > >>> 30 37 00 20 20 20 20 20 20 7. > >>> [root@ZFS-AF ~]# sg_inq --hex --len=64 ses1 > >>> 00 0d 00 05 02 33 00 40 02 4c 53 49 20 20 20 20 20 ....3.@.LSI > >>> 10 53 41 53 33 78 33 36 20 20 20 20 20 20 20 20 20 SAS3x36 > >>> 20 30 37 30 31 78 33 36 2d 36 36 2e 37 2e 31 2e 31 > >>> 0701x36-66.7.1.1 > >>> 30 37 00 20 20 20 20 20 20 7. > >>> [root@ZFS-AF ~]# sg_inq --hex --len=64 ses2 > >>> SCSI INQUIRY failed on ses2, res=-1 > >>> [root@ZFS-AF ~]# sg_inq --hex --len=64 ses3 > >>> SCSI INQUIRY failed on ses3, res=-1 > >>> [root@ZFS-AF ~]# sg_inq --hex --len=64 ses4 > >>> 00 0d 00 05 02 33 00 40 02 4c 53 49 20 20 20 20 20 ....3.@.LSI > >>> 10 53 41 53 33 78 32 38 20 20 20 20 20 20 20 20 20 SAS3x28 > >>> 20 30 37 30 31 78 32 38 2d 36 36 2e 37 2e 31 2e 31 > >>> 0701x28-66.7.1.1 > >>> 30 37 00 20 20 20 20 20 20 7. > >>> [root@ZFS-AF ~]# sg_inq --hex --len=64 ses5 > >>> 00 0d 00 05 02 33 00 40 02 4c 53 49 20 20 20 20 20 ....3.@.LSI > >>> 10 53 41 53 33 78 34 38 20 20 20 20 20 20 20 20 20 SAS3x48 > >>> 20 30 37 30 31 78 34 38 2d 36 36 2e 37 2e 31 2e 31 > >>> 0701x48-66.7.1.1 > >>> 30 37 00 20 20 20 20 20 20 7. > >>> [root@ZFS-AF ~]# > >>> > >>> > >>> And here's dmesg after fresh reboot: > >> > >> Well, that's weird. Your firmware versions look OK, though you might > >> want to upgrade mpr0 just to be consistent. The next thing I would > >> check, if I were you, would be devctl messages. Edit /etc/syslog.conf > >> and change devd's loglevel to INFO, then HUP syslogd. Now every > >> devctl message should get logged in /var/log/devd.log. That will tell > >> you more precisely than dmesg whether there are any arrival or > >> departure events. > >> > >> -Alan > > > > Huh, I never noticed the 6 vs. 8; curiously, mpr0 and mpr1 are the two > > connected to the front expander... and where I've never seen an issue. > Tho > > perhaps I scrambled which cards are serving was which in my testing - I > also > > moved mpr2 to sit on the other CPU's PCI bus. > > > > I've added the devd log, although I haven't been able to trigger the > event > > yet anyway. > > Tried to assert hw.mpr.2.debug_level, however it seems like hw.mpr > doesn't > > exist. > > hw.mpr.debug_level is a tunable which, if set at boot time, will > affect all mpr cards. What you want is dev.mpr.2.debug_level, a > runtime-controllabel sysctl. > > > > > Finally, I haven't the slightest clue how to update the firmware; the > Avago > > site only has a product brochure for the 3008 anyway :( > > It's fairly annoying. First, you must figure out which card you have. > 3008 is the name of your chip. Your card is probably a 9300-9i. If > so, go to this URL and click on firmware. If you download > "Installer_P10_for_UEFI" then you can install it through the EFI > shell. But they also have an installer that runs in FreeBSD. To use > that, download both "Installer_P10_for_FreeBSD" AND > "Installer_P10_for_MSDOS_and_Windows". Unzip the latter and extract > the .bin file. Then unzip the former and run the executable contained > within, providing the path to the .bin file obtained from the latter. > You'll need a reboot afterwards. > > > http://www.avagotech.com/products/server-storage/host-bus-adapters/sas-9300-8i#downloads > > -Alan > _______________________________________________ > freebsd-scsi@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-scsi > To unsubscribe, send any mail to "freebsd-scsi-unsubscribe@freebsd.org" >