From owner-freebsd-scsi@freebsd.org Mon Sep 11 23:52:45 2017 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EDA1FE0841E for ; Mon, 11 Sep 2017 23:52:45 +0000 (UTC) (envelope-from kevin.bowling@kev009.com) Received: from mail-io0-x233.google.com (mail-io0-x233.google.com [IPv6:2607:f8b0:4001:c06::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B77357CB86 for ; Mon, 11 Sep 2017 23:52:45 +0000 (UTC) (envelope-from kevin.bowling@kev009.com) Received: by mail-io0-x233.google.com with SMTP id y123so36841123iod.0 for ; Mon, 11 Sep 2017 16:52:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kev009.com; s=google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=7mTleEpXNo2xcWkagGw5NCfKoi04aAiKQYt0bufK+Iw=; b=qMy/Np5G6T7NVS2mBs1HXtDyFpD1Q1Qk0E6aIUOK2Bve5SRDj6ORl9iABKzTnhrS7P Nsx9rDj5cGrpufCk06fCiyxzhlkhfoXBZ+mmSLcnnS0t6MoNkcRuWOi16q5T6dBhNNOq ks13yOhXTlESD3/w9PGZSEDvwsur8bsImzj3I= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=7mTleEpXNo2xcWkagGw5NCfKoi04aAiKQYt0bufK+Iw=; b=Fp9CqM4NIHPbhUgho5vfoxkaCejV1z2K8544PlJut6zjt2G3OopbZRLsecAsAZmglA pmmUZub3/mVzmZR6/GvnD81Ur7tbEo5Wp0LQR2vbSD0BPyknfBj4ScKD+97BoIx++HMK aBRaKx5yJEgG5edVUul+utHelwXFGHKxWP2Q0VP0gMCMznTR41Bk2oljUkxBAoqqmG8c k37pIViS32tkPjH14yhmIjipBnKwgO70PTfRYPAd1ML2FmD6Qjxs+60B3ZRJz44Hmr0q f0PyxMBqsw9jg6awG0I3SQsvrIElg73sL0Wq17pzvLE2gnAgRkkpVfRJ/EUoMw3F8haD ZQbg== X-Gm-Message-State: AHPjjUgxWS7JluIb9/8IgcdBO5tTd8JM6zaS0GQSH29ON6JDB6ZkYufX mvmrFlV2rBSSzMJXaFZmuGRGLKYg3B/n X-Google-Smtp-Source: AOwi7QDzRlxozFzMRrSwc6SyIpXhWU3AGckroo8/OLETbri8uW25nhePmiWno20eCKuqjCLhfErcW4TQ3NzfXuC7sYY= X-Received: by 10.202.79.206 with SMTP id d197mr14526632oib.192.1505173965102; Mon, 11 Sep 2017 16:52:45 -0700 (PDT) MIME-Version: 1.0 Received: by 10.157.57.200 with HTTP; Mon, 11 Sep 2017 16:52:44 -0700 (PDT) In-Reply-To: <63b26ecc55d5f7a3152af6c26aba18a2@mail.gmail.com> References: <1112cc2edb666ce7fa9c72c11cdd284c@nitrology.com> <63b26ecc55d5f7a3152af6c26aba18a2@mail.gmail.com> From: Kevin Bowling Date: Mon, 11 Sep 2017 16:52:44 -0700 Message-ID: Subject: Re: Disk reordering on LSI SAS2008/mps(4) To: Stephen Mcconnell Cc: FreeBSD-scsi , John Baldwin Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Sep 2017 23:52:46 -0000 https://svnweb.freebsd.org/base?view=revision&revision=323384 fixed it for us On Mon, Aug 28, 2017 at 8:08 AM, Stephen Mcconnell wrote: > I'm assuming that the dubug_level is in hex, right? If it is than the run > where debug_level is 0x583 should be showing some Mapping debug output, > but I don't see any. Do you have mapping enabled in the controller? You > can see the mapping flags in IOC Page 8 in the Flags field. Do you have a > way to look at the controller pages? You'd need either lsiutil or maybe > mpsutil will work (Scott Long wrote mpsutil and I don't know anything > about it). > > If you don't have mapping enabled, you won't be guaranteed that the > devices will be discovered in the same order over a controller reset or > reboot. > > Steve > >> -----Original Message----- >> From: owner-freebsd-scsi@freebsd.org [mailto:owner-freebsd- >> scsi@freebsd.org] On Behalf Of Kevin Bowling >> Sent: Sunday, August 27, 2017 8:51 PM >> To: FreeBSD-scsi >> Cc: John Baldwin >> Subject: Re: Disk reordering on LSI SAS2008/mps(4) >> >> Note that we only see this bug with EARLY_AP_STARTUP enabled >> >> Regards, >> >> On Fri, Aug 25, 2017 at 2:11 PM, Jason Wolfe wrote: >> >> > Attachments are useful. >> > >> > >> > On 2017-08-25 13:58, Jason Wolfe wrote: >> > >> >> Hi! >> >> >> >> We've been having an issue where we see some disk reordering on boot >> >> on HEAD from mid July on LSI controllers, maybe 5% of the time. We >> >> brought mps current as of r322364 with no change behavior. >> >> >> >> I have a few logs attached with various debug output. In all cases >> >> I've seen the pass ordering to be proper, and cam does try to resolve >> >> the da ordering, but the device it tries to reassign to is already >> >> taken. Attached is the full output, and listing some relevant bits >> >> below for the casual reader. Being that the functionality in >> >> scsi_da.c has been fairly static, and it's attempting to reassign, it >> >> seems more likely we are running into something in mps here. The >> >> targets always look to be proper. >> >> >> >> The various settings of hw.mps.use_phy_num (-1/0/1) don't change the >> >> behavior, and neither does hw.mps.enable_ssu=0. We have machines over >> >> various FW versions (15/16) that see the issue. I'm wondering if the >> >> fact that we see this issue over soft reboots means that the firmware >> >> isn't coming into play. To confirm, we are booting from the >> >> controller, so the LSI BIOS is enabled. >> >> >> >> mps0@pci0:3:0:0: class=0x010700 card=0x040015d9 > chip=0x00721000 >> >> rev=0x03 hdr=0x00 >> >> vendor = 'LSI Logic / Symbios Logic' >> >> device = 'SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon]' >> >> class = mass storage >> >> subclass = SAS >> >> >> >> reorder-verbose.txt: >> >> boot_verbose="YES" >> >> hw.mps.debug_level="71" >> >> >> >> da0 at mps0 bus 0 scbus0 target 17 lun 0 >> >> cam_periph_alloc: attempt to re-allocate valid device da0 rejected >> >> flags 0x102 refcount 4 >> >> da1 at mps0 bus 0 scbus0 target 8 lun 0 >> >> daasync: Unable to attach to new device due to status 0x6 >> >> da2 at mps0 bus 0 scbus0 target 9 lun 0 ... >> >> da8 at mps0 bus 0 scbus0 target 15 lun 0 >> >> da9 at mps0 bus 0 scbus0 target 16 lun 0 >> >> da10 at mps0 bus 0 scbus0 target 18 lun 0 >> >> da11 at mps0 bus 0 scbus0 target 19 lun 0 >> >> >> >> pass0 at mps0 bus 0 scbus0 target 8 lun 0 >> >> pass1 at mps0 bus 0 scbus0 target 9 lun 0 ... >> >> pass9 at mps0 bus 0 scbus0 target 17 lun 0 >> >> pass10 at mps0 bus 0 scbus0 target 18 lun 0 >> >> pass11 at mps0 bus 0 scbus0 target 19 lun 0 >> >> >> >> >> >> >> >> >> >> reorder-mps-mapping.txt: >> >> hw.mps.debug_level="583" >> >> >> >> da0 at mps0 bus 0 scbus0 target 19 lun 0 >> >> da1 at mps0 bus 0 scbus0 target 8 lun 0 >> >> da2 at mps0 bus 0 scbus0 target 9 lun 0 >> >> ... >> >> da9 at mps0 bus 0 scbus0 target 16 lun 0 >> >> da10 at mps0 bus 0 scbus0 target 17 lun 0 >> >> da11 at mps0 bus 0 scbus0 target 18 lun 0 >> >> cam_periph_alloc: attempt to re-allocate valid device da0 rejected >> >> flags 0x106 refcount 6 >> >> daasync: Unable to attach to new device due to status 0x6 >> >> >> >> ses0: da1,pass0: Element descriptor: 'Slot 01' >> >> ses0: da1,pass0: SAS Device Slot Element: 1 Phys at Slot 0 >> >> ses0: da0,pass11: Element descriptor: 'Slot 12' >> >> ses0: da0,pass11: SAS Device Slot Element: 1 Phys at Slot 11 >> >> >> >> >> >> Luckily we have found a way to fairly easily repro it over a few >> >> hours, so we are open to any suggestions. >> >> >> >> Thanks! >> >> Jason >> > >> > >> > _______________________________________________ >> > freebsd-scsi@freebsd.org mailing list >> > https://lists.freebsd.org/mailman/listinfo/freebsd-scsi >> > To unsubscribe, send any mail to > "freebsd-scsi-unsubscribe@freebsd.org" >> > >> _______________________________________________ >> freebsd-scsi@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-scsi >> To unsubscribe, send any mail to "freebsd-scsi-unsubscribe@freebsd.org"