From owner-svn-src-all@freebsd.org Tue Aug 27 21:27:18 2019 Return-Path: Delivered-To: svn-src-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 93F8FC835F; Tue, 27 Aug 2019 21:27:18 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 46J25L3BW6z4MdP; Tue, 27 Aug 2019 21:27:17 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from compute6.internal (compute6.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id 56D82220A7; Tue, 27 Aug 2019 17:27:17 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Tue, 27 Aug 2019 17:27:17 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsco.org; h= content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; s=fm1; bh=U XcEkHr1ApfNEEzlOWdbaPubBAY0CQVg6uoL2durN6w=; b=KbkNkQ1fTk6hdw4qO ICmk4VPbXQB4SJilMEmiBIx91UcfvQPL6DfHZ/7LPu4k16zUqL8SF4UXOy+k2Hkr jKuZ357em8LwAyZZOTdUiAo6r0MAnR2f45C7I3Ewuv4OrzrXXkusqeU4tIv7eYlO l6LbuYyH32SaMX9UpcOWCpiSDkJRF0K4DUoBjIpZW9UBuWctplslb5pi+Kug2iHY e/xwxwhD/+SUf+TeqlNATLlsZAMsUcdfxuvrhoOgGMJGrEf3IEtfT9zjVfXA/o52 hauyY92J6a+enxlczdC5eEx0caJPRkkhf9UV4nrMplK6m8tg0bv3kvfd9FKTGgiD iQHJQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; bh=UXcEkHr1ApfNEEzlOWdbaPubBAY0CQVg6uoL2durN 6w=; b=ZX8GmmZ7aOLOdgVEN6heVWBXKW8GEk3LZEEkkH+NnrFlwJXmtLej5C2Sq glDAe7AZYajG7tr9BztlTfN/FTXbgLPoV+X4s/I5mpXC+rTcodjQIBBtP+s93b4O p/6WehUIELnbQIhsNFP4nJZBPO06cGEEx2aFdKco4Lc2ZdldcgZFE/0D3OlDnW9Q 0LOKPV9V9aXZ8y4T0C60CF3SDn/BPQgXMcbGKjcl0Um+LAjmuE3J+VQCEJVri5Pr XR82XRBbpwjECDSDpUhAQ1IvUqukJp7hoYkihXSSkB15EqkBnq1m1eGgjQwuA2lv f5emxRXcaBLbDLG4w8NdEpcvaQDkQ== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduvddrudehkedguddutdcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecunecujfgurheptggguffhjgffgffkfhfvofesth hqmhdthhdtjeenucfhrhhomhepufgtohhtthcunfhonhhguceoshgtohhtthhlsehsrghm shgtohdrohhrgheqnecuffhomhgrihhnpehfrhgvvggsshgurdhorhhgnecukfhppeekrd egiedrkeelrddvudefnecurfgrrhgrmhepmhgrihhlfhhrohhmpehstghothhtlhesshgr mhhstghordhorhhgnecuvehluhhsthgvrhfuihiivgeptd X-ME-Proxy: Received: from [192.168.0.114] (unknown [8.46.89.213]) by mail.messagingengine.com (Postfix) with ESMTPA id 8D5F980059; Tue, 27 Aug 2019 17:27:16 -0400 (EDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.11\)) Subject: Re: svn commit: r351550 - head/sys/cam/scsi From: Scott Long In-Reply-To: <3c2aa0be-3d42-881e-87e1-675499a7bc5f@FreeBSD.org> Date: Tue, 27 Aug 2019 15:27:15 -0600 Cc: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <201908271641.x7RGf6LC075849@repo.freebsd.org> <99271565-F168-48C8-90E0-749417C7C974@samsco.org> <3c2aa0be-3d42-881e-87e1-675499a7bc5f@FreeBSD.org> To: Alexander Motin X-Mailer: Apple Mail (2.3445.104.11) X-Rspamd-Queue-Id: 46J25L3BW6z4MdP X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-6.95 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; REPLY(-4.00)[]; NEURAL_HAM_SHORT(-0.95)[-0.948,0] X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Aug 2019 21:27:18 -0000 Excellent work, thank you! Scott > On Aug 27, 2019, at 2:57 PM, Alexander Motin wrote: >=20 > Some FreeNAS user reported panic after updating to newer version. On > the screenshot provided were several BUSY statuses for SATA disk on > mps(4), followed by panic "Attempt to remove out-of-bounds index -1 = from > queue ...". In his case I blame ancient LSI firmware or some broken > hardware, but I was able to reproduce the panic on FreeBSD head debug > kernel by hacking mps(4) driver to always report BUSY (appeared except > IDENTIFY and REPORT LUNS). To diagnose it I inserted assertion into > xpt_free_ccb(), checking ccb->ccb_h.pinfo.index for values used for > requests still in send queue. Not sure it is to be persistent, but in > this case it lead me directly to this place. >=20 > On 27.08.2019 16:23, Scott Long wrote: >> This is very concerning, and I wonder if it=E2=80=99s the cause of = the mystery use-after-free / double-complete that I=E2=80=99ve seen for = years and have never been able to catch. Can you say more about how you = found it? >>=20 >> Scott >>=20 >>=20 >>> On Aug 27, 2019, at 10:41 AM, Alexander Motin = wrote: >>>=20 >>> Author: mav >>> Date: Tue Aug 27 16:41:06 2019 >>> New Revision: 351550 >>> URL: https://svnweb.freebsd.org/changeset/base/351550 >>>=20 >>> Log: >>> Always check cam_periph_error() status for ERESTART. >>>=20 >>> Even if we do not expect retries, we better be sure, since otherwise = it >>> may result in use after free kernel panic. I've noticed that it = retries >>> SCSI_STATUS_BUSY even with SF_NO_RECOVERY | SF_NO_RETRY. >>>=20 >>> MFC after: 1 week >>> Sponsored by: iXsystems, Inc. >>>=20 >>> Modified: >>> head/sys/cam/scsi/scsi_xpt.c >>>=20 >>> Modified: head/sys/cam/scsi/scsi_xpt.c >>> = =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D >>> --- head/sys/cam/scsi/scsi_xpt.c Tue Aug 27 15:42:08 2019 = (r351549) >>> +++ head/sys/cam/scsi/scsi_xpt.c Tue Aug 27 16:41:06 2019 = (r351550) >>> @@ -1684,8 +1684,9 @@ probe_device_check: >>> case PROBE_TUR_FOR_NEGOTIATION: >>> case PROBE_DV_EXIT: >>> if (cam_ccb_status(done_ccb) !=3D CAM_REQ_CMP) { >>> - cam_periph_error(done_ccb, 0, >>> - SF_NO_PRINT | SF_NO_RECOVERY | SF_NO_RETRY); >>> + if (cam_periph_error(done_ccb, 0, SF_NO_PRINT | >>> + SF_NO_RECOVERY | SF_NO_RETRY) =3D=3D = ERESTART) >>> + goto outr; >>> } >>> if ((done_ccb->ccb_h.status & CAM_DEV_QFRZN) !=3D 0) { >>> /* Don't wedge the queue */ >>> @@ -1735,8 +1736,9 @@ probe_device_check: >>> struct ccb_scsiio *csio; >>>=20 >>> if (cam_ccb_status(done_ccb) !=3D CAM_REQ_CMP) { >>> - cam_periph_error(done_ccb, 0, >>> - SF_NO_PRINT | SF_NO_RECOVERY | SF_NO_RETRY); >>> + if (cam_periph_error(done_ccb, 0, SF_NO_PRINT | >>> + SF_NO_RECOVERY | SF_NO_RETRY) =3D=3D = ERESTART) >>> + goto outr; >>> } >>> if ((done_ccb->ccb_h.status & CAM_DEV_QFRZN) !=3D 0) { >>> /* Don't wedge the queue */ >>>=20 >>=20 >=20 > --=20 > Alexander Motin >=20