From owner-svn-src-all@freebsd.org Tue Aug 27 20:57:57 2019 Return-Path: Delivered-To: svn-src-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 43758DF73C; Tue, 27 Aug 2019 20:57:57 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-yb1-xb29.google.com (mail-yb1-xb29.google.com [IPv6:2607:f8b0:4864:20::b29]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46J1RS1kRwz4L2N; Tue, 27 Aug 2019 20:57:55 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: by mail-yb1-xb29.google.com with SMTP id o82so42920ybg.10; Tue, 27 Aug 2019 13:57:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:to:cc:references:from:autocrypt:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=qh9Ia3lhy4FNxr2uqNDgPQoDs2Vy+2zJVJUCTvN+gu0=; b=d1kTYW8Gvd5fEJkF8HKnWmFaVfR++WwZRYUtdBzXewnHih+Us31e5stwreWHQ2kkF0 l6VQL3BKgdUDc3bWOHMMVmkWBiGX5OV+QYjg/ozVGuDSdRCxO1FmJRxV9PcshR+SfWIG irYaH8DVyJ/NJvudTZUQPDEoLohnXFH8VgJ9TjT53WM697klEPK/dXSCvawCbRtLxEU2 cWla63jLrhqVAQqoao8CwpHXD+peT5cEzfcBkzG5JUiTaOmwH4AjXdFh7axYDXlaTx21 L+M925+cR6fznhAuW2Yei+ut/piZsKKOY0OigB607iJxAEGRn9a6DppEEduNZKu2Q4tK NZGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:to:cc:references:from:autocrypt :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=qh9Ia3lhy4FNxr2uqNDgPQoDs2Vy+2zJVJUCTvN+gu0=; b=gdu3pdRVHx2hGB0TAAOEQWcM9ZY8mstC7ahYrmG6crUeLcGMvlTY61FYI4u6Xn/dgy 0f6SWoww895peR2nFb///gie61z+eXgWqUyvcXxKs+YyqrW1WE0UA7upCfetUcox3Jry CqRTEr+syG3nVECdVXCZknGk3ZU3GDOxpfMTTUs3/lOh1l+RdWQ/FGDVf1ss+GiQsfLS bHakYyNrvfhKWOodcPA7WF+XHs30I3UcifF4qApi64kSNjajpbP74I676HvqxorD+hQl jxY8shAQyUD8d4yHRjo03+cOEQ4ne50n9YjWrOV5ihwkwyiaWUYl8Q0PicF55nuF9Ev6 a9eQ== X-Gm-Message-State: APjAAAVwTSOTO6sq8isuy8of6SVpm/pIU5ru8eQweqA/N/9YCBVryr9u NOM6lnD183ajCkR5TjsU00yg78ERxw8= X-Google-Smtp-Source: APXvYqxtbTY22dgDumRU9LEophnn9TIO2b9Wntgf3NoYLbZxMj7kwUTdUcnXUCG0jBbgm99K0pbUhg== X-Received: by 2002:a5b:b4d:: with SMTP id b13mr590679ybr.308.1566939474661; Tue, 27 Aug 2019 13:57:54 -0700 (PDT) Received: from mavoffice.ixsystems.com ([12.189.233.129]) by smtp.gmail.com with ESMTPSA id 199sm135160ywo.95.2019.08.27.13.57.53 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 27 Aug 2019 13:57:54 -0700 (PDT) Sender: Alexander Motin Subject: Re: svn commit: r351550 - head/sys/cam/scsi To: Scott Long Cc: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org References: <201908271641.x7RGf6LC075849@repo.freebsd.org> <99271565-F168-48C8-90E0-749417C7C974@samsco.org> From: Alexander Motin Autocrypt: addr=mav@FreeBSD.org; prefer-encrypt=mutual; keydata= mQENBFOzxAwBCADkPrax0pI2W/ig0CK9nRJJwsHitAGEZ2HZiFEuti+6/4UVxj81yr4ak/4g 9bKUyC7rMEAp/ZHNhd+MFCPAAcHPvtovnfykqE/vuosCS3wlSLloix2iKVLks0CwbLHGAyne 46lTQW74Xl/33c3W1Z6d8jD9gVFT/xaVzZ0U9xdzOmsYAZaAj4ki0tuxO9F7L+ct9grRe7iP g8t9hai7BL4ee3VRwk2JXnKb7UvBiVITKYWKz1jRvZIrjPokgEcCLOSlv7x/1kjuFnj3xWZU 7HSFFT8J93epBbrSSCsYsppIk2fZH41kaaFXsMQfTPH8wkeM6qwrvOh4HiQM08R+9tThABEB AAG0IUFsZXhhbmRlciBNb3RpbiA8bWF2QEZyZWVCU0Qub3JnPokBVwQTAQoAQQIbAwULCQgH AwUVCgkICwUWAwIBAAIeAQIXgAIZARYhBOmM88TmnMPNDledVYMYw5VbqyJ/BQJZYMKuBQkN McyiAAoJEIMYw5VbqyJ/tuUIAOG3ONOSNYqjK4eTZ1TVh9jdUBAhWk5nhDFnODN49Wj0AbYm 7aIqy8O1hnCDSZG5LttjSAo3UfXJZDKQM0BLb0gpRMBnAYqO6tdolLNqAbPGJBnGoPjsh24y 6KcbDaNnis+lD4GwPXwQM+92wZGhCUFElPV9NciZGVS65TNIgk7X+yEjjhD1MSWKKijZ1r9Z zIt4OzUTxxNOvzdlABZS88nNRdJkatOQJPmFdd1mpP6UzTNCiLUo1pIqOEtJgvVVDYq5WHY6 tciWWYdmZG/tIBexJmv2mV2OLVjXR6ZeKmntVH14H72/wRHJuYHQC+r5SVRcWWayrThsY6jZ Yr4+raS5AQ0EU7PEDAEIAOZgWf2cJIu+58IzP2dkXE/urj3tr4OqrB/yHGWUf71Lz6D0Fi6Z AXgDtmcFLGPfMyWuLAvSM+xmoguk7zC4hRBYvQycmIhuqBq1jO1Wp/Z+lpoPM/1cDYLn8Flv mI/c40MhUZh345DA4jYWWaZNjQHUWVQ1fPf595vdVVMPT/abE8E5DaF6fSkRmqFTmfYRkfbt 3ytU8NdUapDcJVY7cEP2nJBVNZPnOIObR/ZIgSxjjrG5o34yXoqeup8JvwEv+/NylzzuyXEZ R1EdEIzQ/a1nh/0j4NXtzZEqKW4aTWlmSqb6wN8jh1OSOOqkYsfnE3nfxcZbxi4IRoNQYlm5 9R8AEQEAAYkBPAQYAQoAJgIbDBYhBOmM88TmnMPNDledVYMYw5VbqyJ/BQJZYMLYBQkNMczM AAoJEIMYw5VbqyJ/TqgH/RQHClkvecE0262lwKoP/m0Mh4I5TLRgoJJn8S7G1BnqohYJkiLq A6xe6urGD7OqdNAl12UbrjWbdJV+zvea3vJoM4MZuYiYrGaXWxzFXqWJcPwMU9sAh8MRghHu uC5vgPb45Tnftw9/+n0i8GfVhQhOqepUGdQg4NPcXviSkoAvig6pp9Lcxisn0groUQKt15Gc sS9YcQWg3j9Hnipc6Mu416HX98Fb113NHJqc2geTHLkRyuBFOoyIqB6N9GKjzOAIzxxsVdl9 TevwGsrp4M4/RFzWbSgsbOnbE7454lmuVZGfReEjnUm8RHp9Q2UWKXlp3exlZjvOp/uVEpCg lz65AQ0EU7PEDAEIAOZgWf2cJIu+58IzP2dkXE/urj3tr4OqrB/yHGWUf71Lz6D0Fi6ZAXgD tmcFLGPfMyWuLAvSM+xmoguk7zC4hRBYvQycmIhuqBq1jO1Wp/Z+lpoPM/1cDYLn8FlvmI/c 40MhUZh345DA4jYWWaZNjQHUWVQ1fPf595vdVVMPT/abE8E5DaF6fSkRmqFTmfYRkfbt3ytU 8NdUapDcJVY7cEP2nJBVNZPnOIObR/ZIgSxjjrG5o34yXoqeup8JvwEv+/NylzzuyXEZR1Ed EIzQ/a1nh/0j4NXtzZEqKW4aTWlmSqb6wN8jh1OSOOqkYsfnE3nfxcZbxi4IRoNQYlm59R8A EQEAAYkBPAQYAQoAJgIbDBYhBOmM88TmnMPNDledVYMYw5VbqyJ/BQJZYMLYBQkNMczMAAoJ EIMYw5VbqyJ/TqgH/RQHClkvecE0262lwKoP/m0Mh4I5TLRgoJJn8S7G1BnqohYJkiLqA6xe 6urGD7OqdNAl12UbrjWbdJV+zvea3vJoM4MZuYiYrGaXWxzFXqWJcPwMU9sAh8MRghHuuC5v gPb45Tnftw9/+n0i8GfVhQhOqepUGdQg4NPcXviSkoAvig6pp9Lcxisn0groUQKt15GcsS9Y cQWg3j9Hnipc6Mu416HX98Fb113NHJqc2geTHLkRyuBFOoyIqB6N9GKjzOAIzxxsVdl9Tevw Gsrp4M4/RFzWbSgsbOnbE7454lmuVZGfReEjnUm8RHp9Q2UWKXlp3exlZjvOp/uVEpCglz4= Message-ID: <3c2aa0be-3d42-881e-87e1-675499a7bc5f@FreeBSD.org> Date: Tue, 27 Aug 2019 16:57:52 -0400 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.0 MIME-Version: 1.0 In-Reply-To: <99271565-F168-48C8-90E0-749417C7C974@samsco.org> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 46J1RS1kRwz4L2N X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=d1kTYW8G; dmarc=none; spf=pass (mx1.freebsd.org: domain of mavbsd@gmail.com designates 2607:f8b0:4864:20::b29 as permitted sender) smtp.mailfrom=mavbsd@gmail.com X-Spamd-Result: default: False [-6.10 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; NEURAL_HAM_SHORT(-0.98)[-0.984,0]; RCVD_IN_DNSWL_NONE(0.00)[9.2.b.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-2.92)[ip: (-9.34), ipnet: 2607:f8b0::/32(-2.86), asn: 15169(-2.33), country: US(-0.05)]; FORGED_SENDER(0.30)[mav@FreeBSD.org,mavbsd@gmail.com]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[mav@FreeBSD.org,mavbsd@gmail.com]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Aug 2019 20:57:57 -0000 Some FreeNAS user reported panic after updating to newer version. On the screenshot provided were several BUSY statuses for SATA disk on mps(4), followed by panic "Attempt to remove out-of-bounds index -1 from queue ...". In his case I blame ancient LSI firmware or some broken hardware, but I was able to reproduce the panic on FreeBSD head debug kernel by hacking mps(4) driver to always report BUSY (appeared except IDENTIFY and REPORT LUNS). To diagnose it I inserted assertion into xpt_free_ccb(), checking ccb->ccb_h.pinfo.index for values used for requests still in send queue. Not sure it is to be persistent, but in this case it lead me directly to this place. On 27.08.2019 16:23, Scott Long wrote: > This is very concerning, and I wonder if it’s the cause of the mystery use-after-free / double-complete that I’ve seen for years and have never been able to catch. Can you say more about how you found it? > > Scott > > >> On Aug 27, 2019, at 10:41 AM, Alexander Motin wrote: >> >> Author: mav >> Date: Tue Aug 27 16:41:06 2019 >> New Revision: 351550 >> URL: https://svnweb.freebsd.org/changeset/base/351550 >> >> Log: >> Always check cam_periph_error() status for ERESTART. >> >> Even if we do not expect retries, we better be sure, since otherwise it >> may result in use after free kernel panic. I've noticed that it retries >> SCSI_STATUS_BUSY even with SF_NO_RECOVERY | SF_NO_RETRY. >> >> MFC after: 1 week >> Sponsored by: iXsystems, Inc. >> >> Modified: >> head/sys/cam/scsi/scsi_xpt.c >> >> Modified: head/sys/cam/scsi/scsi_xpt.c >> ============================================================================== >> --- head/sys/cam/scsi/scsi_xpt.c Tue Aug 27 15:42:08 2019 (r351549) >> +++ head/sys/cam/scsi/scsi_xpt.c Tue Aug 27 16:41:06 2019 (r351550) >> @@ -1684,8 +1684,9 @@ probe_device_check: >> case PROBE_TUR_FOR_NEGOTIATION: >> case PROBE_DV_EXIT: >> if (cam_ccb_status(done_ccb) != CAM_REQ_CMP) { >> - cam_periph_error(done_ccb, 0, >> - SF_NO_PRINT | SF_NO_RECOVERY | SF_NO_RETRY); >> + if (cam_periph_error(done_ccb, 0, SF_NO_PRINT | >> + SF_NO_RECOVERY | SF_NO_RETRY) == ERESTART) >> + goto outr; >> } >> if ((done_ccb->ccb_h.status & CAM_DEV_QFRZN) != 0) { >> /* Don't wedge the queue */ >> @@ -1735,8 +1736,9 @@ probe_device_check: >> struct ccb_scsiio *csio; >> >> if (cam_ccb_status(done_ccb) != CAM_REQ_CMP) { >> - cam_periph_error(done_ccb, 0, >> - SF_NO_PRINT | SF_NO_RECOVERY | SF_NO_RETRY); >> + if (cam_periph_error(done_ccb, 0, SF_NO_PRINT | >> + SF_NO_RECOVERY | SF_NO_RETRY) == ERESTART) >> + goto outr; >> } >> if ((done_ccb->ccb_h.status & CAM_DEV_QFRZN) != 0) { >> /* Don't wedge the queue */ >> > -- Alexander Motin