From nobody Fri Sep 1 15:46:00 2023 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Rcj701Y8cz4rYgM; Fri, 1 Sep 2023 15:46:04 +0000 (UTC) (envelope-from kevans@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Rcj700xbmz3HBL; Fri, 1 Sep 2023 15:46:04 +0000 (UTC) (envelope-from kevans@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1693583164; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2bJx0Xg6Juv2q4XkURGZaROnUNOOudfZ7B9flSGgekU=; b=L8OKzx/cfR9yf87ZbHWlH2guFpGpQkMADIg+NiYVNJqstRg4fpjURVXBqaFwF9e8PPAyiL QNk1Dt1SktdzudnCoyZR+KyUGIlSpq0JxR15YKXRsl81W56Tts10EB4Gx0qd1vQqfvQz54 PJmQPrXrFrKoNRMckSEdf1m7DbJE3o55ss2wKXpCUwt1StV7N31qlib5+CPADZhXUhpWBG e4BNM7qq1nE14YSDEGtnOROj4cG6qUmBg6WfWngcdojtJGOsZjgH7QN8raP9smvUIPL52q 8Zu6ReJFxKBIk0DyVC5ZuXt87HvFIcdhKvSnb5Gz2RooJTxr0zDUlrIeQWusgA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1693583164; a=rsa-sha256; cv=none; b=KTGej8yVCXpKCZTg0vVhxExdNmE8eXWkH4Smg6pttbwWTHikJtan1MHNhq2cfjhEqp3KVi VMOLjm6UohSLpZcDNTkYccZ7K4o+Oec47ljB/XYiK5V72VVbO/Gdn9PD0TMSShnAeJjFzV Si9V0o1D1Hsn3o+dqxMDrZUSgrs70FvaYcH6N+lBP6wgXVFFtlXsINIsyhqG+KwIghC1C3 lvixYhtKGZS51aVoDV/yYXgfLdbOaP6u0h9ook/nEYc5hiqjpHCsMyPjRDR+Z6YjAMRUMJ 3Ej2hbzYmSSqFpDv16m3GqDwFrpl1600Dc4lAUyi5rvjhdg+13xDH4ENmwVH5g== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1693583164; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2bJx0Xg6Juv2q4XkURGZaROnUNOOudfZ7B9flSGgekU=; b=ECVLUTCLlezVIOHzlIoApf9dNOEAPncdVuP3a1G1qUJkWQ89sJbF5JN6B0bJ7/3WOcK/6y z/fwolSe0z3TEldTQRKKNJ7BgwB+M2xXyaQQGuJpQF9PErjuCRQkFzVJjeGjgIzGM7z/CR ZKWVsetbO7IubOkn2ZyQ+l9tVzoTCLkn9TAshm3smYZxahZoD9HmLUc6lXdkjeknX2h/oI d5LouTki4iZDVtFRLAKPz3WTLSOAKWltALPhnPgDBkkG2Rm5ttbOWJ6Uo2UUegCp7BbSWY jEJoOn5wdDceVRMXjdAiA6Kz6HVlL7yn0JE0clrtCha24JAqILWcw6ST4MLo1Q== Received: from [10.9.4.95] (unknown [209.182.120.176]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: kevans/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4Rcj6z3H9Fz3C; Fri, 1 Sep 2023 15:46:03 +0000 (UTC) (envelope-from kevans@FreeBSD.org) Message-ID: <7b12cc47-0e41-ee8c-2165-9e81874c3490@FreeBSD.org> Date: Fri, 1 Sep 2023 10:46:00 -0500 List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.13.0 Subject: Re: git: 315ee00fa961 - main - zfs: merge openzfs/zfs@804414aad To: Alexander Motin , Martin Matuska Cc: src-committers@freebsd.org, dev-commits-src-all@freebsd.org, dev-commits-src-main@freebsd.org, Cy Schubert References: <202308270509.37R596B5048298@gitrepo.freebsd.org> <65269e7a-4c3f-95ff-3e81-91b76e023fbd@FreeBSD.org> Content-Language: en-US From: Kyle Evans In-Reply-To: <65269e7a-4c3f-95ff-3e81-91b76e023fbd@FreeBSD.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 9/1/23 08:41, Alexander Motin wrote: > On 31.08.2023 22:18, Kyle Evans wrote: >> It seems to have clearly been stomped on by uma trashing. Encountered >> while running a pkgbase build, I think while it was in the packaging >> phase. I note in particular in that frame: >> >> (kgdb) p/x lwb->lwb_issued_timestamp >> $4 = 0xdeadc0dedeadc0de >> >> So I guess it was freed sometime during one of the previous two >> zio_nowait() calls. > > Thank you, Kyle.  If the source lines are resolved correctly and it > really crashes on lwb_child_zio access, then I do see there a possible > race condition, even though I think it would involve at least 2 or may > be even 3 different threads. > Oh, sorry- yes, it was the access to lwb_child_zio there. > I've just created this new PR to address it: > https://github.com/openzfs/zfs/pull/15233 > > If you'll be able to test it, include also the two previous: > https://github.com/openzfs/zfs/pull/15227 > https://github.com/openzfs/zfs/pull/15228 > > Thank you for something actionable, it really feels much better! :) > Perfect, thanks! I haven't been able to reproduce it since the first time, but your explanation sounds plausible to me. I'm not a ZFS developer, but it's not clear to me how I didn't end up tripping over other assertions, though; e.g., in zil_lwb_flush_vdevs_done: 1442 ASSERT3S(lwb->lwb_state, ==, LWB_STATE_WRITE_DONE); 1443 lwb->lwb_state = LWB_STATE_FLUSH_DONE; lwb_state seems to only be set to LWB_STATE_WRITE_DONE in zil_lwb_write_done (lwb_write_zio's completion routine). I would've thought all three of these were executed synchronously in __zio_execute(), which would presumably put us in LWB_STATE_ISSUED at the time of completing the lwb_root_zio? Thanks, Kyle Evans