Date: Thu, 31 Aug 2023 10:53:50 -0700 From: Cy Schubert <Cy.Schubert@cschubert.com> To: Alexander Motin <mav@FreeBSD.org> Cc: Drew Gallatin <gallatin@freebsd.org>, Gleb Smirnoff <glebius@freebsd.org>, Martin Matuska <mm@freebsd.org>, src-committers@freebsd.org, dev-commits-src-all@freebsd.org, dev-commits-src-main@freebsd.org Subject: Re: git: 315ee00fa961 - main - zfs: merge openzfs/zfs@804414aad Message-ID: <20230831175350.981F1D5@slippy.cwsent.com> In-Reply-To: <1db726d4-32c9-e1b8-51d6-981aa51b7825@FreeBSD.org> References: <202308270509.37R596B5048298@gitrepo.freebsd.org> <ZO_aOaf-eGiCMCKy@cell.glebi.us> <c09c92df-90f5-8c94-4125-9e33262bc686@FreeBSD.org> <07faf861-9186-47d1-992a-91d483ea4e9c@app.fastmail.com> <1db726d4-32c9-e1b8-51d6-981aa51b7825@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
In message <1db726d4-32c9-e1b8-51d6-981aa51b7825@FreeBSD.org>, Alexander Motin writes: > On 31.08.2023 08:45, Drew Gallatin wrote: > > On Wed, Aug 30, 2023, at 8:01 PM, Alexander Motin wrote: > >> It is the first time I see a panic like this. I'll think about it > >> tomorrow. But I'd appreciate any information on what is your workload > >> and what are you doing related to ZIL (O_SYNC, fsync(), sync=always, > >> etc) to trigger it? What is your pool configuration? > > > > I'm not Gleb, but this was something at $WORK, so I can perhaps help. > > I've included the output of zpool status, and all non-default settings > > in the zpool. Note that we don't use a ZIL device. > > You don't use SLOG device. ZIL is always with you, just embedded in > this case. > > I tried to think about this for couple hours and still can't see how can > this happen. zil_sync() should not call zil_free_lwb() unless the lwb > is in LWB_STATE_FLUSH_DONE. To get into LWB_STATE_FLUSH_DONE lwb should > first delete all lwb_vdev_tree entries in zil_lwb_write_done(). And no > new entries should be added during/after zil_lwb_write_done() due to set > zio dependencies. > > I've made a patch tuning some assertions for this context: > https://github.com/openzfs/zfs/pull/15227 . If the issue is > reproducible, could you please apply it and try again? May be it give > us any more clues. One thing that circumvents my two problems is reducing poudriere bulk jobs from 8 to 5 on my 4 core machines. -- Cheers, Cy Schubert <Cy.Schubert@cschubert.com> FreeBSD UNIX: <cy@FreeBSD.org> Web: https://FreeBSD.org NTP: <cy@nwtime.org> Web: https://nwtime.org e^(i*pi)+1=0 =L From nobody Thu Aug 31 19:07:40 2023 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Rc9fn0NGcz4rVj1; Thu, 31 Aug 2023 19:08:17 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-yw1-x1132.google.com (mail-yw1-x1132.google.com [IPv6:2607:f8b0:4864:20::1132]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Rc9fm5mcGz4cG3; Thu, 31 Aug 2023 19:08:16 +0000 (UTC) (envelope-from mavbsd@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-yw1-x1132.google.com with SMTP id 00721157ae682-58df8cab1f2so13546837b3.3; Thu, 31 Aug 2023 12:08:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1693508896; x=1694113696; darn=freebsd.org; h=content-transfer-encoding:in-reply-to:subject:from:references:cc:to :content-language:user-agent:mime-version:date:message-id:sender :from:to:cc:subject:date:message-id:reply-to; bh=lpt1FAxULub6RCUc0kYz0qfHAY9tHKICJrDBf6/dWsw=; b=McHxFpkpjy+YsfzRAGkEhp7K2WrhqdFoPdME6h+aOBzyLrEaxgMpPzJfdpkZp5FJ3B zmkDdsOzV9q03f1dvUxwceux9QBiQdw9j99k11hk4+ur7Q7IhvpLfdHR3ESl+SUXkQsV rr8jxH4hrln1v+FZ8m05iF0FOIupStBRhmnKNV0HpSKB/zvXvqHSjUuqIIfpvlOXoGoH GB4MTy77BgJyiJ2wotos45JNGWdN2nFRMUeXWK+7MS0wYWJPdkImvLiMdoo+BHqxa7nD EMTgNmd/dCdw753au/IK0cuP3rGvvbRMsK5fw0oZOmaI09SOkZmPtQpHqPOitkqro+Nq BKRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693508896; x=1694113696; h=content-transfer-encoding:in-reply-to:subject:from:references:cc:to :content-language:user-agent:mime-version:date:message-id:sender :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=lpt1FAxULub6RCUc0kYz0qfHAY9tHKICJrDBf6/dWsw=; b=jCeaVyTx5WKWGJci2j0oVoL28iLpOIj9bkGiWPc+4dIaZ0IuNw8HGk8j7tKOtG9tjP xsD6mvUVordpuWoxx0kOerPHTHHSa+aAf85k0g80p42H6DqtJc6ZDkGf/cXuosBCu9Kc /jUx2NDnzrXbmxFacygINSag5m72F/Egc8dWFfl8K8CdZw8Bi5EnG+dI+nlnxubf6ABH yUDtxjHurMFG0DkBzsZyV2e/YrbEl1EmUIT5LuE7VKmeXicsXgUneGXLaD6WB9iwk7oY si2Cb2C7SNZzRssWC/929kxw2DIk6DY2k3AIC+1kRBrGB7i86Awdq5lLZy8lpq2ZVgOU nxoA== X-Gm-Message-State: AOJu0YztXKO77DNzv0t3W2la9eg2CpBUp0sOnunWXnSECTpqvEycHFsE 0FN36eGkQq4q2833+5LwUHOoVfwk/UDbkw== X-Google-Smtp-Source: AGHT+IF7jF+t6pgD0LGrthE/28Mp90CgoUE8Go9otFJXIjRvhlCG6zHvnps0b2euUASlr4qtVGxoqQ== X-Received: by 2002:a0d:eb10:0:b0:586:cf7:2207 with SMTP id u16-20020a0deb10000000b005860cf72207mr308022ywe.14.1693508896063; Thu, 31 Aug 2023 12:08:16 -0700 (PDT) Received: from [10.230.45.5] ([38.32.73.2]) by smtp.gmail.com with ESMTPSA id s4-20020a817704000000b0057399b3bd26sm607153ywc.33.2023.08.31.12.08.14 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 31 Aug 2023 12:08:15 -0700 (PDT) Message-ID: <a5c51f3f-8c7f-8bd5-f718-72bc33fe22ed@FreeBSD.org> Date: Thu, 31 Aug 2023 15:07:40 -0400 List-Id: Commit messages for all branches of the src repository <dev-commits-src-all.freebsd.org> List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: <mailto:dev-commits-src-all+help@freebsd.org> List-Post: <mailto:dev-commits-src-all@freebsd.org> List-Subscribe: <mailto:dev-commits-src-all+subscribe@freebsd.org> List-Unsubscribe: <mailto:dev-commits-src-all+unsubscribe@freebsd.org> Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.11.0 Content-Language: en-US To: Cy Schubert <Cy.Schubert@cschubert.com>, Gleb Smirnoff <glebius@FreeBSD.org> Cc: Drew Gallatin <gallatin@freebsd.org>, Martin Matuska <mm@freebsd.org>, src-committers@freebsd.org, dev-commits-src-all@freebsd.org, dev-commits-src-main@freebsd.org References: <202308270509.37R596B5048298@gitrepo.freebsd.org> <ZO_aOaf-eGiCMCKy@cell.glebi.us> <c09c92df-90f5-8c94-4125-9e33262bc686@FreeBSD.org> <07faf861-9186-47d1-992a-91d483ea4e9c@app.fastmail.com> <1db726d4-32c9-e1b8-51d6-981aa51b7825@FreeBSD.org> <20230831175350.981F1D5@slippy.cwsent.com> From: Alexander Motin <mav@FreeBSD.org> Subject: Re: git: 315ee00fa961 - main - zfs: merge openzfs/zfs@804414aad In-Reply-To: <20230831175350.981F1D5@slippy.cwsent.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4Rc9fm5mcGz4cG3 On 31.08.2023 13:53, Cy Schubert wrote: > One thing that circumvents my two problems is reducing poudriere bulk jobs > from 8 to 5 on my 4 core machines. Cy, I have no real evidences to think it is related, other than your panics look like some memory corruptions, but could you try is patch: https://github.com/openzfs/zfs/pull/15228 . If it won't do the trick, then I am out of ideas without additional input. Gleb, you may try to add this too, just as a choice between impossible and improbable. -- Alexander Motin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20230831175350.981F1D5>