From nobody Thu Aug 31 16:50:19 2023 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Rc6cJ6jRwz4rMt1; Thu, 31 Aug 2023 16:50:56 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-yw1-x112e.google.com (mail-yw1-x112e.google.com [IPv6:2607:f8b0:4864:20::112e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Rc6cJ4WhYz4PrJ; Thu, 31 Aug 2023 16:50:56 +0000 (UTC) (envelope-from mavbsd@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-yw1-x112e.google.com with SMTP id 00721157ae682-58fae4a5285so12214017b3.0; Thu, 31 Aug 2023 09:50:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1693500655; x=1694105455; darn=freebsd.org; h=content-transfer-encoding:in-reply-to:subject:from:references:cc:to :content-language:user-agent:mime-version:date:message-id:sender :from:to:cc:subject:date:message-id:reply-to; bh=hPWIbPEOOm59r0FCudD+AHyX6x6YMDZgphrVpsUlGXA=; b=Weq9sJd56wZ6nkyiqbzmXmWfwV/loZugEXyL8wJzfpHRQ4Q4wHurQXRulFWQwy3l3Z TY+RTzyWMnPZnaFE6dC0cUjK6BDSqb4ErAxeRj4ZKibYSw/ZQCyGbFVzo0LdjW91YBsE foZdS44ZOxvEcxSk5k0rRinaG/O49eAtmab1EusB67OovgZAB8yFftd/LLOtMYr23tHu oWhPYlQd6gIU806PklLRQtD63GLlOVPmzeigxFvoPHdeAc/pJPTisAFWP48PqlGWD9al 4s6vegEJWStaOFroOY70MmBncK0jPxHv1ZjqX98ehfSG0/A7wKLmvZ/RTKoTJU4fLVEX TqPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693500655; x=1694105455; h=content-transfer-encoding:in-reply-to:subject:from:references:cc:to :content-language:user-agent:mime-version:date:message-id:sender :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=hPWIbPEOOm59r0FCudD+AHyX6x6YMDZgphrVpsUlGXA=; b=Y+u5ygcDpR7oVozQ29CuiYHIjm8IHnTwm9bESCSz6LWRqUvz4oEpLHBVffIOyz4feK Fq6t9ZjbJhlHGrpj9cmjGFxJFTOxGAxIfp6hOQn9PrCbYvFwEIfJIWI0BSkumtuFSUuK 70h3TriV3ek9rivfBDNwsH3WPu+xkCnhj111DLWkn/kwtHphLpcI4Oq8lkOSb1T7FQ/Q zAK242H/L82pXubRdhyzNvl0nz3lQukz6iYehizceERO4IDtYjwnAWwekS3ZgK2dvWY4 n835/0sBRjtT+qnd/XuWJI4sERZRHbWajhPE1W/iB+73enpC28kk6mjPyLqqRwQz362P t/Sg== X-Gm-Message-State: AOJu0YwGp8JRj83BeZ5i3jYN4aVzDnT9SfqeEsrBV9PqdZXNJyp7IHWp 3lZjPKWPsV+iuAyX3huGeYMHO6LaTLRUug== X-Google-Smtp-Source: AGHT+IGhyCW6hHqNwF9v7Vk1sj6CsbkFyZO7h+HJc7DxjOk2LSwnNs9zgedEMPgSsfQYhzVGoHbtGg== X-Received: by 2002:a0d:d5d4:0:b0:583:a590:cd66 with SMTP id x203-20020a0dd5d4000000b00583a590cd66mr5769002ywd.4.1693500655170; Thu, 31 Aug 2023 09:50:55 -0700 (PDT) Received: from [10.230.45.5] ([38.32.73.2]) by smtp.gmail.com with ESMTPSA id l62-20020a0dc941000000b005925765aa30sm524766ywd.135.2023.08.31.09.50.54 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 31 Aug 2023 09:50:54 -0700 (PDT) Message-ID: <1db726d4-32c9-e1b8-51d6-981aa51b7825@FreeBSD.org> Date: Thu, 31 Aug 2023 12:50:19 -0400 List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.11.0 Content-Language: en-US To: Drew Gallatin , Gleb Smirnoff , Martin Matuska Cc: src-committers@freebsd.org, dev-commits-src-all@freebsd.org, dev-commits-src-main@freebsd.org References: <202308270509.37R596B5048298@gitrepo.freebsd.org> <07faf861-9186-47d1-992a-91d483ea4e9c@app.fastmail.com> From: Alexander Motin Subject: Re: git: 315ee00fa961 - main - zfs: merge openzfs/zfs@804414aad In-Reply-To: <07faf861-9186-47d1-992a-91d483ea4e9c@app.fastmail.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4Rc6cJ4WhYz4PrJ On 31.08.2023 08:45, Drew Gallatin wrote: > On Wed, Aug 30, 2023, at 8:01 PM, Alexander Motin wrote: >> It is the first time I see a panic like this.  I'll think about it >> tomorrow.  But I'd appreciate any information on what is your workload >> and what are you doing related to ZIL (O_SYNC, fsync(), sync=always, >> etc) to trigger it?  What is your pool configuration? > > I'm not Gleb, but this was something at $WORK, so I can perhaps help. > I've included the output of zpool status, and all non-default settings > in the zpool.  Note that we don't use a ZIL device. You don't use SLOG device. ZIL is always with you, just embedded in this case. I tried to think about this for couple hours and still can't see how can this happen. zil_sync() should not call zil_free_lwb() unless the lwb is in LWB_STATE_FLUSH_DONE. To get into LWB_STATE_FLUSH_DONE lwb should first delete all lwb_vdev_tree entries in zil_lwb_write_done(). And no new entries should be added during/after zil_lwb_write_done() due to set zio dependencies. I've made a patch tuning some assertions for this context: https://github.com/openzfs/zfs/pull/15227 . If the issue is reproducible, could you please apply it and try again? May be it give us any more clues. -- Alexander Motin