From nobody Wed Aug 23 18:40:49 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RWFRN4pTTz4rDjS for ; Wed, 23 Aug 2023 18:41:20 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-ot1-x32e.google.com (mail-ot1-x32e.google.com [IPv6:2607:f8b0:4864:20::32e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RWFRN0WpNz4kkL for ; Wed, 23 Aug 2023 18:41:20 +0000 (UTC) (envelope-from mavbsd@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20221208 header.b=crrKO+8I; spf=pass (mx1.freebsd.org: domain of mavbsd@gmail.com designates 2607:f8b0:4864:20::32e as permitted sender) smtp.mailfrom=mavbsd@gmail.com; dmarc=none Received: by mail-ot1-x32e.google.com with SMTP id 46e09a7af769-6bcd4b5ebbaso4409945a34.1 for ; Wed, 23 Aug 2023 11:41:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692816079; x=1693420879; h=content-transfer-encoding:in-reply-to:subject:from:references:to :content-language:user-agent:mime-version:date:message-id:sender :from:to:cc:subject:date:message-id:reply-to; bh=WEDuVo/nTnBb/NjtYKhAGXQdHgGMu8fauo6swjY36KY=; b=crrKO+8IlSoQYQB8TErKb5OHNdBq4oDtpiK2Dj0vZXeTGFXagP/bN5XTUV/sD733Ed JyQDqgVNgra+XJtCyGPmuTHN1nbWNntX0d3F03Q3ebE4Pnch/ppQ71x0p0wUAcB3M5LU /8I5293WdvkonHamojzguiJBj03NSvbIM0PDT50UU0Q95Qdd68EyAnCybo6sbU6Gq09c f3i5B6jZ+oOP5DeunR/XA8GQwTBM6Inqioqteu8WP36i1rmZMhAuSDjHUuTBm2jwcutz +pJByP4hXEvZ/+ryLharO/bDqxx99wrKXyJ3b+w1fEF0X7Vd55QdnmYf1H0ycPpfwMr1 TGGA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692816079; x=1693420879; h=content-transfer-encoding:in-reply-to:subject:from:references:to :content-language:user-agent:mime-version:date:message-id:sender :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=WEDuVo/nTnBb/NjtYKhAGXQdHgGMu8fauo6swjY36KY=; b=WzhLTWUf7NdFgNHl69hbfaAWHuvqERB29YFxY8lBGlg52ZxxkWVIjfnxDVUw9gxhrB tfJ6ctJihWoJbtjYIDVcPAh9okvVeSsl+aZXDpUTX4FxNKXerO7+MfxfPTbtEu7YpDon lalSXCrhsRsAVzse+Ja1VN9t/p8gMZKBzqy4Ey6bsLhz6QgCrKBI97BMaxpQI324orHI Wwoq6atkdCTJ0ios4MXfFfkVbEUFwyXz4H0gVegRdHfD109nPWvgn3Pt0JDrrXoWgjLf 5ggeJRiyHCAcS/M+qqP2W3L12KA6oyYPYemNz3rUoyqBYmfvREB3nF5SgT3ip3c6+DZC /F6A== X-Gm-Message-State: AOJu0YxgoJrg01FbjxhNr5LInFZ5M2GsPgv19wE+05aqwwcZ0s4e5p+5 VvVN5rrXUcc4u+J03fy5boLE35lM7JA= X-Google-Smtp-Source: AGHT+IEYFtW8LeKCvMkcRdYqAKdkf5erwSGwUZ7T+mXClLlnw9Op64oVbDcFhfNtDnWLiChf4LtMDg== X-Received: by 2002:a05:6358:248b:b0:132:d42f:8e19 with SMTP id m11-20020a056358248b00b00132d42f8e19mr12863746rwc.31.1692816078751; Wed, 23 Aug 2023 11:41:18 -0700 (PDT) Received: from [10.230.45.5] ([38.32.73.2]) by smtp.gmail.com with ESMTPSA id x137-20020a0dd58f000000b005837b48d16csm3471433ywd.84.2023.08.23.11.41.18 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 23 Aug 2023 11:41:18 -0700 (PDT) Message-ID: Date: Wed, 23 Aug 2023 14:40:49 -0400 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.11.0 Content-Language: en-US To: Mark Millard , Current FreeBSD References: <4FFAE432-21FE-4462-9162-9CC30A5D470A.ref@yahoo.com> <4FFAE432-21FE-4462-9162-9CC30A5D470A@yahoo.com> From: Alexander Motin Subject: Re: ZFS deadlock in 14 In-Reply-To: <4FFAE432-21FE-4462-9162-9CC30A5D470A@yahoo.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spamd-Result: default: False [-3.20 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FORGED_SENDER(0.30)[mav@FreeBSD.org,mavbsd@gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20221208]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::32e:from]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DMARC_NA(0.00)[freebsd.org]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; FROM_NEQ_ENVFROM(0.00)[mav@FreeBSD.org,mavbsd@gmail.com]; RCPT_COUNT_TWO(0.00)[2]; DKIM_TRACE(0.00)[gmail.com:+]; TO_DN_ALL(0.00)[]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; FREEMAIL_TO(0.00)[yahoo.com,freebsd.org]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_COUNT_TWO(0.00)[2] X-Spamd-Bar: --- X-Rspamd-Queue-Id: 4RWFRN0WpNz4kkL On 22.08.2023 14:24, Mark Millard wrote: > Alexander Motin wrote on > Date: Tue, 22 Aug 2023 16:18:12 UTC : > >> I am waiting for final test results from George Wilson and then will >> request quick merge of both to zfs-2.2-release branch. Unfortunately >> there are still not many reviewers for the PR, since the code is not >> trivial, but at least with the test reports Brian Behlendorf and Mark >> Maybee seem to be OK to merge the two PRs into 2.2. If somebody else >> have tested and/or reviewed the PR, you may comment on it. > > I had written to the list that when I tried to test the system > doing poudriere builds (initially with your patches) using > USE_TMPFS=no so that zfs had to deal with all the file I/O, I > instead got only one builder that ended up active, the others > never reaching "Builder started": > Top was showing lots of "vlruwk" for the cpdup's. For example: > > . . . > 362 0 root 40 0 27076Ki 13776Ki CPU19 19 4:23 0.00% cpdup -i0 -o ref 32 > 349 0 root 53 0 27076Ki 13776Ki vlruwk 22 4:20 0.01% cpdup -i0 -o ref 31 > 328 0 root 68 0 27076Ki 13804Ki vlruwk 8 4:30 0.01% cpdup -i0 -o ref 30 > 304 0 root 37 0 27076Ki 13792Ki vlruwk 6 4:18 0.01% cpdup -i0 -o ref 29 > 282 0 root 42 0 33220Ki 13956Ki vlruwk 8 4:33 0.01% cpdup -i0 -o ref 28 > 242 0 root 56 0 27076Ki 13796Ki vlruwk 4 4:28 0.00% cpdup -i0 -o ref 27 > . . . > > But those processes did show CPU?? on occasion, as well as > *vnode less often. None of the cpdup's was stuck in > > Removing your patches did not change the behavior. Mark, to me "vlruwk" looks like a limit on number of vnodes. I was not deep in that area at least recently, so somebody with more experience there could try to diagnose it. At very least it does not look related to the ZIL issue discussed in this thread, at least with the information provided, so I am not surprised that the mentioned patches do not affect it. -- Alexander Motin