From nobody Mon Jul 8 14:09:14 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WHmFn1wTwz5QHYW for ; Mon, 08 Jul 2024 14:09:17 +0000 (UTC) (envelope-from des@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WHmFn1P2Nz4qMp; Mon, 8 Jul 2024 14:09:17 +0000 (UTC) (envelope-from des@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1720447757; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SWrmYTl/I6AP59QUEbw8XhYmsbL/1+I5nxD63QZc+Pc=; b=T+cgOPk+0haEIgg8Z7x403vQ0oZLUWtJbzjqTfr0zY+Z5eA3QyHwXr3NIU4SnPMMlwTBuq AuDmUqH3+aCpZNMVArRKnF6l/sPi0NoUdx7Lr8IScAzkorzjOmYzzAdfV92iHY+AZH5QKo H7ds7A9NWmAdRK8hFU/v7KP7ijD1OLNCplIw3agiYFHxYEGvlxmYB3feOtt63nbll4xPOl 6PJRV7ArHfRlLtv8mo+JAbzGr6ycn5XAhRFT+U+vRdDV/ieqvBvgawfer2d2mm+4J5IiNG OxkNCdYT0LnU4am7NipJOCx4uvLcBdctoj5ZGDG4diUlyxhY2KihDDuogKg4ng== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1720447757; a=rsa-sha256; cv=none; b=M86ZeF5IkfdQcrYzWRvnJegt0t6ebJjGXn81vVKDkyRWRJBjiZWyMJCQWVdUud2iMnfFrm Zh/IWFGnQbKN3UdWyLkJmU5iyeGJp7FZnOZoxiYmqijxyBOWQunvjCf+Gxm2ItPo5yRgmb hLWoQuQJHXjZZ+mn5++7kI9b6E0VKoTTzVz9/bDFhCHZo1wDvmdGW8eWsR0koACGuUbCBW 43fHmbFNpBrvItET5x2d8Iz8pr+4g65Zlya8AKExukY8awptHzCBlGjFbRbgFJGHvA/xL8 vjrTov+Ns8gqge5NKTE8Zjby3e3iK7qkpNF9zGT5S2yXdZ8fd6M0P1yd8C0n5w== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1720447757; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SWrmYTl/I6AP59QUEbw8XhYmsbL/1+I5nxD63QZc+Pc=; b=oUEaBXVqavHk2iaPtabmpM1iEWUnMeAwkHdbrVK03eQTDAaNyqIcrtXnagIhajFsWDC4zT 7+0nG17utWrNtJLPMKwHAgSPkqrKdw4y/jNVsFsetoV/L8uxXoUb17wnHVl7RgTESNS1dn sQrLSTN9YxlkmsERGLvtWVG3dj2BuaE3ex6AaYe7JfibRNi0mpYKvRdrhWpQkAnxIeM2xR oy6McGWIDAfxqBbrYm4NyIQmH7YJ3f07Ug+PY2Kaqgc/WBYng3EVAj6tC0+0Ttlfp3FUvk rVOTO8WBb7HLFEetzg78ImBuqaoIoZGhavkWN7V1d1QScccIE46my9ruVFAhEw== Received: from ltc.des.dev (unknown [91.174.26.112]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: des) by smtp.freebsd.org (Postfix) with ESMTPSA id 4WHmFn0HyZzZJs; Mon, 8 Jul 2024 14:09:17 +0000 (UTC) (envelope-from des@freebsd.org) Received: by ltc.des.dev (Postfix, from userid 1001) id CF5AD10117; Mon, 08 Jul 2024 16:09:14 +0200 (CEST) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: Steven Friedrich Cc: Dave Cottlehuber , FreeBSD Stable Subject: Re: freebsd hdmi audi In-Reply-To: <8d3efa2f-a073-4c02-ae32-3c8d824dc9b7@gmail.com> (Steven Friedrich's message of "Fri, 5 Jul 2024 20:22:06 -0400") References: <3d40cb00-9fbb-4809-b6d9-fa40944cc62c@app.fastmail.com> <86frsp17jc.fsf@ltc.des.dev> <86frsn8pb5.fsf@ltc.des.dev> <8d3efa2f-a073-4c02-ae32-3c8d824dc9b7@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Date: Mon, 08 Jul 2024 16:09:14 +0200 Message-ID: <86bk387z79.fsf@ltc.des.dev> List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-stable@freebsd.org Sender: owner-freebsd-stable@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Steven Friedrich writes: > FreeBSD is missing some magic that Linux has. Perhaps FreeBSD could > find the magic bits. OK, I'll look around, maybe they're in my other coat. DES --=20 Dag-Erling Sm=C3=B8rgrav - des@FreeBSD.org From nobody Wed Jul 10 23:22:00 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WKDVM4ykVz5RMLn for ; Wed, 10 Jul 2024 23:25:15 +0000 (UTC) (envelope-from naddy@mips.inka.de) Received: from mail.inka.de (mail.inka.de [IPv6:2a04:c9c7:0:1073:217:a4ff:fe3b:e77c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4WKDVL5GYRz4SZB for ; Wed, 10 Jul 2024 23:25:14 +0000 (UTC) (envelope-from naddy@mips.inka.de) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of naddy@mips.inka.de designates 2a04:c9c7:0:1073:217:a4ff:fe3b:e77c as permitted sender) smtp.mailfrom=naddy@mips.inka.de Received: from mips.inka.de (naddy@[127.0.0.1]) by mail.inka.de with uucp (rmailwrap 0.5) id 1sRggK-003exT-Fe; Thu, 11 Jul 2024 01:25:04 +0200 Received: from lorvorc.mips.inka.de (localhost [127.0.0.1]) by lorvorc.mips.inka.de (8.18.1/8.18.1) with ESMTP id 46ANM0VS007882 for ; Thu, 11 Jul 2024 01:22:00 +0200 (CEST) (envelope-from naddy@lorvorc.mips.inka.de) Received: (from naddy@localhost) by lorvorc.mips.inka.de (8.18.1/8.18.1/Submit) id 46ANM03g007881 for freebsd-stable@freebsd.org; Thu, 11 Jul 2024 01:22:00 +0200 (CEST) (envelope-from naddy) Date: Thu, 11 Jul 2024 01:22:00 +0200 From: Christian Weisgerber To: freebsd-stable@freebsd.org Subject: mac_do: gid rule fails Message-ID: List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-stable@freebsd.org Sender: owner-freebsd-stable@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.76 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.46)[-0.456]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:202113, ipnet:2a04:c9c7::/32, country:DE]; MIME_TRACE(0.00)[0:+]; FREEFALL_USER(0.00)[naddy]; MISSING_XM_UA(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_NONE(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; DMARC_NA(0.00)[inka.de]; ARC_NA(0.00)[] X-Rspamd-Queue-Id: 4WKDVL5GYRz4SZB I noticed that mac_do(4) and mdo(1) were recently added to 14-STABLE and decided to give them a try. A UID-based rule works: $ sysctl security.mac.do security.mac.do.rules: uid=1000:any security.mac.do.enabled: 1 $ id -u 1000 $ mdo id uid=0(root) gid=0(wheel) groups=0(wheel),5(operator) However, a GID rule fails: $ sysctl security.mac.do.rules security.mac.do.rules: gid=1000:any $ id -g 1000 $ mdo id mdo: failed to call setuid: Operation not permitted Is that a misunderstanding on my part, am I doing something wrong, or is there a bug? 14.1-STABLE as of e729e750806d3873d5de24cce3b47cc054145985. -- Christian "naddy" Weisgerber naddy@mips.inka.de From nobody Thu Jul 11 05:25:21 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WKNV74JQFz5QHZm for ; Thu, 11 Jul 2024 05:25:35 +0000 (UTC) (envelope-from freebsd@oldach.net) Received: from nuc.oldach.net (hmo.in-vpn.de [IPv6:2001:67c:1407:60::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "hmo.in-vpn.de", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WKNV45ZFwz42fl for ; Thu, 11 Jul 2024 05:25:32 +0000 (UTC) (envelope-from freebsd@oldach.net) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=pass (policy=quarantine) header.from=oldach.net; spf=pass (mx1.freebsd.org: domain of freebsd@oldach.net designates 2001:67c:1407:60::1 as permitted sender) smtp.mailfrom=freebsd@oldach.net Received: from nuc.oldach.net (localhost [127.0.0.1]) by nuc.oldach.net (8.18.1/8.18.1) with ESMTPS id 46B5PMhu066384 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Thu, 11 Jul 2024 07:25:22 +0200 (CEST) (envelope-from freebsd@oldach.net) Received: (from hmo@localhost) by nuc.oldach.net (8.18.1/8.18.1) id 46B5PLFD066383; Thu, 11 Jul 2024 07:25:21 +0200 (CEST) (envelope-from freebsd@oldach.net) Message-Id: <202407110525.46B5PLFD066383@nuc.oldach.net> Subject: Re: mac_do: gid rule fails In-Reply-To: from Christian Weisgerber at "11 Jul 2024 01:22:00" To: naddy@mips.inka.de (Christian Weisgerber) Date: Thu, 11 Jul 2024 07:25:21 +0200 (CEST) Cc: freebsd-stable@freebsd.org From: freebsd@oldach.net (Helge Oldach) X-No-Archive: Yes List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-stable@freebsd.org Sender: owner-freebsd-stable@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Greylist: inspected by milter-greylist-4.6.4 (nuc.oldach.net [0.0.0.0]); Thu, 11 Jul 2024 07:25:22 +0200 (CEST) for IP:127.0.0.1 DOMAIN:localhost HELO:nuc.oldach.net FROM:freebsd@oldach.net RCPT: X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.75 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.95)[-0.955]; DMARC_POLICY_ALLOW(-0.50)[oldach.net,quarantine]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; ASN(0.00)[asn:29670, ipnet:2001:67c:1400::/45, country:DE]; ARC_NA(0.00)[]; TO_DN_NONE(0.00)[]; MIME_TRACE(0.00)[0:+]; FROM_NO_DN(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; RCPT_COUNT_TWO(0.00)[2]; R_DKIM_NA(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MISSING_XM_UA(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org] X-Rspamd-Queue-Id: 4WKNV45ZFwz42fl Christian Weisgerber wrote on Thu, 11 Jul 2024 01:22:00 +0200 (CEST): > However, a GID rule fails: > > $ sysctl security.mac.do.rules > security.mac.do.rules: gid=1000:any > $ id -g > 1000 > $ mdo id > mdo: failed to call setuid: Operation not permitted > > Is that a misunderstanding on my part, am I doing something wrong, > or is there a bug? mdo will execute as root user (by default) but you are not entitled to execute mdo as root. Kind regards Helge From nobody Thu Jul 11 11:47:17 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WKXyg2Bllz5QTcc for ; Thu, 11 Jul 2024 11:47:23 +0000 (UTC) (envelope-from void@f-m.fm) Received: from fhigh1-smtp.messagingengine.com (fhigh1-smtp.messagingengine.com [103.168.172.152]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4WKXyf2w3xz4hS5 for ; Thu, 11 Jul 2024 11:47:22 +0000 (UTC) (envelope-from void@f-m.fm) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm2 header.b=Hbyu6wlW; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=YIPyAZhh; dmarc=pass (policy=none) header.from=f-m.fm; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 103.168.172.152 as permitted sender) smtp.mailfrom=void@f-m.fm Received: from compute7.internal (compute7.nyi.internal [10.202.2.48]) by mailfhigh.nyi.internal (Postfix) with ESMTP id B305E11417EE for ; Thu, 11 Jul 2024 07:47:20 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute7.internal (MEProxy); Thu, 11 Jul 2024 07:47:20 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm2; t=1720698440; x=1720784840; bh=hI1YGC+qZ4 Ly5s2Ogxj+UcQGgwf+Z94lgIxV/OWGEhQ=; b=Hbyu6wlWp8WgjDOI8QKevQgCoT a3mbsewS0Q8RP/8ftw6AKwfHMeKqm+FWkfhv3Mp0fAIO/JRCMTdskS5dam3eCeR4 bUk7t15qxaPvGNaQWP78OUVsTHtoly6pQMy5vZ2Giq+ABytK61/rBm6daakQ/0T1 WrEHKiQVMKlEY8fZNaNw6uoeo9pHaYChbudka8QD08IjrIHwuYMmLHPSwynu5EKT R/9MeScSv+iF/cvHbu/HJ83Be8UDo8p2c4TSHZUeo8cnQhkOeIxGU790koAR5Nnp vyGABTDN8kydhOe7+Z9oC59I8ZJTk8VIkbsrnEojww3824E6TQic8GbKMw/Q== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; t=1720698440; x=1720784840; bh=hI1YGC+qZ4Ly5s2Ogxj+UcQGgwf+ Z94lgIxV/OWGEhQ=; b=YIPyAZhhQ1WKfAXDGJ3/bRaCmqXjMpfKiXytpYG8Qbvw Bu29Vzz4uM4VR9E3PkMuJuCWz9Fk8qCdHK6rwi5sR+CuZL9tG01ydJHWRFuCl+RT /3PJWSRuWPJPPKnPpV/ZOv1SOZ4stE6alCsTK4zw+aKKqV/Qv8yY9A9wo84yoJEH /IQNYMcwjuerNFfKpbZdFfhvzcORlev2Miec5Wro4CWfvADpSnGHdS3my7sqsRwO 2DNk6/a4NUVkLJrCr11meE3GfOwILr2bMlB/dgbFbaGgGA879NS2owqggmGBfEsP DyfTuyRvjHmD/NWr85IWuP4jQxwv7zP0DiX7eB4Gqw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeftddrfeeggdeghecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfhfgggtuggjsehttdertd dttddvnecuhfhrohhmpehvohhiugcuoehvohhiugesfhdqmhdrfhhmqeenucggtffrrght thgvrhhnpeekleduvdelhfeileefgffghfffkedtheellefgudfgvdegkeejjedutdehhe fgueenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehv ohhiugesfhdqmhdrfhhm X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Thu, 11 Jul 2024 07:47:19 -0400 (EDT) Date: Thu, 11 Jul 2024 12:47:17 +0100 From: void To: freebsd-stable@freebsd.org Subject: Re: new error messages after upgrade 14.0-p5 to 14.1-p1 amd64 Message-ID: Mail-Followup-To: freebsd-stable@freebsd.org References: List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-stable@freebsd.org Sender: owner-freebsd-stable@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.89 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; MID_RHS_NOT_FQDN(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; NEURAL_HAM_SHORT(-0.29)[-0.286]; R_SPF_ALLOW(-0.20)[+ip4:103.168.172.128/27]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm2,messagingengine.com:s=fm2]; RCVD_IN_DNSWL_LOW(-0.10)[103.168.172.152:from]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_FROM(0.00)[f-m.fm]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_TRACE(0.00)[0:+]; RCPT_COUNT_ONE(0.00)[1]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; FREEMAIL_ENVFROM(0.00)[f-m.fm]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; TO_DN_NONE(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; ASN(0.00)[asn:209242, ipnet:103.168.172.0/24, country:US]; RCVD_VIA_SMTP_AUTH(0.00)[]; MISSING_XM_UA(0.00)[]; DWL_DNSWL_NONE(0.00)[messagingengine.com:dkim] X-Rspamd-Queue-Id: 4WKXyf2w3xz4hS5 Hi Rick, >> >> The nfs server is 14-stable and the exported dir is zfs with the sharenfs property set. >> The client (14.1-p1) is a VM on a different machine but the same network. >> >> I'll try putting rpcbind_enable=YES in /etc/rc.conf >Probably won't make any difference. > Sorry for hte slow response - this vm is rarely rebooted apart from updates. And so it was rebooted to apply 14.1-p1 => 14.1-p2 via freebsd-update: Updating /var/run/os-release done. Clearing /tmp. Starting syslogd. No core dumps found. Starting rpcbind. NFS access cache time=60 rpc.umntall: 10.0.1.102: MOUNTPROG: RPC: Port mapper failure - RPC: Timed out rpc.umntall: 10.0.1.102: MOUNTPROG: RPC: Port mapper failure - RPC: Timed out Mounting late filesystems:. Security policy loaded: MAC/ntpd (mac_ntpd) Starting ntpd. It's really odd. -- From nobody Thu Jul 11 15:17:40 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WKddX3R8xz5QqG5 for ; Thu, 11 Jul 2024 15:17:52 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-pg1-x52b.google.com (mail-pg1-x52b.google.com [IPv6:2607:f8b0:4864:20::52b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WKddW684fz53Wy for ; Thu, 11 Jul 2024 15:17:51 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b="a/gPyCTy"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of rick.macklem@gmail.com designates 2607:f8b0:4864:20::52b as permitted sender) smtp.mailfrom=rick.macklem@gmail.com Received: by mail-pg1-x52b.google.com with SMTP id 41be03b00d2f7-75cda3719efso668498a12.3 for ; Thu, 11 Jul 2024 08:17:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1720711068; x=1721315868; darn=freebsd.org; h=content-transfer-encoding:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=PeuG8F5M2ZEgFQpY3PZyQ8D3VMb0lySwb79Lyvrso+0=; b=a/gPyCTyMOTZ/wD6TLzMPjQsvPRpY1PQyS18dblHBQkpUNEiwexeVa/yxCMVuz7FDN Rn4CliBA3cNQBkXHTxqZXQPQzoH+Fj0B4Ies4NnlaQezcrzPKhxrA9yLGH5x0SVjT1F5 Rwt0+TFxnEQ0TV1M7xQXOd9CcXmvvScyqsSaQn89QzwFgts6zhvSVn0yyZwqTBljsh/b xZm6d7gYD6ZgIlPbroK9jp6bMLKgfWbJFng978d8EHJOFYo0gGXpopQey4k906m3W6eA ky/YzsSDN1+Ex1f3WBIcROKiOriceXVjHz+9bHhd8wJYQdSyfjF47JypPavgsKkwYovV vWLw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720711068; x=1721315868; h=content-transfer-encoding:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PeuG8F5M2ZEgFQpY3PZyQ8D3VMb0lySwb79Lyvrso+0=; b=EsSnH8X9B8GQBEE3TbGqFmPUeOssS9YYDc3vqHlR1FWD0bBXtcyryug3DUjjZAltUG iFjsP8MQLo5fZGvyQWPzRbo/vnUq4ALfVIJ2mYTBV9osL1jC2Wr1YRgoXimP8kUvHh9V 5nKFsez+jcIsQpMv3z+7VIhq8hwJW4uJ7Lh1QxJn4X6sVpiPOTaduFelGKJG2XXRZwbM R+VGovkrBbWVMof1bejMB49Y9Vlh6fPIe5wuf4afJW2hdwb1SyOHXw7KHydB8JnFC2RK Vbj35ldoKgWOrcjWKcmww39F86e2r4Qx+pSZtP2kZaeQZ1qh0+3rqK+26UUFQTIi7r4L 7Xgw== X-Gm-Message-State: AOJu0Yxo+x4vOVN/eAHXkJS8p41VV+xgMGo4SxZlnWprTXfOfRlPwuQZ l2y/1R1uAe8vMvypbcxRLwoNutQU/rGdxlkq5JaxJFUgOtM9nLXddU5LDZltUE6Cde51HOzhr77 0fi6Dxqtp7cOB7g7Njm3RGotpAbz9g04= X-Google-Smtp-Source: AGHT+IHPoDTudctyNYGqKQRrTWJE80XMngIvVL+m/tjkzN0SEkohd/yAVk4Ao+ehPCiZS8+PhXgqBaL1YERMYlIv5rE= X-Received: by 2002:a05:6a20:1589:b0:1c2:8eb7:19cd with SMTP id adf61e73a8af0-1c2984c871emr10329289637.42.1720711068089; Thu, 11 Jul 2024 08:17:48 -0700 (PDT) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-stable@freebsd.org Sender: owner-freebsd-stable@FreeBSD.org MIME-Version: 1.0 References: In-Reply-To: From: Rick Macklem Date: Thu, 11 Jul 2024 08:17:40 -0700 Message-ID: Subject: Re: new error messages after upgrade 14.0-p5 to 14.1-p1 amd64 To: freebsd-stable@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.99 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.991]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; FREEMAIL_ENVFROM(0.00)[gmail.com]; TAGGED_FROM(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; FROM_HAS_DN(0.00)[]; MISSING_XM_UA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::52b:from]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; TO_DN_NONE(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-Rspamd-Queue-Id: 4WKddW684fz53Wy On Thu, Jul 11, 2024 at 4:47=E2=80=AFAM void wrote: > > Hi Rick, > > > >> > >> The nfs server is 14-stable and the exported dir is zfs with the share= nfs property set. > >> The client (14.1-p1) is a VM on a different machine but the same netwo= rk. > >> > >> I'll try putting rpcbind_enable=3DYES in /etc/rc.conf > >Probably won't make any difference. > > > > Sorry for hte slow response - this vm is rarely rebooted apart from > updates. And so it was rebooted to apply 14.1-p1 =3D> 14.1-p2 via freebsd= -update: > > Updating /var/run/os-release done. > Clearing /tmp. > Starting syslogd. > No core dumps found. > Starting rpcbind. > NFS access cache time=3D60 > rpc.umntall: 10.0.1.102: MOUNTPROG: RPC: Port mapper failure - RPC: Timed= out > rpc.umntall: 10.0.1.102: MOUNTPROG: RPC: Port mapper failure - RPC: Timed= out > Mounting late filesystems:. > Security policy loaded: MAC/ntpd (mac_ntpd) > Starting ntpd. > > It's really odd. Did you look at the bug I pointed at in my previous email? https://lists.freebsd.org/archives/freebsd-current/2024-June/006075.html It basically says that 14.1 is buggy if you do not specify a netmask or wid= th for ifconfig for IP4 addresses. The bug is fixed in stable/14, but exists in 14.1. So, make sure that there is a netmask or width on the IP4 address for all y= our ifconfig lines (probably in /etc/rc.conf). rick > -- > From nobody Fri Jul 12 13:13:42 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WLBrc0nJdz5QNLm for ; Fri, 12 Jul 2024 13:14:24 +0000 (UTC) (envelope-from void@f-m.fm) Received: from fout4-smtp.messagingengine.com (fout4-smtp.messagingengine.com [103.168.172.147]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4WLBrb29mbz46dX for ; Fri, 12 Jul 2024 13:14:23 +0000 (UTC) (envelope-from void@f-m.fm) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm2 header.b="pp/C3W2d"; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=brE9Vs2g; dmarc=pass (policy=none) header.from=f-m.fm; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 103.168.172.147 as permitted sender) smtp.mailfrom=void@f-m.fm Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailfout.nyi.internal (Postfix) with ESMTP id A5AEE1388187 for ; Fri, 12 Jul 2024 09:14:21 -0400 (EDT) Received: from imap46 ([10.202.2.96]) by compute6.internal (MEProxy); Fri, 12 Jul 2024 09:14:21 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm2; t=1720790061; x=1720876461; bh=e6HYESfDc1 gwYfjKhxnrXqlro5WDpJ0qb/GJczlEBjY=; b=pp/C3W2d5gczwjvOS7RtwSaL4s jHDOWIlqceh9wYU/wNqnazqLGMMVi0HNaJyRvNz5ILnOqsLE3oN5MgeAJA3TpDBb uwpLV7xGvy/GmXPb6LIZE4OUtbj+HKqVZEGpKjxfbB+MHDT8dNwUNfMpuJehZDT9 jLV97eFiPWbO2kjPzYVumNkcUPP1c0fqAxm+MNjPijJqLvuhUiJQCADVKw9dJWCo TUpd6krSfutrwm0GODtRwvkXmRMMJsGgsd5714km3mnkvWq7yMKdwCOg5Nmzj9Qo dDBiprb5oU1fmbPVkj8nf6a0lULOrvxvK1UyMinJlPV09I5aRD6MGQXyrJ0Q== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; t=1720790061; x=1720876461; bh=e6HYESfDc1gwYfjKhxnrXqlro5WD pJ0qb/GJczlEBjY=; b=brE9Vs2gU+PP8bJw5LsWmP8d/XkfgNRp4k5hVxAbH7oe zhAvylHfAoSexLdKzPZLG/ihNQ7NDE+iBqP0H+vqJ1l8nJ6iyQmvLD9HlXEWXIAp LV5VqbR5gqjDcT+Gvp8C+oCB0gG1gMtDTUkIMHwIyNlAydpJy2VpLD/x7A8lkNmv YqjFGJyklOMGTCWCSBhHcgK3OoWHgFfYH3HR+yn7h7Aderpdk31g0MxlB+PBAlTl vSVLxVw6cvWaguIOgdUesjA8Zw/C+ylkHfJ127G0jf4i9ogcSn9d3WqZRNg2o3YL /JiU0+b0DOLxjdoGWm4Suid5R21ZoE548gDG/pFOJA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeftddrfeeigdehiecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecunecujfgurhepofgfggfkjghffffhvffutgesthdtre dtreertdenucfhrhhomhepvhhoihguuceovhhoihgusehfqdhmrdhfmheqnecuggftrfgr thhtvghrnhepieeiheeutdduieekudeikedvffeggfefvedvfeehffeguddthedttedvhf evkefhnecuffhomhgrihhnpehfrhgvvggsshgurdhorhhgnecuvehluhhsthgvrhfuihii vgeptdenucfrrghrrghmpehmrghilhhfrhhomhepvhhoihgusehfqdhmrdhfmh X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mailuser.nyi.internal (Postfix, from userid 501) id 71F742A20085; Fri, 12 Jul 2024 09:14:20 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.11.0-alpha0-568-g843fbadbe-fm-20240701.003-g843fbadb List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-stable@freebsd.org Sender: owner-freebsd-stable@FreeBSD.org MIME-Version: 1.0 Message-Id: In-Reply-To: References: Date: Fri, 12 Jul 2024 14:13:42 +0100 From: void To: stable@freebsd.org Subject: Re: new error messages after upgrade 14.0-p5 to 14.1-p1 amd64 Content-Type: text/plain X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.64 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.55)[-0.551]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm2,messagingengine.com:s=fm2]; R_SPF_ALLOW(-0.20)[+ip4:103.168.172.128/27]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[103.168.172.147:from]; XM_UA_NO_VERSION(0.01)[]; RCPT_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:209242, ipnet:103.168.172.0/24, country:US]; MIME_TRACE(0.00)[0:+]; DWL_DNSWL_NONE(0.00)[messagingengine.com:dkim]; FREEMAIL_FROM(0.00)[f-m.fm]; FREEMAIL_ENVFROM(0.00)[f-m.fm]; ARC_NA(0.00)[]; MLMMJ_DEST(0.00)[stable@freebsd.org]; RCVD_COUNT_THREE(0.00)[3]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_TLS_LAST(0.00)[]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[stable@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+] X-Rspamd-Queue-Id: 4WLBrb29mbz46dX Hi Rick, There's no freebsd machine I've ever had (any version) that has had the ip specified without the netmask keyword in /etc/rc.conf. Until https://lists.freebsd.org/archives/freebsd-current/2024-June/006075.html I didn't even know it was possible to configure networking and have it work properly without a netmask. So, I'm at a loss. No harm though, it still works; it just emits the error twice. Maybe the error will go away in a later patchlevel. -- From nobody Sun Jul 14 02:02:17 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WM7rT6GZtz5Rb6g for ; Sun, 14 Jul 2024 02:02:33 +0000 (UTC) (envelope-from wollman@hergotha.csail.mit.edu) Received: from hergotha.csail.mit.edu (tunnel82308-pt.tunnel.tserv4.nyc4.ipv6.he.net [IPv6:2001:470:1f06:ccb::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "garrett.wollman.name", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WM7rS5lSVz4hxP for ; Sun, 14 Jul 2024 02:02:32 +0000 (UTC) (envelope-from wollman@hergotha.csail.mit.edu) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bimajority.org (policy=none); spf=pass (mx1.freebsd.org: domain of wollman@hergotha.csail.mit.edu designates 2001:470:1f06:ccb::2 as permitted sender) smtp.mailfrom=wollman@hergotha.csail.mit.edu Received: from hergotha.csail.mit.edu (localhost [127.0.0.1]) by hergotha.csail.mit.edu (8.18.1/8.18.1) with ESMTPS id 46E22JVs007795 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Sat, 13 Jul 2024 22:02:22 -0400 (EDT) (envelope-from wollman@hergotha.csail.mit.edu) Received: (from wollman@localhost) by hergotha.csail.mit.edu (8.18.1/8.18.1/Submit) id 46E22HP7007794; Sat, 13 Jul 2024 22:02:17 -0400 (EDT) (envelope-from wollman) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-stable@freebsd.org Sender: owner-freebsd-stable@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <26259.12713.114036.564205@hergotha.csail.mit.edu> Date: Sat, 13 Jul 2024 22:02:17 -0400 From: Garrett Wollman To: freebsd-stable@freebsd.org Subject: Possible bug in zfs send or pipe implementation? X-Mailer: VM 8.2.0b under 29.2 (amd64-portbld-freebsd13.2) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.4 (hergotha.csail.mit.edu [0.0.0.0]); Sat, 13 Jul 2024 22:02:22 -0400 (EDT) X-Spam-Status: No, score=-0.8 required=5.0 tests=ALL_TRUSTED, HEADER_FROM_DIFFERENT_DOMAINS,T_SCC_BODY_TEXT_LINE autolearn=disabled version=4.0.0 X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on hergotha.csail.mit.edu X-Spamd-Bar: - X-Spamd-Result: default: False [-1.90 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-0.999]; NEURAL_HAM_SHORT(-1.00)[-0.997]; FORGED_SENDER(0.30)[wollman@bimajority.org,wollman@hergotha.csail.mit.edu]; R_SPF_ALLOW(-0.20)[+ip6:2001:470:1f06:ccb::2]; MIME_GOOD(-0.10)[text/plain]; DMARC_POLICY_SOFTFAIL(0.10)[bimajority.org : SPF not aligned (relaxed), No valid DKIM,none]; RCPT_COUNT_ONE(0.00)[1]; TO_DN_NONE(0.00)[]; FREEFALL_USER(0.00)[wollman]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; MIME_TRACE(0.00)[0:+]; FROM_HAS_DN(0.00)[]; R_DKIM_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; FROM_NEQ_ENVFROM(0.00)[wollman@bimajority.org,wollman@hergotha.csail.mit.edu]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[] X-Rspamd-Queue-Id: 4WM7rS5lSVz4hxP I'm migrating an old file server to new hardware using syncoid. Every so often, the `zfs send` process gets stuck with the following kstacks: 7960 108449 zfs - mi_switch sleepq_catch_signals sleepq_wait_sig _sleep pipe_write zfs_file_write_impl zfs_file_write dump_record dmu_dump_write do_dump dmu_send_impl dmu_send_obj zfs_ioc_send zfsdev_ioctl_common zfsdev_ioctl devfs_ioctl vn_ioctl devfs_ioctl_f 7960 126072 zfs send_traverse_threa mi_switch sleepq_catch_signals sleepq_wait_sig _cv_wait_sig bqueue_enqueue_impl send_cb traverse_visitbp traverse_visitbp traverse_visitbp traverse_dnode traverse_visitbp traverse_visitbp traverse_visitbp traverse_visitbp traverse_visitbp traverse_visitbp traverse_dnode traverse_visitbp 7960 126074 zfs send_merge_thread mi_switch sleepq_catch_signals sleepq_wait_sig _cv_wait_sig bqueue_enqueue_impl send_merge_thread fork_exit fork_trampoline 7960 126075 zfs send_reader_thread mi_switch sleepq_catch_signals sleepq_wait_sig _cv_wait_sig bqueue_enqueue_impl send_reader_thread fork_exit fork_trampoline Near as I can tell, the thread first thread is trying to write serialized data data to the output pipe and is blocked. The other threads are stuck because the write process isn't making progress. The process reading from the pipe (which is just a progress meter) is sitting in select() waiting for the pipe to become ready, so either zfs_file_write() is doing something wrong, or the pipe implementation has lost a selwakeup() somewhere. (Or, possibly but unlikely, the progress meter has lost the read end of the pipe from its read fd_set.) Unfortunately, neither fstat nor procstat print any useful information about the state of the pipe, so I can only try to deduce what's going on from the observable behavior. -GAWollman From nobody Sun Jul 14 02:42:32 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WM8ks466Kz5Pwdw for ; Sun, 14 Jul 2024 02:42:45 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-qv1-xf35.google.com (mail-qv1-xf35.google.com [IPv6:2607:f8b0:4864:20::f35]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WM8ks08phz4myx for ; Sun, 14 Jul 2024 02:42:45 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-qv1-xf35.google.com with SMTP id 6a1803df08f44-6b760dc7e08so6099066d6.3 for ; Sat, 13 Jul 2024 19:42:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1720924963; x=1721529763; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=IONACmGFLI6llEjTq8f6ypuTAj82b9ceNJ1xngFFSdY=; b=PE5U/VwD/wlcHBoLORiqqD7CEygqbLdZPm6CwFZzSHQb/yeJ6Df9P57e6C6J9KgSPq kJiC9nH9bacgna/y7waIC4aEB/D5O5f0beZ7gj6oTN8qrSBW3tYKWYQstYs7eRn6ZC7T bJ0SlPXkx/c/z47erDmOqG+waKqcsCUIM/GbAtHLoLGgMqzbH8+G81sgkjEuLzWg8JQu rG78ZuO0ecaXbBBCPq8ILujk+SRy4iQphbfoiHDlbK9mcc/CnhybEiWc2gYlCCaFUhjW nQXCqMyGpjAxa6giQ8y31/Lm8PnxFfjOpUh+vUQt9AOc4EhfLDtUYvax3oR1odLv81IK rIHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720924963; x=1721529763; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=IONACmGFLI6llEjTq8f6ypuTAj82b9ceNJ1xngFFSdY=; b=YNHsmbXm8X1XA3KsdgsjJf9ql8vExvNo7nc8MTHTsJG3sY3gNvCYQo7HvWfm549A0D BwPHdGwx7Dc2p8ETrXvK0WOHDpDcnQSn1vrSK+VDAfolCCpq+cW8KATI6PIVXIcYVs3G 0imILOczlb2HL0cWDvkdOYM3bfHSqE9jksNR8UDOw5gOJIloXC10s+bZ0BrYwdTkkW8H JD6sVr+TB/2YY3XM8GWcCdSv5Il8+3fms8NjU/qKDE3hpLnNcA2H0LS+B9Z34ZLZU8fC 8TctGbwUKLYYTa6gjGcx+Y5G6m4a/NLz8GVSU6Dd/LdvRUiT7eGOapQuHFpHyH4A6TM/ wrKQ== X-Gm-Message-State: AOJu0Yw7rFWfza+xZ2w305gJIvjffcGncZLg9bPmCm+ULJD2z06lTECb f5g7exxzhcp1/K8K2qPx0+cSmIVRWlikbqBiko8zRTWQv4gnIJsQYnNTOE+sv2lzgguCDCdDhIE QB7jiR0uiYxckn795crBYCM9Jf4eg X-Google-Smtp-Source: AGHT+IEWhmWSnhLJIw09iOLUYZCBSNvqL2PWQ1sFic3nY5q0GEurXR0rjnU5/jC0nOjA8zxK/73PhQF+ctszAtryjIw= X-Received: by 2002:a05:6214:19c8:b0:6b4:f761:f0b8 with SMTP id 6a1803df08f44-6b61bc7f095mr239621086d6.8.1720924963380; Sat, 13 Jul 2024 19:42:43 -0700 (PDT) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-stable@freebsd.org Sender: owner-freebsd-stable@FreeBSD.org MIME-Version: 1.0 References: <26259.12713.114036.564205@hergotha.csail.mit.edu> In-Reply-To: <26259.12713.114036.564205@hergotha.csail.mit.edu> From: Rick Macklem Date: Sat, 13 Jul 2024 19:42:32 -0700 Message-ID: Subject: Re: Possible bug in zfs send or pipe implementation? To: Garrett Wollman Cc: freebsd-stable@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4WM8ks08phz4myx On Sat, Jul 13, 2024 at 7:02=E2=80=AFPM Garrett Wollman wrote: > > I'm migrating an old file server to new hardware using syncoid. Every > so often, the `zfs send` process gets stuck with the following > kstacks: > > 7960 108449 zfs - mi_switch sleepq_cat= ch_signals sleepq_wait_sig _sleep pipe_write zfs_file_write_impl zfs_file_w= rite dump_record dmu_dump_write do_dump dmu_send_impl dmu_send_obj zfs_ioc_= send zfsdev_ioctl_common zfsdev_ioctl devfs_ioctl vn_ioctl devfs_ioctl_f > 7960 126072 zfs send_traverse_threa mi_switch sleepq_cat= ch_signals sleepq_wait_sig _cv_wait_sig bqueue_enqueue_impl send_cb travers= e_visitbp traverse_visitbp traverse_visitbp traverse_dnode traverse_visitbp= traverse_visitbp traverse_visitbp traverse_visitbp traverse_visitbp traver= se_visitbp traverse_dnode traverse_visitbp > 7960 126074 zfs send_merge_thread mi_switch sleepq_cat= ch_signals sleepq_wait_sig _cv_wait_sig bqueue_enqueue_impl send_merge_thre= ad fork_exit fork_trampoline > 7960 126075 zfs send_reader_thread mi_switch sleepq_cat= ch_signals sleepq_wait_sig _cv_wait_sig bqueue_enqueue_impl send_reader_thr= ead fork_exit fork_trampoline > > Near as I can tell, the thread first thread is trying to write > serialized data data to the output pipe and is blocked. The other > threads are stuck because the write process isn't making progress. # ps axHl should show you what wchan's the processes are waiting on and that might give you a clue w.r.t. what is happening? If is easy to build a kernel from sources and boot that, you could try defi= ning PIPE_NODIRECT in sys/kern/sys_pipe.c and see if that avoids the hangs? rick > > The process reading from the pipe (which is just a progress meter) is > sitting in select() waiting for the pipe to become ready, so either > zfs_file_write() is doing something wrong, or the pipe implementation > has lost a selwakeup() somewhere. (Or, possibly but unlikely, the > progress meter has lost the read end of the pipe from its read > fd_set.) Unfortunately, neither fstat nor procstat print any useful > information about the state of the pipe, so I can only try to deduce > what's going on from the observable behavior. > > -GAWollman > From nobody Sun Jul 14 03:19:50 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WM9Ym5lG6z5Q1B2 for ; Sun, 14 Jul 2024 03:19:56 +0000 (UTC) (envelope-from wollman@hergotha.csail.mit.edu) Received: from hergotha.csail.mit.edu (tunnel82308-pt.tunnel.tserv4.nyc4.ipv6.he.net [IPv6:2001:470:1f06:ccb::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "garrett.wollman.name", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WM9Yl63dqz4ryp for ; Sun, 14 Jul 2024 03:19:55 +0000 (UTC) (envelope-from wollman@hergotha.csail.mit.edu) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bimajority.org (policy=none); spf=pass (mx1.freebsd.org: domain of wollman@hergotha.csail.mit.edu designates 2001:470:1f06:ccb::2 as permitted sender) smtp.mailfrom=wollman@hergotha.csail.mit.edu Received: from hergotha.csail.mit.edu (localhost [127.0.0.1]) by hergotha.csail.mit.edu (8.18.1/8.18.1) with ESMTPS id 46E3JpAi008417 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sat, 13 Jul 2024 23:19:53 -0400 (EDT) (envelope-from wollman@hergotha.csail.mit.edu) Received: (from wollman@localhost) by hergotha.csail.mit.edu (8.18.1/8.18.1/Submit) id 46E3Joxo008416; Sat, 13 Jul 2024 23:19:50 -0400 (EDT) (envelope-from wollman) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-stable@freebsd.org Sender: owner-freebsd-stable@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <26259.17366.276955.824313@hergotha.csail.mit.edu> Date: Sat, 13 Jul 2024 23:19:50 -0400 From: Garrett Wollman To: Rick Macklem Cc: freebsd-stable@freebsd.org Subject: Re: Possible bug in zfs send or pipe implementation? In-Reply-To: References: <26259.12713.114036.564205@hergotha.csail.mit.edu> X-Mailer: VM 8.2.0b under 29.2 (amd64-portbld-freebsd13.2) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.4 (hergotha.csail.mit.edu [0.0.0.0]); Sat, 13 Jul 2024 23:19:53 -0400 (EDT) X-Spam-Status: No, score=-1.8 required=5.0 tests=ALL_TRUSTED, HEADER_FROM_DIFFERENT_DOMAINS,NICE_REPLY_A,T_SCC_BODY_TEXT_LINE autolearn=disabled version=4.0.0 X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on hergotha.csail.mit.edu X-Spamd-Bar: - X-Spamd-Result: default: False [-1.89 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; NEURAL_HAM_SHORT(-0.99)[-0.986]; FORGED_SENDER(0.30)[wollman@bimajority.org,wollman@hergotha.csail.mit.edu]; R_SPF_ALLOW(-0.20)[+ip6:2001:470:1f06:ccb::2]; MIME_GOOD(-0.10)[text/plain]; DMARC_POLICY_SOFTFAIL(0.10)[bimajority.org : SPF not aligned (relaxed), No valid DKIM,none]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[wollman]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; TO_DN_SOME(0.00)[]; MIME_TRACE(0.00)[0:+]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_TO(0.00)[gmail.com]; R_DKIM_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_NEQ_ENVFROM(0.00)[wollman@bimajority.org,wollman@hergotha.csail.mit.edu]; TAGGED_RCPT(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[] X-Rspamd-Queue-Id: 4WM9Yl63dqz4ryp < said: > # ps axHl > should show you what wchan's the processes are waiting on and that might > give you a clue w.r.t. what is happening? zfs is waiting to write into the pipe and pv (the progress meter) is waiting in select. > If is easy to build a kernel from sources and boot that, you could try defining > PIPE_NODIRECT in sys/kern/sys_pipe.c and see if that avoids the hangs? It's easy to build a kernel from sources, but not easy to reboot the server -- it's being retired shortly, and because of time constraints I need to get it drained before the next scheduled outage. -GAWollman From nobody Sun Jul 14 03:50:55 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WMBFn1bDcz5Q4LP for ; Sun, 14 Jul 2024 03:51:09 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-pg1-x535.google.com (mail-pg1-x535.google.com [IPv6:2607:f8b0:4864:20::535]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WMBFm74Tyz3xFr for ; Sun, 14 Jul 2024 03:51:08 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-pg1-x535.google.com with SMTP id 41be03b00d2f7-656d8b346d2so1945774a12.2 for ; Sat, 13 Jul 2024 20:51:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1720929067; x=1721533867; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=GMALvBdxy4CDLqz+e70twUUwjinKEpbs2V5jQybaiJk=; b=dhUfLm47RfJATvXhweJhI6lf0y9UWR6q7JHxDcHlF72tGjW1G3y9DqcmE6VOiMNNGw M1IqIFUpGy0ew5k5VKSS5aDdU0oMK0S0mnfPh7r0VpnXcrXHL/Bel1YtJoxEsDHnM7hA zk5PQbgzaWE4OmfEdHc2mztWMrr5AtcaUHuBb3MfyRXYtYS/t5J0obHhDA5yBKgiTS8i 1Qj0r7KU/z1S8nA/GXanr/GgJ/tYMSSGxy0FcNEMYYV2xNcgpfhBdPdsKVG3P8xpkeox d4FpVz1Hb2HGuhDKvcnrnmmLsr80xRd5LMFwULj5caRi1+rQT3xdv6CF8KfTGsCTUaTt Gf4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720929067; x=1721533867; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GMALvBdxy4CDLqz+e70twUUwjinKEpbs2V5jQybaiJk=; b=qIGTUYE7l930DytcQnfGmYSkLptr9qG/BvpOGox6gpjd3GYvRcLLfXoQ9n/BL4xjo1 ZC9jYBwHDPMZxEKGbRueSMroWlviXZyRsDowt961I3bIeVeq+uG4gBeAJ5sbrnLgiBwg NmXHsGKXbVv7ZnqEylqjZAcWrfcCQpznijnp5aZiWwcbWpeiZ/wmoIak3JwxC9MN1Rrq zSUeCVvHkh2IMvNN+b5crPBl3EmZDrKQVhZPpUF3ziwPvfR+ZGTNNlxiJ72P+2Tza/1N uNQEn9tW0hLTXMtfYPmOeg25mWlxjo8NJJQsSieJrV6l2tOO8BDvfldUb3b6TgR2UIqk CHBw== X-Gm-Message-State: AOJu0YzZz1ISxcOYFwr2eHGhUqKeCibVwNCn+Iz93PJK9w105127kTzD 01swCjd3I5xY5oYl8iOxf+5b6WwdiqTHtVgXvMea9gP1022mgVU0WyGSTsFcsSGeebK7iSVciiH 8l0lZPrPL/j/RShv34pDzTqjTcbj0 X-Google-Smtp-Source: AGHT+IHWTZiSMRPYY3hwKrHP+tTGAF6p5oc9YE8L8WROUPRZQG2BzoOdOfouA5+7eccQXyrmQqFUeVd5LhANIyF5R08= X-Received: by 2002:a05:6a20:2588:b0:1c0:f193:8387 with SMTP id adf61e73a8af0-1c2981fbf26mr18806427637.11.1720929066876; Sat, 13 Jul 2024 20:51:06 -0700 (PDT) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-stable@freebsd.org Sender: owner-freebsd-stable@FreeBSD.org MIME-Version: 1.0 References: <26259.12713.114036.564205@hergotha.csail.mit.edu> <26259.17366.276955.824313@hergotha.csail.mit.edu> In-Reply-To: <26259.17366.276955.824313@hergotha.csail.mit.edu> From: Rick Macklem Date: Sat, 13 Jul 2024 20:50:55 -0700 Message-ID: Subject: Re: Possible bug in zfs send or pipe implementation? To: Garrett Wollman Cc: freebsd-stable@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4WMBFm74Tyz3xFr On Sat, Jul 13, 2024 at 8:19=E2=80=AFPM Garrett Wollman wrote: > > < said: > > > # ps axHl > > should show you what wchan's the processes are waiting on and that migh= t > > give you a clue w.r.t. what is happening? > > zfs is waiting to write into the pipe and pv (the progress meter) is > waiting in select. Just to clarify it, are you saying zfs is sleeping on "pipewr"? (There is also a msleep() for "pipbww" in pipe_write().) rick > > > If is easy to build a kernel from sources and boot that, you could try = defining > > PIPE_NODIRECT in sys/kern/sys_pipe.c and see if that avoids the hangs? > > It's easy to build a kernel from sources, but not easy to reboot the > server -- it's being retired shortly, and because of time constraints > I need to get it drained before the next scheduled outage. > > -GAWollman > From nobody Sun Jul 14 03:58:07 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WMBQ52r7Jz5Q59B for ; Sun, 14 Jul 2024 03:58:21 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-ot1-x334.google.com (mail-ot1-x334.google.com [IPv6:2607:f8b0:4864:20::334]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WMBQ371blz40N8 for ; Sun, 14 Jul 2024 03:58:19 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=LkvK89vT; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of rick.macklem@gmail.com designates 2607:f8b0:4864:20::334 as permitted sender) smtp.mailfrom=rick.macklem@gmail.com Received: by mail-ot1-x334.google.com with SMTP id 46e09a7af769-7036e383089so1993094a34.2 for ; Sat, 13 Jul 2024 20:58:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1720929499; x=1721534299; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=vABtcgD+ZpCc1GgO9wCHzsJu4c+LLmYLG1m/fF/DBfo=; b=LkvK89vTX3jG3wetkTp4iEBEAdheBq5ZpP0qMpn+KpVmCzLy4JieV3sdhWBa+4Qv+j PpClrlKRnsTmRKfF/M5lQAyr3S56Niu1s4GNl2J2r1s3E6Q7kcEXagnvaUv/zIvMfzXB 1evmss7gGcEyRyP+FGA+NW/VgYlcRLph4Gqv1co6pnSR6L/cElSSv0BywcDFAebORagf TLG83yGDhvtV6fv51wlJL3lI+KUQEN62x1RgW2eD49tOpTDV/5A8VPRi2uGIzFrFU9iW ks30Nis3nIohXmrxphF1yK6IYImKO7or9y7xMwfOi0fqRlxfzey/YPls6noENEm/5IQD 3Xdg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720929499; x=1721534299; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vABtcgD+ZpCc1GgO9wCHzsJu4c+LLmYLG1m/fF/DBfo=; b=tU+HDdPFl8n4cqkecomDVhbGTzmmvDJMDqsvcC/4XkNp3+zNn//YfYlegibWuQwUex V3CFtuBrxHXcAf+7rWVqMfUBESpHet8yKJkvnu3SwB3WxqZ1XyGI4Z5NE747yKSrfN7Y MNn4+8PQDou88Ci3n1mqX+7UQ64UG+VjxxLIRnBMdzwDCEOv2zOOgaCdIWDNWBE8nMQ3 a/RtBKa99Lkg21YNSCT6dyL5LFwJBqPo2x//4VWiFsZOGLfRYBiodhmxf379C7ysVueU 9cD9WSDwx6dd9Wh8o2JxGvUdL/Jj9q3ezOGeXOuB4tIl6HeEk1MDElC10UR1ocWFUiSC LcEw== X-Gm-Message-State: AOJu0YyKF5WLIOjh/L1+4L8iXMmLQxHiwQfvT59PRCM9gcMGZEzSF1B7 7l5EW4DpeH5w0q+YjEVTAA2nauRvLNepHWKCwkIfVJYxn2d+X3NxRbh2RtFEiGAmcdch2LoSJe7 KxJWB9+bFYcG+4JQdkmd78pBv/Yuc X-Google-Smtp-Source: AGHT+IFLmBUh/bm9yrfPm62d8psSe2HjoxMc4JKumdM/fG4JcKPhilEceRsI7NTsjqkewQI7SSkcpnu1yhMZqYwBKhs= X-Received: by 2002:a9d:7b52:0:b0:703:6578:8e2b with SMTP id 46e09a7af769-70375b40067mr18016392a34.23.1720929498624; Sat, 13 Jul 2024 20:58:18 -0700 (PDT) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-stable@freebsd.org Sender: owner-freebsd-stable@FreeBSD.org MIME-Version: 1.0 References: <26259.12713.114036.564205@hergotha.csail.mit.edu> <26259.17366.276955.824313@hergotha.csail.mit.edu> In-Reply-To: From: Rick Macklem Date: Sat, 13 Jul 2024 20:58:07 -0700 Message-ID: Subject: Re: Possible bug in zfs send or pipe implementation? To: Garrett Wollman Cc: freebsd-stable@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.99 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.994]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; ARC_NA(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; TAGGED_FROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FROM_HAS_DN(0.00)[]; MISSING_XM_UA(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MID_RHS_MATCH_FROMTLD(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_COUNT_ONE(0.00)[1]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::334:from] X-Rspamd-Queue-Id: 4WMBQ371blz40N8 On Sat, Jul 13, 2024 at 8:50=E2=80=AFPM Rick Macklem wrote: > > On Sat, Jul 13, 2024 at 8:19=E2=80=AFPM Garrett Wollman wrote: > > > > < said: > > > > > # ps axHl > > > should show you what wchan's the processes are waiting on and that mi= ght > > > give you a clue w.r.t. what is happening? > > > > zfs is waiting to write into the pipe and pv (the progress meter) is > > waiting in select. > Just to clarify it, are you saying zfs is sleeping on "pipewr"? If I am reading the code correctly, if it sleeping on "pipewr", it is out of space and that is controlled via: kern.ipc.maxpipekva and you can see what it is using by looking at kern.ipc.pipekva (Unfortunately, I don't think you can change kern.ipc.maxpipekva on the fly= . It looks like it is a loader tunable, so you'd need to reboot to make it larger.) Anyhow, you can take a look at the sysctls. They might help? There is quite a detailed comment in sys/kern/sys_pipe.c related to this. rick > (There is also a msleep() for "pipbww" in pipe_write().) > > rick > > > > > > If is easy to build a kernel from sources and boot that, you could tr= y defining > > > PIPE_NODIRECT in sys/kern/sys_pipe.c and see if that avoids the hangs= ? > > > > It's easy to build a kernel from sources, but not easy to reboot the > > server -- it's being retired shortly, and because of time constraints > > I need to get it drained before the next scheduled outage. > > > > -GAWollman > > From nobody Sun Jul 14 10:13:17 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WMLlJ4pWzz5QlFZ for ; Sun, 14 Jul 2024 10:13:48 +0000 (UTC) (envelope-from Alexander@Leidinger.net) Received: from mailgate.Leidinger.net (bastille.leidinger.net [89.238.82.207]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature ECDSA (P-256) client-digest SHA256) (Client CN "mailgate.leidinger.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WMLlJ1Zvtz4bs4 for ; Sun, 14 Jul 2024 10:13:48 +0000 (UTC) (envelope-from Alexander@Leidinger.net) Authentication-Results: mx1.freebsd.org; none List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-stable@freebsd.org Sender: owner-freebsd-stable@FreeBSD.org MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=leidinger.net; s=outgoing-alex; t=1720952015; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=0War64JkLCQqXcTJLCxMp4kYEeqDBa4E2nmr/TEbvlo=; b=LjylkgdR96oOw2IG2+ddgD1xbn/+ryaDVLpVG0wW+lU845ms/nva5nppJEyD5Agb+/UEO+ 2ZeuB3ZegRjufxpQFR0qxyP/a110AczKFLpFAuBdcbzDx4vOD+JMQKOx5sjIXituagiHiw PAd6T0+eMTjW+XCUzA7C4fGk8a8Eg0aZKTOcA/3ycZwxbsVeLo6Fr/Cp0yI9asJG/xuGWt y+KENQ7vpL6pb2VZYnsieTfR3icN870d6j3mHodtuQttTi6n6VNLkofTPr28ZzCMd2YH2u JYKnMPwR9k7f0htGP5SPV2n+z5z9g0RfY0CBP/bfmpBedG5h4rX8Q0ahSgkV+w== Date: Sun, 14 Jul 2024 12:13:17 +0200 From: Alexander Leidinger To: Garrett Wollman Cc: Rick Macklem , freebsd-stable@freebsd.org Subject: Re: Possible bug in zfs send or pipe implementation? In-Reply-To: <26259.17366.276955.824313@hergotha.csail.mit.edu> References: <26259.12713.114036.564205@hergotha.csail.mit.edu> <26259.17366.276955.824313@hergotha.csail.mit.edu> Message-ID: <2a8e7892280dbc5be1d34a53a5f36587@Leidinger.net> Organization: No organization, this is a private message. Content-Type: multipart/signed; protocol="application/pgp-signature"; boundary="=_26d7ff65f42eda061c5d904c2ac65c3b"; micalg=pgp-sha256 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_RCPT(0.00)[]; ASN(0.00)[asn:34240, ipnet:89.238.64.0/18, country:DE] X-Rspamd-Queue-Id: 4WMLlJ1Zvtz4bs4 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --=_26d7ff65f42eda061c5d904c2ac65c3b Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII; format=flowed Am 2024-07-14 05:19, schrieb Garrett Wollman: > < said: > >> # ps axHl >> should show you what wchan's the processes are waiting on and that >> might >> give you a clue w.r.t. what is happening? > > zfs is waiting to write into the pipe and pv (the progress meter) is > waiting in select. > >> If is easy to build a kernel from sources and boot that, you could try >> defining >> PIPE_NODIRECT in sys/kern/sys_pipe.c and see if that avoids the hangs? > > It's easy to build a kernel from sources, but not easy to reboot the > server -- it's being retired shortly, and because of time constraints > I need to get it drained before the next scheduled outage. As you need to stop the process anyway: I always use mbuffer with "-m 50M" (50M buffer, and with "-v 2" you will also get some progress info) between the sender and the receiver. This may or may not change the behavior your see (given you have some time constraint which prevents certain investigative stuff and gives priority to get the job done). Bye, Alexander. -- http://www.Leidinger.net Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF http://www.FreeBSD.org netchild@FreeBSD.org : PGP 0x8F31830F9F2772BF --=_26d7ff65f42eda061c5d904c2ac65c3b Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc; size=833 Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEER9UlYXp1PSd08nWXEg2wmwP42IYFAmaTpMsACgkQEg2wmwP4 2IYLkA/+IBCmIfp4hksBbQ//nAF8vvJ+HEPZIaCOBBPXDb+MnE637lLMBcqwrdrl mQtlTM8GpBrb5hJnMc/zy2RJkkVR+tuCP2hbIPaj/hi0xfbxzQ+t80DSY+pSib8x TMOBavGcpoJfn8CWQ4ZadWNpK+cUfQ0K/twqgD2IjCpW/Dj7lLuF71PgPcddo4RF v0VOJwCZi6oX9BqJ4FglX5x9vVTlyjzksOEha47SJKnFrLahPISSw+PUGWM9KKeW XxQOBQPgXdpaizQzM28h2I3wIBbRgBwjOZLeS36IewGT0eYzsJC5EMpwin04ulNR x5C/3fMvUxG+olmQZ55GNX9qJens0RZzEX7rynu8+mTAJIRZWMN//7Y8WhxzXwDY 2JJ4xan2hefMKAJoKl6VQCDkR/1LVQkQ6tW6IfbAt16mfVBOtS6Rh1oKMhQbpl/C JV05xakd1i//3WetqjS+5uvMdtNKuhYbytjn7kydCWDgIm4OIQb5MOSqquIVCh0x BotYf1L0ygSMvHrdmXyqVugjd6v+YMZr+36tvgdbXqHkaEZkMG3yrCoxIrTjGI4J LeMxQCKbh98KWoTAKJqbvkXIn4fL0lPeFlAyga5KZHkbIwaAAVMlAk4mcPc6zFGK TGqysTven06EOFRgdarEZYm/gJK8MQvajC7dBpTE90IodI7dpcU= =B1Ev -----END PGP SIGNATURE----- --=_26d7ff65f42eda061c5d904c2ac65c3b-- From nobody Sun Jul 14 12:18:08 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WMPjr6k4Xz5QVvp for ; Sun, 14 Jul 2024 12:27:44 +0000 (UTC) (envelope-from pmc@citylink.dinoex.sub.org) Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "uucp.dinoex.sub.de", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WMPjq0Y5Fz4ngV; Sun, 14 Jul 2024 12:27:42 +0000 (UTC) (envelope-from pmc@citylink.dinoex.sub.org) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of pmc@citylink.dinoex.sub.org designates 2a0b:f840::12 as permitted sender) smtp.mailfrom=pmc@citylink.dinoex.sub.org; arc=pass ("uucp.dinoex.org:s=M20221114:i=1") Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]) by uucp.dinoex.org (8.18.1/8.18.1) with ESMTPS id 46ECRA1T089511 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sun, 14 Jul 2024 14:27:10 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) ARC-Seal: i=1; a=rsa-sha256; d=uucp.dinoex.org; s=M20221114; t=1720960033; cv=none; b=M4sFqowk8+AdyQDl6rqbWvKSIwz2z0oUHwkKJzsneKE8R37ZisuAtG2SrUYn8w+rwnS7mqb1vbGmPFcQ0WFk6SunDTh8A5ZygqLfbcGZ0Od3UVlzgkFJ9ofz1EiLsdiD4T3HDcJyQJ+ArOojhUug69XaWHYsiGyGoT+goe1Br+Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=uucp.dinoex.org; s=M20221114; t=1720960033; c=relaxed/simple; bh=GFhM+5Qklq7PY/VlhzLPRhQ+0jIsukoB0ZlhotPaV34=; h=Received:Received:Received:Received:X-Authentication-Warning:Date: From:To:Cc:Subject:Message-ID:MIME-Version:Content-Type: Content-Disposition:X-Milter:X-Greylist; b=d2AKwb+gzRh+0aSzAqvFPKi2ZP2b9lNYegPz9ODFgwSwcdkBpHYELsTQyR8YeOps3RDb/oN3iSMtvwupDMCL1uDzVreo8LDmDuCuPauuG3Ctae7qsaDS+QxLCkE/fViJSqg9lAyRUUBnOPrN8uxoZZ8VFlwRQR9cF7UmmkaBzew= ARC-Authentication-Results: i=1; uucp.dinoex.org Received: (from uucp@localhost) by uucp.dinoex.org (8.18.1/8.18.1/Submit) with UUCP id 46ECRAlW089510; Sun, 14 Jul 2024 14:27:10 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) Received: from disp.intra.daemon.contact (disp-e.intra.daemon.contact [IPv6:fd00:0:0:0:0:0:0:112]) by admn.intra.daemon.contact (8.18.1/8.18.1) with ESMTPS id 46ECJsaw025345 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=OK); Sun, 14 Jul 2024 14:19:55 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) Received: from disp.intra.daemon.contact (localhost [127.0.0.1]) by disp.intra.daemon.contact (8.18.1/8.18.1) with ESMTPS id 46ECI8fj052460 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sun, 14 Jul 2024 14:18:09 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) Received: (from pmc@localhost) by disp.intra.daemon.contact (8.18.1/8.18.1/Submit) id 46ECI8Rw052459; Sun, 14 Jul 2024 14:18:08 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) X-Authentication-Warning: disp.intra.daemon.contact: pmc set sender to pmc@citylink.dinoex.sub.org using -f Date: Sun, 14 Jul 2024 14:18:08 +0200 From: Peter To: freebsd-stable@freebsd.org Cc: cperciva@freebsd.org Subject: Re: Change to FreeBSD release scheduling etc. Message-ID: List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-stable@freebsd.org Sender: owner-freebsd-stable@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Milter: Spamilter (Reciever: uucp.dinoex.org; Sender-ip: 0:0:2a0b:f840::; Sender-helo: uucp.dinoex.org;) X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]); Sun, 14 Jul 2024 14:27:13 +0200 (CEST) X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.26 / 15.00]; ARC_ALLOW(-1.00)[uucp.dinoex.org:s=M20221114:i=1]; NEURAL_HAM_LONG(-1.00)[-1.000]; FAKE_REPLY(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.96)[-0.963]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; ASN(0.00)[asn:205376, ipnet:2a0b:f840::/32, country:DE]; RCPT_COUNT_TWO(0.00)[2]; MISSING_XM_UA(0.00)[]; R_DKIM_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; DMARC_NA(0.00)[sub.org]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_FIVE(0.00)[5]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; RCVD_TLS_LAST(0.00)[]; HAS_XAW(0.00)[] X-Rspamd-Queue-Id: 4WMPjq0Y5Fz4ngV Folks, there was recently a message from Colin Percival, concerning the release schedule. And now I see on the main webpage a new release 13.4 announced to appear soon, while the upgrade to 13.3 was only recently. I read statements concerning the rationale behind this. These statements circle around the feasibility from a workflow perspective. I do understand that developers do not want to wait a year for their ready-made new features to roll out. On the other hand I see a serious issue from a reliability perspective. A new release is a rough time. For some 2-3 months, issues do pop up, and, depending on the complexity of a configuration, unexpected things do no longer work in production - mostly regressions. Some examples from 13.3 would be PR 276862, 278338 or the [in]famous 275594. If time allows, I tried to identify these during BETA - but this appears to be mostly futile. So for some issues (like, less recently, D19488 delaying the system startup for 5 minutes) I might just decide to fix them locally, or live with it - which nevertheless adds to the ripples and disturbances during the months after a new release. In short, it takes me about 3 months to catch the surprizes[*] and fix (or find out how to cope with) the concerning issues, regressions et al. that come along with a new release. Up to now that would then give another 9 months during which the systems can be operated in a plan-of-record fashion, until the next release starts the hassle again. With the new concept this does seriousely change. So, while I understand the agenda, I would like to bring into conscience that this is not without unpleasant effects at another side. Thinking about what could be done for remedy, what comes to my mind most easily is, simply support two most recent releases, so people get the option to skip each other upgrade. That doesn't look like much work, basically just backporting occasional security fixes. So much as a spontaneous suggestion. regards, PMc [*] Example surprize of today for 13.3-RELEASE: kernel crash from sysctl_iflist(), invoked by dhcpcd, whilst shutting down vimage jails. Apparently the involved [re]moving of netifs somehow conflicts with other things. From nobody Sun Jul 14 13:58:34 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WMRlD6VWvz5QgY7 for ; Sun, 14 Jul 2024 13:59:04 +0000 (UTC) (envelope-from ml@netfence.it) Received: from soth.netfence.it (mailserver.netfence.it [78.134.96.152]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mailserver.netfence.it", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WMRlD2Z3Dz41sd for ; Sun, 14 Jul 2024 13:59:04 +0000 (UTC) (envelope-from ml@netfence.it) Authentication-Results: mx1.freebsd.org; none Received: from [10.1.2.18] (mailserver.netfence.it [78.134.96.152]) (authenticated bits=0) by soth.netfence.it (8.18.1/8.17.2) with ESMTPSA id 46EDwYLH034562 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Sun, 14 Jul 2024 15:58:35 +0200 (CEST) (envelope-from ml@netfence.it) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netfence.it; s=202404; t=1720965517; bh=QOHInCTGdUqVA/YYoblQ5E6IMqOEyYgS6zwWAsdVN/w=; h=Date:Subject:To:References:From:Cc:In-Reply-To; b=TDA3KYZAskCX22w5immEnSunL0G76aSzL+xG9Qqmxc9BmDspQoGlk3eXKbL62Gnmk 3jhodzwRJkrp6VAJk8d310g/5VIjhtEDiopdKjxgynXRGHLqw5VIWGI/x+3/uADARj BBu8iY8jNgHNUvk5o4r+75KydbrTsPaDY0rRyN/c= X-Authentication-Warning: soth.netfence.it: Host mailserver.netfence.it [78.134.96.152] claimed to be [10.1.2.18] Message-ID: <0f3ac4f8-e5ee-4ac4-a7c2-793035d9cde9@netfence.it> Date: Sun, 14 Jul 2024 15:58:34 +0200 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-stable@freebsd.org Sender: owner-freebsd-stable@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Change to FreeBSD release scheduling etc. Content-Language: en-US To: pmc@citylink.dinoex.sub.org References: From: Andrea Venturoli Cc: stable@freebsd.org In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.86 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:35612, ipnet:78.134.0.0/17, country:IT] X-Rspamd-Queue-Id: 4WMRlD2Z3Dz41sd On 7/14/24 14:18, Peter wrote: Hello. First off, please bear in mind that I'm not being polemic. > there was recently a message from Colin Percival, concerning the > release schedule. That message was confusing to me: then I realized had I probably read it too hastily. It talks about "a release every 3 months" and that alarmed me a lot: that, along with "the support duration for individual point releases" remaining until "next point release + 3 months" would basically mean "upgrade every 3 months", which is infeasible in many situations. However, when I looked at the schedule at the end of the mail, I saw that those 3 months alternate between major releases, so what I thought doesn't hold. It's actually 6 months. Personally I still find it acceptable to upgrade every 6 months (not optimal, but mostly acceptable). > A new release is a rough time. For some 2-3 months, issues do pop > up... Up to now that would then give another 9 months... I agree with that: needing to upgrade once a years instead of twice would be a lot better. Of course there are also reasons that make me say the new schedule is in some way a good news, but IMVHO they don't balance the work that sysadmins are expeced to endure. > Thinking about what could be done for remedy, what comes to my mind > most easily is, simply support two most recent releases, so people > get the option to skip each other upgrade. That or some LTS version, like we had in the past. Unfortunately this was debated a lot already and I don't have any hope thereof. I'm glad to hear someone else is having my same concerns. Currently this issues are on hold here, but soon we'll have to think about what to do. Try and cope (and convince customers to pay for more work)? Try and maintain our own branches with backported security fixes (with the added difficulties for the port tree)? Move to another OS (still unlikely after how much was invested, but this is another straw)? I'd be happy to hear if others have any idea or what they plan to do. bye & Thanks av. From nobody Sun Jul 14 17:32:24 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WMXTV6VVgz5R3hm for ; Sun, 14 Jul 2024 17:32:30 +0000 (UTC) (envelope-from wollman@hergotha.csail.mit.edu) Received: from hergotha.csail.mit.edu (tunnel82308-pt.tunnel.tserv4.nyc4.ipv6.he.net [IPv6:2001:470:1f06:ccb::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "garrett.wollman.name", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WMXTV2235z4R7V for ; Sun, 14 Jul 2024 17:32:30 +0000 (UTC) (envelope-from wollman@hergotha.csail.mit.edu) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bimajority.org (policy=none); spf=pass (mx1.freebsd.org: domain of wollman@hergotha.csail.mit.edu designates 2001:470:1f06:ccb::2 as permitted sender) smtp.mailfrom=wollman@hergotha.csail.mit.edu Received: from hergotha.csail.mit.edu (localhost [127.0.0.1]) by hergotha.csail.mit.edu (8.18.1/8.18.1) with ESMTPS id 46EHWPdB015428 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sun, 14 Jul 2024 13:32:27 -0400 (EDT) (envelope-from wollman@hergotha.csail.mit.edu) Received: (from wollman@localhost) by hergotha.csail.mit.edu (8.18.1/8.18.1/Submit) id 46EHWPhg015427; Sun, 14 Jul 2024 13:32:25 -0400 (EDT) (envelope-from wollman) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-stable@freebsd.org Sender: owner-freebsd-stable@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <26260.2984.961319.782123@hergotha.csail.mit.edu> Date: Sun, 14 Jul 2024 13:32:24 -0400 From: Garrett Wollman To: Rick Macklem Cc: freebsd-stable@freebsd.org Subject: Re: Possible bug in zfs send or pipe implementation? In-Reply-To: References: <26259.12713.114036.564205@hergotha.csail.mit.edu> <26259.17366.276955.824313@hergotha.csail.mit.edu> X-Mailer: VM 8.2.0b under 29.2 (amd64-portbld-freebsd13.2) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.4 (hergotha.csail.mit.edu [0.0.0.0]); Sun, 14 Jul 2024 13:32:28 -0400 (EDT) X-Spam-Status: No, score=-1.8 required=5.0 tests=ALL_TRUSTED, HEADER_FROM_DIFFERENT_DOMAINS,NICE_REPLY_A,T_SCC_BODY_TEXT_LINE autolearn=disabled version=4.0.0 X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on hergotha.csail.mit.edu X-Spamd-Bar: - X-Spamd-Result: default: False [-1.90 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FORGED_SENDER(0.30)[wollman@bimajority.org,wollman@hergotha.csail.mit.edu]; R_SPF_ALLOW(-0.20)[+ip6:2001:470:1f06:ccb::2]; MIME_GOOD(-0.10)[text/plain]; DMARC_POLICY_SOFTFAIL(0.10)[bimajority.org : SPF not aligned (relaxed), No valid DKIM,none]; RCVD_COUNT_TWO(0.00)[2]; FREEFALL_USER(0.00)[wollman]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; TO_DN_SOME(0.00)[]; MIME_TRACE(0.00)[0:+]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_TO(0.00)[gmail.com]; R_DKIM_NA(0.00)[]; FROM_NEQ_ENVFROM(0.00)[wollman@bimajority.org,wollman@hergotha.csail.mit.edu]; FROM_HAS_DN(0.00)[]; TAGGED_RCPT(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[] X-Rspamd-Queue-Id: 4WMXTV2235z4R7V < said: > Just to clarify it, are you saying zfs is sleeping on "pipewr"? > (There is also a msleep() for "pipbww" in pipe_write().) It is sleeping on pipewr, yes. [wollman@nfs-prod-11 ~]$ sysctl kern.ipc.pipekva kern.ipc.pipekva: 536576 [wollman@nfs-prod-11 ~]$ sysctl kern.ipc.maxpipekva kern.ipc.maxpipekva: 2144993280 It's not out of KVA, it's just waiting for the `pv` process to wake up and read more data. `pv` is single-threaded and blocked on "select". It doesn't always get stuck in the same place, which is why I'm suspecting a lost wakeup somewhere. -GAWollman From nobody Sun Jul 14 22:14:44 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WMflN6MG4z5RWsp for ; Sun, 14 Jul 2024 22:14:56 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-io1-xd2c.google.com (mail-io1-xd2c.google.com [IPv6:2607:f8b0:4864:20::d2c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WMflN4Yfzz55dk; Sun, 14 Jul 2024 22:14:56 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-io1-xd2c.google.com with SMTP id ca18e2360f4ac-7fedb09357aso198445639f.2; Sun, 14 Jul 2024 15:14:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1720995295; x=1721600095; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=LFIOPX4B/MPsQviTAnc5CYWLYQdbUC2pWO+49/o0Dc4=; b=Rsh2HFIt+LjuGEbx0MgZXdb0ZS1xMQdpAZM7sH1QH1htAPMLCa3HPx/SzQoXQvkPia 5Y9yz5y7xSnAKVhEfR5WaJuiIHIGKo3gDe22Uu6R5jOVxh5qK5fBjONIJIhVF6fM9wSg 3dcPEg5+5ZCmDPCjaIj/m1veMvmo9cP1I622DwRApljTqiKVXi36D8FNOs4NZ+C6gs8H f0roBrt+s8DhKCcbEIsZvJMDFixelObXthrEvW1yW4Bk47NNPt5TfFnnjW2RBNSVESFD m1nxxPYrXCZQmEhMdRyZk4tOsHpY2eCCvQUnS7bXqjTefueHNZk7uj/2SSViYMqjl/VB AHnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720995295; x=1721600095; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=LFIOPX4B/MPsQviTAnc5CYWLYQdbUC2pWO+49/o0Dc4=; b=a3ZXM2qq8JF6hr94zo4p+sSx5mxlr30MchNTYjyv7A6K5Vm13OMaV9b6dKTjadlmOU pZZlickiQ0c2uM68jobMFKjXQbbm6ns/HcBTG5kREb95CwpdAzKcp5L+jGSTtlWGVR5C N6MzRTOrj6zdSrmRWknq2gahn4o50fbwI6Psq/m2km1wM06kvp9f/xKpj9Nbd31dzSgY Fs1E0C/usu60m6swLs3UfXEP7T5U/TaoUPGQ/mS0C2Kj0TzNLl3RQSJJHgovdznNcEav 0md+WqrTqWPma+6buVMyDMr4xM/yaeGUoD+ZdvdXHIcLiodDSXKV7vlwjcGjIKO7Ilmo qF7g== X-Forwarded-Encrypted: i=1; AJvYcCWP01MofGAnvAeISbp8uoMtcvOPvAkqCKO5VjPi4E3ZrEVykM5fNJJcBtqUzaa7w18f6L57ufABk2I+OqrFWkP3+Q== X-Gm-Message-State: AOJu0Yyxaf6OQ/B2nzA3/wSbmrSZQ9mH2oxxWjAdEMBcoYAj9uGHF+Qk 3IuBzbAPceEGBSEHyqSezPeyZ2kjvWNkqzVSB0ecij8q3aTYP3sor2Tt4yErZ+yjcNHcsXviCZv a9qYZkMWoGucMsBSSdtnGD2yVwA== X-Google-Smtp-Source: AGHT+IEDZu71htSgNqq/4PqBDAcTuH50o3ybksfNyw8R8qIEBYX5wvnao8kLxz8AyNQk+h78wlZzBlY35V5C1WhUOk4= X-Received: by 2002:a6b:f20c:0:b0:804:657d:92fc with SMTP id ca18e2360f4ac-804657d94a6mr1515371839f.18.1720995295198; Sun, 14 Jul 2024 15:14:55 -0700 (PDT) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-stable@freebsd.org Sender: owner-freebsd-stable@FreeBSD.org MIME-Version: 1.0 References: <26259.12713.114036.564205@hergotha.csail.mit.edu> <26259.17366.276955.824313@hergotha.csail.mit.edu> <26260.2984.961319.782123@hergotha.csail.mit.edu> In-Reply-To: <26260.2984.961319.782123@hergotha.csail.mit.edu> From: Rick Macklem Date: Sun, 14 Jul 2024 15:14:44 -0700 Message-ID: Subject: Re: Possible bug in zfs send or pipe implementation? To: Garrett Wollman Cc: freebsd-stable@freebsd.org, Mark Johnston Content-Type: multipart/alternative; boundary="0000000000003b04a7061d3c7185" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4WMflN4Yfzz55dk --0000000000003b04a7061d3c7185 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Sun, Jul 14, 2024 at 10:32=E2=80=AFAM Garrett Wollman wrote: > < > said: > > > Just to clarify it, are you saying zfs is sleeping on "pipewr"? > > (There is also a msleep() for "pipbww" in pipe_write().) > > It is sleeping on pipewr, yes. > > [wollman@nfs-prod-11 ~]$ sysctl kern.ipc.pipekva > kern.ipc.pipekva: 536576 > [wollman@nfs-prod-11 ~]$ sysctl kern.ipc.maxpipekva > kern.ipc.maxpipekva: 2144993280 > > It's not out of KVA, it's just waiting for the `pv` process to wake up > and read more data. `pv` is single-threaded and blocked on "select". > > It doesn't always get stuck in the same place, which is why I'm > suspecting a lost wakeup somewhere. > This snippet from sys/kern/sys_pipe.c looks a little suspicious to me... /* * Direct copy, bypassing a kernel buffer. */ } else if ((size =3D rpipe->pipe_pages.cnt) !=3D 0) { if (size > uio->uio_resid) size =3D (u_int) uio->uio_resid; PIPE_UNLOCK(rpipe); error =3D uiomove_fromphys(rpipe->pipe_pages.ms, rpipe->pipe_pages.pos, size, uio); PIPE_LOCK(rpipe); if (error) break; nread +=3D size; rpipe->pipe_pages.pos +=3D size; rpipe->pipe_pages.cnt -=3D size; if (rpipe->pipe_pages.cnt =3D=3D 0) { rpipe->pipe_state &=3D ~PIPE_WANTW; wakeup(rpipe); } If it reads uio_resid bytes which is less than pipe_pages.cnt, no wakeup() occurs. I'd be tempted to try getting rid of the "if (rpipe->pipe_pages.cnt =3D=3D = 0)" and do the wakeup() unconditionally, to see if it helps? Because if the application ("pv" in this case) doesn't do another read() on the pipe before calling select(), no wakeup() is going to occur, because here's what pipe_write() does... /* * We have no more space and have something to offer, * wake up select/poll. */ pipeselwakeup(wpipe); wpipe->pipe_state |=3D PIPE_WANTW; pipeunlock(wpipe); error =3D msleep(wpipe, PIPE_MTX(rpipe), PRIBIO | PCATCH, "pipewr", 0); pipelock(wpipe, 0); if (error !=3D 0) break; continue; Note that, once in msleep(), no call to pipeselwakeup() will occur until it gets woken up. I think the current code assumes that the reader ("pv" in this case) will read all the data out of the pipe before calling select() again. Does it do that? rick ps: I've added markj@ as a cc, since he seems to have been the last guy involved in sys_pipe.c. > -GAWollman > > --0000000000003b04a7061d3c7185 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


<= div class=3D"gmail_quote">
On Sun, Jul= 14, 2024 at 10:32=E2=80=AFAM Garrett Wollman <wollman@bimajority.org> wrote:
<<On Sat, 13 Jul 2024 20:50:55= -0700, Rick Macklem <rick.macklem@gmail.com> said:

> Just to clarify it, are you saying zfs is sleeping on "pipewr&quo= t;?
> (There is also a msleep() for "pipbww" in pipe_write().)

It is sleeping on pipewr, yes.

[wollman@nfs-prod-11 ~]$ sysctl kern.ipc.pipekva
kern.ipc.pipekva: 536576
[wollman@nfs-prod-11 ~]$ sysctl kern.ipc.maxpipekva
kern.ipc.maxpipekva: 2144993280

It's not out of KVA, it's just waiting for the `pv` process to wake= up
and read more data.=C2=A0 `pv` is single-threaded and blocked on "sele= ct".

It doesn't always get stuck in the same place, which is why I'm
suspecting a lost wakeup somewhere.
This snippet from sys/kern/sys_= pipe.c looks a little suspicious to me...=C2=A0
=C2=A0 /*<= /span>
* Direct copy, bypassing a kernel buffer.
*/
} else if ((size =3D rpipe->pipe_pages.cnt) != =3D 0) {
if (size > uio->uio_resid)
<= /span>size =3D (u_int) uio->uio_resid;
PIPE_UNLOCK(rp= ipe);
error =3D uiomove_fromphys(rpipe->pipe_pages.ms,
=C2=A0 =C2=A0 rpi= pe->pipe_pages.pos, size, uio);
PIPE_LOCK(rpipe);
if (error)
break;
nr= ead +=3D size;
rpipe->pipe_pages.pos +=3D size;
rpipe->pipe_pages.cnt -=3D size;
if (rp= ipe->pipe_pages.cnt =3D=3D 0) {
rpipe->pipe_state= &=3D ~PIPE_WANTW;
= wakeup(rpipe);
= }
If it reads uio_resid bytes which is less = than pipe_pages.cnt, no
wakeup() occurs= .
I'd be tempted to try getting rid= of the "if (rpipe->pipe_pages.cnt =3D=3D 0)"
and do the wakeup() unconditionally, to see if it helps= ?

Because if the application ("pv" in this case) doesn't do= another read() on the
pipe before call= ing select(), no wakeup() is going to occur, because here's
what pipe_write() does...
/*
* We have no more space and have something to offer,
= * wake up select/poll.
*/
pipeselwakeup(wpipe);

wpipe->pipe_state |=3D PIPE_WANTW;=
pipeunlock(wpipe);
error =3D msleep(wpipe, PI= PE_MTX(rpipe),
=C2=A0 = =C2=A0 PRIBIO | PCATCH, "pipewr", 0);
pipelock(wpipe, 0);
if (error !=3D 0)
break;
continue;
Note that, once in msleep(), no call to pipeselwakeup= () will occur until
it gets woken up.

I = think the current code assumes that the reader ("pv" in this case= ) will
read all the data out of the pipe before calling select() = again.
Does it do that?

rick
p= s: I've added markj@ as a cc, since he seems to have been the last guy = involved
=C2=A0 =C2=A0 in sys_pipe.c.


-GAWollman

--0000000000003b04a7061d3c7185-- From nobody Sun Jul 14 23:35:57 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WMhXz2nXlz5Pwnp for ; Sun, 14 Jul 2024 23:36:03 +0000 (UTC) (envelope-from wollman@hergotha.csail.mit.edu) Received: from hergotha.csail.mit.edu (tunnel82308-pt.tunnel.tserv4.nyc4.ipv6.he.net [IPv6:2001:470:1f06:ccb::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "garrett.wollman.name", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WMhXy1hW6z41ds for ; Sun, 14 Jul 2024 23:36:02 +0000 (UTC) (envelope-from wollman@hergotha.csail.mit.edu) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bimajority.org (policy=none); spf=pass (mx1.freebsd.org: domain of wollman@hergotha.csail.mit.edu designates 2001:470:1f06:ccb::2 as permitted sender) smtp.mailfrom=wollman@hergotha.csail.mit.edu Received: from hergotha.csail.mit.edu (localhost [127.0.0.1]) by hergotha.csail.mit.edu (8.18.1/8.18.1) with ESMTPS id 46ENZwPw017828 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sun, 14 Jul 2024 19:36:00 -0400 (EDT) (envelope-from wollman@hergotha.csail.mit.edu) Received: (from wollman@localhost) by hergotha.csail.mit.edu (8.18.1/8.18.1/Submit) id 46ENZwud017827; Sun, 14 Jul 2024 19:35:58 -0400 (EDT) (envelope-from wollman) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-stable@freebsd.org Sender: owner-freebsd-stable@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <26260.24797.484878.694868@hergotha.csail.mit.edu> Date: Sun, 14 Jul 2024 19:35:57 -0400 From: Garrett Wollman To: Rick Macklem Cc: freebsd-stable@freebsd.org Subject: Re: Possible bug in zfs send or pipe implementation? In-Reply-To: References: <26259.12713.114036.564205@hergotha.csail.mit.edu> <26259.17366.276955.824313@hergotha.csail.mit.edu> X-Mailer: VM 8.2.0b under 29.2 (amd64-portbld-freebsd13.2) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.4 (hergotha.csail.mit.edu [0.0.0.0]); Sun, 14 Jul 2024 19:36:00 -0400 (EDT) X-Spam-Status: No, score=-1.8 required=5.0 tests=ALL_TRUSTED, HEADER_FROM_DIFFERENT_DOMAINS,NICE_REPLY_A,T_SCC_BODY_TEXT_LINE autolearn=disabled version=4.0.0 X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on hergotha.csail.mit.edu X-Spamd-Bar: - X-Spamd-Result: default: False [-1.30 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.40)[-0.402]; FORGED_SENDER(0.30)[wollman@bimajority.org,wollman@hergotha.csail.mit.edu]; R_SPF_ALLOW(-0.20)[+ip6:2001:470:1f06:ccb::2]; MIME_GOOD(-0.10)[text/plain]; DMARC_POLICY_SOFTFAIL(0.10)[bimajority.org : SPF not aligned (relaxed), No valid DKIM,none]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[wollman]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; TO_DN_SOME(0.00)[]; MIME_TRACE(0.00)[0:+]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_TO(0.00)[gmail.com]; R_DKIM_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_NEQ_ENVFROM(0.00)[wollman@bimajority.org,wollman@hergotha.csail.mit.edu]; TAGGED_RCPT(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[] X-Rspamd-Queue-Id: 4WMhXy1hW6z41ds I'm currently running syncoid with `--quiet`, which removes `pv` from the pipeline. It seems to be moving along, but of course I can't easily tell what it's doing. Without `pv` in the way, the process reading the pipe is `lzop` instead, which doesn't try to do any fancy select() stuff, it's just a normal filter reading data from stdin and writing compressed data to stdout, in sequence. -GAWollman